Kerberized NFSv4 HOWTO

This page is a running documentation page for setting up Kerberized NFSv4. As I set up my network (and I guess in some cases after I set up my network ...), I'll document what I'm doing so that other people don't have to go through this mess again later. This page is a work in progress.

Words to the effect of:

• Initially wanted setup between nyus and home network
• nyus lives a fair ways off on the network (over untrusted terrain, anyway)
• NFSv3 not working out so well; Kerberos supported or not supported depending on who you ask
• NFSv3 requires RO export hacks (see link 1), and did not want to export RO to the world

Steps:

• JOSHUAWISE.COM does not have a Kerberos realm yet
• Set up a Kerberos realm
• Test it by Kerberizing SSH
• Try setting up NFS
• Kerberize client?

This assumes that you've already set up a Kerberos realm and that you've already also set up your NFS server. As usual, I'll be using the Kerberos realm JOSHUAWISE.COM for testing, and I'll be using the server nyus.joshuawise.com. I'll be setting up the client on shebang.

• Verify that kernel has NFSv4 support.

  joshua@shebang:/usr/src/linux-2.6.17.7$ grep nfs4 /proc/filesystems
  nodev   nfs4

• Install krb5 stuff. You may do different things depending on your distro.

  joshua@shebang:~$ sudo apt-get install krb5-clients krb5-user krb5-doc krb5-config
  words

  Prompt shows up: Enter the hostnames of Kerberos servers in the JOSHUAWISE.COM Kerberos realm separated by spaces. How the heck did it know that this is the JOSHUAWISE.COM realm?? I put in kerberos.joshuawise.com, which is a CNAME for joshuawise.com, which is my KDC. I put in kerberos.joshuawise.com for the administrative server, too. Dumped me back to a prompt.

• Verify that the realm is set up properly in /etc/krb5.conf:

  [libdefaults]
          default_realm = JOSHUAWISE.COM

  Add in the a setting for my realm in the [domain_realm] section:

          joshuawise.com = JOSHUAWISE.COM
          .joshuawise.com = JOSHUAWISE.COM

• Try kinit'ing:

  joshua@shebang:~$ kinit
  Password for joshua@JOSHUAWISE.COM:
  joshua@shebang:~$

• Make a directory for nyus:

  joshua@shebang:~$ sudo mkdir /nfs/nyus

• Try mounting it to see what I did wrong: (this seemed too damn easy!)

  joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
  Warning: rpc.idmapd appears not to be running.
           All uids will be mapped to the nobody uid.
  mount: wrong fs type, bad option, bad superblock on nyus.joshuawise.com:/,
         missing codepage or other error
         In some cases useful info is found in syslog - try
         dmesg | tail  or so

• Turn on idmapd and gssd. Edit /etc/default/nfs-common and change the variable set lines that are currently empty so that they look like this:

  # Do you want to start the idmapd daemon? It is only needed for NFSv4.
  NEED_IDMAPD=yes
  
  # Do you want to start the gssd daemon? It is required for Kerberos mounts.
  NEED_GSSD=yes

• Make those settings take effect:

  joshua@shebang:~$ sudo /etc/init.d/nfs-common restart

• Try mounting again:

  joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
  mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only
  mount: cannot mount block device nyus.joshuawise.com:/ read-only

  WARNING! WARNING! WARNING! mount IS LYING! IGNORE ITS LIES! THEY SERVE ONLY TO CONFUSE YOU! “write-protected”, especially when followed by “cannot mount read-only”, really means “the mount failed when I tried to mount it read-write for an unspecified reason”, not that it is actually write-protected. Take a look in syslog to see what the issue was:

  Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No such file or directory while beginning keytab scan for keytab '/etc/krb5.keytab'
  Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No usable keytab entries found in keytab '/etc/krb5.keytab'
  Jan 21 22:16:17 shebang rpc.gssd[6676]: Do you have a valid keytab entry for nfs/<your.host>@<YOUR.REALM> in keytab file /etc/krb5.keytab ?
  Jan 21 22:16:17 shebang rpc.gssd[6676]: Continuing without (machine) credentials - nfs4 mounts with Kerberos will fail
  Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu
  Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu

• Two issues here. Issue one – no keytab. Resolve this by kadminning up and getting the NFS key for nyus.

  joshua@shebang:~$ sudo kadmin
  Authenticating as principal root/admin@JOSHUAWISE.COM with password.
  Password for root/admin@JOSHUAWISE.COM: <type my password here>
  kadmin:  ktadd nfs/nyus.joshuawise.com
  Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
  Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
  kadmin:  ^Djoshua@shebang:~$

  If you get issues here to the effect of:

  kadmin: Operation requires ``change-password'' privilege while changing nfs/nyus.joshuawise.com@JOSHUAWISE.COM's key

then you have not restarted the Kerberos admin server after adding the */admin * ACL. This can be frustratingly difficult to discover, especially when you thought you restarted it earlier after setting that the first time around, and you thought that anyway it should've discovered it by now, and what is it doing changing a key anyway – shouldn't it be just adding it to the keytab?, and you're still not certain that that's what are supposed to do there, and ... Well, you get the picture.

• Restart gssd. /etc/init.d/nfs-common restart
• Issue two – NYUS.RES.cmu.edu != nyus.joshuawise.com. Add to /etc/hosts:

  128.2.162.192 nyus.joshuawise.com

• Retry the mount:

  joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
  mount: Connection timed out

  Now what's wrong?!... shebang is behind NAT, but who cares? dmesg reports:

  [17341332.176000] RPC: AUTH_GSS upcall timed out.
  [17341332.176000] Please check user daemon is running!

  OK, so I restarted rpc.gssd using rpc.gssd -vvvf, and got this:

  joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
  handling krb5 upcall
  Using keytab file '/etc/krb5.keytab'
  INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM' are good until 1169473613
  using FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM as credentials cache for machine creds
  using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM
  creating context using fsuid 0 (save_uid 0)
  creating tcp client for server nyus.joshuawise.com
  creating context with server nfs@nyus.joshuawise.com

  long pause...

  WARNING: Failed to create krb5 context for user with uid 0 for server nyus.joshuawise.com
  WARNING: Failed to create krb5 context for user with uid 0 with credentials cache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM for server nyus.joshuawise.com
  WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server nyus.joshuawise.com
  doing error downcall
  destroying client clnt12
  mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only

  Note: It appears that rpc.gssd died because I control-c'ed a mount. After I restarted rpc.gssd again, I got some new messages in addition to the old ones:

  handling krb5 upcall
  WARNING: failed reading uid from krb5 upcall pipe: No such file or directory
  WARNING: can't create tcp rpc_clnt for server nyus.joshuawise.com for user with uid 0: RPC: Remote system error - Connection timed out

  OK, well, I should've checked nyus's syslog:

  Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: gss_accept_sec_context failed
  Jan 21 23:37:04 localhost rpc.svcgssd[8000]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): Miscellaneous failure - Key version number for principal in key table is incorrect
  Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: failed to write message

• Verify and fix key versions. A key version, evidently, is also known as a “kvno”. When I did the “ktadd” on shebang, it incremented the kvno, which meant that shebang had the only up to date principal! When I did the ktadd again on nyus to update it, of course, I just broke shebang and dashnine. To prove that this is how it broke, I can do this:

  joshua@shebang:~$ sudo klist -e -k -t /etc/krb5.keytab
  Keytab name: FILE:/etc/krb5.keytab
  KVNO Timestamp         Principal
  ---- ----------------- --------------------------------------------------------
     4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (Triple DES cbc mode with HMAC/sha1)
     4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (DES cbc mode with CRC-32)
  joshua@shebang:~$ sudo kadmin
  Authenticating as principal root/admin@JOSHUAWISE.COM with password.
  Password for root/admin@JOSHUAWISE.COM:
  kadmin:  getprinc nfs/nyus.joshuawise.com@JOSHUAWISE.COM
  Principal: nfs/nyus.joshuawise.com@JOSHUAWISE.COM
  <blah blah>
  Number of keys: 2
  Key: vno 7, Triple DES cbc mode with HMAC/sha1, no salt
  Key: vno 7, DES cbc mode with CRC-32, no salt

  I just ended up ktremove'ing the nyus keys from shebang, and ktadding to get the latest on nyus.

• Make a keytab on shebang. I tturns out that I didn't need nyus's key on shebang, I just needed some key. So, I did this:

  joshua@shebang:~$ sudo kadmin
  Authenticating as principal root/admin@JOSHUAWISE.COM with password.
  Password for root/admin@JOSHUAWISE.COM:
  kadmin:  addprinc -randkey host/shebang.joshuawise.com
  WARNING: no policy specified for host/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy
  Principal "host/shebang.joshuawise.com@JOSHUAWISE.COM" created.
  kadmin:  addprinc -randkey nfs/shebang.joshuawise.com
  WARNING: no policy specified for nfs/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy
  Principal "nfs/shebang.joshuawise.com@JOSHUAWISE.COM" created.
  kadmin:  ktadd host/shebang.joshuawise.com
  Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
  Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
  kadmin:  ktadd nfs/shebang.joshuawise.com
  Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
  Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
  kadmin:  ^Djoshua@shebang:~$

• Restart gssd to use the new keytab.

  joshua@shebang:~$ sudo killall rpc.gssd
  joshua@shebang:~$ sudo killall rpc.gssd
  rpc.gssd: no process killed
  joshua@shebang:~$ sudo rpc.gssd
  joshua@shebang:~$

• Mount nyus.

  joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
  joshua@shebang:~$

• Test the security of Kerberized NFS.

  joshua@shebang:~$ kinit
  Password for joshua@JOSHUAWISE.COM:
  joshua@shebang:~$ ls /nfs/nyus
  bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
  joshua@shebang:~$ touch /nfs/nyus/foo
  touch: cannot touch `/nfs/nyus/foo': Permission denied
  joshua@shebang:~$ touch /nfs/nyus/home/joshua/foo
  joshua@shebang:~$ sudo rm /nfs/nyus/home/joshua/foo # because root has not authenticated as joshua
  rm: remove write-protected regular empty file `/nfs/nyus/home/joshua/foo'? y
  rm: cannot remove `/nfs/nyus/home/joshua/foo': Permission denied
  joshua@shebang:~$ rm /nfs/nyus/home/joshua/foo
  joshua@shebang:~$

• Add nyus to /etc/fstab:

  nyus.joshuawise.com:/ /nfs/nyus nfs4	user,exec,rsize=8192,wsize=8192,intr,rw,sec=krb5i       0	0

  ...and give it a shot...

  joshua@shebang:~$ kdestroy
  joshua@shebang:~$ sudo umount /nfs/nyus
  joshua@shebang:~$ mount /nfs/nyus
  joshua@shebang:~$ ls /nfs/nyus
  ls: /nfs/nyus: Permission denied
  joshua@shebang:~$ kinit
  Password for joshua@JOSHUAWISE.COM:
  joshua@shebang:~$ ls /nfs/nyus
  bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
  joshua@shebang:~$ kdestroy
  joshua@shebang:~$ ls /nfs/nyus
  bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz

• Presto! A mostly working Kerberized NFSv4 setup, in only 19 mind-bogglingly overcomplicated steps! There, now that wasn't so bad, was it?

Mode notes:

• Install pam_krb5: joshua@nyus:~$ sudo apt-get install libpam-krb5
• Add pam_krb5 to /etc/pam.d/common-auth as first line: auth sufficient pam_krb5.so ignore_root
• Add pam_krb5 to /etc/pam.d/common-session as first line: session optional pam_krb5.so ignore_root
• Add pam_krb5 to /etc/pam.d/common-account as first line: account required pam_krb5.so ignore_root
• Add pam_krb5 to /etc/pam.d/common-password as first line: password optional pam_krb5.so ignore_root
• Move away allowed keys to force a password login: joshua@nyus:~/.ssh$ mv authorized_keys2 authorized_keys2-old
• Change password to something that is not my UNIX password: joshua@shebang:~$ kpasswd
• Try connecting: joshua@shebang:~$ ssh nyus
• Observe that it uses the new Kerberos password.
• Change the password using PAM passwd: joshua@nyus:~$ passwd
• Connect again and observe that the password has been changed.
• Observe that 'klist' shows that you have a TGT after logging in over SSH using the PAM module.
• You now have working pam_krb5. passwd is a little messy:

  joshua@nyus:~/.ssh$ passwd
  Password:
  Changing password for joshua
  (current) UNIX password:
  Enter new password:
  Enter it again:
  Enter new UNIX password:
  Retype new UNIX password:
  passwd: password updated successfully

Here's some stuff I've referenced while setting this system up.

• Kerberized NFS V3 & V4 Server Set-Up – I didn't read it all the way through, but it looks very useful.