====== Kerberized NFSv4 HOWTO ====== This page is a **running documentation page for setting up Kerberized NFSv4**. As I set up my network (and I guess in some cases after I set up my network ...), I'll document what I'm doing so that other people don't have to go through this mess again later. This page is a work in progress. ===== Why Kerberized NFSv4? ===== Words to the effect of: * Initially wanted setup between ''nyus'' and home network * ''nyus'' lives a fair ways off on the network (over untrusted terrain, anyway) * NFSv3 not working out so well; Kerberos supported or not supported depending on who you ask * NFSv3 requires RO export hacks (see link 1), and did not want to export RO to the world Steps: * ''JOSHUAWISE.COM'' does not have a Kerberos realm yet * Set up a Kerberos realm * Test it by Kerberizing SSH * Try setting up NFS * Kerberize client? ===== Setting up a client ===== This assumes that you've already set up a Kerberos realm and that you've already also set up your NFS server. As usual, I'll be using the Kerberos realm ''JOSHUAWISE.COM'' for testing, and I'll be using the server ''nyus.joshuawise.com''. I'll be setting up the client on ''shebang''. * Verify that kernel has NFSv4 support. joshua@shebang:/usr/src/linux-2.6.17.7$ grep nfs4 /proc/filesystems nodev nfs4 * Install krb5 stuff. You may do different things depending on your distro. joshua@shebang:~$ sudo apt-get install krb5-clients krb5-user krb5-doc krb5-config wordsPrompt shows up: **Enter the hostnames of Kerberos servers in the JOSHUAWISE.COM Kerberos realm separated by spaces.** How the heck did it know that this is the ''JOSHUAWISE.COM'' realm?? I put in ''kerberos.joshuawise.com'', which is a CNAME for ''joshuawise.com'', which is my KDC. I put in ''kerberos.joshuawise.com'' for the administrative server, too. Dumped me back to a prompt. * Verify that the realm is set up properly in /etc/krb5.conf: [libdefaults] default_realm = JOSHUAWISE.COMAdd in the a setting for my realm in the [domain_realm] section: joshuawise.com = JOSHUAWISE.COM .joshuawise.com = JOSHUAWISE.COM * Try kinit'ing: joshua@shebang:~$ kinit Password for joshua@JOSHUAWISE.COM: joshua@shebang:~$ * Make a directory for ''nyus'': joshua@shebang:~$ sudo mkdir /nfs/nyus * Try mounting it to see what I did wrong: (this seemed too damn easy!) joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus Warning: rpc.idmapd appears not to be running. All uids will be mapped to the nobody uid. mount: wrong fs type, bad option, bad superblock on nyus.joshuawise.com:/, missing codepage or other error In some cases useful info is found in syslog - try dmesg | tail or so * Turn on ''idmapd'' and ''gssd''. Edit ''/etc/default/nfs-common'' and change the variable set lines that are currently empty so that they look like this:# Do you want to start the idmapd daemon? It is only needed for NFSv4. NEED_IDMAPD=yes # Do you want to start the gssd daemon? It is required for Kerberos mounts. NEED_GSSD=yes * Make those settings take effect: joshua@shebang:~$ sudo /etc/init.d/nfs-common restart * Try mounting again: joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only mount: cannot mount block device nyus.joshuawise.com:/ read-only **WARNING! WARNING! WARNING! ''mount'' IS LYING! IGNORE ITS LIES! THEY SERVE ONLY TO CONFUSE YOU!** "write-protected", especially when followed by "cannot mount read-only", really means "the mount failed when I tried to mount it read-write for an unspecified reason", not that it is actually write-protected. Take a look in syslog to see what the issue was: Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No such file or directory while beginning keytab scan for keytab '/etc/krb5.keytab' Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No usable keytab entries found in keytab '/etc/krb5.keytab' Jan 21 22:16:17 shebang rpc.gssd[6676]: Do you have a valid keytab entry for nfs/@ in keytab file /etc/krb5.keytab ? Jan 21 22:16:17 shebang rpc.gssd[6676]: Continuing without (machine) credentials - nfs4 mounts with Kerberos will fail Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu * Two issues here. Issue one -- no keytab. Resolve this by kadminning up and getting the NFS key for nyus. joshua@shebang:~$ sudo kadmin Authenticating as principal root/admin@JOSHUAWISE.COM with password. Password for root/admin@JOSHUAWISE.COM: kadmin: ktadd nfs/nyus.joshuawise.com Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab. kadmin: ^Djoshua@shebang:~$If you get issues here to the effect of:kadmin: Operation requires ``change-password'' privilege while changing nfs/nyus.joshuawise.com@JOSHUAWISE.COM's key then you have not restarted the Kerberos admin server after adding the ''*/admin *'' ACL. This can be frustratingly difficult to discover, especially when you thought you restarted it earlier after setting that the //first// time around, and you thought that anyway it should've discovered it by now, and what is it doing changing a key //anyway// -- shouldn't it be just adding it to the keytab?, and you're still not certain that that's what are supposed to do there, and ... Well, you get the picture. * Restart gssd. ''/etc/init.d/nfs-common restart'' * Issue two -- ''NYUS.RES.cmu.edu'' != ''nyus.joshuawise.com''. Add to /etc/hosts: 128.2.162.192 nyus.joshuawise.com * Retry the mount:joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus mount: Connection timed outNow what's wrong?!... shebang **is** behind NAT, but who cares? dmesg reports:[17341332.176000] RPC: AUTH_GSS upcall timed out. [17341332.176000] Please check user daemon is running!OK, so I restarted ''rpc.gssd'' using ''rpc.gssd -vvvf'', and got this:joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus handling krb5 upcall Using keytab file '/etc/krb5.keytab' INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM' are good until 1169473613 using FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM as credentials cache for machine creds using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM creating context using fsuid 0 (save_uid 0) creating tcp client for server nyus.joshuawise.com creating context with server nfs@nyus.joshuawise.com**long pause...** WARNING: Failed to create krb5 context for user with uid 0 for server nyus.joshuawise.com WARNING: Failed to create krb5 context for user with uid 0 with credentials cache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM for server nyus.joshuawise.com WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server nyus.joshuawise.com doing error downcall destroying client clnt12 mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-onlyNote: It appears that rpc.gssd died because I control-c'ed a mount. After I restarted rpc.gssd again, I got some new messages in addition to the old ones:handling krb5 upcall WARNING: failed reading uid from krb5 upcall pipe: No such file or directory WARNING: can't create tcp rpc_clnt for server nyus.joshuawise.com for user with uid 0: RPC: Remote system error - Connection timed outOK, well, I should've checked ''nyus'''s syslog:Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: gss_accept_sec_context failed Jan 21 23:37:04 localhost rpc.svcgssd[8000]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): Miscellaneous failure - Key version number for principal in key table is incorrect Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: failed to write message * Verify and fix key versions. A key version, evidently, is also known as a "kvno". When I did the "ktadd" on shebang, it incremented the kvno, which meant that shebang had the only up to date principal! When I did the ktadd again on nyus to update it, of course, I just broke shebang and dashnine. To prove that this is how it broke, I can do this:joshua@shebang:~$ sudo klist -e -k -t /etc/krb5.keytab Keytab name: FILE:/etc/krb5.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (Triple DES cbc mode with HMAC/sha1) 4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (DES cbc mode with CRC-32) joshua@shebang:~$ sudo kadmin Authenticating as principal root/admin@JOSHUAWISE.COM with password. Password for root/admin@JOSHUAWISE.COM: kadmin: getprinc nfs/nyus.joshuawise.com@JOSHUAWISE.COM Principal: nfs/nyus.joshuawise.com@JOSHUAWISE.COM Number of keys: 2 Key: vno 7, Triple DES cbc mode with HMAC/sha1, no salt Key: vno 7, DES cbc mode with CRC-32, no saltI just ended up ktremove'ing the nyus keys from shebang, and ktadding to get the latest on nyus. * Make a keytab on shebang. I tturns out that I didn't need nyus's key on shebang, I just needed some key. So, I did this:joshua@shebang:~$ sudo kadmin Authenticating as principal root/admin@JOSHUAWISE.COM with password. Password for root/admin@JOSHUAWISE.COM: kadmin: addprinc -randkey host/shebang.joshuawise.com WARNING: no policy specified for host/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy Principal "host/shebang.joshuawise.com@JOSHUAWISE.COM" created. kadmin: addprinc -randkey nfs/shebang.joshuawise.com WARNING: no policy specified for nfs/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy Principal "nfs/shebang.joshuawise.com@JOSHUAWISE.COM" created. kadmin: ktadd host/shebang.joshuawise.com Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab. kadmin: ktadd nfs/shebang.joshuawise.com Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab. kadmin: ^Djoshua@shebang:~$ * Restart gssd to use the new keytab.joshua@shebang:~$ sudo killall rpc.gssd joshua@shebang:~$ sudo killall rpc.gssd rpc.gssd: no process killed joshua@shebang:~$ sudo rpc.gssd joshua@shebang:~$ * Mount ''nyus''.joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus joshua@shebang:~$ * Test the security of Kerberized NFS.joshua@shebang:~$ kinit Password for joshua@JOSHUAWISE.COM: joshua@shebang:~$ ls /nfs/nyus bin boot dev etc home initrd initrd.img lib lost+found media mnt nfs opt proc root sbin srv sys tmp usr var vmlinuz joshua@shebang:~$ touch /nfs/nyus/foo touch: cannot touch `/nfs/nyus/foo': Permission denied joshua@shebang:~$ touch /nfs/nyus/home/joshua/foo joshua@shebang:~$ sudo rm /nfs/nyus/home/joshua/foo # because root has not authenticated as joshua rm: remove write-protected regular empty file `/nfs/nyus/home/joshua/foo'? y rm: cannot remove `/nfs/nyus/home/joshua/foo': Permission denied joshua@shebang:~$ rm /nfs/nyus/home/joshua/foo joshua@shebang:~$ * Add nyus to /etc/fstab:nyus.joshuawise.com:/ /nfs/nyus nfs4 user,exec,rsize=8192,wsize=8192,intr,rw,sec=krb5i 0 0...and give it a shot...joshua@shebang:~$ kdestroy joshua@shebang:~$ sudo umount /nfs/nyus joshua@shebang:~$ mount /nfs/nyus joshua@shebang:~$ ls /nfs/nyus ls: /nfs/nyus: Permission denied joshua@shebang:~$ kinit Password for joshua@JOSHUAWISE.COM: joshua@shebang:~$ ls /nfs/nyus bin boot dev etc home initrd initrd.img lib lost+found media mnt nfs opt proc root sbin srv sys tmp usr var vmlinuz joshua@shebang:~$ kdestroy joshua@shebang:~$ ls /nfs/nyus bin boot dev etc home initrd initrd.img lib lost+found media mnt nfs opt proc root sbin srv sys tmp usr var vmlinuz * Presto! A mostly working Kerberized NFSv4 setup, in only 19 mind-bogglingly overcomplicated steps! There, now that wasn't so bad, was it? ===== Setting up PAM ===== Mode notes: * Install pam_krb5: ''joshua@nyus:~$ sudo apt-get install libpam-krb5'' * Add pam_krb5 to /etc/pam.d/common-auth as first line: ''auth sufficient pam_krb5.so ignore_root'' * Add pam_krb5 to /etc/pam.d/common-session as first line: ''session optional pam_krb5.so ignore_root'' * Add pam_krb5 to /etc/pam.d/common-account as first line: ''account required pam_krb5.so ignore_root'' * Add pam_krb5 to /etc/pam.d/common-password as first line: ''password optional pam_krb5.so ignore_root'' * Move away allowed keys to force a password login: ''joshua@nyus:~/.ssh$ mv authorized_keys2 authorized_keys2-old'' * Change password to something that is not my UNIX password: ''joshua@shebang:~$ kpasswd'' * Try connecting: ''joshua@shebang:~$ ssh nyus'' * Observe that it uses the new Kerberos password. * Change the password using PAM passwd: ''joshua@nyus:~$ passwd'' * Connect again and observe that the password has been changed. * Observe that 'klist' shows that you have a TGT after logging in over SSH using the PAM module. * You now have working pam_krb5. passwd is a little messy: joshua@nyus:~/.ssh$ passwd Password: Changing password for joshua (current) UNIX password: Enter new password: Enter it again: Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully ===== Links ===== Here's some stuff I've referenced while setting this system up. * [[http://zumastor.blogspot.com/2006/11/kerberized-nfs-v3-v4-server-set-up.html|Kerberized NFS V3 & V4 Server Set-Up]] -- I didn't read it all the way through, but it looks very useful.