|
Next revision
|
Previous revision
|
kerberos-nfs [2011/06/28 09:10]
joshuawise created |
kerberos-nfs [2007/01/23 03:53] (current)
|
| bits | ====== Kerberized NFSv4 HOWTO ====== |
| | |
| | This page is a **running documentation page for setting up Kerberized NFSv4**. As I set up my network (and I guess in some cases after I set up my network ...), I'll document what I'm doing so that other people don't have to go through this mess again later. This page is a work in progress. |
| | |
| | ===== Why Kerberized NFSv4? ===== |
| | Words to the effect of: |
| | * Initially wanted setup between ''nyus'' and home network |
| | * ''nyus'' lives a fair ways off on the network (over untrusted terrain, anyway) |
| | * NFSv3 not working out so well; Kerberos supported or not supported depending on who you ask |
| | * NFSv3 requires RO export hacks (see link 1), and did not want to export RO to the world |
| | |
| | Steps: |
| | * ''JOSHUAWISE.COM'' does not have a Kerberos realm yet |
| | * Set up a Kerberos realm |
| | * Test it by Kerberizing SSH |
| | * Try setting up NFS |
| | * Kerberize client? |
| | |
| | ===== Setting up a client ===== |
| | This assumes that you've already set up a Kerberos realm and that you've already also set up your NFS server. As usual, I'll be using the Kerberos realm ''JOSHUAWISE.COM'' for testing, and I'll be using the server ''nyus.joshuawise.com''. I'll be setting up the client on ''shebang''. |
| | |
| | * Verify that kernel has NFSv4 support. <code>joshua@shebang:/usr/src/linux-2.6.17.7$ grep nfs4 /proc/filesystems |
| | nodev nfs4</code> |
| | * Install krb5 stuff. You may do different things depending on your distro. <code>joshua@shebang:~$ sudo apt-get install krb5-clients krb5-user krb5-doc krb5-config |
| | words</code>Prompt shows up: **Enter the hostnames of Kerberos servers in the JOSHUAWISE.COM Kerberos realm separated by spaces.** How the heck did it know that this is the ''JOSHUAWISE.COM'' realm?? I put in ''kerberos.joshuawise.com'', which is a CNAME for ''joshuawise.com'', which is my KDC. I put in ''kerberos.joshuawise.com'' for the administrative server, too. Dumped me back to a prompt. |
| | * Verify that the realm is set up properly in /etc/krb5.conf: <code>[libdefaults] |
| | default_realm = JOSHUAWISE.COM</code>Add in the a setting for my realm in the [domain_realm] section:<code> joshuawise.com = JOSHUAWISE.COM |
| | .joshuawise.com = JOSHUAWISE.COM</code> |
| | * Try kinit'ing: <code>joshua@shebang:~$ kinit |
| | Password for joshua@JOSHUAWISE.COM: |
| | joshua@shebang:~$</code> |
| | * Make a directory for ''nyus'': <code>joshua@shebang:~$ sudo mkdir /nfs/nyus</code> |
| | * Try mounting it to see what I did wrong: (this seemed too damn easy!) <code>joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus |
| | Warning: rpc.idmapd appears not to be running. |
| | All uids will be mapped to the nobody uid. |
| | mount: wrong fs type, bad option, bad superblock on nyus.joshuawise.com:/, |
| | missing codepage or other error |
| | In some cases useful info is found in syslog - try |
| | dmesg | tail or so</code> |
| | * Turn on ''idmapd'' and ''gssd''. Edit ''/etc/default/nfs-common'' and change the variable set lines that are currently empty so that they look like this:<code># Do you want to start the idmapd daemon? It is only needed for NFSv4. |
| | NEED_IDMAPD=yes |
| | |
| | # Do you want to start the gssd daemon? It is required for Kerberos mounts. |
| | NEED_GSSD=yes</code> |
| | * Make those settings take effect: <code>joshua@shebang:~$ sudo /etc/init.d/nfs-common restart</code> |
| | * Try mounting again: <code>joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus |
| | mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only |
| | mount: cannot mount block device nyus.joshuawise.com:/ read-only</code> **WARNING! WARNING! WARNING! ''mount'' IS LYING! IGNORE ITS LIES! THEY SERVE ONLY TO CONFUSE YOU!** "write-protected", especially when followed by "cannot mount read-only", really means "the mount failed when I tried to mount it read-write for an unspecified reason", not that it is actually write-protected. Take a look in syslog to see what the issue was: <code>Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No such file or directory while beginning keytab scan for keytab '/etc/krb5.keytab' |
| | Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No usable keytab entries found in keytab '/etc/krb5.keytab' |
| | Jan 21 22:16:17 shebang rpc.gssd[6676]: Do you have a valid keytab entry for nfs/<your.host>@<YOUR.REALM> in keytab file /etc/krb5.keytab ? |
| | Jan 21 22:16:17 shebang rpc.gssd[6676]: Continuing without (machine) credentials - nfs4 mounts with Kerberos will fail |
| | Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu |
| | Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu</code> |
| | * Two issues here. Issue one -- no keytab. Resolve this by kadminning up and getting the NFS key for nyus. <code>joshua@shebang:~$ sudo kadmin |
| | Authenticating as principal root/admin@JOSHUAWISE.COM with password. |
| | Password for root/admin@JOSHUAWISE.COM: <type my password here> |
| | kadmin: ktadd nfs/nyus.joshuawise.com |
| | Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. |
| | Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab. |
| | kadmin: ^Djoshua@shebang:~$</code>If you get issues here to the effect of:<code>kadmin: Operation requires ``change-password'' privilege while changing nfs/nyus.joshuawise.com@JOSHUAWISE.COM's key</code> |
| | then you have not restarted the Kerberos admin server after adding the ''*/admin *'' ACL. This can be frustratingly difficult to discover, especially when you thought you restarted it earlier after setting that the //first// time around, and you thought that anyway it should've discovered it by now, and what is it doing changing a key //anyway// -- shouldn't it be just adding it to the keytab?, and you're still not certain that that's what are supposed to do there, and ... Well, you get the picture. |
| | * Restart gssd. ''/etc/init.d/nfs-common restart'' |
| | * Issue two -- ''NYUS.RES.cmu.edu'' != ''nyus.joshuawise.com''. Add to /etc/hosts: <code>128.2.162.192 nyus.joshuawise.com</code> |
| | * Retry the mount:<code>joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus |
| | mount: Connection timed out</code>Now what's wrong?!... shebang **is** behind NAT, but who cares? dmesg reports:<code>[17341332.176000] RPC: AUTH_GSS upcall timed out. |
| | [17341332.176000] Please check user daemon is running!</code>OK, so I restarted ''rpc.gssd'' using ''rpc.gssd -vvvf'', and got this:<code>joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus |
| | handling krb5 upcall |
| | Using keytab file '/etc/krb5.keytab' |
| | INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM' are good until 1169473613 |
| | using FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM as credentials cache for machine creds |
| | using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM |
| | creating context using fsuid 0 (save_uid 0) |
| | creating tcp client for server nyus.joshuawise.com |
| | creating context with server nfs@nyus.joshuawise.com</code>**long pause...**<code> |
| | WARNING: Failed to create krb5 context for user with uid 0 for server nyus.joshuawise.com |
| | WARNING: Failed to create krb5 context for user with uid 0 with credentials cache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM for server nyus.joshuawise.com |
| | WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server nyus.joshuawise.com |
| | doing error downcall |
| | destroying client clnt12 |
| | mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only</code>Note: It appears that rpc.gssd died because I control-c'ed a mount. After I restarted rpc.gssd again, I got some new messages in addition to the old ones:<code>handling krb5 upcall |
| | WARNING: failed reading uid from krb5 upcall pipe: No such file or directory |
| | WARNING: can't create tcp rpc_clnt for server nyus.joshuawise.com for user with uid 0: RPC: Remote system error - Connection timed out</code>OK, well, I should've checked ''nyus'''s syslog:<code>Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: gss_accept_sec_context failed |
| | Jan 21 23:37:04 localhost rpc.svcgssd[8000]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): Miscellaneous failure - Key version number for principal in key table is incorrect |
| | Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: failed to write message</code> |
| | * Verify and fix key versions. A key version, evidently, is also known as a "kvno". When I did the "ktadd" on shebang, it incremented the kvno, which meant that shebang had the only up to date principal! When I did the ktadd again on nyus to update it, of course, I just broke shebang and dashnine. To prove that this is how it broke, I can do this:<code>joshua@shebang:~$ sudo klist -e -k -t /etc/krb5.keytab |
| | Keytab name: FILE:/etc/krb5.keytab |
| | KVNO Timestamp Principal |
| | ---- ----------------- -------------------------------------------------------- |
| | 4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (Triple DES cbc mode with HMAC/sha1) |
| | 4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (DES cbc mode with CRC-32) |
| | joshua@shebang:~$ sudo kadmin |
| | Authenticating as principal root/admin@JOSHUAWISE.COM with password. |
| | Password for root/admin@JOSHUAWISE.COM: |
| | kadmin: getprinc nfs/nyus.joshuawise.com@JOSHUAWISE.COM |
| | Principal: nfs/nyus.joshuawise.com@JOSHUAWISE.COM |
| | <blah blah> |
| | Number of keys: 2 |
| | Key: vno 7, Triple DES cbc mode with HMAC/sha1, no salt |
| | Key: vno 7, DES cbc mode with CRC-32, no salt</code>I just ended up ktremove'ing the nyus keys from shebang, and ktadding to get the latest on nyus. |
| | * Make a keytab on shebang. I tturns out that I didn't need nyus's key on shebang, I just needed some key. So, I did this:<code>joshua@shebang:~$ sudo kadmin |
| | Authenticating as principal root/admin@JOSHUAWISE.COM with password. |
| | Password for root/admin@JOSHUAWISE.COM: |
| | kadmin: addprinc -randkey host/shebang.joshuawise.com |
| | WARNING: no policy specified for host/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy |
| | Principal "host/shebang.joshuawise.com@JOSHUAWISE.COM" created. |
| | kadmin: addprinc -randkey nfs/shebang.joshuawise.com |
| | WARNING: no policy specified for nfs/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy |
| | Principal "nfs/shebang.joshuawise.com@JOSHUAWISE.COM" created. |
| | kadmin: ktadd host/shebang.joshuawise.com |
| | Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. |
| | Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab. |
| | kadmin: ktadd nfs/shebang.joshuawise.com |
| | Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. |
| | Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab. |
| | kadmin: ^Djoshua@shebang:~$</code> |
| | * Restart gssd to use the new keytab.<code>joshua@shebang:~$ sudo killall rpc.gssd |
| | joshua@shebang:~$ sudo killall rpc.gssd |
| | rpc.gssd: no process killed |
| | joshua@shebang:~$ sudo rpc.gssd |
| | joshua@shebang:~$</code> |
| | * Mount ''nyus''.<code>joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus |
| | joshua@shebang:~$</code> |
| | * Test the security of Kerberized NFS.<code>joshua@shebang:~$ kinit |
| | Password for joshua@JOSHUAWISE.COM: |
| | joshua@shebang:~$ ls /nfs/nyus |
| | bin boot dev etc home initrd initrd.img lib lost+found media mnt nfs opt proc root sbin srv sys tmp usr var vmlinuz |
| | joshua@shebang:~$ touch /nfs/nyus/foo |
| | touch: cannot touch `/nfs/nyus/foo': Permission denied |
| | joshua@shebang:~$ touch /nfs/nyus/home/joshua/foo |
| | joshua@shebang:~$ sudo rm /nfs/nyus/home/joshua/foo # because root has not authenticated as joshua |
| | rm: remove write-protected regular empty file `/nfs/nyus/home/joshua/foo'? y |
| | rm: cannot remove `/nfs/nyus/home/joshua/foo': Permission denied |
| | joshua@shebang:~$ rm /nfs/nyus/home/joshua/foo |
| | joshua@shebang:~$</code> |
| | * Add nyus to /etc/fstab:<code>nyus.joshuawise.com:/ /nfs/nyus nfs4 user,exec,rsize=8192,wsize=8192,intr,rw,sec=krb5i 0 0</code>...and give it a shot...<code>joshua@shebang:~$ kdestroy |
| | joshua@shebang:~$ sudo umount /nfs/nyus |
| | joshua@shebang:~$ mount /nfs/nyus |
| | joshua@shebang:~$ ls /nfs/nyus |
| | ls: /nfs/nyus: Permission denied |
| | joshua@shebang:~$ kinit |
| | Password for joshua@JOSHUAWISE.COM: |
| | joshua@shebang:~$ ls /nfs/nyus |
| | bin boot dev etc home initrd initrd.img lib lost+found media mnt nfs opt proc root sbin srv sys tmp usr var vmlinuz |
| | joshua@shebang:~$ kdestroy |
| | joshua@shebang:~$ ls /nfs/nyus |
| | bin boot dev etc home initrd initrd.img lib lost+found media mnt nfs opt proc root sbin srv sys tmp usr var vmlinuz</code> |
| | * Presto! A mostly working Kerberized NFSv4 setup, in only 19 mind-bogglingly overcomplicated steps! There, now that wasn't so bad, was it? |
| | ===== Setting up PAM ===== |
| | Mode notes: |
| | |
| | * Install pam_krb5: ''joshua@nyus:~$ sudo apt-get install libpam-krb5'' |
| | * Add pam_krb5 to /etc/pam.d/common-auth as first line: ''auth sufficient pam_krb5.so ignore_root'' |
| | * Add pam_krb5 to /etc/pam.d/common-session as first line: ''session optional pam_krb5.so ignore_root'' |
| | * Add pam_krb5 to /etc/pam.d/common-account as first line: ''account required pam_krb5.so ignore_root'' |
| | * Add pam_krb5 to /etc/pam.d/common-password as first line: ''password optional pam_krb5.so ignore_root'' |
| | * Move away allowed keys to force a password login: ''joshua@nyus:~/.ssh$ mv authorized_keys2 authorized_keys2-old'' |
| | * Change password to something that is not my UNIX password: ''joshua@shebang:~$ kpasswd'' |
| | * Try connecting: ''joshua@shebang:~$ ssh nyus'' |
| | * Observe that it uses the new Kerberos password. |
| | * Change the password using PAM passwd: ''joshua@nyus:~$ passwd'' |
| | * Connect again and observe that the password has been changed. |
| | * Observe that 'klist' shows that you have a TGT after logging in over SSH using the PAM module. |
| | * You now have working pam_krb5. passwd is a little messy: <code>joshua@nyus:~/.ssh$ passwd |
| | Password: |
| | Changing password for joshua |
| | (current) UNIX password: |
| | Enter new password: |
| | Enter it again: |
| | Enter new UNIX password: |
| | Retype new UNIX password: |
| | passwd: password updated successfully</code> |
| | |
| | ===== Links ===== |
| | Here's some stuff I've referenced while setting this system up. |
| | |
| | * [[http://zumastor.blogspot.com/2006/11/kerberized-nfs-v3-v4-server-set-up.html|Kerberized NFS V3 & V4 Server Set-Up]] -- I didn't read it all the way through, but it looks very useful. |
