<!--
.. title: Kerberized NFSv4 HOWTO
.. slug: kerberos-nfs
.. date: 2007-01-22 22:53:17 UTC-04:00
.. tags: favorites
.. category: 
.. link: 
.. description: 
.. why-favorite: somewhat out of date, but may still help you with some of the common pitfalls
.. type: text
-->

[TOC]

This page is a **running documentation page for setting up Kerberized NFSv4**. As I set up my network (and I guess in some cases after I set up my network ...), I'll document what I'm doing so that other people don't have to go through this mess again later. This page is a work in progress.

## Why Kerberized NFSv4?

Words to the effect of:

* Initially wanted setup between `nyus` and home network
* `nyus` lives a fair ways off on the network (over untrusted terrain, anyway)
* NFSv3 not working out so well; Kerberos supported or not supported depending on who you ask
* NFSv3 requires RO export hacks (see link 1), and did not want to export RO to the world

Steps:

* `JOSHUAWISE.COM` does not have a Kerberos realm yet
* Set up a Kerberos realm
* Test it by Kerberizing SSH
* Try setting up NFS
* Kerberize client?

## Setting up a client

This assumes that you've already set up a Kerberos realm and that you've already also set up your NFS server. As usual, I'll be using the Kerberos realm `JOSHUAWISE.COM` for testing, and I'll be using the server `nyus.joshuawise.com`. I'll be setting up the client on `shebang`.

* Verify that kernel has NFSv4 support.

```text
joshua@shebang:/usr/src/linux-2.6.17.7$ grep nfs4 /proc/filesystems
nodev   nfs4
```

* Install krb5 stuff. You may do different things depending on your distro.

```text
joshua@shebang:~$ sudo apt-get install krb5-clients krb5-user krb5-doc krb5-config
words
```
Prompt shows up: **Enter the hostnames of Kerberos servers in the JOSHUAWISE.COM Kerberos realm separated by spaces.** How the heck did it know that this is the `JOSHUAWISE.COM` realm?? I put in `kerberos.joshuawise.com`, which is a CNAME for `joshuawise.com`, which is my KDC. I put in `kerberos.joshuawise.com` for the administrative server, too. Dumped me back to a prompt.

* Verify that the realm is set up properly in /etc/krb5.conf:

```text
[libdefaults]
        default_realm = JOSHUAWISE.COM
```
Add in the a setting for my realm in the [domain_realm] section:

```text
        joshuawise.com = JOSHUAWISE.COM
        .joshuawise.com = JOSHUAWISE.COM
```

* Try kinit'ing:

```text
joshua@shebang:~$ kinit
Password for joshua@JOSHUAWISE.COM:
joshua@shebang:~$
```

* Make a directory for `nyus`:

```text
joshua@shebang:~$ sudo mkdir /nfs/nyus
```

* Try mounting it to see what I did wrong: (this seemed too damn easy!)

```text
joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
Warning: rpc.idmapd appears not to be running.
         All uids will be mapped to the nobody uid.
mount: wrong fs type, bad option, bad superblock on nyus.joshuawise.com:/,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
```

* Turn on `idmapd` and `gssd`. Edit `/etc/default/nfs-common` and change the variable set lines that are currently empty so that they look like this:

```text
# Do you want to start the idmapd daemon? It is only needed for NFSv4.
NEED_IDMAPD=yes

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=yes
```

* Make those settings take effect:

```text
joshua@shebang:~$ sudo /etc/init.d/nfs-common restart
```

* Try mounting again:

```text
joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only
mount: cannot mount block device nyus.joshuawise.com:/ read-only
```
**WARNING! WARNING! WARNING! `mount` IS LYING! IGNORE ITS LIES! THEY SERVE ONLY TO CONFUSE YOU!** "write-protected", especially when followed by "cannot mount read-only", really means "the mount failed when I tried to mount it read-write for an unspecified reason", not that it is actually write-protected. Take a look in syslog to see what the issue was:

```text
Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No such file or directory while beginning keytab scan for keytab '/etc/krb5.keytab'
Jan 21 22:16:17 shebang rpc.gssd[6676]: ERROR: No usable keytab entries found in keytab '/etc/krb5.keytab'
Jan 21 22:16:17 shebang rpc.gssd[6676]: Do you have a valid keytab entry for nfs/<your.host>@<YOUR.REALM> in keytab file /etc/krb5.keytab ?
Jan 21 22:16:17 shebang rpc.gssd[6676]: Continuing without (machine) credentials - nfs4 mounts with Kerberos will fail
Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu
Jan 21 22:17:34 shebang rpc.gssd[6676]: WARNING: Failed to obtain machine credentials for connection to server NYUS.RES.cmu.edu
```

* Two issues here. Issue one -- no keytab. Resolve this by kadminning up and getting the NFS key for nyus.

```text
joshua@shebang:~$ sudo kadmin
Authenticating as principal root/admin@JOSHUAWISE.COM with password.
Password for root/admin@JOSHUAWISE.COM: <type my password here>
kadmin:  ktadd nfs/nyus.joshuawise.com
Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal nfs/nyus.joshuawise.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:  ^Djoshua@shebang:~$
```
If you get issues here to the effect of:

```text
kadmin: Operation requires ``change-password'' privilege while changing nfs/nyus.joshuawise.com@JOSHUAWISE.COM's key
```

then you have not restarted the Kerberos admin server after adding the `*/admin *` ACL. This can be frustratingly difficult to discover, especially when you thought you restarted it earlier after setting that the *first* time around, and you thought that anyway it should've discovered it by now, and what is it doing changing a key *anyway* -- shouldn't it be just adding it to the keytab?, and you're still not certain that that's what are supposed to do there, and ... Well, you get the picture.
* Restart gssd. `/etc/init.d/nfs-common restart`
* Issue two -- `NYUS.RES.cmu.edu` != `nyus.joshuawise.com`. Add to /etc/hosts:

```text
128.2.162.192 nyus.joshuawise.com
```

* Retry the mount:

```text
joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
mount: Connection timed out
```
Now what's wrong?!... shebang **is** behind NAT, but who cares? dmesg reports:

```text
[17341332.176000] RPC: AUTH_GSS upcall timed out.
[17341332.176000] Please check user daemon is running!
```
OK, so I restarted `rpc.gssd` using `rpc.gssd -vvvf`, and got this:

```text
joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
handling krb5 upcall
Using keytab file '/etc/krb5.keytab'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM' are good until 1169473613
using FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM
creating context using fsuid 0 (save_uid 0)
creating tcp client for server nyus.joshuawise.com
creating context with server nfs@nyus.joshuawise.com
```
**long pause...**

```text
WARNING: Failed to create krb5 context for user with uid 0 for server nyus.joshuawise.com
WARNING: Failed to create krb5 context for user with uid 0 with credentials cache FILE:/tmp/krb5cc_machine_JOSHUAWISE.COM for server nyus.joshuawise.com
WARNING: Failed to create krb5 context for user with uid 0 with any credentials cache for server nyus.joshuawise.com
doing error downcall
destroying client clnt12
mount: block device nyus.joshuawise.com:/ is write-protected, mounting read-only
```
Note: It appears that rpc.gssd died because I control-c'ed a mount. After I restarted rpc.gssd again, I got some new messages in addition to the old ones:

```text
handling krb5 upcall
WARNING: failed reading uid from krb5 upcall pipe: No such file or directory
WARNING: can't create tcp rpc_clnt for server nyus.joshuawise.com for user with uid 0: RPC: Remote system error - Connection timed out
```
OK, well, I should've checked `nyus`'s syslog:

```text
Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: gss_accept_sec_context failed
Jan 21 23:37:04 localhost rpc.svcgssd[8000]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): Miscellaneous failure - Key version number for principal in key table is incorrect
Jan 21 23:37:04 localhost rpc.svcgssd[8000]: WARNING: failed to write message
```

* Verify and fix key versions. A key version, evidently, is also known as a "kvno". When I did the "ktadd" on shebang, it incremented the kvno, which meant that shebang had the only up to date principal! When I did the ktadd again on nyus to update it, of course, I just broke shebang and dashnine. To prove that this is how it broke, I can do this:

```text
joshua@shebang:~$ sudo klist -e -k -t /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (Triple DES cbc mode with HMAC/sha1)
   4 01/21/07 22:24:33 nfs/nyus.joshuawise.com@JOSHUAWISE.COM (DES cbc mode with CRC-32)
joshua@shebang:~$ sudo kadmin
Authenticating as principal root/admin@JOSHUAWISE.COM with password.
Password for root/admin@JOSHUAWISE.COM:
kadmin:  getprinc nfs/nyus.joshuawise.com@JOSHUAWISE.COM
Principal: nfs/nyus.joshuawise.com@JOSHUAWISE.COM
<blah blah>
Number of keys: 2
Key: vno 7, Triple DES cbc mode with HMAC/sha1, no salt
Key: vno 7, DES cbc mode with CRC-32, no salt
```
I just ended up ktremove'ing the nyus keys from shebang, and ktadding to get the latest on nyus.
* Make a keytab on shebang. I tturns out that I didn't need nyus's key on shebang, I just needed some key. So, I did this:

```text
joshua@shebang:~$ sudo kadmin
Authenticating as principal root/admin@JOSHUAWISE.COM with password.
Password for root/admin@JOSHUAWISE.COM:
kadmin:  addprinc -randkey host/shebang.joshuawise.com
WARNING: no policy specified for host/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy
Principal "host/shebang.joshuawise.com@JOSHUAWISE.COM" created.
kadmin:  addprinc -randkey nfs/shebang.joshuawise.com
WARNING: no policy specified for nfs/shebang.joshuawise.com@JOSHUAWISE.COM; defaulting to no policy
Principal "nfs/shebang.joshuawise.com@JOSHUAWISE.COM" created.
kadmin:  ktadd host/shebang.joshuawise.com
Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:  ktadd nfs/shebang.joshuawise.com
Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal nfs/shebang.joshuawise.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:  ^Djoshua@shebang:~$
```

* Restart gssd to use the new keytab.

```text
joshua@shebang:~$ sudo killall rpc.gssd
joshua@shebang:~$ sudo killall rpc.gssd
rpc.gssd: no process killed
joshua@shebang:~$ sudo rpc.gssd
joshua@shebang:~$
```

* Mount `nyus`.

```text
joshua@shebang:~$ sudo mount -t nfs4 -o sec=krb5i,rw nyus.joshuawise.com:/ /nfs/nyus
joshua@shebang:~$
```

* Test the security of Kerberized NFS.

```text
joshua@shebang:~$ kinit
Password for joshua@JOSHUAWISE.COM:
joshua@shebang:~$ ls /nfs/nyus
bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
joshua@shebang:~$ touch /nfs/nyus/foo
touch: cannot touch `/nfs/nyus/foo': Permission denied
joshua@shebang:~$ touch /nfs/nyus/home/joshua/foo
joshua@shebang:~$ sudo rm /nfs/nyus/home/joshua/foo # because root has not authenticated as joshua
rm: remove write-protected regular empty file `/nfs/nyus/home/joshua/foo'? y
rm: cannot remove `/nfs/nyus/home/joshua/foo': Permission denied
joshua@shebang:~$ rm /nfs/nyus/home/joshua/foo
joshua@shebang:~$
```

* Add nyus to /etc/fstab:

```text
nyus.joshuawise.com:/ /nfs/nyus nfs4	user,exec,rsize=8192,wsize=8192,intr,rw,sec=krb5i       0	0
```
...and give it a shot...

```text
joshua@shebang:~$ kdestroy
joshua@shebang:~$ sudo umount /nfs/nyus
joshua@shebang:~$ mount /nfs/nyus
joshua@shebang:~$ ls /nfs/nyus
ls: /nfs/nyus: Permission denied
joshua@shebang:~$ kinit
Password for joshua@JOSHUAWISE.COM:
joshua@shebang:~$ ls /nfs/nyus
bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
joshua@shebang:~$ kdestroy
joshua@shebang:~$ ls /nfs/nyus
bin  boot  dev  etc  home  initrd  initrd.img  lib  lost+found  media  mnt  nfs  opt  proc  root  sbin  srv  sys  tmp  usr  var  vmlinuz
```

* Presto! A mostly working Kerberized NFSv4 setup, in only 19 mind-bogglingly overcomplicated steps! There, now that wasn't so bad, was it?
## Setting up PAM

Mode notes:

* Install pam_krb5: `joshua@nyus:~$ sudo apt-get install libpam-krb5`
* Add pam_krb5 to /etc/pam.d/common-auth as first line: `auth sufficient pam_krb5.so ignore_root`
* Add pam_krb5 to /etc/pam.d/common-session as first line: `session optional pam_krb5.so ignore_root`
* Add pam_krb5 to /etc/pam.d/common-account as first line: `account required pam_krb5.so ignore_root`
* Add pam_krb5 to /etc/pam.d/common-password as first line: `password optional pam_krb5.so ignore_root`
* Move away allowed keys to force a password login: `joshua@nyus:~/.ssh$ mv authorized_keys2 authorized_keys2-old`
* Change password to something that is not my UNIX password: `joshua@shebang:~$ kpasswd`
* Try connecting: `joshua@shebang:~$ ssh nyus`
* Observe that it uses the new Kerberos password.
* Change the password using PAM passwd: `joshua@nyus:~$ passwd`
* Connect again and observe that the password has been changed.
* Observe that 'klist' shows that you have a TGT after logging in over SSH using the PAM module.
* You now have working pam_krb5. passwd is a little messy:

```text
joshua@nyus:~/.ssh$ passwd
Password:
Changing password for joshua
(current) UNIX password:
Enter new password:
Enter it again:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
```

## Links

Here's some stuff I've referenced while setting this system up.

* [Kerberized NFS V3 & V4 Server Set-Up](http://zumastor.blogspot.com/2006/11/kerberized-nfs-v3-v4-server-set-up.html) -- I didn't read it all the way through, but it looks very useful.