NFSv4/Linux Hacking

Bugs, questions to nfsv4@linux-nfs.org. Fixes for this document to bfields@citi.umich.edu.

This is an attempt to provide pointers to the basic information necessary to start hacking the Linux NFSv4 implementation. I assume that you know C and know the basics of administering a Linux box (so I assume, for example, that you know how to build and install a new kernel). I don't assume a knowledge of kernel internals.

setting up NFSv4

There are instructions for setting it up from CITI's NFSv4 project pages. Get the very latest and play around with it. Note that you don't need two machines to test both the client and the server--just install both on one machine, and mount locally.

understanding NFSv4

The authoritative source is RFC 3530. Don't read it! It's too long. But keep a copy of it and of the RFC's it references on hand to refer to when you need to understand something specific.

The best way to watch NFSv4 at work is to run NFSv4 while watching your network with a packet sniffer. Use Wireshark: it's widely available and has up-to-date support for NFSv4. Once again, your traffic doesn't have to be going over a "real" network for this to work; if your client and server are on the same machine, just sniff the loopback interface ("lo").

Wireshark also has a companion program, tshark, with a text-only interface.

I usually adjust the Wireshark preferences to give the "Packet Details" panel the full height of the window. You may also need to set:

	Protocols->TCP->"Allow subdissector to desegment TCP streams"
	Protocols->IP ->"Reassemble fragmented IP datagrams"
	Protocols->RPC->"Desegment all RPC-over-TCP messages"
	Protocols->RPC->"Defragment all RPC-over-TCP messages"

In addition to providing filters in the capture dialog or (with the -f option) on the commandline, Wireshark also gives you some help constructing filters after-the-fact: right-click on an element in the middle pane and play around with the "prepare" and "match" menus. One additional hint: right-clicking on an element and then choosing "expand tree" recursively expands everything in that element.

reading kernel code

The best way to understand how some part of the kernel works is usually just to read the code.

The best way to get the code is to install git, then run:

	# get linus's mainline tree:
	git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
	# if you also want a few nfs-related trees:
	git remote add -f trond git://linux-nfs.org/pub/linux/nfs-2.6.git
	git remote add -f bfields git://linux-nfs.org/~bfields/linux.git
	git remote add -f bhalevy git://linux-nfs.org/~bhalevy/linux-pnfs.git

Then you can check out different versions with:

	git checkout v2.6.25
	git checkout v2.6.26-rc1
	git remote show trond    # what branches does Trond have?
	git checkout trond/devel # Checkout the tip of Trond's "devel" branch

and download new updates with:

	git fetch origin
	git fetch trond
	git fetch bfields
	git fetch bhalevy

(But note this does not affect your working directory; if you want to see what's new on some branch, you'll need to run "git checkout" again.)

"git grep" is useful for finding your way around, but you may also want to set up a good text editor integrated with a database of code cross-references. I use cscope and vim. The cscope home page has instructions on using cscope with vim and emacs, and instructions on using cscope on a large project like the kernel without waiting forever for the indices to build. This allows you to follow the flow of control easily by popping quickly from the use of a function to its definition and back.

Take notes. As an example, I keep some rough notes on the kernel. In many cases they're too rough to be of use to someone else, but they help me organize my thoughts while I'm learning something new.

It's easy to get lead astray if one attempts to understand large subsystems all at once. Instead, try to keep in mind one small goal (e.g., to fix a bug, to learn how to use a certain interface).

Robert Love's "Linux Kernel Development" gives a good overview if read side-by-side with the kernel code. "Linux Device Drivers" is also good, as is "Understanding the Linux Kernel". See also lwn.net's kernel coverage.

NFS Debugging

You can use the rpcdebug command (included in nfs-utils) to get additional debugging information dumped in your logs.

(To see the code that produces this, see include/linux/sunrpc/debug.h, include/linux/nfs_fs.h, include/linux/nfsd/debug.h, the NFSDDBG_FACILITY defines at the top of each .c file, and the dprintk()'s sprinkled throughout.

generating patches

Patches are the basic unit of communication with other kernel hackers. They should be readable by humans, not just by the patch command. To this end:

Use the "-u" option to diff. I also like the "-p" option, which adds a guess at the current C function before each chunk.
Make patches that are short and that only do one thing. This may mean that, after writing a bunch of new code to implement a new feature, you need to spend some time breaking up the code into smaller patches which introduce the new feature in easier-to-understand chunks.
When dealing with a long series of patches, make sure that each individual patch introduces no compile-time or run-time regressions.

See also Andrew Morton's The Perfect Patch, and see Documentation/SubmittingPatches in your friendly local kernel tree.

Generating the patches can be done with diff -urNp if you have two clean trees to compare; things get more complicated if you mistakenly do a "make" in one of the two trees, or if you start having to deal with long series of patches.

There are a several ways people deal with this; in rough order by my preference:

I use git, which is what the upstream kernel source is kept in. See the tutorial and user's manual for an introduction. Specifically, the chapter Rewriting history and maintaining patch series may be useful.
Andrew Morton's patch scripts may be simpler to get started with than git. See the included "docco.txt" file for instructions. Or try quilt, which is based on Andrew Morton's patch scripts.
tricks with lndir (or similar tricks with "cp -al" and hard links): very simple, but it doesn't automate quite as much as Andrew Morton's scripts--eventually you'll need to write additional scripts to maintain patch comments and so on.

Mailing Lists

It's helpful to at least skim the following mailing lists:

The linux-kernel@vger.kernel.org: traffic is several hundred posts a day, so don't subscribe unless you have some filtering set up to help you cope.
nfs@lists.sourceforge.net: nfs/linux stuff.
nfsv4@ietf.org: ietf NFSv4 working group.
nfsv4@linux-nfs.org: discussion of the linux implementation of nfsv4.

Miscellaneous

Use the -s option to make; this eliminates most of its output so that you can see (potentially important) compiler warnings more easily.