Cluster Coherent NFS

Statelessness allowed NFS Versions 2 and 3 servers to export shared storage in parallel with ease and opened the door to parallel, high-bandwidth, industry standard NAS that scales in fine-grained increments.

NFSv4 servers don't have it so easy. They have their own state to manage — like OPEN and LOCK — but the protocol does not support distributing it among multiple servers, making it difficult to export shared storage in parallel.

CITI is developing Linux kernel intefaces to allow stateful kernel lock managers such as the NFSv4 server and LOCKD to communicate with cluster file systems to enable exporting a multiple symmetric NFS service. See the Cluster Coherent NFS Wiki for more detail.

Polyserve and IBM have been the major sponsors for this project.

• Polyserve
• Statement of Work, 2004-07-01 to 2005-06-30
• Status, 2004-09
• Status, 2004-12
• IBM
• Statement of Work, 2005-07-01 to 2006-03-31
• Status, 2006-10-26
• Statement of Work, 2006-10-01 to 2007-09-30
• Status, 2008-01-10
• Status, 2008-11-03

CITI is improving support for exporting NFS from Linux cluster file systems in other ways, and also riding herd on Linux issues related to multi-protocol file servers. Specific areas of concern in this Google-sponsored project include POSIX locks, the Linux lease mechanism, RPC reply caching, and ACL interoperability.

Exporting a cluster file system with NFS improves availability, load balancing, and I/O bandwidth by allowing simultaneous NFS exports of the same filesystem from multiple cluster nodes, and by providing for migration and failover between nodes. There is marked interest in engineering NFS for cluster file systems among users and developers of out-of-tree cluster filesystems (such as GFPS and Lustre), but the cooperation and consensus of the kernel community (and acceptance of patches) depend on demonstrating working implementations with in-tree filesystems.

There are currently two in-tree cluster filesystems: OCFS2 and GFS2. OCFS2 lacks support for NFS exports and for POSIX locks, so we are focusing our attention on GFS2. However, GFS2 is also still young and these features are not yet stable; this complicates our work.

We have identified three areas requiring the attention of file system developers: POSIX file locks, leases, and RPC reply caching. In addition, CITI is addressing some of the issues that arise when two file servers (nfsd and Samba) attempt to export the same cluster filesystem.

• Google
• Statement of Work, 2008-02-01 to 2009-01-31
• Progress report, 2008-08-08