pNFS/PVFS2 Documentation

Linux pNFS/PVFS2 Installation Instructions

Step 1: pNFS Kernel

• Ensure you have access to a built pNFS kernel source tree.

Step 2: Compile, Boot, and Verify Kernel and NFSv4.

• Compile and install kernel on every client, data server, and MDS. Instructions are located here.
• Ensure NFSv4 is working properly. For example, export and mount a local directory through the loopback device.

Step 3: Ensure you have a working PVFS2 file system.

Step 4: Export the pvfs2 file system via NFSv4 on the MDS.

Step 5: Install pvfs2 layout driver.

• Compile and install PVFS2 server code. Execute the following instructions in the pvfs2 layout driver directory:

  ./configure -with-kernel=/cvs/pnfs -prefix=/usr/local/bin/pvfs2-layout --disable-server --disable-karma
  where
  /cvs/pnfs is the compiled pNFS kernel source tree.
  /usr/local/bin/pvfs2-server is the installation target.
  
  make
  make install
  make KMOD_DIR=/usr/local/bin/pvfs2-layout kmod_install
  

• More PVFS2 specific instructions can be found here.

Step 6: Create /etc/pvfs2tab file on every client.

File contains

tcp://(mds serrver name):3334/pvfs2-fs

Step 7: Mount PVFS2 file system using pNFS

• Execute the following commands on the client:

  insmod /usr/local/bin/pvfs2-layout/pvfs2-pnfs.ko
  /usr/local/bin/pvfs2-layout/sbin/pvfs2-client -p /usr/local/bin/pvfs2-layout/sbin/pvfs2-client-core
  mount -t nfs4 (mds server name):/ /mnt/nfs4/
  

Step 7: Stopping pvfs2 layout driver.

• Execute the following commands on the client:

umount /mnt/pnfs
killall /usr/local/bin/pvfs2-layout/sbin/pvfs2-client
rmmod pvfs2_pnfs

Layout Driver Policies

Several /proc/sys/pvfs2-pnfs/ variables exist to control the behavior of the PVFS2 layout driver. They are:

• blocksize - Size (in bytes) of reads and writes to storage nodes. Currently the maximum size is 32768.
• write/read_threshold - Threshold (in bytes) under which writes/reads will be sent to the NFSv4 server. Value is -1 to always use direct I/O through the layout driver.
• use_pagecache - Value of 0 to avoid using page cache, all writes/reads go straight to PVFS2 storage nodes (this is the default behavior of unmodified PVFS2). Value of 1 to use page cache. This enables PVFS2 with a data cache, readahead, and a writeback cache.

Debugging Help

• NFS debugging:

  echo 32767 > /proc/sys/sunrpc/nfsd_debug
  echo 32767 > /proc/sys/sunrpc/nfs_debug
  

• pNFS layout driver debugging:

  // Before layout driver is loaded
  export PNFS_DEBUGMASK=all
  
  // Uses log file /tmp/pnfs-client.log
  

• PVFS2 debugging, see PVFS2 website

Known Problems

• Sometimes interrupting an application while writing/reading will cause the application to hang.

Please help improve these instructions, email dhildebz@umich.edu