pNFS/PVFS2 Documentation

Linux pNFS/PVFS2 Installation Instructions

Note: If you get confused at any point, verify your setup with the diagram below.

Step 1: Download and Untar

• Download and untar pNFS kernel, PVFS2 Server, and PVFS2 Layout Driver.

Step 2: Compile, Boot, and Verify Kernel and NFSv4

• Compile and install pNFS kernel on every client, data server, and MDS. Instructions are located here under Instructions.
• Ensure NFSv4 is working properly. For example, export and mount a local directory through the loopback device.

Step 3: Compile and Verify PVFS2 storage system

• Compile and install the PVFS2 server code. Follow the instructions on the PVFS2 website.
• Example:

  ./configure -with-kernel=/path/to/kernel -prefix=/usr/local/bin/pvfs2-server
  make
  make install
  make KMOD_DIR=/usr/local/bin/pvfs2-server kmod_install
  

• To verify, create a pvfs2 file system and access it. All instructions are on the PVFS2 website
• Example (easiest to run all commands as root):

  // Define PVFS2 file system and create config files
  // Run on MDS
  /usr/local/bin/pvfs2-server/bin/pvfs2-genconfig /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf
  
  // Copy /etc/pvfs2 directory to every data server
  // Create /etc/pvfs2tab file with contents:
  tcp://"mds_server_hostname":3334/pvfs2-fs /mnt/pvfs2 pvfs2 default 0 0
  
  // Create storage space
  // Run on each data server and MDS
  /usr/local/bin/pvfs2-server/sbin/pvfs2-server /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-hostname -f
  
  // Start server
  // Run on each data server and MDS
  /usr/local/bin/pvfs2-server/sbin/pvfs2-server /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-hostname
  
  // Test filesystem
  // Run on MDS and ensure positive result
  /usr/local/bin/pvfs2-server/bin/pvfs2-ping -m /mnt/pvfs2
  
  // Load kernel interface
  // Run on MDS
  insmod /usr/local/bin/pvfs2-server/pvfs2.ko
  /usr/local/bin/pvfs2-server/sbin/pvfs2-client -p /usr/lo:cal/bin/pvfs2-server/sbin/pvfs2-client-core
  mount -t pvfs2 tcp://"mds_server_hostname":3334/pvfs2-fs /mnt/pvfs2
  
  // Now run ls, cat, etc in /mnt/pvfs2 to ensure you have successfully
  // created a pvfs2 file system.
  

Step 4: Export PVFS2 storage system with NFSv4

• Create /etc/exports file on MDS.

  /mnt/pvfs2  *(rw,sync,fsid=0,insecure,no_subtree_check)
  

• Run commands on MDS:

  rpc.mountd
  rpc.nfsd 8
  exportfs -r
  

Step 5: On clients, compile and install PVFS2 layout driver.

• Example:

  ./configure -with-kernel=/cvs/pnfs -prefix=/usr/local/bin/pvfs2-layout --disable-server
  make
  make install
  make KMOD_DIR=/usr/local/bin/pvfs2-layout kmod_install
  

Step 6: Verify you can mount exported PVFS2 file system via standard NFSv4

• On client:

  mount -t nfs4 "mds_server_name":/ /mnt/nfs4/
  

• If successful:

  umount /mnt/nfs4/
  

Step 7: Mount PVFS2 file system using pNFS

• Copy /etc/pvfs2tab file to every client
• Example:

  // On client
  insmod /usr/local/bin/pvfs2-layout/pvfs2-pnfs.ko
  /usr/local/bin/pvfs2-layout/sbin/pvfs2-client -p /usr/local/bin/pvfs2-layout/sbin/pvfs2-client-core
  mount -t nfs4 "mds_server_name":/ /mnt/nfs4/
  

System Configuration Diagram

Layout Driver Policies

Several /proc/sys/pvfs2-pnfs/ variables exist to control the behavior of the PVFS2 layout driver. They are:

• blocksize - Size (in bytes) of reads and writes to storage nodes.
• write/read_threshold - Threshold (in bytes) under which writes/reads will be sent to the NFSv4 server. Value is -1 to always use direct I/O through the layout driver.
• use_pagecache - Value of 0 to avoid using page cache, all writes/reads go straight to PVFS2 storage nodes (this is the default behavior of unmodified PVFS2). Value of 1 to use page cache. This enables PVFS2 with a data cache, readahead, and a writeback cache.

Debugging Help

• NFS debugging:

  echo 32767 > /proc/sys/sunrpc/nfsd_debug
  echo 32767 > /proc/sys/sunrpc/nfs_debug
  

• pNFS layout driver debugging:

  // Before layout driver is loaded
  export PNFS_DEBUGMASK=all
  
  // Uses log file /tmp/pnfs-client.log
  

• PVFS2 debugging, see PVFS2 website

Known Problems

• Sometimes interrupting an application while writing/reading will cause the application to hang.

Please help improve these instructions, email dhildebz@umich.edu