10 Mbps Advanced Packet Vault

Operations Guide

Center for Information Technology Integration
August 2001

Introduction

This document assists in configuring, compiling, installing, and running the 10 Mbps Advanced Packet Vault (APV), developed under Department of Justice Award No. 2000-DT-CX-K001. The reader is assumed to be familiar with OpenBSD and its components as well as administering a conventional OpenBSD system.

Project Overview

During the first year of our three-year project, we have developed a production 10 Mbps APV. The Vault writes all captured IP packets to long-term magnetic tape storage for later analysis and for evidentiary purposes. Performance and reliability have been achieved by using high-capacity commodity hardware, a secure and reliable open-source operating system, and mass storage. The goal is to permit the vault to archive all traffic found on a fully-loaded 10 Mbps network segment, and to permit the vault to run for extended periods of time without constant supervision. See Antonelli et al [ACF01] for details.

Hardware Environment

The APV implementation discussed herein has been developed and tested on a high-performance commodity Intel host platform running OpenBSD 2.9. The platform is composed of the following components:
 
Intel STL2 Server Board Dual 866 Mhz Pentium III processors, dual PCI (33 Mhz/32 bit, 66 Mhz/64 bit)
Adaptec AIC-7899 U160 (on-board STL2) Dual-port SCSI controller; provides separate Ultra160 and Ultra Wide SCSI channels
Intel EtherExpress Pro100+ (on-board STL2) 10/100 Mbps PCI Ethernet controller
512 MB RAM (on-board STL2) Two Kingston KVR133X72RC3/256-IS PC-133 256 MB memory modules 
Adaptec 29160 SCSI Controller single-channel Ultra 160 SCSI controller
35 GB SCSI Disk Seagate ST336704LWV Cheetah 35 GB SCSI3 disk
15 GB IDE Disk Western Digital WD153AA Caviar 15 GB 5400 RPM Fast ATA/Enhanced IDE compatible 
Qualstar Tape Library Qualstar TLS-8211 tape library subsystem with one HP Model 230 LTO tape drive, eleven media slots, a media changer, and an I/O port

The SCSI disk is connected to the Ultra 160 channel of the on-board SCSI controller, and the tape library changer and LTO tape drive are connected to the Adaptec 29160 SCSI controller. The SCSI disk is used exclusively for buffering packets before they are written to tape, storing packets retrieved from tape, and for the APV database; the IDE disk is used for all other permanenent storage. Not shown in the above table are standard components including a monitor, keyboard, mouse, and floppy and CD-ROM drives.

OpenBSD 2.9 currently does not utilize the second processor, but this is promised.

Software Environment

We recommend that you read the entire procedure before beginning any installation to get an overall understanding of the process.

Installing OpenBSD

Before making the APV kernel and system program modifications, you need to make sure that you are running a stable version of OpenBSD 2.9. We recommend upgrading to the latest OPENBSD_2_9 patch branch via Anonymous CVS; follow the procedures outlined in http://www.openbsd.org/stable.html to upgrade your OpenBSD kernel and system programs to the latest "stable" release.   To upgrade to OpenBSD versions greater than 2.9, please see the the OpenBSD upgrade-minifaq at http://www.openbsd.org/faq/upgrade-minifaq.html.  See also the general procedures outlined in the OpenBSD documentation http://www.openbsd.org/faq.

We recommend the following IDE disk partition sizes. The SCSI disk partition will be created later.
 
/ 1 GB
/tmp 1
/var 2
/usr 4
/home 5
swap 1

When choosing which installation packages to install, we recommend you select all packages; when asked if you ever plan to run X, we recommend you respond affirmatively.

After verifying correct system operation, you may proceed by making APV kernel and system program modifications; obtaining, building, and installing prerequisite packages; and compiling the APV user programs.
 

Unpacking the APV Source Distribution

Choose a directory into which the APV source distribution is to be unpacked.  For example, to unpack the distribution into directory /usr/apv type the following with root authority:

cd /usr
tar zxf apv10.tar.gz

In the remainder of this document, file and directory names beginning with apv are assumed to refer to the root of the APV source distribution (/usr/apv in this example).

The top level of the APV software distribution is organized as follows:
 
bin APV OpenBSD system program modifications.
bpf Utility routines for reading BPF-format data.
crypto Utility routines for encrypting and decrypting data.
crypto/rijndael Utility routines for AES.
decrypt User program to decrypt data from the vault.
des Utility routines for DES.
doc APV documentation and OpenBSD configuration files.
dump User program to encrypt data.
listen User program to read data from network.
pilot Scripts for running the APV.
sys APV OpenBSD kernel modifications
util Utility routines.

Building a modified OpenBSD Kernel

APV kernel code changes and additions follow the conventional OpenBSD directory structure and compilation methodology.
Installing APV Source Distribution Kernel Files
The APV source distribution contains a skeleton kernel source tree rooted at apv/sys. It includes modified kernel source files in directories apv/sys/net/ and apv/sys/scsi/. The following kernel files have been modified:
 
apv/sys/net/bpf.c
apv/sys/net/bpf.h
apv/sys/net/bpfdesc.h
Enable direct copy of BPF packets to disk, bypassing user space copies.
apv/sys/scsi/ch.c Enable bar- and ASC/ASCQ codes to be returned from Qualstar TLS-8211 changer in response to CHIOGSTATUS ioctl.
apv/sys/scsi/scsiconf.c Obtain correct response to SCSI inquiry of Qualstar TLS-8211 changer.

The APV source distribution contains a patch file apv/sys/kernel_patch that will apply the above changes.  Apply the patch by entering the following commands with root authority:

cd /usr/src/sys
patch <apv/sys/kernel_patch

Kernel Compilation Options
The following options must be enabled:
 
CITI_PACKET_VAULT Enable CITI BPF, SCSI, and changer modifications.
SCSI_2_DEF Required for tape library.
MFS Memory file system.

We had to disable the IPv6 options for our STL2 server board's on-board Ethernet controller to work. If you have a different motherboard, you may not need to disable these options, otherwise disable the following:
 
INET6 Disable IPv6 kernel support.
PULLDOWN_TEST Requires IPv6.

The following pseudo-devices must be disabled:
 
gif Requires IPv6.

The APV source distribution includes appropriately modified configuration files apv/sys/arch/i386/conf/VAULT4 and apv/sys/conf/GENERIC4, which should be copied to the corresponding places in the kernel build directory /usr/src/sys.

Building the Kernel
Use standard procedures to configure, check dependencies, and build a kernel; enter the following commands with root authority (NOTE: you'll probably want to keep a backup of a known good kernel. Just in case...):

cd /usr/src/sys/arch/i386/conf
config ./VAULT4
cd ../compile/VAULT4
make depend && make && cp bsd /

Later you will be asked to reboot your system to activate the new kernel.

Updating /usr/include

Two modified header files must be copied from the kernel build directory to /usr/include.  Issue the following commands with root authority:

cd /usr/src/sys/net
cp bpf.h bpfdesc.h /usr/include/net
 

Building modified OpenBSD System Utilities

The following utilities need to be modified and recompiled:
 
bin/chio Display tape bar- and ASC/ASCQ codes.

The APV source distribution contains a patch file apv/bin/user_patch that will apply the above changes.  Apply the patch and build and install a modified chio by entering the following commands with root authority:

cd /usr/src/bin/
patch <apv/bin/user_patch
cd chio
make
cp obj/chio /bin
 

Obtaining and Building Prerequisites

Before using the APV code, it is necessary to obtain and install several prerequisite packages. All are publicly available via the Web.
Berkeley DB
One of the APV user programs (pkt_dump) needs to maintain an in-memory database structure. This is implemented using the Berkeley DB database system from Sleepycat Software, an Open Source product. You can download the source from Sleepycat Software at http://www.sleepycat.com. We recommend you download and use version 3.2.9 of the software (available at the time of this writing at http://www.sleepycat.com/update/3.2.9/db-3.2.9.tar.gz), as the APV was built and tested with this version. Refer to the instructions in the downloaded code to build and install the library.
gnupg
The cryptographic organization of the APV includes a requirement for asymmetric encryption. This is implemented using the GNU Privacy Guard, which is freely available under the GPL. You can download a binary built as an OpenBSD 2.9 package from ftp://ftp.openbsd.org/pub/OpenBSD/2.9/packages/i386/gnupg-1.0.6.tgz. To install the package after downloading, issue the following command with root authority:

pkg_add gnupg-1.0.6.tgz
 

tcl
One of the APV scripts requires the TCL interpreter.  You can download a binary built as an OpenBSD 2.9 package from ftp://ftp.openbsd.org/pub/OpenBSD/2.9/packages/i386/tcl-8.3.2.tgz. To install the package after downloading, issue the following command with root authority:

pkg_add tcl-8.3.2.tgz

If you have the ports collection installed, you can install tcl just by issuing the following commands as root:

cd /usr/ports/lang/tcl/; make install

and making a symbolic link:

ln -s /usr/local/bin/tclsh8.3 /usr/local/bin/tclsh
 

gnutar
The default tar command that comes with OpenBSD does not properly return an error code when it has encountered an error. The gnu version of tar is currently available in the OpenBSD distributions area. You can download the src from ftp://ftp.openbsd.org/pub/OpenBSD/distfiles/tar-1.13.19.tar.gz. You need to untar, configure, and build the code as follows:

cd /usr/local/src
tar xzf tar-1.13.19.tar.gz
cd tar-1.13.19
./configure
make

After successfully building the program, run the following with root authority, which will install the gnu tar program as /usr/local/bin/tar.

make install

Compiling the APV User Programs

A Makefile is provided at the top level of the APV source distribution in the directory apv. Compile the APV user program components by issuing the following commands:

cd apv
make
 

Preparing to run the APV

Several steps are necessary to prepare an APV machine to run for the first time. All commands should be executed with root authority.
Set Up Directories and Filesystems
Use standard procedures to create a disk partition of minimum size 35 GB on the SCSI disk and add an appropriate entry to /etc/fstab referencing the new partition:

NOTE:  For host platforms whose components differ from ours (see the "Hardware Environment" section above) the name of the SCSI disk device may be differ from /dev/sd0a!

newfs /dev/sd0a
ed /etc/fstab
$a
/dev/sd0a /scratch0 ffs rw 1 2
.
w
q

Create directories in and links to the partition:

mkdir /scratch0
mount /scratch0
cd /scratch0
mkdir volumes retrieved fileDB

Create a directory /mfs, which the vault will use as a mount point for the mfs filesystem:

mkdir /mfs
 

Remove Daemons and Periodic Tasks
OpenBSD starts several programs that remain active while the system is running. Under certain conditions they could consume resources needed by the APV or serve as targets for attacks against the APV. Edit /etc/rc.conf and set the following entries to NO to turn them off:

sendmail_flags=NO
inetd=NO
lpd=NO

Later you will be asked to reboot your system to activate the above changes.

OpenBSD runs several tasks periodically, and they will consume resources needed by the APV. Issue the following command as root:

crontab -e

and delete the lines specifying daily, weekly, and monthly maintenance.

NOTE: it is important that these scripts be run periodically, especially /etc/daily. We recommend that these scripts be run manually, at a time when the APV is not running.

Ethernet Device Name
The Ethernet device name is defined as fxp0 in the APV software distribution. For host platforms whose components differ from ours (see the "Hardware Environment" section above) the name of the Ethernet device may be different, and will require a change to a script.

To determine the name of the network interface, use the command

netstat -in

to list the available network interfaces. Once this has been determined, change the apv/pilot/pilot.tcl script to use the new name:

ed apv/pilot/pilot.tcl
/set interface/
s/fxp0/newname/
w
q
 

Panic Action
We recommend configuring the system to reboot automatically on a kernel panic.  Enter the following commands with root authority:

ed /etc/sysctl.conf
/ddb.panic/
s/^/#/
w
q
 

Reboot System
Reboot your system to start the new kernel and remove the unneccessary daemons. Enter the following command with root authority:

reboot

Inspect the boot log by typing

dmesg

and examining the output for signs that the SCSI subsystem has recognized the tape library changer (ch0 device) and the library tape drive (st0 device).

If all is well, log in under the new kernel and issue the following command with root authority:

chio status

You should see output of the form

picker 0: 
slot 0: <ACCESS,FULL> [PVT=A0000002                        ]
slot 1: <ACCESS,FULL> [PVT=A0000004                        ]
slot 2: <ACCESS>
slot 3: <ACCESS>
slot 4: <ACCESS,EXCEPT,FULL> [PVT=] [ASC=0x83,ASCQ=0x09]
slot 5: <ACCESS>
slot 6: <ACCESS,FULL> [PVT=A0000000                        ]
slot 7: <ACCESS>
slot 8: <ACCESS>
slot 9: <ACCESS>
slot 10: <ACCESS>
portal 0: <INEAB,EXENAB,ACCESS>
drive 0: <FULL> [PVT=A0000003                        ]
The strings following "PVT=" are the bar-code labels of the corresponding tapes. The ASC/ASCQ codes for the tape in slot 4 indicate the tape has no bar-code. The ASC/ASCQ codes and their meanings are described in Section 14.3.4 of the TLS-6000 SCSI-2 Interface Manual [Qua01] available from Qualstar.
Load Tapes
Fill the tape library with fresh tapes to which unique bar-codes have been affixed, and prepare a supply of bar-coded tapes to be used to replenish the library while running.

Create Master Key

You will need to create a vault master key (actually, a public keypair), which is used in the cryptographic organization of the vault. Data may be retrieved en masse from the vault archives with the master key. Consequently, the master key is itself protected by a pass phrase that you choose at the time of generation. You should keep both the master key and the passphrase protected in a safe place; loss of either will prevent retrieval of any data from the vault archives.

The master key is identified by the User-ID you select for it (see below). It is important that you select a name that is distinct from that of all other vaults. For this reason, specify an email address using the fully-qualified domain name of the host platform.

Become root, and ensure that any previous keypairs are deleted:

/bin/rm -rf /root/.gnupg

Create the master key:

gpg
gpg --gen-key

The first invocation initializes things, the second creates tke keypair. When asked, select the default kind of key (DSA and ElGamal); the default key size (1024 bits); a key lifetime of 0 (does not expire); and for the key User-ID specify real name "apv10", email address "apv10@apv.your.domain.name, and comment "kv1". Next, choose a passphrase to protect the key. The passphrase will be required for retrieving vault data.

After the keys are generated, you can display some information about them using:

gpg --list-public-keys
gpg --list-secret-keys
 

Starting the APV

The APV code is divided into two subsystems. The listener subsystem reads packets, encrypts them, and stores them to disk in units of 1 GB volumes. The archiver subsystem reads volumes from the disk and writes them to magnetic tape. The subsystems are largely independent, and are started and stopped independently. To start the APV, first start the listener by issuing the following commands with root authority in a console or X window:

cd apv/pilot
./apvstart -l

Start the archiver by issuing the following commands with root authority in another window:

cd apv/pilot
./apvstart -a
 

Stopping the APV

The usual way of stopping the APV involves shutting down the listener subsystem while the archiver subsystem is left running. This is accomplished by typing Ctrl-C in the listener window. This shuts down the flow of packets into the APV while the archiver continues to write volumes to tape. The APV may be restarted at any time by restarting the listener subsystem:

cd apv/pilot
./apvstart -l

The archiver will suspend when there are no more volumes to write to the tape, and will resume when more volumes appear.

The archiver may be stopped by typing Ctrl-C in its window. The recommended way of stopping the archiver is to stop the listener first, waiting until all data have been written to tape, and then stopping the archiver. If the archiver is currently writing a volume to tape when it is requested to stop, it will not stop immediately but will finish writing the volume to the tape first. This process can take several minutes; the archiver will display progress messages in its window. At exit, the archiver unloads the tape from the drive and stores it in the library. Any volumes waiting to be written to tape will be written to a new tape when the archiver is next started.

To assist in determining whether a volume is being written or if any volumes are waiting to be written, execute the following commands with root authority:

cd apv/util
./apvsync status

A volume with status "FILLED" is waiting to be written to tape, but the write operation has not begun; status "DRAINING" indicates the volume is being written to tape.

Monitoring the APV

The startapv command starts four main processes: These processes write output to files in /var/log named, respectively, listen.log, pkt_dump.log, pilot.log, and archiver.log. Running tail -f on each of these files in four separate windows will give a good picture of the operation of the vault.  All log file entries are self-explanatory, with the exception of listen.log, whose entries are written one per segment and may be interpreted as follows:
 
 
dt Time over which this segment was collected (secs)
pr:t(+s) Packets read (t = total for this listener, s = total for segment)
br:t(+s) Bytes read (t = total for this listener, s = total for segment)
bs:t(+s) Bytes processed (t = total for this listener, s = total for segment)
pd:t(+s) Packets dropped (t = total for this listener, s = total for segment)
bd:t(+s) Bytes dropped (t = total for this listener, s = total for segment)
Bps Bytes/sec processed for this segment
Mbps Mbits/sec processed for thie segment.
mf MFS space free, MB

To make sure the logs are regularly rotated, lines should be added to the file /etc/newsyslog.conf; an example is provided in apv/doc/newsyslog.conf. See man newsyslog for more information.

It may also be interesting to watch the output of df /mfs and df /scratch0. In normal operation, the space used in each filesystem should be relatively stable. However if, for example, network traffic significantly exceeds 10 Mbps, /mfs will fill up as pkt_dump lags behind, and /scratch0 will fill up as archiver.pl lags behind.

Database

The APV maintains a flat-file database /scratch0/fileDB/firstgen describing the volumes written to tape. Each volume is described by a record written to the end of the file after it has been committed to tape. The record contains the following fields:
 
volid The volume ID of the volume.
tapeid The ID (bar-code number) of the tape on which the volume was written.
sequence# Sequence number of the volume on the tape (the first volume of the tape has sequence number one).
starttime Epoch at which the first packet of the first segment in the volume was returned by BPF.
endtime Epoch at which the last packet of the last segment in the volume was returned by BPF.
segments The number of segments stored in the volume.
lis_pkts Number of packets in the volume returned by BPF.
lis_pkts_drop Number of packets in the volume dropped by BPF.
lis_bytes Number of bytes in the volume read by BPF.
lis_bytes_snap Number of bytes in the volume returned by BPF.
lis_bytes_drop Number of bytes dropped from the volume by BPF.
lis_bytes_per_sec Bytes/sec processed by the listener for the volume, averaged over all segments.
dmp_pkts Number of packets read by the dumper for the volume.
dmp_pkts_written Number of packets written by the dumper to the volume.
dmp_bytes Number of bytes read by the dumper for the volume.
dmp_bytes_written Number of bytes written by the dumper to the volume.
dmp_bytes_per_sec Bytes/sec processed by the dumper for the volume, averaged over all segments.

Recovering Data from the APV

To recover data from the APV, determine from which tape(s) data are to be recovered. Manual inspection of the database /scratch0/fileDB/firstgen can, for example, determine the tapeid's of tapes that contain volumes gathered between epochs of interest. In the future an automated tool will be provided to assist in this task.

Data may be recovered from a tape by issuing the following commands with root authority:

cd apv/pilot
./retrieve.pl [--tapeid=id --firstvol=first --lastvol=last]

where tapeid is the ID from the label of the desired tape, firstvol is the sequence number of the first volume to be recovered (there will be at most 100 volumes per tape; the first volume has sequence number one), and lastvol is the number of the last volume. If the last two parameters are elided, all volumes on the tape are restored.

Alternatively, if the retrieve.pl command is given without the tapeid parameter, you will be prompted to enter the tape id and, optionally, the starting and ending volume numbers. When the specified tape has been processed, you will be prompted for another.

retrieve.pl will demand the passphrase used to encrypt the vault master key; this must be entered twice to prevent accidental mistyping. Terminal echo is turned off.

The specified tape will be loaded, and the specified volume(s) will be copied from tape, decrypted, and stored in directory /scratch0/retrieved. Each volume directory will consist of a number of decrypted segment packet files (named :n.d, where n denotes the epoch at which the last packet was returned by BPF for the segment, and d disambiguates files written at the same epoch) and segment descriptor files (named x:n.d). Each segment packet file is in tcpdump format and may be viewed with the command

tcpdump -r :n.d

The associated descriptor file is a text file giving more details about the data in the segment:
 
starttime Epoch at which the first packet was returned by BPF.
endtime Epoch at which the last packet was returned by BPF.
lis_pkts Number of packets returned by BPF.
lis_pkts_drop Number of packets dropped by BPF.
lis_bytes Number of bytes read by BPF.
lis_bytes_snap Number of bytes returned by BPF.
lis_bytes_drop Number of bytes dropped by BPF.
lis_bytes_per_sec Bytes/sec processed by the listener.
dmp_pkts Number of packets read by the dumper.
dmp_pkts_written Number of packets written by the dumper.
dmp_bytes Number of bytes read by the dumper.
dmp_bytes_written Number of bytes written by the dumper.
dmp_bytes_per_sec Bytes/sec processed by the dumper.

It will not be possible to use the APV for writing volumes to tape while it is being used to recover data, although the APV listener subsystem can be running concurrently. However, resource contention between the listener and the retrieval process could force the listener to stop.

Understanding and Recovering from APV Errors

Exceptional APV conditions and suggested recovery procedures are summarized below.

Packet Overrun

If the packet input rate exceeds 10 Mbps it is possible for the APV to drop packets. This will be indicated in the listener log file /var/log/listen.log by non-zero segment pd: (packets dropped) and bd: (bytes dropped) "+" entries, and by volume summary entries in the database. Such nonzero entries serve as evidence that the vault did not capture all packets on the network while collecting data for the specified segment, but vault operations are not interrupted by such overruns.

If such overruns are noticed, you may want to review the use of system resources (see "Remove Daemons and Periodic Tasks") to verify that periodic tasks or daemons are not the cause of lost packets.

MFS/UFS Exhaustion

The listener subsystem will stop if less than 32 MB are found to be available in MFS after a segment has been written there. This event is noted by a message in the listener log file /var/log/listen.log. This is usually caused by consistent heavy network traffic with which the packet dumper cannot keep up. The archiver subsystem continues to run, but the APV will no longer be collecting packets from the network.

In a similar vein, the listener subsystem will stop if less than 1 GB are found to be available in UFS. Again, this is noted in the log and the APV will stop collecting packets from the network. This may be caused by the archiver waiting for scratch tapes, or continuous heavy traffic with which the archiver cannot keep up.

If either of these conditions are encountered, more monitoring of the system resources may indicate where the problem lies. In either case, the listener subsystem may be restarted after the cause has been determined and the backlog has been alleviated.

Tape Exhaustion

If the archiver needs a scratch (blank) tape from the tape library in order to archive APV data, the archiver will request that new tapes be loaded via a message in /var/log/archiver.log and in the window where the archiver was started. Once fresh scratch tapes have been loaded into the library, hit enter in the archiver window to begin using them. While scratch tapes are not available, APV data will be buffered in UFS. If UFS space becomes exhausted before fresh tapes are added, the APV will stop collecting packets from the network as explained above.

Unexpected Failures

Unexpected failures in the listener, packet dumper, or archiver programs, as well as the scripts that drive them, cause an explanatory message to appear in the appropriate log file. The affected subsystem will stop. If the listener subsystem fails, the APV will stop collecting packets from the network. If the archiver subsystem fails, packets will be buffered in UFS. If UFS space becomes exhausted before the cause of the archiver subsystem has been identified and repaired, the APV will stop collecting packets from the network as explained above.

Host Platform Crashes

If the host platform experiences a kernel crash, OpenBSD will reboot automatically.  After the system has rebooted, log in and execute the following commands with root authority before restarting any APV components:

cd apv/pilot
./reconcile.pl

These commands will reconcile the state of the buffered volumes in UFS with the archiver.  When the archiver is next restarted, these buffered volumes will be written to tape.

Advanced configuration

A number of options may be passed to the vault at the time it is run to change its behavior. However, the APV may fail to operate reliably under some circumstances if it is used in other than the default configuration, so this is recommended only for testing purposes.

Examination of the apvstart script will show that the main job of apvstart -l is to run the shell script pilot.sh, which in turn runs pilot.tcl. Both pilot.sh and pilot.tcl accept a number of command-line options; run pilot.tcl --help to get a list. You can use any of these options by appending them to the line of apvstart in which pilot.sh is run.

In particular, command-line options can be used to choose any of three different file formats for the encrypted files that the vault produces: conversation format (the default), endpoint format, and the format used by the prototype vault; see [ACF01] for a description of the differences between these formats.

Resource requirements

On a fully saturated 10Mbps network, the listen script can produce a new 16-megabyte file in /mfs/raw about every one and a half seconds. These files are processed by pkt_dump, placed in UFS grouped together into 1 GB volumes, and written to disk by the archiver. We fit one hundred volumes on a tape, so a new tape can be written a little more often than once a day. Sufficient memory and disk space must be available to handle this flow of data reliably. In more detail:

Memory space

The system should have 512 MB of memory, with 256 MB devoted to MFS. The listener subsystem will stop (and packets will be dropped) if less than 32 MB are available in MFS. At a constant 10 Mbps load, pkt_dump is capable of avoiding a buildup of unencrypted data in MFS. At constant loads above 10 Mbps, however, MFS space eventually will be exhausted. MFS is designed to buffer short-term loads above 10 Mbps.

Disk space

UFS disk space
The system should have 35 GB of disk storage space for UFS. The listener subsystem will stop (and packets will be dropped) if less than 1 GB is available in UFS. At a constant 10 Mbps load, the archiver is capable of avoiding a buildup of volumes in UFS. At constant loads above 10 Mbps, however, UFS space eventually will be exhausted.

Conversation and endpoint modes add different amounts of extra data to each packet. In the worst case, pkt_dump in conversation mode might produce 102 MB in UFS per minute, and in endpoint mode might produce 135 MB per minute. However, the data won't be written to tape until an entire 1 GB volume is completed. The above memory and disk space recommendations will ensure there is enough space to buffer several volumes.

Database disk space
A database record is added for each volume. Each record is approximately 140 bytes. Assuming worst case from above, there should be a volume created every 7.5 minutes. At this rate, 192 volumes are created each day. This comes to approximately 27KB of database growth per day.
Log disk space
The logs produced by the packet vault should at worst use under 400 KB per hour, and their size is limited by newsyslog, which is run every hour by cron. Make sure the limits set in /etc/newsyslog.conf are reasonable; the sample version of this file provided in apv/doc/ limits the total size of the logs produced by the vault to 30 MB. (See also the "Monitoring the APV" section above.)

Tape consumption

In conversation mode, it is theoretically possible for the vault to fill up a tape every 16-17 hours; 12-13 hours are possible in endpoint mode. The library has eleven tape slots, so a manual procedure will suffice to clear written tapes and install fresh tapes into the library, if followed diligently. When the supply of fresh tapes runs out the archiver will display a message in its console window and in its log. Volumes will be buffered in UFS until fresh tapes are added. When UFS space drops below 1 GB, the listener will stop, and packets will be dropped.

To assist with tape management, use the following commands.

Tape Status
Use the following command to show the status of the tapes in the library:

cd apv/pilot
./tapestatus.pl

In the output of this command, "WRITTEN" means the tape has been written to and can be removed; "SCRATCH" denotes a fresh tape available for writing; "N/A" means an empty or unavailable slot.

Removing Filled Tapes
Use the following command to remove a tape from the Qualstar 8211 tape library:

chio move slot n portal 0

The tapestatus.pl output will identify the slot numbers, n, of tapes that have been written. Use chio to move each tape in turn to the I/O port; press and hold the "*" key on the library front panel and then press "MENU" to open the I/O port and eject the tape. Press the "*" and "MENU" key sequence again to close the port.

Ingesting Fresh Tapes
Use the following command to put a fresh tape into the Qualstar 8211 tape library:

chio move portal 0 slot n

where n is the number of an empty slot identified as "N/A" by the output of tapestatus.pl. For each fresh tape in turn press and hold the "*" key on the library front panel and then press "MENU" to open the I/O port, and insert the tape into the port, after which the port will close automatically. Use chio to move the tape from the I/O port to an available slot.

Bulk Moves
Rather than using the I/O port, you may open the front door of the Qualstar 8211 tape library and remove and insert the desired tapes en masse. When the door is closed the library changer will scan all slots. Concurrent tape drive operations should be unaffected. We recommend using the I/O port. NOTE: an alarm buzzer will sound when you reach into the library.

References

[ACF01] Charles J. Antonelli, Kevin Coffman, and J. Bruce Fields, "A 10 Mbps Advanced Packet Vault," Center for Information Technology Integration, University of Michigan, 2001 (in preparation).

[Qua01] Qualstar Corporation, "TLS-6000 SCSI-2 Interface Manual," 501205 Revision A. http://www.qualstar.com, under "Technical Services."