File Systems on Eagle
User home directories as well as some system directories are
located in a Network File Service (NFS) that is accessible from
all nodes of Eagle.
Because the NFS servers are outside Eagle, you home directory does not provide
the fastest performance. For fast file access on Eagle, see
"Global Work Space", below.
Each user has a default storage limit of 500 MB in their home
directory. In addition to the ".yesterday" backup described
below, home directories in NFS are copied to tape backup four times a
week.
To find your quota and usage
in NFS, use the following command
$ lsquota
Filesystem Size Used Avail Use%
/autofs/spin/home/trey 1000 479 521 47
Each home directory has a default set of subdirectories:
| public |
This directory is world readable. Use it to make files available to
other CCS users. |
| private |
This directory is only accessible by the user. |
| .yesterday |
This directory contains a read-only copy of all the rest of the home
directory, including other subdirectories, as of the day before. The copy
is generated very early each morning. If you accidentally delete any of your
files, you can simply copy versions from the day before out of ".yesterday".
You cannot directly create or remove files from the ".yesterday" directory.
|
| bin |
This directory is a location for user-generated executables. It is
not in your "PATH" by default, however. You can add it or one if
its subdirectories to your "PATH" in your ".profile" or
".cshrc" file. |
| www |
Documents kept in this directory are available over the World-Wide
Web provided the permissions allow readability. These files can be accessed
from "http://www.ccs.ornl.gov/~user". |
Global work space is available on Eagle for temporary files used
across multiple nodes and for staging large files from and to HPSS. The space is not backed up, and it may be purged
to help ensure that adequate work space is available for new jobs. Files
that have not been accessed for more than a week are considered eligible
for purging.
The global work space on Eagle uses IBM's General Parallel File System
(GPFS). GPFS provides high bandwidth by striping files across multiple servers
and by using native communication over the SP Switch. The following
area is available through GPFS.
/tmp/work/$USER
All users have a directory in "/tmp/work" created for them
when their accounts are created. This area is a GPFS filesytem and is intended
as work space for Eagle applications. GPFS is not backed up, so
you need to archive any important output from GPFS to one of the other file
systems for permanent storage.
"/tmp/work" is purged on a regular basis to help ensure
that adequate work space is available for new jobs. Files that have not been
accessed for more than a week are considered eligible for purging.
In addition to old files being purged, each user is subject to a 100
GB maximum usage in "/tmp/work/$USER". The
100 GB should be thought of as a high water mark that you cannot exceed
on your biggest run even if there is additional space available in "/tmp/work".
If your research demands that you exceed 100 GB of work space,
you should contact "consult@ccs.orn.gov" prior to submitting the job, and space
will be made available to you in an alternate directory if possible.
For easier porting between Eagle and the systems at NERSC, we also define the environment variable
"$SCRATCH" to point to "/tmp/work/$USER".
For more information about GPFS, see the online documentation from IBM,
available at the following URL.
http://publib.boulder.ibm.com/clresctr/windows/public/gpfsbooks.html
The High-Performance Storage System (HPSS) provides archival
storage. It is "high performance" relative to other archival systems,
not relative to native file systems like GPFS. Large permanent files
should be moved directly from GPFS, presumably where they were
created, to HPSS.
You access HPSS through the "hsi" and "htar" interfaces, which
are available on all Eagle nodes.
Because they use DCE authentication, "hsi" and "htar" require
no password and can thus be used within batch scripts.
The CCS is moving to one-time passwords (OTPs) for authentication. Once this
occurs, "hsi" will require an OTP for each connection, so it will no
longer work within batch scripts. Please contact
"consult@ccs.ornl.gov" if you need password-free access to
"hsi" on Eagle.
For more information on HPSS and "hsi", type "hsi
help" on Eagle or see the online
documentation, available at the following URL.
For more information on "htar", see "man htar" on Eagle or see the online
documentation, available at the following URL.
HPSS is unavailable during maintenance, which typically
occurs Wednesday mornings, about 7-10 AM ET.
To assist users in managing their work during maintenance
activities, the utility "hsmq" has been developed to provide a way
of queuing HPSS requests. This utility is especially useful when you
would like to enter commands in your batch script to copy output files
to HPSS and want to be sure that the transfer takes place even if the
job ends on Wednesday morning when HPSS is down. See "man hsmq" for more information.
Though the "/tmp/work" directories are shared across
Eagle, "/tmp" and all its other subdirectories are local to each
node. The local storage on each node is very limited, and each uses
a single pair of mirrored disks. GPFS, however, is served from striped
arrays of disks. Therefore, for large files, GPFS is faster
than local "/tmp". For these reasons, we recommend the
use of "/tmp/work" instead of "/tmp".
If you do use "/tmp", we strongly urge you to remove all your
files at the end of each job. We purge
"/tmp" on a regular basis so that users
can expect roughly 2GB of "/tmp" space on each node at the
start of each job. This expectation is easier to meet if all users
clean up any "/tmp" use at the end of each job.
phoenix
| ram
| cheetah
| eagle
|
