File Systems on Cheetah
Contents
User home directories as well as some system directories are
located in a Network File Service (NFS) that is accessible from
all nodes of Cheetah.
Because the NFS servers are outside Cheetah, you home directory does not provide
the fastest performance. For fast file access on Cheetah, see
"Global Work Space", below.
Each user has a default storage limit of 500 MB in their home
directory. In addition to the ".yesterday" backup described
below, home directories in NFS are copied to tape backup four times a
week.
To find your quota and usage
in NFS, use the following command
$ lsquota
Filesystem Size Used Avail Use%
/autofs/spin/home/trey 1000 479 521 47
Each home directory has a default set of subdirectories:
| public |
This directory is world readable. Use it to make files available to
other CCS users. |
| private |
This directory is only accessible by the user. |
| .yesterday |
This directory contains a read-only copy of all the rest of the home
directory, including other subdirectories, as of the day before. The copy
is generated very early each morning. If you accidentally delete any of your
files, you can simply copy versions from the day before out of ".yesterday".
You cannot directly create or remove files from the ".yesterday" directory.
|
| bin |
This directory is a location for user-generated executables. It is
not in your "PATH" by default, however. You can add it or one if
its subdirectories to your "PATH" in your ".profile" or
".cshrc" file. |
| www |
Documents kept in this directory are available over the World-Wide
Web provided the permissions allow readability. These files can be accessed
from "http://www.ccs.ornl.gov/~user". |
A small number of Cheetah 32-way nodes have 160GB of free, local disk space, where the
local space on one node is not directly accessible from any other node.
Some nodes do not have any local scratch space. See the Loadleveler page for information on how to get at the
local scratch space.
Files may only reside in local space for the life of a job. When a
job exits, any files remaining in it will be purged. Permanent files should
be moved to HPSS or staged to HPSS through the global work space. Also, any files that must be available
to multiple nodes during your job cannot reside in local space; use global work space instead.
An environment variable, "$NODE_JOBDIR", is defined at the
beginning of each job, pointing to areas in scratch space on each node of
the job. Though the value of "$NODE_JOBDIR" is the same for all the
nodes of a particular job, it points to a different space on each node.
See the Cheetah LoadLeveler page for more information.
Do not use "/tmp" directly for temporary storage of
files. When "/tmp" fills up, the node becomes unusable.
Global work space is available on Cheetah for temporary files used
across multiple nodes and for staging large files from and to HPSS. The space is not backed up, and it may be purged
to help ensure that adequate work space is available for new jobs. Files
that have not been accessed for more than a week are considered eligible
for purging.
The global work space on Cheetah uses IBM's General Parallel File System
(GPFS). GPFS provides high bandwidth by striping files across multiple servers
and by using native communication over the High-Performance Switch. The following
area is available through GPFS.
/tmp/work/$USER
All users have a directory in "/tmp/work" created for them
when their accounts are created. This area is a GPFS filesytem and is intended
as work space for Cheetah applications. GPFS is not backed up, so
you need to archive any important output from GPFS to one of the other file
systems for permanent storage.
"/tmp/work" is purged on a regular basis to help ensure
that adequate work space is available for new jobs. Files that have not been
accessed for more than a week are considered eligible for purging.
In addition to old files being purged, each user is subject to a 150
GB maximum usage in "/tmp/work/$USER". The
150 GB should be thought of as a high water mark that you cannot exceed
on your biggest run even if there is additional space available in "/tmp/work".
If your research demands that you exceed 150 GB of work space,
you should contact "consult@ccs.orn.gov" prior to submitting the job, and space
will be made available to you in an alternate directory if possible.
For easier porting between Cheetah and the systems at NERSC, we also define the environment variable
"$SCRATCH" to point to "/tmp/work/$USER".
For more information about GPFS, see the online documentation from IBM,
available at the following URL.
http://publib.boulder.ibm.com/clresctr/windows/public/gpfsbooks.html
The High-Performance Storage System (HPSS) provides archival
storage. It is "high performance" relative to other archival systems, not
relative to native file systems like GPFS. Large permanent files should be
moved directly from GPFS, presumably where they were created, to HPSS.
You access HPSS through the "hsi" and "htar"
interfaces, which are available
on all Cheetah nodes. Because they use DCE authentication,
"hsi" and "htar"
require no password and can thus be used within batch scripts.
The CCS is moving to one-time passwords (OTPs) for authentication. Once this
occurs, "hsi" will require an OTP for each connection, so it will no
longer work within batch scripts. Please contact
"consult@ccs.ornl.gov" if you need password-free access to
"hsi" on Cheetah.
For more information on HPSS and "hsi", type "hsi help"
on Cheetah or see the online documentation, available at the following URL.
For more information on "htar", see "man htar" on
Cheetah or see the online documentation, available at the
following URL.
http://www.ccs.ornl.gov/Storage/htar
HPSS is unavailable during maintenance, which typically occurs Wednesday
mornings, about 7-10 AM ET.
To assist users in managing their work during maintenance activities,
the utility "hsmq" has been developed to provide a way of queuing HPSS requests.
This utility is especially useful when you would like to enter commands
in your batch script to copy output files to HPSS and want to be sure that
the transfer takes place even if the job ends when HPSS is down. See
"man hsmq" on Cheetah for more information.
phoenix
| ram
| cheetah
| eagle
|
