NCCS | User Info | search  

File Systems on Colt


Contents


Distributed File System

Colt will eventually use the Distributed Computing Environment (DCE) for user authentication. With DCE, all user information is centralized in the "registry", so each system in a DCE "cell" does not require a separate copy of each user's login information ("/etc/passwd"). Integrated with DCE is the Distributed File System (DFS). Though DCE is not yet available, DFS is accessible from all nodes of Colt through NFS mounts. Because of the NFS intermediation, users must take special steps to write to DFS.

When you first log in, and at regular (~monthly) intervals afterwards, you will need to issue the following commands on Colt. You should not have to issue these commands every time you log in (unless you only log in once a month).

$ kinit
Password for user@dce.ccs.ornl.gov: password
$ dfs_login
DCE password for user: password
...

Because the DFS servers are external to Colt, DFS does not provide the highest performance. For fast file access on Colt, see PFS, below.

All user home directories are kept in DFS, and each user has a default storage limit of 500 MB. In addition to the "yesterday" backup described below, home directories in DFS are copied to tape backup four times a week. Each home directory has a default set of subdirectories:

publicThis directory is world readable. Use it to make files available to other users of Colt.
privateThis directory is only accessible by the user.
yesterdayThis directory contains a read-only copy of all the rest of the home directory, including other subdirectories, as of the day before. The copy is generated at 4 AM each morning. If you accidentally delete any of your DFS files, you can simply copy versions from the day before out of "yesterday".
binThis directory is a default location for user-generated executables. It is not in your "PATH" by default, however. You can add it or one if its subdirectories to your "PATH" in your ".profile" or ".cshrc" file.
wwwIn the future, we plan to make all documents kept in this directory available over the World-Wide Web.

For more information on DCE and DFS, see online documentation from IBM, available at the following URL.

http://www-4.ibm.com/software/network/dce/library/publications/dceaix.html


Parallel File System

The Parallel File System (PFS) is a temporary-storage area with shared access from all Colt nodes. For large files, PFS is the highest-performance file system available on Colt. The PFS servers are nodes of Colt, and data transfer goes exclusively over the Quadrics Elan interconnect.

PFS is intended as work space for Colt applications. PFS is not backed up, so you need to copy any important output from PFS to one of the other file systems for permanent storage. Currently, there is only one PFS partition, "/pfs1". Additional partitions, with different topologies, are planned for the near future.


High-Performance Storage System

The High-Performance Storage System (HPSS) provides archival storage. It is "high performance" relative to other archival systems, not relative to native file systems like PFS. Large permanent files should be moved directly from PFS, presumably where they were created, to HPSS.

You access HPSS through the "hsi" interface. This interface is available on all AlphaServer nodes. When used with DCE authentication, "hsi" requires no password and can thus be used within batch scripts. DCE is not yet supported on Colt, however. (But neither is a batch system!)

For more information on HPSS and "hsi", type "hsi help" or see the online documentation kept at SDSC, available at the following URL.

http://www.sdsc.edu/Storage/hsi/


What about "/tmp"?

"/scratch" is scratch space local to each node for use by running jobs. "/tmp" is local to each node, but for system use only. The "/tmp" local storage on each node is very limited and used for system administration, so we strongly suggest that you do not use it. Use "/scratch" instead if you want local disk, and use "/pfs*" or DFS if you want disk available to all processes in a parallel job.


phoenix | ram | cheetah | eagle
ornl | nccs | ccs | computers | disclaimer

URL http://www.ccs.ornl.gov/colt/filesystems.html
Updated: Thursday, 10-Oct-2002 09:00:43 EDT
consult@ccs.ornl.gov