|
Phoenix Software: Languages, Libraries, Applications, and Tools
This page lists tools and software available on the Cray X1
system at ORNL. If you have any further questions or comments, please send
them to consult@ccs.ornl.gov.
Note: To find the list of current modules loaded by Cray
use: module avail.
A very good source of information
is CrayDoc which Cray updates
with each new Programming Environment release. Here you can find
information on the compilers, the open software, the scientific
libraries, and more.
For software installed by CCS,
look in directories of the form
/apps/<software_name>/prod/<machine>.
Here,machine is x1_mp2* where the x1 represents Cray's
processor and mp2* represents the OS.
- ftn,
- Cray Fortran compiler installed in /opt/PE/bin/ftn
For more information on the Fortran compiler, consider the following:
- cc
- Cray C compiler installed in /opt/PE/bin/cc
- CC
- Cray C++ compiler installed in the
/opt/PE/bin/CC.
For more information on the Cray C compilers, consider the following:
Perl is a high-level programming
language. It is primarily derived from the C programming language.
It has process, file and text manipulation facilities that make it
particularly well-suited for tasts involving quick prototyping, system
utilities, software tools, system management tasks, database access,
graphical programming and web programming. The Perl 5.6.1 interpreter is
installed in /opt/open/open/bin/perl.
For more information, consult the
Perl home page.
Tcl (Tool Control Language) is an
interpreted scripting language
that is mostly used for building graphical user interfaces.
Tk is a graphical toolkit that gives
one access to many of
the graphics primitives of the underlying windowing
system, as well as providing access to many types of widgets,
such as buttons, scrollbars, menus, and text boxes.
Versions 8.3 and 8.4 of Tcl and
Tk are installed on the Cray in /opt/open/open/lib/.
For more information, consult
Tcl/Tk Information.
There are also man pages on Tcl/Tk like man tk
The Advanced Computational Testing and Simulation (ACTS) Toolkit
is a set of DOE-developed software tools that make it easier for
programmers to write high performance scientific applications for
parallel computers.
NOTE: Most ACTS Toolkit routines are currently NOT installed on Phoenix.
The ones that are installed have shown very poor performance. Work is
underway to obtain better tuned versions.
We're expecting a Cray-version of PETSc in the
near future.
ACTS is divided into 4 areas in which there are several software tools
each. Currently, we only have some of the ACTS Toolkit installed. In
the list below, those tools listed with a link are installed.
- Numerical Tools
- Code Development Tools
- Global Arrays
- Overture
- POOMA
- Code Execution Tools
- CUMULVS
- Globus
- PAWS
- SILOON
- TAU
- Library Development Tools
- ATLAS & PHiPAC
- Nexus
- PADRE
- PETE
See
The ACTS Toolkit for more information.
AZTEC is a massively parallel iterative solver library for
solving sparse linear systems. Aztec is easy-to-use and efficient.
Its simplicity is attained through its use of a global distributed
matrix.
The Aztec library is installed on the Cray at
/opt/public/lib/libaztec.a. The Aztec header files are located
in /opt/public/include and named
az_aztec.h, az_aztec_defs.h, and az_aztecf.h.
For more information see the
AZTEC homepage
The Basic Linear Algebra Communication Subprograms (BLACS), it is a
library constructed by the ScaLAPACK team. It is intended for use in numerical
programming where communication involves two-dimensional portions of arrays.
BLACS is built on top of the MPI libraries.
The BLACS library is typically used when using the ScaLAPACK library.
Cray provides BLACS in its CrayLibSci.
For example, to compile and link your program (file.c of file.f) to the
BLACS libraries, use the following:
ftn -o file file.f -lsci
or
cc -o file file.c -lsci
For more information see the
BLACS Home Page.
BLAS,
the Basic Linear Algebra Subprograms, are portable and efficient routines
that perform vector and matrix operations. The Level 1 BLAS do
vector-vector operations, the Level 2 BLAS do matrix-vector operations,
and the Level 3 BLAS do matrix-matrix operations. Because of their efficiency,
BLAS routines are commonly used in the development of high-quality
linear-algebra software such as LAPACK and LINPACK.
CrayLibSci contains all the BLAS routines.
For example, to compile and link your program (file.c of file.f)
using the BLAS libraries, use the following:
ftn -o file file.f -lsci
or
cc -o file file.c -lsci
For more information see the
BLAS Home Page .
CrayLibSci is a library
of mathematical subroutines that have been highly optimized for the Cray family
of machines and so run much faster than equivalent routines from other
libraries.
On the Cray, the CrayLibSci is loaded with the PE module,
it is installed in /opt/ctl/craylibsci/craylibsci.
You can link to libsci.a simply with -lsci .
The CrayLibSci contains several sets of routines, including
the 3 levels of BLAS, the BLACS, LAPACK, ScaLAPCK, many FFTs, and a few other
math functions such as Bessel, Trigonometric and Erf functions.
At the Cray Documentation
website, look for "Man Page Collection: Scientific Library Routines."
The CraySciLib contains many Fast Fourier Transform routines,
for the FFTs of periodic and other symmetric sequences.
It includes complex, real, sine,
cosine, and quarter-wave transforms. When compiling your code
be sure to link with -lsci .
You can type "man -k fft" to find out about all the
available fft routines on Phoenix.
At the Cray Documentation
website, look for "Man Page Collection: Scientific Library Routines."
FFTPACK is a package of Fortran subprograms for the fast Fourier
transform of periodic and other symmetric sequences. It includes
complex, real, sine, cosine, and quarter-wave transforms. The library
is in /usr/local/lib and named libfftpack.a. Note
that ESSL also contains FFT routines.
For more information see the
FFTPACK home
page.
The FFTW routines, version 2.1.5 are available in
/opt/public/lib and the include files are in /opt/public/include.
If you wish to see the source or use the
3.0.1 version, you can go to /apps/FFTW/version/ia64_rh73/
For additional information see the
FFTW home page.
HDF stands for Hierarchical Data Format. It is designed to be a portable file format for scientific data management. The HDF software includes I/O libraries and tools for analyzing, visualizing, and converting scientific data.
HDF5 is available on phoenix as a module. Use "module avail" to see the available versions and builds. The command "module load" can be used to load the needed HDF module. By default the modules will set up an environment for building MSP mode executables.
Once an HDF module has been loaded the following examples can be used to compile and link your program (hdf5example.f90) to the HDF libraries.
MSP mode:
ftn -o hdf5example.x hdf5example.f90 -lz -lsz -lhdf5_fortran -lhdf5
SSP mode:
ftn -o fileexample.x fileexample.f90 -Ossp -p$FORTRAN_MODULE_PATH/ssp -lz -lsz -lhdf5_fortran -lhdf5
For more information see the
HDF home page.
LAPACK provides routines for solving systems of simultaneous
linear equations, least-squares solutions of linear systems of equations,
eigenvalue problems, and singular value problems. The associated matrix
factorizations (LU, Cholesky, QR, SVD, Schur, generalized Schur) are also
provided, as are related computations such as reordering of the Schur
factorizations and estimating condition numbers. Dense and banded matrices
are handled, but not general sparse matrices. In all areas, similar
functionality is provided for real and complex matrices, in both single
and double precision.
The LAPACK routines are built upon the BLAS routines. They are all
included within Cray's CrayLibSci library.
For example, you may compile your file.f file as
follows:
ftn -o file file.f -lsci
For more information on LAPACK, consult the
LAPACK Home Page.
At the Cray Documentation
website, look for "Man Page Collection: Scientific Library Routines."
The Fortran and C intrinsic library (libfi) consists of libraries
of tuned mathematical intrinsic functions. Be sure to compile your code
with -lfi .
For more information see
Cray Documentation and
find the latest document entitled "Man Page Collection: Fortran and
C/C++ Intrinsic Procedures".
The Message Passing Toolkit Libraries is a subroutine library
on Cray for MPI.
For more information see the
Cray Documentation
website, and look for "Cray Message Passing Toolkit Release Overview"
and "Man Page Collection: Message Passing Interface (MPI)." Also
consider MPI Standard
Network Common Data Form (NetCDF) is an interface for array-oriented
data access and a library that provides an implementation of the
interface.
Version 3.5 is installed. The NetCDF executable is
located at /opt/public/bin. The NetCDF libraries are
available in /opt/public/lib.
You can access the C include files using the
following compiler option:
-I/opt/public/include
See the /apps/netcdf/prod/x1* directories for all the various
builds done on the X1.
More information is available at
NetCDF Online Info
and with the online man pages for
netcdf, ncdump, ncgen, ncview
The Performance API (PAPI) is a standard application programming
interface for accessing hardware performance counters available on most
modern microprocessors. PAPI provides two interfaces to the underlying
counter hardware; a simple, high-level interface for the acquisition
of simple measurements and a fully programmable, low level interface
directed towards users with more sophisticated needs.
On phoenix, the PAPI library has been installed in
/apps/PAPI/prod/x1_msp,
and
/apps/PAPI/prod/x1_ssp.
For more information see the
PAPI Homepage
Parallel Basic Linear Algebra Subroutines (PBLAS) for distributed
memory MIMD computers. The PBLAS
routines are also in the ScaLAPACK library.
If you use ScaLAPACK from CraySciLib, then you will automatically
get the PBLAS in the same library. Be sure to compile your
code with
-lsci .
For more information see
ScaLAPACK is a library of high-performance linear algebra
routines for distributed-memory message-passing MIMD computers and
networks of workstations supporting PVM and/or MPI.
ScaLAPACK is included within the CrayLibSci library subroutines.
Be sure to link with -lsci when you compile your code.
See the
ScaLAPACK Home Page for more information.
SHMEM is a fairly complete implementation of the SHMEM API
popularized by Cray Research for their T3D/E systems.
For more information see the
Cray Documentation website,
and look for "Man Page Collection: Shared Memory Access (SHMEM)."
sprng provides both FORTRAN and C (also C++) interfaces for the use of
the parallel random number generators. The current version has all the
generators in one library.
SPRNG is located in /opt/public/lib/libsprng.a
Please consult the documentation and the interface at
SPRNG home.
coreinfo is a GNU-based component of gdb (see below),
which gives direct information on coredumps.
Use it with: coreinfo executable_name core_filename
gdb is a free, GNU general-purpose debugger modified for Cray
systems. For more information on gdb, type man gdb .
GDB is located in
/opt/gnu/bin/gdb.
The CrayPat performance analysis tool is the primary high-level tool
for identifying opportunities for optimization on the Cray X1 system.
CrayPat provides access to a wide variety of performance experiments
that measure how an executable program consumes resources while it is
running, as well as several different user interfaces that provide
access to the experiment and reporting functions.
Note: The CrayPat GUI interface, CrayPat PerfShell interface, and pat
command line interface are deferred. The remainder of this section
deals with the pat_build, pat_hwpc, and pat_report components of
CrayPat, which can be executed as stand-alone programs.
For more information, see the pat(1), pat_build(1), pat_hwpc(1), and
pat_report(1) man pages. Also see
Cray Documentation and
find the latest document entitled
"Optimizing Applications on the Cray X1 System."
TotalView is a source-level, X-windows GUI debugger for single-
and multi-process programs. It works with Fortran and Fortran 90, C,
and C++. It can be used for parallel programs that run under POE with
hand-coded MPI calls. It also has facilities for multi-process
thread-based parallel programs such as OpenMP.
NOTE: From the TotalView Release Overview, Installation Guide, and User's Guide
Addendum for Cray X1TM Systems manual. Section 1.5 Limitations:
TotalView currently does not use aprun to launch the executable to be debugged,
and it explicitly disables the auto-aprun feature. As a consequence, an application
launched from the TotalView command line will be placed on a system node, not
an application node, and it will share the node's processors with operating
system processes.
As a workaround you may use aprun to launch the application first, then use
TotalView to attach to the process after it is running.
The specific instructions on how to fire up totalview to debug mpi tasks that
are started by aprun are these:
1) submit a job using PBS, the same you have been doing all the time
2) use "psview" to identify the APID and the job executable name
after you see it's running
3) type "totalview", and a totalview window will pop up
4) go to "file" and click on "new program", another window
will pop up
5) put in the name of the job name shown in psview in the first box
6) put in the APID in the process id box
7) click local in the third box
8) click OK, and the debug session will start
In order to run a PARALLEL (interactive) job with totalview on the
Cray, you do this:
env CRAY_AUTO_APRUN_OPTIONS="-n3" totalview <program> [totalview
options] [-a <program options>]
Where obviously the number following the -n is the number of processors
you want.
The Totalview parallel debugger is located at
/opt/totalview/totalview/bin/totalview.
At the
Cray Documentation website,
see:
- Cray's TotalView 6.1 (May 2003)
- Cray's TotalView 6.1 - User's Guide Addendum (May 2003)
Also consider
Etnus TotalView docs
Hierarchical Storage Interface (hsi) is a friendly interface for the
users of HPSS. For more information on hsi, type "hsi help" on
one of the login nodes or click on the link below.
Other HSI resources
PBS is the "Portable Batch System" job scheduler used on the Cray system.
For additional PBS information, please use
|
