dCache: access to mass storage system (like CASTOR)
By using the dcache software, developed
jointly at DESY and
FERMILAB, it is possible to read and write files in a special area,
where files
are automatically copied to tape and deleted from disk once they are not
used anymore. If one accesses the file again, it is automatically copied
back from tape to disk. Files on tape cannot be removed.
In addition we have dCache disk-space assigned where files are always
kept on disk and are not copied to tape. It is for fast access to large
mDST, microDST and MC files.
Files on dCache should be typically a GByte. It is not for small files.
To access dCache files first make sure, that you have included module
load dcache in your .bashrc or .tcshrc file (or have set up the environment according
to the section Create your environment below).
For tape based long term files users can create directories here:
/grid/fzk.de/mounts/pnfs/compass/users/
(But please remember: Files on tape cannot be removed)
You can use standard commands to create and access the directories
e.g. ls, mkdir, rmdir, rm and to move single files
within dCache mv.
However, to access files special commands are required:
copy single files to and from dcache:
dccp -d 1 /tmp/FILE /grid/fzk.de/mounts/pnfs/compass/users/USERNAME/FILE
From root you can directly read or write dCache files, if you put
dcache: in front of the filename. Example:
dcache:/grid/fzk.de/mounts/pnfs/compass/users/heinsius/hist.root
Phast versions 7.023 and above have dcache support built in. Only for older
versions one needs to put dcache: in front of the filename.
Example for newer Phast:
phast -c /tmp/ -N 1000 -u 0 \
-h /grid/fzk.de/mounts/pnfs/compass/users/heinsius/hist.root \
/grid/fzk.de/mounts/pnfs/compass/data/mDST/2004/W23-2/mDST-35619-2-7.root
To export files you can use this procedure (wildcards in filename allowed):
module load dcache-preload
scp -P 22 filename-in-dcache node:.
scp -P 22 2nd-filename-in-dcache node:.
module unload dcache-preload
More information can be found here.
Software
Standard software is installed in the
/grid/fzk.de/mounts/nfs/software/compass
directory. The correct environment variables are set by the module
commands.
After running
source /grid/fzk.de/home/compass/sl4_test_setup
you can use the following commands (see also .bashrc below):
- module help usage information for the module package
- module list show used packages
- module avail show available/installed packages
- module load root load variables for root
- module display root show variables which are defined for root
- module switch root/4.03 root/4.04 change the root version
in the current environment
If other versions of root, cernlib, clhep, gcc etc. are required,
please send mail to
gridka-compass at lists.kit.edu.
Hints on compiling CORAL and PHAST can be found in the section "How to create your environment" below.
Partially obsolete information on the installation of COMPASS specitic software can be found here.
Batch system
The batch system is PBS. For more information check
the GridKa PBS page
Commands
qsub -q long -l max_test_run=1 job-script
qstat -au user-name
qdel job-id
To limit the maximum number of COMPASS jobs running in parallel
please always use the qsub option -l max_test_run=1
Otherwise the system will become unuseable for everybody.
One should wait a few seconds before submitting one job after another.
You can check the maximum number jobs with qstat -Q.
Queues
- short: 1 h
- long: 1-10h
- e-long: 10-48h
- test: for quick tests only. Do not submit more than one
test job here.
It is recommended to use the short and long queue.
LSF emulation
If you are used to lsf put the line "module load lsf" at the end of your
.bashrc file. You will get lsf like commands:
bjobs bkill bpeek bqueues bsub
Usage
The jobs
are distributed according to a "fair share" algorithm. Each day a new
fair share is calculated based on the usage of the last days and weighted
by the total usage of the last half year.
Currently COMPASS overuses its CPU-time share. So please think before
using a lot of CPU time.
Please note that the batch nodes are on a private network. I.e. in general no
access to machines outside fzk.de is possible.
COMPASS VO (virtual
organization)
First you have to apply for a certificate by your local authority. For
the procedure please check:
LCG user
registration. Users registered at CERN can get certificates here:
CERN certificate site.
The CERN
certificates will also grant you access to CERN web applications and
allow mail signing.
Next step is to add your identification to the COMPASS VO. The COMPASS VO
is maintained at a VOMS server at Gridka. Please access the following web
page:
VOMS
user creation
Before accessing the webpage make sure that your certificate is correctly
imported into the web browser. To create a certificate which can be
imported use the following command:
openssl pkcs12 -export -out usercert.p12 -name 'GridKa-Zertifikat' \\
-inkey userkey.pem -in usercert.pem
Family name and given name should be filled automatically, please check.
Please fill in addition the Institute, phone number and email (Email is
required). And select I have read and agree to the VO's Usage
Rules.
In case of certificate troubles you can go
here to check if you have correctly installed your certificate in your
web browser.
Create your environment
For setting an environment which should work under SL3 (on compass.fzk.de) and SL4 (on the login node gridka25 and on the worker nodes) you can put the line
source /grid/fzk.de/home/compass/sl4_test_setup
into your .bashrc (and get rid of everything else)
For SL3 it will load the following versions:
- cern/2004-sl3 (CERNLIB)
- shift/1.7.1.5-sl3 (libshift etc.)
- clhep/1.8.2.1-sl3 (CLHEP library)
- mysql/4-sl3 (mySQL headers)
- dcache/1.2.35 (dcap for accessing dCache)
- root/4.04
For SL4 it will use:
- root/5.14.00e
- dcache/1.2.42
- cern/2005
- clhep/1.9.3.1
- shift/2.1.4.10
- mysql/5.0.45
- utils/sl4 (e.g. mkdirhier)
There is a precompiled CORAL for SL4 in
/grid/fzk.de/mounts/nfs/software/compass/SL4_test/coral/prod-010707/coral
If you want to compile your own CORAL under SL4 on gridka25, please note that there is still something wrong about
version 2.1.4.10 the shift library and headers and you will get error messages ( undefined reference to rfio...)
The current workaround is to use shift/1.7.1.5-sl4:
module switch shift shift/1.7.1.5-sl4
Then CORAL should compile. For running it, however, you have to switch back:
module switch shift shift/2.1.4.10
Otherwise it will complain about missing libshift.so.2...
For compiling phast, using the option NO_RFIO=1 should be sufficient and no switching of shift versions should be necessary.
Last update 6.10.2008 by Andreas Mutter
