Difference: MassSe (1 vs. 13)

Installation

tar  -xzf  gstream.tar.gz
cd  gstream
./configure  [--prefix=some_installation_path]
make
make  install

export LCG_GFAL_VO=your_vo

setenv LCG_GFAL_VO your_vo

export DEST=your_favourite_storage_element1,your_storage_element2,...

setenv DEST your_favourite_storage_element1,your_storage_element2,...

export TMPDIR=/tmp

setenv TMPDIR /tmp

Mass file handling on the Grid (the GSTREAM library)

The GSTREAM library for read/write C++ streams to Storage Elements

export DEST=your_favourite_storage_element1,your_storage_element2,...

setenv DEST your_favourite_storage_element1,your_storage_element2,...

GSTREAM toolkit extensions for command line mass file manipulations

Some helper scripts are also shipped together with the GSTREAM library. The wrapper scripts lcg-cr.sh, lcg-rep.sh, lcg-cp.sh, lcg-del.sh provide wrappers around the commands lcg-cr, lcg-rep, lcg-cp, lcg-del, such that whole directory trees can also be handled. Wrappers around other lcg- commands may be also written based on them (in this case, we would be grateful to be shared with them). These scripts provide text outputs of the planned series of lfc- and lcg- commands. This text output can be passed to the script runplan.sh, which executes the command list. Every command is included in a checker loop: execution is given up after 5 unsuccessful tries. If any interruption occures (e.g. because of failure), the runplan.sh script can be started again, and will continue from the command, where it got interrupted.

Practical examples:

> lcg-cr.sh  /home/user_name/source  /grid/your_vo/user_name/destination  your_favourite_se  your_vo  > runplan.dat
> runplan.sh  runplan.dat

> lcg-rep.sh  /grid/your_vo/user_name/source  destination_se  your_vo  > runplan.dat
> runplan.sh  runplan.dat

> lcg-cp.sh  /grid/your_vo/user_name/source  /home/user_name/dest  your_vo  > runplan.dat
> runplan.sh  runplan.dat

> lcg-del.sh  /grid/your_vo/user_name/target  target_se  your_vo  > runplan.dat
> runplan.sh runplan.dat

The wrapper scripts may be improved in the future. We thank to Kálmán Kővári for this toolkit. There are also other useful scripts in the package. Please have a look a them.

-- AndrasLaszlo - 29 Jun 2008

A WMSX toolkit for command line mass file manipulations

A WMSX toolkit for read/write C++ streams to Storage Elements

setenv LCG_GFAL_VO your_vo

-- AndrasLaszlo - 17 Sep 2007

Mass file handling on the Grid (a WMSX extension)

• A WMSX toolkit for command line mass file manipulations.

The wrapper scripts lcg-cr.sh, lcg-rep.sh, lcg-cp.sh, lcg-del.sh provide wrappers around the commands lcg-cr, lcg-rep, lcg-cp, lcg-del, such that whole directory trees can also be handled. Wrappers around other lcg- commands may be also written based on them (in this case, we would be grateful to be shared with them). These scripts provide text outputs of the planned series of lfc- and lcg- commands. This text output can be passed to the script runplan.sh, which executes the command list. Every command is included in a checker loop: execution is given up after 5 unsuccessful tries. If any interruption occures (e.g. because of failure), the runplan.sh script can be started again, and will continue from the command, where it got interrupted.

Practical examples:

> lcg-cr.sh  /home/user_name/source  /grid/your_vo/user_name/destination  your_favourite_se  your_vo  > runplan.dat
> runplan.sh  runplan.dat

> lcg-rep.sh  /grid/your_vo/user_name/source  destination_se  your_vo  > runplan.dat
> runplan.sh  runplan.dat

> lcg-cp.sh  /grid/your_vo/user_name/source  /home/user_name/dest  your_vo  > runplan.dat
> runplan.sh  runplan.dat

> lcg-del.sh  /grid/your_vo/user_name/target  target_se  your_vo  > runplan.dat
> runplan.sh runplan.dat

The wrapper scripts may be improved in the future.

• A WMSX toolkit for read/write C++ streams to Storage Elements.

As one does generally not want to always stage out the data files from _SE_-s onto a local disk by hand, and then process it, it is recommended to have read/write streams. The C++ library, shipped with WMSX implements such a library. (gstream, igstream, ogstream stream classes, like the usual C++ STL fstream, ifstream, ofstream file input/output stream classes; the letter 'g' standing for 'grid'). It does nothing else, but treats the file as a normal file, unless its name begins with the string /grid/. In this case, it stages out the datafile in question onto a local (or AFS) area, and then treats the local file as a normal file. Sooner or later this solution has to be replaced with a GFAL based C++ library.

One commonly faces the problem that the file not only has to be processed, but it also has to be passed through a filter programme. Therefore I also wrote pipe streams for grid storage (igpstream, ogpstream), which are based on the ipstream and opstream classes of the library at http://pstreams.sourceforge.net (note the LGPL license!).

Practical examples:

#include "gstream.h"

int main(int argc, char *argv[])
{
    // Open the datafile for reading.
    igstream igfile("/grid/cms/alaszlo/some_datafile.dat");
        // Extract data from your datafile with 'igstream::operator>>' or with 'igstream::read(char*, int)'.
    // Close the datafile.
    igfile.close();

    // Open the datafile for writing.
    ogstream ogfile("/grid/cms/alaszlo/some_datafile.dat");
        // Write data to your datafile with 'ogstream::operator<<' or with 'ogstream::write(char*, int)'.
    // Close the datafile.
    ogfile.close();

    // Open the datafile for reading, through a filter program.
    igpstream igpfile("/grid/cms/alaszlo/some_datafile.dat.gz", "gunzip --stdout %f");
        // Extract data from your datafile with 'igpstream::operator>>' or with 'igpstream::read(char*, int)'.
    // Close the datafile.
    igpfile.close();

    // Open the datafile for writing, through a filter program.
    ogpstream ogpfile("/grid/cms/alaszlo/some_datafile.dat.gz", "gzip - > %f");
        // Write data to your datafile with 'ogpstream::operator<<' or with 'ogpstream::write(char*, int)'.
    // Close the datafile.
    ogpfile.close();

    return 0;
}

Before usage, one has to export the following environmental variables:

export LCG_GFAL_VO=your_vo

setenv :CG_GFAL_VO your_vo

export DEST=your_favourite_storage_element

setenv DEST your_favourite_storage_element

Setting the environmental variable TMPDIR is optional. This specifies the local (or AFS) directory, where the datafiles are staged out (therefore, it has to have large disk space!). E.g.:

export TMPDIR=/tmp

setenv TMPDIR /tmp

If not specified, the current working directory ($PWD) is used, as this is recommended for grid jobs (the working nodes have large disk spaces).