Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 39 to 39 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
The possible job submission procedures are described in the next sections. | ||||||||
Line: 117 to 117 | ||||||||
If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.
The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.)
\ No newline at end of file | ||||||||
Added: | ||||||||
> > |
The backend conceptThe backend is the system which is actually taking care about your job submission. This section needs expansion...Graphical User InterfaceA simple graphical interface is written to help job flow monitoring. Start with:wmsx-gui.sh-- AndrasLaszlo - 10 Jul 2009 |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 39 to 39 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
The possible job submission procedures are described in the next sections. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 35 to 35 | ||||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < | The possible job submission procedures ars described in the next sections. | |||||||
> > | The possible job submission procedures are described in the next sections. | |||||||
Submission of single jobs with traditional JDL files | ||||||||
Line: 74 to 75 | ||||||||
| ||||||||
Changed: | ||||||||
< < | The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl , which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed, which shall be discussed in the followings. | |||||||
> > | The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.wjdl , which describes your job. If the COMMAND.wjdl is not present, default job specifications are assumed, which shall be discussed in the followings. | |||||||
The outputs of the jobs are written into generated directories under the out directory of the specified working directory of the provider.
WMSX JDL files | ||||||||
Changed: | ||||||||
< < | For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl , to customize the properties of your job. | |||||||
> > | For each COMMAND in the ArgList file, there may be a WMSX JDL file COMMAND.wjdl , to customize the properties of your job. | |||||||
The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:
| ||||||||
Line: 91 to 92 | ||||||||
| ||||||||
Changed: | ||||||||
< < | If the JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.). | |||||||
> > | If the WMSX JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.). | |||||||
In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension). |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 21 to 21 | ||||||||
After the start of the provider, directories log , debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. The files jobids.all , jobids.running , jobids.failed and jobids.done under the working directory contain the unique job identifiers of the sent Grid jobs. | ||||||||
Added: | ||||||||
> > | Note: On certain platforms, the file and directory names, given as command line arguments to wmsx-provider.sh or wmsx-requestor.sh , are only properly recognized when specified as absolute paths. | |||||||
RequestorRequestor is an application to submit commands to the Provider system for execution. Start with: |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 97 to 97 | ||||||||
In most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be called COMMAND_preexec and COMMAND_postexec . They must be executable. They will be run directly before submission and after job output is retrieved, respectively. | ||||||||
Changed: | ||||||||
< < | COMMAND_preexec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments. | |||||||
> > | COMMAND_preexec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framework), and with all the given arguments from ArgList as following arguments. | |||||||
COMMAND_postexec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framework), and with all the given arguments from ArgList as following arguments. | ||||||||
Changed: | ||||||||
< < | The retrieved outputs of your job will always be in the tarball out.tar.gz under the job output directory. | |||||||
> > | The retrieved outputs of your job will always be in the tarball out.tar.gz under the generated job output directory. | |||||||
If COMMAND_postexec returns with 0 nothing further happens. If returns with 1, the script COMMAND_chain is called, if present, which can be used to launch further jobs. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 33 to 33 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
The possible job submission procedures ars described in the next sections. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 12 to 12 | ||||||||
Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with: | ||||||||
Changed: | ||||||||
< < | wmsx-provider.sh -v myworkdir | |||||||
> > | wmsx-provider.sh myworkdir [-v] | |||||||
where: | ||||||||
Deleted: | ||||||||
< < |
| |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
After the start of the provider, directories log , debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. The files jobids.all , jobids.running , jobids.failed and jobids.done under the working directory contain the unique job identifiers of the sent Grid jobs. | ||||||||
Line: 29 to 29 | ||||||||
wmsx-requestor.sh -option | ||||||||
Changed: | ||||||||
< < | A complete list of options is available with -h. Some options are: | |||||||
> > | A complete list of options is available with -h . Some options are: | |||||||
| ||||||||
Line: 81 to 81 | ||||||||
For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl , to customize the properties of your job.
The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only: | ||||||||
Added: | ||||||||
> > |
| |||||||
| ||||||||
Deleted: | ||||||||
< < |
| |||||||
|
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 45 to 45 | ||||||||
Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is: | ||||||||
Changed: | ||||||||
< < | wmsx-requestor.sh -j example.jdl -o StdOutFile -r resultDir | |||||||
> > | wmsx-requestor.sh -j example.jdl -r resultDir [ -o StdOutFile ] | |||||||
Where the options are:
| ||||||||
Deleted: | ||||||||
< < |
| |||||||
| ||||||||
Added: | ||||||||
> > |
| |||||||
Automated mass submission of jobs via ArgList files |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSXBasic usage | ||||||||
Added: | ||||||||
> > | This is a program for mass job management on the Grid. | |||||||
There are two program you will be using: Provider and Requestor.
Provider | ||||||||
Line: 18 to 19 | ||||||||
| ||||||||
Changed: | ||||||||
< < | After the start of the provider, directories log , debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. | |||||||
> > | After the start of the provider, directories log , debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. The files jobids.all , jobids.running , jobids.failed and jobids.done under the working directory contain the unique job identifiers of the sent Grid jobs. | |||||||
Requestor |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 75 to 75 | ||||||||
The outputs of the jobs are written into generated directories under the out directory of the specified working directory of the provider. | ||||||||
Changed: | ||||||||
< < | WMSX JDL files. | |||||||
> > | WMSX JDL files | |||||||
For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl , to customize the properties of your job.
The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only: |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 18 to 18 | ||||||||
| ||||||||
Added: | ||||||||
> > | After the start of the provider, directories log , debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. | |||||||
RequestorRequestor is an application to submit commands to the Provider system for execution. Start with: | ||||||||
Line: 37 to 39 | ||||||||
The possible job submission procedures ars described in the next sections. | ||||||||
Changed: | ||||||||
< < | Traditional submission of single jobs with traditional JDL files | |||||||
> > | Submission of single jobs with traditional JDL files | |||||||
Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is: | ||||||||
Line: 71 to 73 | ||||||||
The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl , which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed, which shall be discussed in the followings. | ||||||||
Added: | ||||||||
> > | The outputs of the jobs are written into generated directories under the out directory of the specified working directory of the provider. | |||||||
WMSX JDL files.For eachCOMMAND in the ArgList file, there may be a JDL file COMMAND.jdl , to customize the properties of your job. |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 8 to 8 | ||||||||
Provider | ||||||||
Changed: | ||||||||
< < | Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with: | |||||||
> > | Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with: | |||||||
Changed: | ||||||||
< < | wmsx-provider.sh -v /tmp/myworkdir | |||||||
> > | wmsx-provider.sh -v myworkdir | |||||||
where: | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Requestor | ||||||||
Changed: | ||||||||
< < | Requestor is an application to submit commands to the Provider system for execution. Start with: | |||||||
> > | Requestor is an application to submit commands to the Provider system for execution. Start with: | |||||||
wmsx-requestor.sh -option | ||||||||
Line: 35 to 30 | ||||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
| ||||||||
Changed: | ||||||||
< < | Traditional Submission of Single Jobs with Traditional JDL Files | |||||||
> > | Traditional submission of single jobs with traditional JDL files | |||||||
Changed: | ||||||||
< < | Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is: | |||||||
> > | Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is: | |||||||
Changed: | ||||||||
< < | wmsx-requestor.sh -j example.jdl -o /tmp/StdOut -r /tmp/resultDir | |||||||
> > | wmsx-requestor.sh -j example.jdl -o StdOutFile -r resultDir | |||||||
Where the options are:
| ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Changed: | ||||||||
< < | Automated Mass Submission of Jobs via ArgList Files | |||||||
> > | Automated mass submission of jobs via ArgList files | |||||||
Changed: | ||||||||
< < | By using ArgList files, one can submit many independent jobs and handle their outputs in a simple and efficient way. | |||||||
> > | By using ArgList files, one can submit many independent jobs and handle their outputs in a simple and efficient way. | |||||||
Changed: | ||||||||
< < | The ArgList file can contain lines of the following format: | |||||||
> > | The ArgList file can contain lines of the following format: | |||||||
COMMAND parameters | ||||||||
Changed: | ||||||||
< < | where COMMAND refers to a job name, and parameters are the command
line arguments of the executable of your job. | |||||||
> > | where COMMAND refers to a job name, and parameters are the command line arguments of the executable of your job. | |||||||
To submit the jobs, use: | ||||||||
Changed: | ||||||||
< < |
nmaxjobs (optional).
The word COMMAND refers to a WMSX JDL file (not a traditional JDL file),
named COMMAND.jdl , which describes your job. If the COMMAND.jdl
is not present, default job specifications are assumed. | |||||||
> > |
COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl , which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed, which shall be discussed in the followings. | |||||||
WMSX JDL files. | ||||||||
Deleted: | ||||||||
< < | For each COMMAND in the ArgList file, there may be a JDL file
COMMAND.jdl , to customize the properties of your job.
The structure of a WMSX JDL file is similar to traditional JDL files,
however the supported variables are only:
"COMMAND.tar.gz" . (Must be of tar-gz format!)
"." means that they are not wrapped
in a directory. If not specified, defaults to "COMMAND" .
"COMMAND" .
"out" .
"AFS" checks for AFS presence. E.g.:
Software = {"AFS", "g++"}; . If not set, defaults to empty.
COMMAND.tar.gz etc.).
In the followings, things are more easily explained if the notion of
AbsCOMMAND is introduced:
this is simply the full path to the file COMMAND.jdl file (or if not
present, to the the COMMAND.tar.gz file), without the ".jdl" extension
(or without the ".tar.gz" extension, if JDL file is not present).
PreExec and PostExec ScriptsIn most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be calledCOMMAND_preexec and COMMAND_postexec . They must be executable.
They will be run directly before submission and after job output is retrieved,
respectively.
PreExec , if present, is automatically called by the framework
with the AbsCOMMAND as first argument,
and with all the given arguments from ArgList as following arguments.
PostExec , if present, is automatically called by the framework
with the AbsCOMMAND as first argument, the name of the job output
directory as second argument (automatically generated by the framwork),
and with all the given arguments from ArgList as following arguments.
The retrieved outputs of your job will always be in the tarball
OutputDirectory.tar.gz under the job output directory, where
OutputDirectory is what you specified in the JDL file (out by default).
If PostExec returns with 0 nothing further happens.
If PostExec returns with 1, the user script COMMAND_chain is called,
which can be used to launch further jobs.
Job ChainingThe running time of Grid jobs is limited. This is unavoidable for efficient control on resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state should be dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The scriptCOMMAND_chain is the tool to lauch further
jobs, when needed.
If the COMMAND_postexec script returns 1, the script COMMAND_chain
is invoked (must be executable). In this case,
if present, it is automatically called by the framework
with the AbsCOMMAND as first argument, the name
of the job output directory as second argument, and with all the given
arguments from ArgList as following arguments.
The output of COMMAND_chain is interpreted by the framework as ArgList
lines, just as if they were lines from the initial ArgList file.
Therefore, it can be used to lauch
further jobs by a finished job, depending on the decision of the
COMMAND_postexec script. This is called the job chaining.
(The COMMAND_chain may have multiple lines as output. Each line is
interpreted like a line from the ArgList file, so multiple jobs may
also be lauched: the job chain may also fork.) | |||||||
\ No newline at end of file | ||||||||
Added: | ||||||||
> > | For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl , to customize the properties of your job.
The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:
COMMAND.tar.gz etc.).
In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension).
Pre-execution and post-execution scriptsIn most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be calledCOMMAND_preexec and COMMAND_postexec . They must be executable. They will be run directly before submission and after job output is retrieved, respectively.
COMMAND_preexec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments.
COMMAND_postexec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framework), and with all the given arguments from ArgList as following arguments.
The retrieved outputs of your job will always be in the tarball out.tar.gz under the job output directory.
If COMMAND_postexec returns with 0 nothing further happens. If returns with 1, the script COMMAND_chain is called, if present, which can be used to launch further jobs.
Job chainingThe running time of Grid jobs is limited. This is unavoidable for efficient controling of resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter subsequent pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state is dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The scriptCOMMAND_chain is the tool to lauch further jobs, when needed.
If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.
The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.) |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
WMSX | ||||||||
Line: 8 to 8 | ||||||||
Provider | ||||||||
Changed: | ||||||||
< < | is the background process which does the actual work. There can be only one provider runnung on a machine per user. Start with | |||||||
> > | Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with: | |||||||
Changed: | ||||||||
< < | ./wmsx-provider -v /tmp/myworkdir | |||||||
> > | wmsx-provider.sh -v /tmp/myworkdir | |||||||
where: | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Requestor | ||||||||
Changed: | ||||||||
< < | is an application to submit commands to the system for execution. Start with: | |||||||
> > | Requestor is an application to submit commands to the Provider system for execution. Start with: | |||||||
Changed: | ||||||||
< < | ./wmsx-requestor -option | |||||||
> > | wmsx-requestor.sh -option | |||||||
A complete list of options is available with -h. Some options are: | ||||||||
Changed: | ||||||||
< < |
| |||||||
> > |
| |||||||
Changed: | ||||||||
< < | JDL Files | |||||||
> > | The possible job submission procedures ars described in the next sections. | |||||||
Changed: | ||||||||
< < | Can be sumbitted and managed through the framework. A sample usage is: | |||||||
> > | Traditional Submission of Single Jobs with Traditional JDL FilesSingle jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is: | |||||||
Changed: | ||||||||
< < | ./wmsx-requestor -j example.jdl -o /tmp/StdOut -r /tmp/resultDir | |||||||
> > | wmsx-requestor.sh -j example.jdl -o /tmp/StdOut -r /tmp/resultDir | |||||||
Where the options are: | ||||||||
Changed: | ||||||||
< < | ||||||||
> > |
| |||||||
Changed: | ||||||||
< < | ArgList Files | |||||||
> > | Automated Mass Submission of Jobs via ArgList Files | |||||||
Changed: | ||||||||
< < | Can specify multiple jobs to create one "larger" jpb. | |||||||
> > | By using ArgList files, one can submit many independent jobs and handle their outputs in a simple and efficient way. | |||||||
Changed: | ||||||||
< < | ArgList Format: | |||||||
> > | The ArgList file can contain lines of the following format: | |||||||
Added: | ||||||||
> > | ||||||||
COMMAND parameters | ||||||||
Added: | ||||||||
> > | ||||||||
Changed: | ||||||||
< < | ./wmsx-requestor options:
Custom JDL files.for Each command, there may be a file command.jdl, with custom JDL contents. sup ported are:
Pre/PostExecmust be calles COMMAND_preexec and COMMAND_postexec. MUST be executable. Will be run directly before submission and after job output is retrieved. Preexec is called with the name of the command and its full directory as first argument, and with all the given args as following arguments. Postexec is called with the name of the command and its full directory as first argument, , the name of the output directory as second argument, and with all the given args as following arguments. The output of your job will always be in a file "out.tar.gz" in the output directory. If postexec return with 0 nothing further happens. If postexec returns with 1, COMMAND_chain is called. | |||||||
> > | where COMMAND refers to a job name, and parameters are the command
line arguments of the executable of your job. | |||||||
Changed: | ||||||||
< < | Chaining | |||||||
> > | To submit the jobs, use: | |||||||
Changed: | ||||||||
< < | The _chain command's output is interpreted just as a line from the args file. It can therefore be used to start further jobs. | |||||||
> > |
nmaxjobs (optional).
The word COMMAND refers to a WMSX JDL file (not a traditional JDL file),
named COMMAND.jdl , which describes your job. If the COMMAND.jdl
is not present, default job specifications are assumed.
WMSX JDL files.For eachCOMMAND in the ArgList file, there may be a JDL file
COMMAND.jdl , to customize the properties of your job.
The structure of a WMSX JDL file is similar to traditional JDL files,
however the supported variables are only:
"COMMAND.tar.gz" . (Must be of tar-gz format!)
"." means that they are not wrapped
in a directory. If not specified, defaults to "COMMAND" .
"COMMAND" .
"out" .
"AFS" checks for AFS presence. E.g.:
Software = {"AFS", "g++"}; . If not set, defaults to empty.
COMMAND.tar.gz etc.).
In the followings, things are more easily explained if the notion of
AbsCOMMAND is introduced:
this is simply the full path to the file COMMAND.jdl file (or if not
present, to the the COMMAND.tar.gz file), without the ".jdl" extension
(or without the ".tar.gz" extension, if JDL file is not present).
PreExec and PostExec ScriptsIn most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be calledCOMMAND_preexec and COMMAND_postexec . They must be executable.
They will be run directly before submission and after job output is retrieved,
respectively.
PreExec , if present, is automatically called by the framework
with the AbsCOMMAND as first argument,
and with all the given arguments from ArgList as following arguments.
PostExec , if present, is automatically called by the framework
with the AbsCOMMAND as first argument, the name of the job output
directory as second argument (automatically generated by the framwork),
and with all the given arguments from ArgList as following arguments.
The retrieved outputs of your job will always be in the tarball
OutputDirectory.tar.gz under the job output directory, where
OutputDirectory is what you specified in the JDL file (out by default).
If PostExec returns with 0 nothing further happens.
If PostExec returns with 1, the user script COMMAND_chain is called,
which can be used to launch further jobs.
Job ChainingThe running time of Grid jobs is limited. This is unavoidable for efficient control on resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state should be dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The scriptCOMMAND_chain is the tool to lauch further
jobs, when needed.
If the COMMAND_postexec script returns 1, the script COMMAND_chain
is invoked (must be executable). In this case,
if present, it is automatically called by the framework
with the AbsCOMMAND as first argument, the name
of the job output directory as second argument, and with all the given
arguments from ArgList as following arguments.
The output of COMMAND_chain is interpreted by the framework as ArgList
lines, just as if they were lines from the initial ArgList file.
Therefore, it can be used to lauch
further jobs by a finished job, depending on the decision of the
COMMAND_postexec script. This is called the job chaining.
(The COMMAND_chain may have multiple lines as output. Each line is
interpreted like a line from the ArgList file, so multiple jobs may
also be lauched: the job chain may also fork.) | |||||||
\ No newline at end of file |
Line: 1 to 1 | ||||||||
---|---|---|---|---|---|---|---|---|
Added: | ||||||||
> > |
WMSXBasic usageThere are two program you will be using: Provider and Requestor.Provideris the background process which does the actual work. There can be only one provider runnung on a machine per user. Start with./wmsx-provider -v /tmp/myworkdirwhere:
Requestoris an application to submit commands to the system for execution. Start with:./wmsx-requestor -optionA complete list of options is available with -h. Some options are:
JDL FilesCan be sumbitted and managed through the framework. A sample usage is:./wmsx-requestor -j example.jdl -o /tmp/StdOut -r /tmp/resultDirWhere the options are:
ArgList FilesCan specify multiple jobs to create one "larger" jpb. ArgList Format: COMMAND parameters ./wmsx-requestor options:
Custom JDL files.for Each command, there may be a file command.jdl, with custom JDL contents. sup ported are:
Pre/PostExecmust be calles COMMAND_preexec and COMMAND_postexec. MUST be executable. Will be run directly before submission and after job output is retrieved. Preexec is called with the name of the command and its full directory as first argument, and with all the given args as following arguments. Postexec is called with the name of the command and its full directory as first argument, , the name of the output directory as second argument, and with all the given args as following arguments. The output of your job will always be in a file "out.tar.gz" in the output directory. If postexec return with 0 nothing further happens. If postexec returns with 1, COMMAND_chain is called.ChainingThe _chain command's output is interpreted just as a line from the args file. It can therefore be used to start further jobs. |