edu.isi.pegasus.planner.code.generator.condor
public class CondorGenerator extends Abstract
Modifier and Type | Field and Description |
---|---|
static String |
CONDOR_DAGMAN_LOGICAL_NAME
The logical name with which to query the transformation catalog for the
condor_dagman executable, that ends up running the mini dag as one
job.
|
static String |
CONDOR_DAGMAN_NAMESPACE
The namespace to use for condor dagman.
|
static String |
DAGMAN_PROPERTIES_PREFIX
The prefix for DAGMan specific properties
|
static int |
DEFAULT_CHMOD_PRIORITY_KEY
The default priority key associated with chmod jobs.
|
static int |
DEFAULT_CLEANUP_PRIORITY_KEY
The default priority key associated with the cleanup jobs.
|
static int |
DEFAULT_CREATE_DIR_PRIORITY_KEY
The default priority key associated with the create dir jobs.
|
static int |
DEFAULT_INTER_SITE_PRIORITY_KEY
The default priority key associated with the inter site transfer jobs.
|
static String |
DEFAULT_PERIODIC_RELEASE_VALUE
Default value for the periodic_release for a job
|
static String |
DEFAULT_PERIODIC_REMOVE_VALUE
Default value for the periodic_remove for a job
|
static int |
DEFAULT_REPLICA_REG_PRIORITY_KEY
The default priority key associated with the replica registration jobs.
|
static int |
DEFAULT_STAGE_IN_PRIORITY_KEY
The default priority key associated with the stagein jobs.
|
static int |
DEFAULT_STAGE_OUT_PRIORITY_KEY
The default priority key associated with the stage out jobs.
|
static String |
DEFAULT_SUBDAG_CATEGORY_KEY
The default category for the sub dax jobs.
|
private boolean |
mAssignDefaultJobPriorities
Boolean indicating whether to assign job priorities or not.
|
protected ADag |
mConcreteWorkflow
The workflow for which the code has to be generated.
|
private long |
mCondorVersion
The long value of condor version.
|
protected PrintWriter |
mDagWriter
The file handle to the .dag file.
|
protected boolean |
mDone
A boolean indicating whether the files have been generated or not.
|
protected GridStartFactory |
mGridStartFactory
The handle to the GridStart Factory.
|
protected boolean |
mInitializeGridStart
A boolean indicating whether grid start has been initialized or not.
|
protected String |
mPoolClass
Specifies the implementing class for the pool interface.
|
static String |
mSeparator
The nice separator, define once, use often.
|
private SiteStore |
mSiteStore
The handle to the site catalog store.
|
protected CondorStyleFactory |
mStyleFactory
Handle to the Style factory, that is used for this workflow.
|
protected TransformationCatalog |
mTCHandle
Handle to the Transformation Catalog.
|
protected String |
mTempLogFile
The name of the log file in the /tmp directory
|
mBag, mLogger, mPOptions, mProps, mSubmitFileDir
VERSION
Constructor and Description |
---|
CondorGenerator()
The default constructor.
|
Modifier and Type | Method and Description |
---|---|
protected void |
applyStyle(Job job,
PrintWriter writer)
Applies a submit file style to the job, according to the fact whether
the job has to be submitted directly to condor or to a remote jobmanager
via CondorG and GRAM.
|
protected void |
concreteDagEmpty(String filename,
ADag dag)
A callout method that dictates what needs to be done in case the concrete
plan that is generated is empty.
|
protected Job |
constructDAGJob(String name,
String directory,
String dagBasename)
Constructs a job that plans and submits the partitioned workflow,
referred to by a Partition.
|
private Map |
constructMap(String propValue)
Constructs a map with the numbers/values to be passed in the RSL handle
for certain pools.
|
private TransformationCatalogEntry |
constructTCEntryFromEnvironment()
Returns a tranformation catalog entry object constructed from the environment
An entry is constructed if either of the following environment variables
are defined
1) CONDOR_HOME
2) CONDOR_LOCATION
CONDOR_HOME takes precedence over CONDOR_LOCATION
|
private TransformationCatalogEntry |
constructTCEntryFromEnvProfiles(ENV env)
Returns a tranformation catalog entry object constructed from the environment
An entry is constructed if either of the following environment variables
are defined
1) CONDOR_HOME
2) CONDOR_LOCATION
CONDOR_HOME takes precedence over CONDOR_LOCATION
|
private TransformationCatalogEntry |
defaultTCEntry(String site)
Returns a default TC entry to be used in case entry is not found in the
transformation catalog.
|
Collection<File> |
generateCode(ADag dag)
Generates the code for the concrete workflow in Condor DAGMAN and CondorG
input format.
|
void |
generateCode(ADag dag,
Job job)
Generates the code (condor submit file) for a single job.
|
protected boolean |
generateLogFileSymlink(String logFile,
String symlink)
This method generates a symlink to the actual log file written in the
local temp directory.
|
Map<String,String> |
getAdditionalBraindumpEntries(ADag workflow)
Returns a Map containing additional braindump entries that are specific
to a Code Generator.
|
protected String |
getBasename(String prefix,
String suffix)
A covenience method to construct the basename.
|
protected String |
getCategoryDAGManKnobs(PegasusProperties properties)
Write out the DAGMan knobs for each category the user mentions in
the properties.
|
protected String |
getCondorLogInSubmitDirectory()
Returns the path to the condor log file in the submit directory.
|
protected String |
getCondorLogInSubmitDirectory(ADag dag)
Returns the path to the condor log file in the submit directory.
|
protected String |
getCondorLogInTmpDirectory()
Returns the name of Condor log file in a tmp directory that is created
if generation of symlink for condor logs is turned on.
|
protected String |
getDAGMANOutFilename(ADag dag)
Returns the basename of the file, that contains the output of the
dagman while running the dag generated for the workflow.
|
protected int |
getJobPriority(Job job,
int depth)
Computes the priority for a job based on job type and depth in the workflow
|
private String |
gridstart(PrintWriter writer,
Job job,
boolean isGlobusJob)
This function creates the stdio handling with and without gridstart.
|
protected void |
handleCondorVarForJob(Job job)
It updates/adds the condor variables that are got through the Dax with
the values specified in the properties file, pool config file or adds some
variables internally.
|
protected void |
handleEnvVarForJob(Job sinfo)
It updates/adds the environment variables that are got through the Dax with
the values specified in the properties file, pool config file or adds some
variables internally.
|
protected void |
handleGlobusRSLForJob(Job sinfo)
It updates/adds the the Globus RSL parameters got through the dax that are
in Job object.
|
void |
initialize(PegasusBag bag)
Initializes the Code Generator implementation.
|
protected File |
initializeDagFileWriter(String filename,
DagInfo dinfo)
Initializes the file handler to the dag file and writes the header to it.
|
protected boolean |
isInteger(String value)
Returns a boolean indicating whether the value represented is an
integer or not.
|
void |
populatePeriodicReleaseAndRemoveInJob(Job job)
Populates the periodic release and remove values in the job.
|
protected void |
printDagString(String str)
Writes a string to the dag file.
|
void |
reset()
Resets the Code Generator implementation.
|
boolean |
startMonitoring()
Starts monitoring of the workflow by invoking a workflow monitor daemon
tailstatd.
|
protected void |
writeDagFileTail(DagInfo dinfo)
It writes the relations making up the DAG in the dag file and and closes
the file handle to it.
|
protected void |
writeDOTFile(String filename,
ADag dag)
Writes out the DOT file in the submit directory.
|
protected void |
writeJobMapFile(String filename,
ADag dag)
Writes out the job map file in the submit directory.
|
getDAGFilename, getDAGFilename, getFileBaseName, getWriter, writeOutBraindump, writeOutDAXReplicaStore, writeOutNotifications, writeOutStampedeEvents
public static final String DEFAULT_SUBDAG_CATEGORY_KEY
public static final String mSeparator
public static final String DEFAULT_PERIODIC_RELEASE_VALUE
public static final String DEFAULT_PERIODIC_REMOVE_VALUE
public static final String CONDOR_DAGMAN_NAMESPACE
public static final String CONDOR_DAGMAN_LOGICAL_NAME
public static final String DAGMAN_PROPERTIES_PREFIX
public static final int DEFAULT_STAGE_IN_PRIORITY_KEY
public static final int DEFAULT_INTER_SITE_PRIORITY_KEY
public static final int DEFAULT_CREATE_DIR_PRIORITY_KEY
public static final int DEFAULT_CHMOD_PRIORITY_KEY
public static final int DEFAULT_STAGE_OUT_PRIORITY_KEY
public static final int DEFAULT_REPLICA_REG_PRIORITY_KEY
public static final int DEFAULT_CLEANUP_PRIORITY_KEY
protected TransformationCatalog mTCHandle
private SiteStore mSiteStore
protected String mPoolClass
protected PrintWriter mDagWriter
protected String mTempLogFile
protected boolean mDone
protected ADag mConcreteWorkflow
protected CondorStyleFactory mStyleFactory
protected GridStartFactory mGridStartFactory
protected boolean mInitializeGridStart
private long mCondorVersion
private boolean mAssignDefaultJobPriorities
public void initialize(PegasusBag bag) throws CodeGeneratorException
initialize
in interface CodeGenerator
initialize
in class Abstract
bag
- the bag of initialization objects.CodeGeneratorException
- in case of any error occuring code generation.public Collection<File> generateCode(ADag dag) throws CodeGeneratorException
dag
- the concrete workflow.File
objects for the files written
out.CodeGeneratorException
- in case of any error occuring code generation.public void generateCode(ADag dag, Job job) throws CodeGeneratorException
dag
- the dag of which the job is a part of.job
- the Job
object holding the information about
that particular job.CodeGeneratorException
- in case of any error occuring code generation.public boolean startMonitoring() throws org.griphyn.vdl.euryale.VTorInUseException
startMonitoring
in interface CodeGenerator
startMonitoring
in class Abstract
org.griphyn.vdl.euryale.VTorInUseException
- in case the method is called before the
submit files have been generated.public void reset() throws CodeGeneratorException
reset
in interface CodeGenerator
reset
in class Abstract
CodeGeneratorException
- in case of any error occuring code generation.private Map constructMap(String propValue)
propValue
- the value of the property got from the properties file.protected Job constructDAGJob(String name, String directory, String dagBasename)
name
- the name to be assigned to the job.directory
- the submit directory where the submit files for the
partition should reside. this is where the dag file is
createddagBasename
- the basename of the dag file created.private TransformationCatalogEntry defaultTCEntry(String site)
site
- the site for which the default entry is required.private TransformationCatalogEntry constructTCEntryFromEnvironment()
private TransformationCatalogEntry constructTCEntryFromEnvProfiles(ENV env)
env
- the environment profiles.protected String getBasename(String prefix, String suffix)
prefix
- the first half of basenamesuffix
- the latter half of basenameprotected File initializeDagFileWriter(String filename, DagInfo dinfo) throws CodeGeneratorException
filename
- basename of dag file to be written.dinfo
- object containing daginfo of type DagInfo .CodeGeneratorException
- in case of any error occuring code generation.protected String getCategoryDAGManKnobs(PegasusProperties properties)
properties
- the pegasus propertiesprotected void writeDOTFile(String filename, ADag dag) throws CodeGeneratorException
filename
- basename of dot file to be written .dag
- the ADag
object.CodeGeneratorException
- in case of any error occuring code generation.protected void writeJobMapFile(String filename, ADag dag) throws CodeGeneratorException
filename
- basename of dot file to be written .dag
- the ADag
object.CodeGeneratorException
- in case of any error occuring code generation.protected void writeDagFileTail(DagInfo dinfo) throws CodeGeneratorException
dinfo
- object containing daginfo of type DagInfo.CodeGeneratorException
protected void printDagString(String str) throws CodeGeneratorException
str
- The String to be printed to the dag file.CodeGeneratorException
protected String getCondorLogInTmpDirectory()
protected String getCondorLogInSubmitDirectory()
dag
- the concrete workflow.protected String getCondorLogInSubmitDirectory(ADag dag)
dag
- the concrete workflow.public Map<String,String> getAdditionalBraindumpEntries(ADag workflow)
getAdditionalBraindumpEntries
in class Abstract
workflow
- the executable workflowprotected boolean generateLogFileSymlink(String logFile, String symlink)
logFile
- the full path to the log file.symlink
- the full path to the symlink.protected String getDAGMANOutFilename(ADag dag)
dag
- the DAG containing the concrete workflowprotected void concreteDagEmpty(String filename, ADag dag) throws CodeGeneratorException
filename
- Filename of the dag to be written of type String.dag
- the concrete dag that is empty.CodeGeneratorException
- in case of any error occuring code generation.protected void handleCondorVarForJob(Job job) throws CodeGeneratorException
job
- The Job object containing the information about the job.CodeGeneratorException
public void populatePeriodicReleaseAndRemoveInJob(Job job)
periodic_release = False periodic_remove = (JobStatus == 5) && ((CurrentTime - EnteredCurrentStatus) > 14400)where releasevalue is value of condor profile periodic_release and removevalue is value of condor profile periodic_remove
job
- the job object.protected boolean isInteger(String value)
value
- the String passedprotected void applyStyle(Job job, PrintWriter writer) throws CodeGeneratorException
job
- the job on which the style needs to be applied.writer
- the PrintWriter stream to the submit file for the job.CodeGeneratorException
- in case of any error occuring code generation.protected void handleEnvVarForJob(Job sinfo)
job
- The Job object containing the information about the job.protected void handleGlobusRSLForJob(Job sinfo)
job
- The Job object containing the information about the job.protected int getJobPriority(Job job, int depth)
job
- the job whose priority needs to be computeddepth
- the depth in the workflowprivate String gridstart(PrintWriter writer, Job job, boolean isGlobusJob) throws CodeGeneratorException
writer
- is an open stream for the Condor submit file.job
- is the job information structure.isGlobusJob
- is true
, if the job generated a
line universe = globus
, and thus runs remotely.
Set to false
, if the job runs on the submit
host in any way.CodeGeneratorException
- in case of any error occuring code generation.