edu.isi.pegasus.planner.refiner
public class DeployWorkerPackage extends Engine
Modifier and Type | Field and Description |
---|---|
static String |
BASE_BUILD_DIRECTORY_URL
The base directory URL for the builds.
|
static String |
CLEANUP_PREFIX
Constant suffix for the names of the deployment nodes.
|
static String |
COMPLETE_TRANSFORMATION_NAME
The complete TC name for pegasus worker package.
|
static String |
COMPLETE_UNTAR_TRANSFORMATION_NAME
The complete TC name for untar.
|
static String |
DEFAULT_REFINER
The default transfer refiner name.
|
static String |
DEPLOY_WORKER_PREFIX
Constant suffix for the names of the deployment nodes.
|
static String |
DERIVATION_NAME
The logical name of the transformation for the worker package
|
static String |
DERIVATION_NAMESPACE
The derivation namespace for the worker package.
|
static String |
DERIVATION_VERSION
The version number for the derivations for worker package.
|
protected ADag |
mCurrentDag
It is a reference to the Concrete Dag so far.
|
protected Map<String,FileTransfer> |
mFTMap
The FileTransfer map indexed by site id.
|
protected String |
mJobPrefix
The job prefix that needs to be applied to the job file basenames.
|
protected Map<String,Boolean> |
mLocalTransfers
Map that indicates whether we need local setup transfer jobs for a site or
not.
|
private static Map<SysInfo.OS,String> |
mOSToNMIOSReleaseAndVersion
The map storing OS to corresponding NMI OS platforms.
|
private static Pattern |
mPattern
Stores compiled patterns at first use, quasi-Singleton.
|
private static String |
mRegexExpression
Store the regular expressions necessary to parse the basename from the worker
package url to retrieve the version of pegasus.
|
protected Implementation |
mSetupTransferImplementation
The transfer implementation to be used for staging in the data as part
of setup job.
|
protected Map<String,String> |
mSiteToPegasusHomeMap
Maps a site to the the directory where the pegasus worker package has
been untarred during workflow execution.
|
protected boolean |
mTransferWorkerPackage
Boolean indicating whether user wants the worker package to be transferred
or not.
|
protected String |
mUserSpecifiedSourceLocation
The user specified location from where to stage the worker packages.
|
protected boolean |
mUseUserSpecifiedSourceLocation
Boolean indicating whether to use the user specified location or not
|
protected boolean |
mWorkerNodeExecution
Boolean indicating worker node execution.
|
static String |
PACKAGE_NAME
The name of the package in which all the implementing classes are.
|
static String |
PATH_VALUE
The path to be set for create dir jobs.
|
static String |
PEGASUS_VERSION
The version of pegasus matching the planner.
|
static String[][] |
PEGASUS_WORKER_EXECUTABLES
Array storing the names of the executables in the $PEGASUS_HOME/bin directory
Associates the transformation name with the executable basenames
|
static String |
POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
The arguments for pegasus-exitcode when you only want the log files to be rotated.
|
static String |
REFINER_NAME
The name of the refiner for purposes of error logging
|
static String |
TRANSFORMATION_NAME
The logical name of the worker package
|
static String |
TRANSFORMATION_NAMESPACE
The transformation namespace for the worker package
|
static String |
TRANSFORMATION_VERSION
The version number for the worker package.
|
static String |
UNTAR_DERIVATION_NAME
The logical name of the transformation for the untar job.
|
static String |
UNTAR_DERIVATION_NAMESPACE
The derivation namespace for the untar job.
|
static String |
UNTAR_DERIVATION_VERSION
The version number for the derivations for untar job.
|
static String |
UNTAR_PREFIX
Constant suffix for the names of the deployment nodes.
|
static String |
UNTAR_TRANSFORMATION_NAME
The logical name of the worker package
|
static String |
UNTAR_TRANSFORMATION_NAMESPACE
The transformation namespace for the worker package
|
static String |
UNTAR_TRANSFORMATION_VERSION
The version number for the worker package.
|
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
Constructor and Description |
---|
DeployWorkerPackage(PegasusBag bag)
A pratically nothing constructor !
|
Modifier and Type | Method and Description |
---|---|
ADag |
addCleanupNodesForWorkerPackage(ADag dag)
Adds cleanup nodes in the workflow for sites specified.
|
private TransformationCatalogEntry |
addDefaultTCEntry(String site,
String pegasusHome,
SysInfo sysinfo,
boolean useFullPath,
String name,
String executable)
Returns a default TC entry to be used in case entry is not found in the
transformation catalog.
|
protected TransformationCatalogEntry |
addDefaultTCEntryForPegasusWebsite(String site,
String name)
Returns a default TC entry for the pegasus site.
|
ADag |
addSetupNodes(ADag dag)
Adds a setup node per execution site in the workflow that will stage the
worker node executables to the workdirectory on the sites the workflow
has been scheduled to.
|
private Graph |
addSetupNodesWithoutUntarNodes(ADag dag,
Set<String> deploymentSites)
Adds only the stage worker nodes to the workflow.
|
private Graph |
addSetupNodesWithUntarNodes(ADag dag,
Set<String> deploymentSites)
Adds untar nodes to the workflow, in addition to the stage worker nodes
|
protected String |
constructDefaultURLToPegasusWorkerPackage(String name,
SysInfo sysinfo)
Constructs the default URL's for the pegasus worker package.
|
private TransformationCatalogEntry |
defaultUntarTCEntry(SiteCatalogEntry site)
Returns a default TC entry to be used in case entry is not found in the
transformation catalog.
|
protected String |
getBasename(String url)
Returns the basename of the URL using substring.
|
protected String |
getCleanupJobname(ADag dag,
String site)
It returns the name of the untar job, that is to be assigned.
|
protected String |
getDeployJobName(ADag dag,
String site,
boolean localTransfer)
It returns the name of the deployment job, that is to be assigned.
|
protected Set |
getDeploymentSites(ADag dag)
Retrieves the sites for which the deployment jobs need to be created.
|
protected String |
getRootDirectoryNameForPegasus(String url)
Does regex magic to figure out the version of pegasus from the url, and
use it to construct the name of pegasus directory, when worker package
is untarred.
|
private String |
getStagingSite(String site)
Returns the staging site for a particular execution site.
|
protected String |
getUntarJobName(ADag dag,
String site)
It returns the name of the untar job, that is to be assigned.
|
void |
initialize(ADag scheduledDAG)
Initialize with the scheduled graph.
|
static DeployWorkerPackage |
loadDeployWorkerPackage(PegasusBag bag)
Loads the implementing class corresponding to the mode specified by the
user at runtime.
|
protected Job |
makeUntarJob(String site,
String jobName,
String wpBasename)
It creates a untar job , that untars the worker package that is staged
by the setup transfer job.
|
private static Map<SysInfo.OS,String> |
osToOSReleaseAndVersion()
Maps each to OS to a specific OS release for purposes of picking up the
correct worker package for a site.
|
boolean |
runTransferOnLocalSite(Refiner refiner,
String site,
String destinationURL,
int type)
Returns whether to run a transfer job on local site or not.
|
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
public static final String DEPLOY_WORKER_PREFIX
public static final String UNTAR_PREFIX
public static final String CLEANUP_PREFIX
public static final String POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
public static final String[][] PEGASUS_WORKER_EXECUTABLES
private static final String mRegexExpression
public static final String PATH_VALUE
public static final String DEFAULT_REFINER
public static final String TRANSFORMATION_NAMESPACE
public static final String TRANSFORMATION_NAME
public static final String TRANSFORMATION_VERSION
public static final String UNTAR_TRANSFORMATION_NAMESPACE
public static final String UNTAR_TRANSFORMATION_NAME
public static final String UNTAR_TRANSFORMATION_VERSION
public static final String COMPLETE_UNTAR_TRANSFORMATION_NAME
public static final String COMPLETE_TRANSFORMATION_NAME
public static final String DERIVATION_NAMESPACE
public static final String DERIVATION_NAME
public static final String DERIVATION_VERSION
public static final String UNTAR_DERIVATION_NAMESPACE
public static final String UNTAR_DERIVATION_NAME
public static final String UNTAR_DERIVATION_VERSION
public static final String PACKAGE_NAME
public static final String BASE_BUILD_DIRECTORY_URL
public static final String PEGASUS_VERSION
public static final String REFINER_NAME
private static Pattern mPattern
private static Map<SysInfo.OS,String> mOSToNMIOSReleaseAndVersion
protected ADag mCurrentDag
protected String mJobPrefix
protected Implementation mSetupTransferImplementation
protected Map<String,FileTransfer> mFTMap
protected Map<String,Boolean> mLocalTransfers
protected Map<String,String> mSiteToPegasusHomeMap
protected String mUserSpecifiedSourceLocation
protected boolean mUseUserSpecifiedSourceLocation
protected boolean mTransferWorkerPackage
protected boolean mWorkerNodeExecution
public DeployWorkerPackage(PegasusBag bag)
bag
- bag of initialization objectsprivate static Map<SysInfo.OS,String> osToOSReleaseAndVersion()
public static DeployWorkerPackage loadDeployWorkerPackage(PegasusBag bag) throws FactoryException
bag
- bag of initialization objectsFactoryException
- that nests any error that
might occur during the instantiation of the implementation.public void initialize(ADag scheduledDAG)
scheduledDAG
- the scheduled workflow.public boolean runTransferOnLocalSite(Refiner refiner, String site, String destinationURL, int type)
site
- the site handle associated with the destination URL.destURL
- the destination URLtype
- the type of transfer job for which the URL is being constructed.protected String getRootDirectoryNameForPegasus(String url)
url
- the url.public ADag addSetupNodes(ADag dag)
dag
- the scheduled workflow.private Graph addSetupNodesWithUntarNodes(ADag dag, Set<String> deploymentSites)
dag
- the dagdeploymentSites
- the sites for which the worker package has to be deployedprivate Graph addSetupNodesWithoutUntarNodes(ADag dag, Set<String> deploymentSites)
dag
- the dagdeploymentSites
- the sites for which the worker package has to be deployedpublic ADag addCleanupNodesForWorkerPackage(ADag dag)
dag
- the workflowprotected Set getDeploymentSites(ADag dag)
dag
- the dag on which the jobs need to execute.protected String getDeployJobName(ADag dag, String site, boolean localTransfer)
dag
- the workflow so far.site
- the execution pool for which the create directory job
is responsible.localTransfer
- whether the transfer needs to run locally or not.protected String getUntarJobName(ADag dag, String site)
dag
- the workflow so far.site
- the execution pool for which the create directory job
is responsible.protected String getCleanupJobname(ADag dag, String site)
dag
- the workflow so far.site
- the execution pool for which the create directory job
is responsible.protected Job makeUntarJob(String site, String jobName, String wpBasename)
site
- the execution pool for which the create dir job is to be
created.jobName
- the name that is to be assigned to the job.wpBasename
- the basename of the worker package that is staged to remote site.private TransformationCatalogEntry addDefaultTCEntry(String site, String pegasusHome, SysInfo sysinfo, boolean useFullPath, String name, String executable)
site
- the site for which the default entry is required.pegasusHome
- the path to deployed worker packagesysinfo
- the system information of that site.useFullPath
- boolean indicating whether to use just the basename or
the full pathname
- the logical name of the transformationexecutable
- the basename of the executableprotected TransformationCatalogEntry addDefaultTCEntryForPegasusWebsite(String site, String name)
site
- the execution site for which we need a matching static binary.name
- logical name of the transformationprotected String constructDefaultURLToPegasusWorkerPackage(String name, SysInfo sysinfo)
name
- the logical name of the executable, usually worker|binary.sysinfo
- the sysinfo for which the path is required.private TransformationCatalogEntry defaultUntarTCEntry(SiteCatalogEntry site)
site
- the site for which the default entry is required.protected String getBasename(String url)
url
-