edu.isi.pegasus.planner.refiner
public class ReplicaCatalogBridge extends Engine
Modifier and Type | Field and Description |
---|---|
static String |
CACHE_READ_ONLY_KEY
The name of the key that disables writing back to the cache file.
|
static String |
CACHE_REPLICA_CATALOG_IMPLEMENTER
The name of the Replica Catalog Implementer that serves as the source for
cache files.
|
static String |
CACHE_REPLICA_CATALOG_KEY
The name of the source key for Replica Catalog Implementer that serves as
cache
|
static String |
DEFAULT_REGISTRATION_CATEGORY_KEY
Default category for registration jobs
|
static String |
DIRECTORY_REPLICA_CATALOG_IMPLEMENTER
The name of the Replica Catalog Implementer that serves as the source for
cache files.
|
static String |
DIRECTORY_REPLICA_CATALOG_KEY
The name of the source key for Replica Catalog Implementer that serves as
cache
|
private ReplicaStore |
mCacheStore
The replica store in which we store all the results that are queried from
the cache replica catalogs.
|
private ADag |
mDag
The DAG being worked upon.
|
private ReplicaStore |
mDAXReplicaStore
The DAX Replica Store.
|
private boolean |
mDefaultTCRCCreated
A boolean indicating whether the attempt to create a default tc entry
has happened or not.
|
private TransformationCatalogEntry |
mDefaultTCRCEntry
The default tc entry.
|
private ReplicaStore |
mDirectoryReplicaStore
The replica store where we store all the results that are queried from
the input directory specified at the command line.
|
private ReplicaStore |
mInheritedReplicaStore
The inherited Replica Store
|
private boolean |
mRCDown
A boolean variable to desingnate whether the RLI queried was down or not.
|
private ReplicaCatalog |
mReplicaCatalog
The handle to the main Replica Catalog.
|
private ReplicaStore |
mReplicaStore
The replica store in which we store all the results that are queried from
the main replica catalog.
|
protected Set |
mSearchFiles
The Vector of
String objects containing the logical
filenames of the files whose locations are to be searched in the
Replica Catalog. |
private boolean |
mTreatCacheAsRC
A boolean indicating whether the cache file needs to be treated as a
replica catalog or not.
|
static String |
RC_DERIVATION_NAME
The derivation name for the transfer jobs.
|
static String |
RC_DERIVATION_NS
The derivation namespace for the transfer jobs.
|
static String |
RC_DERIVATION_VERSION
The version number for the derivations for registration jobs.
|
static String |
RC_TRANSFORMATION_NAME
The logical name of the transformation used.
|
static String |
RC_TRANSFORMATION_NS
The transformation namespace for the regostration jobs.
|
static String |
RC_TRANSFORMATION_VERSION
The logical name of the transformation used.
|
static String |
REPLICA_CATALOG_URL_KEY
The name of the URL key for the replica catalog impelementer to be picked
up.
|
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
Constructor and Description |
---|
ReplicaCatalogBridge(ADag dag,
PegasusBag bag)
The overloaded constructor.
|
Modifier and Type | Method and Description |
---|---|
protected void |
appendProperty(StringBuffer sb,
String key,
String value)
Appends a property to the StringBuffer, in the java command line format.
|
void |
closeConnection()
To close the connection to replica services.
|
private TransformationCatalogEntry |
defaultTCRCEntry()
Returns a default TC entry to be used in case entry is not found in the
transformation catalog.
|
void |
finalize()
Closes the connection to the rli.
|
private String |
ftToRC(FileTransfer ft)
Converts a
FileTransfer to a RC compatible string representation. |
private String |
generateMappingsFile(String regJob,
Collection files)
Generates the registration mappings in a text file that is generated in the
dax directory (the directory where all the condor submit files are
generated).
|
private String |
generateRepJobArgumentString(SiteCatalogEntry site,
String regJob,
Collection files)
Generates the argument string to be given to the replica registration job.
|
private Profile |
getClassPath(String home)
Returns the classpath for the default rc-client entry.
|
protected String |
getCommandLineProperties(PegasusProperties properties)
Returns the properties that need to be passed to the the rc-client
invocation on the command line .
|
String |
getDefaultRegistrationMaxJobsPropertyKey()
Returns the property key that can be used to set the max jobs for the
default category associated with the registration jobs.
|
ReplicaLocation |
getFileLocs(String lfn)
Returns all the locations as returned from the Replica Lookup Mechanism.
|
Set |
getFilesInReplica()
This returns the files for which mappings exist in the Replica Catalog.
|
private String |
getPathToLocalProxy()
Returns path to the local proxy
|
private ReplicaStore |
getReplicaStoreFromDirectory(String directory)
Loads the mappings from the input directory
|
private ReplicaStore |
getReplicaStoreFromFiles(Set files)
Ends up loading a Replica Store from replica catalog files
|
void |
initialize(ADag dag,
PegasusBag bag)
Intialises the refiner.
|
void |
initialize(ADag dag,
PegasusProperties properties,
PlannerOptions options)
Intialises the refiner.
|
private void |
loadCacheFiles(Set cacheFiles)
Ends up loading the cache files so as to enable the lookup for the transient
files created by the parent jobs.
|
private void |
loadInheritedReplicaStore(Set files)
Ends up loading the inherited replica files.
|
Job |
makeRCRegNode(String regJobName,
Job job,
Collection files)
It constructs the Job object for the registration node, which
registers the materialized files on the output pool in the RLS.
|
private ReplicaLocation |
retrieveFromCache(String lfn)
Retrieves a location from the cache table, that contains the contents
of the cache files specified at runtime.
|
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
public static final String DEFAULT_REGISTRATION_CATEGORY_KEY
public static final String RC_TRANSFORMATION_NS
public static final String RC_TRANSFORMATION_NAME
public static final String RC_TRANSFORMATION_VERSION
public static final String RC_DERIVATION_NS
public static final String RC_DERIVATION_NAME
public static final String RC_DERIVATION_VERSION
public static final String CACHE_REPLICA_CATALOG_IMPLEMENTER
public static final String CACHE_REPLICA_CATALOG_KEY
public static final String CACHE_READ_ONLY_KEY
public static final String DIRECTORY_REPLICA_CATALOG_IMPLEMENTER
public static final String DIRECTORY_REPLICA_CATALOG_KEY
public static final String REPLICA_CATALOG_URL_KEY
private ReplicaCatalog mReplicaCatalog
protected Set mSearchFiles
String
objects containing the logical
filenames of the files whose locations are to be searched in the
Replica Catalog.private boolean mRCDown
private ReplicaStore mReplicaStore
private ReplicaStore mCacheStore
private ReplicaStore mDirectoryReplicaStore
private ReplicaStore mDAXReplicaStore
private ReplicaStore mInheritedReplicaStore
private boolean mTreatCacheAsRC
private TransformationCatalogEntry mDefaultTCRCEntry
private boolean mDefaultTCRCCreated
private ADag mDag
public ReplicaCatalogBridge(ADag dag, PegasusBag bag)
dag
- the workflow that is being worked on.bag
- of initialization objects.public void initialize(ADag dag, PegasusBag bag)
dag
- the workflow that is being worked on.bag
- the bag of Pegasus initialization objectspublic void initialize(ADag dag, PegasusProperties properties, PlannerOptions options)
dag
- the workflow that is being worked on.properties
- the properties passed to the planner.options
- the options passed to the planner at runtime.public void closeConnection()
public void finalize()
public Set getFilesInReplica()
Set
of logical file names as String objects, for
which logical to physical mapping exists.mSearchFiles
public ReplicaLocation getFileLocs(String lfn)
lfn
- The name of the logical file whose PFN mappings are
required.org.griphyn.cPlanner.classes.ReplicaLocation
public String getDefaultRegistrationMaxJobsPropertyKey()
public Job makeRCRegNode(String regJobName, Job job, Collection files)
DagInfo
object.regJobName
- The name of the job which registers the files in the
Replica Location Service.job
- The job whose output files are to be registered in
the Replica Location Service.files
- Collection of FileTransfer
objects
containing the information about source and
destination URLs. The destination
URLs would be our PFNs.private TransformationCatalogEntry defaultTCRCEntry()
private Profile getClassPath(String home)
home
- the home directory where we need to check for lib directory.private String generateRepJobArgumentString(SiteCatalogEntry site, String regJob, Collection files)
site
- the SiteCatalogEntry
objectregJob
- The name of the registration job.files
- Collection of FileTransfer
objects containing the
information about source and destURLs. The destination
URLs would be our PFNs.protected String getCommandLineProperties(PegasusProperties properties)
properties
- the properties objectprotected void appendProperty(StringBuffer sb, String key, String value)
sb
- the StringBuffer to append the property to.key
- the property.value
- the property value.private String generateMappingsFile(String regJob, Collection files)
regJob
- The name of the registration job.files
- Collection of FileTransfer
objects containing the
information about source and destURLs. The destination
URLs would be our PFNs.private String ftToRC(FileTransfer ft)
FileTransfer
to a RC compatible string representation.ft
- the FileTransfer
objectprivate ReplicaLocation retrieveFromCache(String lfn)
lfn
- the logical name of the file.ReplicaLocation
object corresponding to the entry
if found, else null.private void loadInheritedReplicaStore(Set files)
files
- set of paths to the inherited replica files.private void loadCacheFiles(Set cacheFiles)
cacheFiles
- set of paths to the cache files.private ReplicaStore getReplicaStoreFromFiles(Set files)
files
- set of paths to the cache files.private ReplicaStore getReplicaStoreFromDirectory(String directory)
directory
- the directory to load fromprivate String getPathToLocalProxy()