edu.isi.pegasus.planner.cluster
Modifier and Type | Class and Description |
---|---|
private static class |
Horizontal.JobComparator
A job comparator, that allows me to compare jobs according to the
transformation names.
|
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_COLLAPSE_FACTOR
The default collapse factor for collapsing jobs with same logical name
scheduled onto the same execution pool.
|
static String |
DESCRIPTION
A short description about the partitioner.
|
private Map |
mCollapseMap
Map to hold the collapse values for the various execution pools.
|
protected JobAggregatorInstanceFactory |
mJobAggregatorFactory
The handle to the job aggregator factory.
|
private static Comparator |
mJobComparator
A singleton access to the job comparator.
|
private Map |
mJobMap
Map to hold the jobs sorted by the label of jobs in dax.
|
protected LogManager |
mLogger
The handle to the logger object.
|
private PPS |
mPPS
The handle to the provenance store implementation.
|
protected PegasusProperties |
mProps
The handle to the properties object holding all the properties.
|
private Map |
mReplacementTable
Replacement table, that identifies the corresponding fat job for a job.
|
private ADag |
mScheduledDAG
ADag object containing the jobs that have been scheduled by the site
selector.
|
private Map |
mSubInfoMap
A Map to store all the job(Job) objects indexed by their logical ID found in
the dax.
|
private XMLProducer |
mXMLStore
The XML Producer object that records the actions.
|
Constructor and Description |
---|
Horizontal()
The default constructor.
|
Modifier and Type | Method and Description |
---|---|
protected void |
appendAttribute(StringBuffer xmlFeed,
String key,
String value)
Appends an xml attribute to the xml feed.
|
private void |
assimilateJobs()
Puts the jobs in the abstract workflow into the job that is index
by the logical name of the jobs.
|
private List<List<Job>> |
bestFitBinPack(List<Job> jobs,
double maxTime)
Perform best fit bin packing.
|
private void |
collapseJobs(String name,
List jobs,
String partitionID)
Collapses the jobs having the same logical name according to the sites
where they are scheduled.
|
String |
constructID(String partitionID,
int id)
Given an integer id, returns a string id that is used for the clustered
job.
|
private Map |
constructMap(String propValue)
Constructs a map with the numbers/values for the collapsing factors to
collapse the nodes of same type.
|
String |
description()
Returns a textual description of the transfer implementation.
|
void |
determineClusters(Partition partition)
Determine the clusters for a partition.
|
private Comparator<Job> |
getBinPackingComparator()
The comparator is used to sort a collection of jobs in decreasing order
of their run times as specified by the Pegasus.JOB_RUN_TIME property.
|
ADag |
getClusteredDAG()
Returns the clustered workflow.
|
int[] |
getCollapseFactor(String pool,
Job job,
int size)
Returns the collapse factor, that is used to chunk up the jobs of a
particular type on a pool.
|
private String |
getRunTime(Job job) |
ADag |
getWorkflow()
Returns a reference to the workflow that is being refined by the refiner.
|
XMLProducer |
getXMLProducer()
Returns a reference to the XMLProducer, that generates the XML fragment
capturing the actions of the refiner.
|
void |
initialize(ADag dag,
PegasusBag bag)
Initializes the Clusterer impelementation
|
private Comparator |
jobComparator()
Singleton access to the job comparator.
|
protected void |
logRefinerAction(AggregatedJob clusteredJob,
JobAggregator aggregator)
Records the refiner action into the Provenace Store as a XML fragment.
|
void |
parents(String partitionID,
List parents)
Am empty implementation of the callout, as state is maintained
internally to determine the relations between the jobs.
|
private void |
printList(List l)
A utility method to print short description of jobs in a list.
|
private void |
replaceJobs()
The relations/edges are changed in local graph structure.
|
private void |
updateReplacementTable(List jobs,
Job mergedJob)
Updates the replacement table.
|
public static final int DEFAULT_COLLAPSE_FACTOR
public static final String DESCRIPTION
private static Comparator mJobComparator
protected LogManager mLogger
protected PegasusProperties mProps
protected JobAggregatorInstanceFactory mJobAggregatorFactory
private ADag mScheduledDAG
private Map mJobMap
private Map mSubInfoMap
private Map mCollapseMap
private Map mReplacementTable
private XMLProducer mXMLStore
private PPS mPPS
private Comparator jobComparator()
public ADag getWorkflow()
getWorkflow
in interface Refiner
public XMLProducer getXMLProducer()
getXMLProducer
in interface Refiner
public void initialize(ADag dag, PegasusBag bag) throws ClustererException
initialize
in interface Clusterer
dag
- the workflow that is being clustered.bag
- the bag of objects that is useful for initialization.ClustererException
- in case of error.public void determineClusters(Partition partition) throws ClustererException
1) bundle (dictates the number of clustered jobs that are created) 2) collapse (the number of jobs that make a single clustered job)In case of both parameters being associated with the jobs in a group, the bundle parameter overrides collapse parameter.
determineClusters
in interface Clusterer
partition
- the partition for which the clusters need to be
determined.ClustererException
- in case of error.Pegasus.BUNDLE_KEY
,
Pegasus.COLLAPSE_KEY
public void parents(String partitionID, List parents) throws ClustererException
parents
in interface Clusterer
partitionID
- the id of a partition.parents
- the list of String
objects that contain
the id's of the parents of the partition.ClustererException
- in case of error.private void collapseJobs(String name, List jobs, String partitionID)
name
- the logical name of the jobs in the list passed to
this function.jobs
- the list Job
objects corresponding
to the jobs that have the same logical name.partitionID
- the ID of the partition to which the jobs belong.private List<List<Job>> bestFitBinPack(List<Job> jobs, double maxTime)
jobs
- List of jobs sorted in decreasing order of the job runtime.maxTime
- The maximum time for which the clustered job should run.private Comparator<Job> getBinPackingComparator()
public ADag getClusteredDAG() throws ClustererException
getClusteredDAG
in interface Clusterer
ADag
object corresponding to the clustered workflow.ClustererException
- in case of error.public String description()
description
in interface Clusterer
protected void logRefinerAction(AggregatedJob clusteredJob, JobAggregator aggregator)
clusteredJob
- the clustered jobaggregator
- the aggregator that was used to create this clustered jobprotected void appendAttribute(StringBuffer xmlFeed, String key, String value)
xmlFeed
- the xmlFeed to which xml is being writtenkey
- the attribute keyvalue
- the attribute valuepublic int[] getCollapseFactor(String pool, Job job, int size)
pool
- the pool where the chunking up is occuringjob
- the Job
object containing the job that
is to be chunked up together.size
- the number of jobs that refer to the same logical
transformation and are scheduled on the same execution pool.public String constructID(String partitionID, int id)
partitionID
- the id of the partition.id
- the integer id from which the string id has to be
constructed. The id should be unique for all the
clustered jobs that are formed for a particular
partition.private void updateReplacementTable(List jobs, Job mergedJob)
jobs
- the List of jobs that is being replaced.mergedJob
- the mergedJob that is replacing the jobs in the list.private void assimilateJobs()
private Map constructMap(String propValue)
propValue
- the value of the property got from the properties file.private void replaceJobs()
private void printList(List l)
l
- the list of Job
objects