com.asakusafw.runtime.stage

Class AbstractStageClient

java.lang.Object

org.apache.hadoop.conf.Configured

com.asakusafw.runtime.stage.BaseStageClient

com.asakusafw.runtime.stage.AbstractStageClient

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public abstract class AbstractStageClient
extends BaseStageClient

An abstract implementation of Hadoop MapReduce stage client class.

Since:

0.1.0

Version:

0.7.1

Field Summary

Fields

Modifier and Type	Field and Description
static String	METHOD_COMBINER_CLASS The method name of getCombinerClassOrNull().
static String	METHOD_GROUPING_COMPARATOR_CLASS The method name of getGroupingComparatorClassOrNull().
static String	METHOD_PARTITIONER_CLASS The method name of getPartitionerClassOrNull().
static String	METHOD_REDUCER_CLASS The method name of getReducerClassOrNull().
static String	METHOD_SHUFFLE_KEY_CLASS The method name of getShuffleKeyClassOrNull().
static String	METHOD_SHUFFLE_VALUE_CLASS The method name of getShuffleValueClassOrNull().
static String	METHOD_SORT_COMPARATOR_CLASS The method name of getSortComparatorClassOrNull().
static String	METHOD_STAGE_INPUTS The method name of getStageInputs().
static String	METHOD_STAGE_OUTPUT_PATH The method name of getStageOutputPath().
static String	METHOD_STAGE_OUTPUTS The method name of getStageOutputs().
static String	METHOD_STAGE_RESOURCES The method name of getStageResources().

Fields inherited from class com.asakusafw.runtime.stage.BaseStageClient

METHOD_BATCH_ID, METHOD_FLOW_ID, METHOD_STAGE_ID

Constructor Summary

Constructors

Constructor and Description
AbstractStageClient()

Method Summary

Modifier and Type	Method and Description
protected void	configureStage(org.apache.hadoop.mapreduce.Job job, VariableTable variables) Configures the Job object for this stage.
org.apache.hadoop.mapreduce.Job	createJob(org.apache.hadoop.conf.Configuration conf) Creates a new job.
protected int	execute(String[] args) Performs stage execution.
protected Class<? extends org.apache.hadoop.mapreduce.Reducer>	getCombinerClassOrNull() Returns the combiner class.
protected Class<? extends org.apache.hadoop.io.RawComparator>	getGroupingComparatorClassOrNull() Returns the grouping comparator class (for shuffle keys).
protected Class<? extends org.apache.hadoop.mapreduce.Partitioner>	getPartitionerClassOrNull() Returns the partitioner class.
protected Class<? extends org.apache.hadoop.mapreduce.Reducer>	getReducerClassOrNull() Returns the reducer class.
protected Class<? extends org.apache.hadoop.io.Writable>	getShuffleKeyClassOrNull() Returns the shuffle key class.
protected Class<? extends org.apache.hadoop.io.Writable>	getShuffleValueClassOrNull() Returns the shuffle value class.
protected Class<? extends org.apache.hadoop.io.RawComparator>	getSortComparatorClassOrNull() Returns the sort comparator class (for shuffle keys).
protected abstract List<StageInput>	getStageInputs() Returns the stage inputs.
protected abstract String	getStageOutputPath() Returns the base path of stage output.
protected List<StageOutput>	getStageOutputs() Returns the stage outputs.
protected List<StageResource>	getStageResources() Returns the stage resources.

Methods inherited from class com.asakusafw.runtime.stage.BaseStageClient

getBatchId, getDefinitionId, getExecutionId, getFlowId, getOperationId, getStageId, getUser, run

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

METHOD_STAGE_OUTPUT_PATH

public static final String METHOD_STAGE_OUTPUT_PATH

The method name of getStageOutputPath().

See Also:

Constant Field Values

METHOD_STAGE_INPUTS

public static final String METHOD_STAGE_INPUTS

The method name of getStageInputs().

See Also:

Constant Field Values

METHOD_STAGE_OUTPUTS

public static final String METHOD_STAGE_OUTPUTS

The method name of getStageOutputs().

See Also:

Constant Field Values

METHOD_STAGE_RESOURCES

public static final String METHOD_STAGE_RESOURCES

The method name of getStageResources().

See Also:

Constant Field Values

METHOD_SHUFFLE_KEY_CLASS

public static final String METHOD_SHUFFLE_KEY_CLASS

The method name of getShuffleKeyClassOrNull().

See Also:

Constant Field Values

METHOD_SHUFFLE_VALUE_CLASS

public static final String METHOD_SHUFFLE_VALUE_CLASS

The method name of getShuffleValueClassOrNull().

See Also:

Constant Field Values

METHOD_PARTITIONER_CLASS

public static final String METHOD_PARTITIONER_CLASS

The method name of getPartitionerClassOrNull().

See Also:

Constant Field Values

METHOD_COMBINER_CLASS

public static final String METHOD_COMBINER_CLASS

The method name of getCombinerClassOrNull().

See Also:

Constant Field Values

METHOD_SORT_COMPARATOR_CLASS

public static final String METHOD_SORT_COMPARATOR_CLASS

The method name of getSortComparatorClassOrNull().

See Also:

Constant Field Values

METHOD_GROUPING_COMPARATOR_CLASS

public static final String METHOD_GROUPING_COMPARATOR_CLASS

The method name of getGroupingComparatorClassOrNull().

See Also:

Constant Field Values

METHOD_REDUCER_CLASS

public static final String METHOD_REDUCER_CLASS

The method name of getReducerClassOrNull().

See Also:

Constant Field Values

Constructor Detail

AbstractStageClient

public AbstractStageClient()

Method Detail

configureStage

protected void configureStage(org.apache.hadoop.mapreduce.Job job,
                              VariableTable variables)
                       throws IOException,
                              InterruptedException

Configures the Job object for this stage.

Parameters:

job - the target job

variables - current variable table

Throws:

IOException - if failed to configure the job

InterruptedException - if interrupted while configuring Job object

getStageInputs

protected abstract List<StageInput> getStageInputs()

Returns the stage inputs.

Returns:

the stage inputs

See Also:

StageInputDriver

getStageOutputPath

protected abstract String getStageOutputPath()

Returns the base path of stage output.

Returns:

the base path

getStageOutputs

protected List<StageOutput> getStageOutputs()

Returns the stage outputs.

Returns:

the stage outputs

See Also:

StageOutputDriver

getStageResources

protected List<StageResource> getStageResources()

Returns the stage resources.

Returns:

the stage resources

See Also:

StageResourceDriver

getShuffleKeyClassOrNull

protected Class<? extends org.apache.hadoop.io.Writable> getShuffleKeyClassOrNull()

Returns the shuffle key class.

Returns:

the shuffle key class, or null if a reduce operation is not required in this stage

getShuffleValueClassOrNull

protected Class<? extends org.apache.hadoop.io.Writable> getShuffleValueClassOrNull()

Returns the shuffle value class.

Returns:

the shuffle value class, or null if a reduce operation is not required in this stage

getPartitionerClassOrNull

protected Class<? extends org.apache.hadoop.mapreduce.Partitioner> getPartitionerClassOrNull()

Returns the partitioner class.

Returns:

the partitioner class, or null if a reduce operation is not required in this stage

getCombinerClassOrNull

protected Class<? extends org.apache.hadoop.mapreduce.Reducer> getCombinerClassOrNull()

Returns the combiner class.

Returns:

the combiner class, or null if a combine operation is not required in this stage

getSortComparatorClassOrNull

protected Class<? extends org.apache.hadoop.io.RawComparator> getSortComparatorClassOrNull()

Returns the sort comparator class (for shuffle keys).

Returns:

the sort comparator class, or null if a reduce operation is not required in this stage

getGroupingComparatorClassOrNull

protected Class<? extends org.apache.hadoop.io.RawComparator> getGroupingComparatorClassOrNull()

Returns the grouping comparator class (for shuffle keys).

Returns:

the grouping comparator class, or null if a reduce operation is not required in this stage

getReducerClassOrNull

protected Class<? extends org.apache.hadoop.mapreduce.Reducer> getReducerClassOrNull()

Returns the reducer class.

Returns:

the reducer class, or null if a reduce operation is not required in this stage

execute

protected int execute(String[] args)
               throws Exception

Description copied from class: BaseStageClient

Performs stage execution.

Specified by:

execute in class BaseStageClient

Parameters:

args - arguments

Returns:

exit code of the execution

Throws:

Exception - if failed to execute

createJob

public org.apache.hadoop.mapreduce.Job createJob(org.apache.hadoop.conf.Configuration conf)
                                          throws IOException,
                                                 InterruptedException

Creates a new job.

Parameters:

conf - asakusa job configuration

Returns:

the created job

Throws:

IOException - if failed to create a new job

InterruptedException - if interrupted while creating Job object

IllegalArgumentException - if some parameters were null