com.asakusafw.runtime.directio.hadoop

Class HadoopDataSource

java.lang.Object

com.asakusafw.runtime.directio.AbstractDirectDataSource

com.asakusafw.runtime.directio.hadoop.HadoopDataSource

All Implemented Interfaces:

DirectDataSource, org.apache.hadoop.conf.Configurable

public class HadoopDataSource
extends AbstractDirectDataSource
implements org.apache.hadoop.conf.Configurable

An implementation of AbstractDirectDataSource using FileSystem.

Since:

0.2.5

Version:

0.7.0

Constructor Summary

Constructors

Constructor and Description
HadoopDataSource()

Method Summary

Modifier and Type	Method and Description
void	cleanupAttemptOutput(OutputAttemptContext context) Cleanup output on the attempt.
void	cleanupTransactionOutput(OutputTransactionContext context) Cleanup output on the transaction.
void	commitAttemptOutput(OutputAttemptContext context) Commits output on the attempt.
void	commitTransactionOutput(OutputTransactionContext context) Commits output on the transaction.
void	configure(DirectDataSourceProfile profile) Configures this data source.
boolean	delete(String basePath, ResourcePattern resourcePattern, boolean recursive, Counter counter) Deletes resources.
<T> List<DirectInputFragment>	findInputFragments(DataDefinition<T> definition, String basePath, ResourcePattern resourcePattern) Collects input fragments about target resources.
<T> Optional<T>	findProperty(Class<T> propertyType) Returns the property of this data source.
org.apache.hadoop.conf.Configuration	getConf()
List<ResourceInfo>	list(String basePath, ResourcePattern resourcePattern, Counter counter) List resources.
<T> ModelInput<T>	openInput(DataDefinition<T> definition, DirectInputFragment fragment, Counter counter) Opens a fragment for input.
<T> ModelOutput<T>	openOutput(OutputAttemptContext context, DataDefinition<T> definition, String basePath, String resourcePath, Counter counter) Opens a resource for output.
String	path(String basePath) Returns a textually representation of the target path.
String	path(String basePath, ResourcePattern resourcePattern) Returns a textually representation of the target path pattern.
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setupAttemptOutput(OutputAttemptContext context) Prepares output area for the attempt.
void	setupTransactionOutput(OutputTransactionContext context) Prepares output area for the transaction.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopDataSource

public HadoopDataSource()

Method Detail

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

configure

public void configure(DirectDataSourceProfile profile)
               throws IOException,
                      InterruptedException

Description copied from class: AbstractDirectDataSource

Configures this data source.

Specified by:

configure in class AbstractDirectDataSource

Parameters:

profile - profile object

Throws:

IOException - if failed to configure this object

InterruptedException - if interrupted

path

public String path(String basePath,
                   ResourcePattern resourcePattern)

Description copied from interface: DirectDataSource

Returns a textually representation of the target path pattern.

Specified by:

path in interface DirectDataSource

Parameters:

basePath - base path of target resources

resourcePattern - search pattern of target resources from basePath

Returns:

the corresponded path

path

public String path(String basePath)

Description copied from interface: DirectDataSource

Returns a textually representation of the target path.

Specified by:

path in interface DirectDataSource

Parameters:

basePath - base path of target resources

Returns:

the corresponded path

findInputFragments

public <T> List<DirectInputFragment> findInputFragments(DataDefinition<T> definition,
                                                        String basePath,
                                                        ResourcePattern resourcePattern)
                                                 throws IOException,
                                                        InterruptedException

Description copied from interface: DirectDataSource

Collects input fragments about target resources.

Specified by:

findInputFragments in interface DirectDataSource

Type Parameters:

T - target data type

Parameters:

definition - the data definition

basePath - base path of target resources

resourcePattern - search pattern of target resources from basePath

Returns:

the found fragments of resources

Throws:

IOException - if failed to find resources by I/O error

InterruptedException - if interrupted while finding fragments

openInput

public <T> ModelInput<T> openInput(DataDefinition<T> definition,
                                   DirectInputFragment fragment,
                                   Counter counter)
                            throws IOException,
                                   InterruptedException

Description copied from interface: DirectDataSource

Opens a fragment for input.

Specified by:

openInput in interface DirectDataSource

Type Parameters:

T - target data type

Parameters:

definition - the data definition

fragment - target fragment

counter - counter object

Returns:

ModelInput to obtain contents in the fragment

Throws:

IOException - if failed to open the fragment by I/O error

InterruptedException - if interrupted

openOutput

public <T> ModelOutput<T> openOutput(OutputAttemptContext context,
                                     DataDefinition<T> definition,
                                     String basePath,
                                     String resourcePath,
                                     Counter counter)
                              throws IOException,
                                     InterruptedException

Description copied from interface: DirectDataSource

Opens a resource for output.

Specified by:

openOutput in interface DirectDataSource

Type Parameters:

T - target data type

Parameters:

context - current output attempt context

definition - the data definition

basePath - base path of target resource

resourcePath - target resource path from basePath

counter - counter object

Returns:

ModelOutput to put contents to the resource

Throws:

IOException - if failed to open the resource by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

list

public List<ResourceInfo> list(String basePath,
                               ResourcePattern resourcePattern,
                               Counter counter)
                        throws IOException,
                               InterruptedException

Description copied from interface: DirectDataSource

List resources.

Specified by:

list in interface DirectDataSource

Parameters:

basePath - base path of target resources

resourcePattern - search pattern of target resources from basePath

counter - the resource counter

Returns:

the found resources, or an empty list if there are no resources

Throws:

IOException - if failed to list resources by I/O error

InterruptedException - if interrupted

delete

public boolean delete(String basePath,
                      ResourcePattern resourcePattern,
                      boolean recursive,
                      Counter counter)
               throws IOException,
                      InterruptedException

Description copied from interface: DirectDataSource

Deletes resources.

Specified by:

delete in interface DirectDataSource

Parameters:

basePath - base path of target resources

resourcePattern - search pattern of target resources from basePath

recursive - also deletes containers and thier components recursively (only if this datasource supports containers)

counter - the resource counter

Returns:

true if successfully deleted, otherwise false

Throws:

IOException - if failed to delete resources by I/O error

InterruptedException - if interrupted

setupAttemptOutput

public void setupAttemptOutput(OutputAttemptContext context)
                        throws IOException,
                               InterruptedException

Description copied from interface: DirectDataSource

Prepares output area for the attempt.

Specified by:

setupAttemptOutput in interface DirectDataSource

Parameters:

context - current attempt context

Throws:

IOException - if failed by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

commitAttemptOutput

public void commitAttemptOutput(OutputAttemptContext context)
                         throws IOException,
                                InterruptedException

Description copied from interface: DirectDataSource

Commits output on the attempt.

Specified by:

commitAttemptOutput in interface DirectDataSource

Parameters:

context - current attempt context

Throws:

IOException - if failed by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

cleanupAttemptOutput

public void cleanupAttemptOutput(OutputAttemptContext context)
                          throws IOException,
                                 InterruptedException

Description copied from interface: DirectDataSource

Cleanup output on the attempt.

Specified by:

cleanupAttemptOutput in interface DirectDataSource

Parameters:

context - current attempt context

Throws:

IOException - if failed by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

setupTransactionOutput

public void setupTransactionOutput(OutputTransactionContext context)
                            throws IOException,
                                   InterruptedException

Description copied from interface: DirectDataSource

Prepares output area for the transaction.

Specified by:

setupTransactionOutput in interface DirectDataSource

Parameters:

context - current transaction context

Throws:

IOException - if failed by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

commitTransactionOutput

public void commitTransactionOutput(OutputTransactionContext context)
                             throws IOException,
                                    InterruptedException

Description copied from interface: DirectDataSource

Commits output on the transaction.

Specified by:

commitTransactionOutput in interface DirectDataSource

Parameters:

context - current transaction context

Throws:

IOException - if failed by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

cleanupTransactionOutput

public void cleanupTransactionOutput(OutputTransactionContext context)
                              throws IOException,
                                     InterruptedException

Description copied from interface: DirectDataSource

Cleanup output on the transaction.

Specified by:

cleanupTransactionOutput in interface DirectDataSource

Parameters:

context - current transaction context

Throws:

IOException - if failed by I/O error

InterruptedException - if interrupted

See Also:

Output Staging

findProperty

public <T> Optional<T> findProperty(Class<T> propertyType)

Description copied from interface: DirectDataSource

Returns the property of this data source.

Specified by:

findProperty in interface DirectDataSource

Type Parameters:

T - the property type

Parameters:

propertyType - the property type

Returns:

the corresponded property, or empty if it is not defined