com.asakusafw.runtime.directio.hadoop

Class HadoopFileFormatAdapter<T>

java.lang.Object

org.apache.hadoop.conf.Configured

com.asakusafw.runtime.directio.hadoop.HadoopFileFormat<T>

com.asakusafw.runtime.directio.hadoop.HadoopFileFormatAdapter<T>

Type Parameters:

T - the type of target data model

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

public class HadoopFileFormatAdapter<T>
extends HadoopFileFormat<T>

Adapts BinaryStreamFormat into HadoopFileFormat.

Since:

0.2.6

Constructor Summary

Constructors

Constructor and Description
HadoopFileFormatAdapter(BinaryStreamFormat<T> streamFormat) Creates a new instance.
HadoopFileFormatAdapter(BinaryStreamFormat<T> streamFormat, org.apache.hadoop.conf.Configuration configuration) Creates a new instance.

Method Summary

Modifier and Type	Method and Description
ModelInput<T>	createInput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, long offset, long fragmentSize, Counter counter) Creates a new ModelInput for the specified properties.
ModelOutput<T>	createOutput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, Counter counter) Creates a new ModelOutput for the specified properties.
long	getMinimumFragmentSize() Returns the lower limit of fragment size (in bytes).
long	getPreferredFragmentSize() Returns the preferred fragment size (in bytes).
Class<T>	getSupportedType() Returns the supported data model type.

Methods inherited from class com.asakusafw.runtime.directio.hadoop.HadoopFileFormat

createInput

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileFormatAdapter

public HadoopFileFormatAdapter(BinaryStreamFormat<T> streamFormat)

Creates a new instance.

Parameters:

streamFormat - adaption target

Throws:

IllegalArgumentException - if some parameters were null

HadoopFileFormatAdapter

public HadoopFileFormatAdapter(BinaryStreamFormat<T> streamFormat,
                               org.apache.hadoop.conf.Configuration configuration)

Creates a new instance.

Parameters:

streamFormat - adaption target

configuration - initial configuration

Throws:

IllegalArgumentException - if some parameters were null

Method Detail

getSupportedType

public Class<T> getSupportedType()

Description copied from interface: DataFormat

Returns the supported data model type.

Returns:

the supported type

getPreferredFragmentSize

public long getPreferredFragmentSize()
                              throws IOException,
                                     InterruptedException

Description copied from interface: FragmentableDataFormat

Returns the preferred fragment size (in bytes).

Returns:

the preferred fragment size, or -1 as infinite

Throws:

IOException - if failed to compute bytes count

InterruptedException - if interrupted

getMinimumFragmentSize

public long getMinimumFragmentSize()
                            throws IOException,
                                   InterruptedException

Description copied from interface: FragmentableDataFormat

Returns the lower limit of fragment size (in bytes). Each fragment must not smaller than this size.

Returns:

the lower limit of fragment size, or -1 as infinite (never split into fragments)

Throws:

IOException - if failed to compute bytes count

InterruptedException - if interrupted

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 org.apache.hadoop.fs.FileSystem fileSystem,
                                 org.apache.hadoop.fs.Path path,
                                 long offset,
                                 long fragmentSize,
                                 Counter counter)
                          throws IOException,
                                 InterruptedException

Description copied from class: HadoopFileFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class HadoopFileFormat<T>

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

counter - the current counter

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   org.apache.hadoop.fs.FileSystem fileSystem,
                                   org.apache.hadoop.fs.Path path,
                                   Counter counter)
                            throws IOException,
                                   InterruptedException

Description copied from class: HadoopFileFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class HadoopFileFormat<T>

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

counter - the current counter

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted