com.asakusafw.runtime.directio.hadoop

Class HadoopFileFormat<T>

java.lang.Object

org.apache.hadoop.conf.Configured

com.asakusafw.runtime.directio.hadoop.HadoopFileFormat<T>

Type Parameters:

T - the type of target data model

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

AbstractOrcFileFormat, AbstractParquetFileFormat, HadoopFileFormatAdapter, SequenceFileFormat

public abstract class HadoopFileFormat<T>
extends org.apache.hadoop.conf.Configured
implements FragmentableDataFormat<T>

Data model format of InputStream / OutputStream . This implementation class must have a public constructor without any parameters.

Since:

0.2.6

Version:

0.9.1

Constructor Summary

Constructors

Constructor and Description
HadoopFileFormat() Creates a new instance without configuration.
HadoopFileFormat(org.apache.hadoop.conf.Configuration conf) Creates a new instance with configuration.

Method Summary

Modifier and Type	Method and Description
ModelInput<T>	createInput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, Counter counter) Creates a new ModelInput for the specified properties.
abstract ModelInput<T>	createInput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, long offset, long fragmentSize, Counter counter) Creates a new ModelInput for the specified properties.
abstract ModelOutput<T>	createOutput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, Counter counter) Creates a new ModelOutput for the specified properties.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.FragmentableDataFormat

getMinimumFragmentSize, getPreferredFragmentSize

Methods inherited from interface com.asakusafw.runtime.directio.DataFormat

getSupportedType

Constructor Detail

HadoopFileFormat

public HadoopFileFormat()

Creates a new instance without configuration.

HadoopFileFormat

public HadoopFileFormat(org.apache.hadoop.conf.Configuration conf)

Creates a new instance with configuration.

Parameters:

conf - configuration

Method Detail

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 org.apache.hadoop.fs.FileSystem fileSystem,
                                 org.apache.hadoop.fs.Path path,
                                 Counter counter)
                          throws IOException,
                                 InterruptedException

Creates a new ModelInput for the specified properties.

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

counter - the current counter

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

IllegalArgumentException - if this does not support target property sequence, or any parameter is null

Since:

0.9.1

createInput

public abstract ModelInput<T> createInput(Class<? extends T> dataType,
                                          org.apache.hadoop.fs.FileSystem fileSystem,
                                          org.apache.hadoop.fs.Path path,
                                          long offset,
                                          long fragmentSize,
                                          Counter counter)
                                   throws IOException,
                                          InterruptedException

Creates a new ModelInput for the specified properties.

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

counter - the current counter

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

IllegalArgumentException - if this does not support target property sequence, or any parameter is null

createOutput

public abstract ModelOutput<T> createOutput(Class<? extends T> dataType,
                                            org.apache.hadoop.fs.FileSystem fileSystem,
                                            org.apache.hadoop.fs.Path path,
                                            Counter counter)
                                     throws IOException,
                                            InterruptedException

Creates a new ModelOutput for the specified properties.

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

counter - the current counter

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted

IllegalArgumentException - if this does not support property sequence, or any parameter is null