com.asakusafw.directio.hive.parquet

Class AbstractParquetFileFormat<T>

java.lang.Object

org.apache.hadoop.conf.Configured

com.asakusafw.runtime.directio.hadoop.HadoopFileFormat<T>

com.asakusafw.directio.hive.parquet.AbstractParquetFileFormat<T>

Type Parameters:

T - the data model type

All Implemented Interfaces:

TableInfo.Provider, DataFormat<T>, FragmentableDataFormat<T>, StripedDataFormat<T>, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

ParquetFileFormat

public abstract class AbstractParquetFileFormat<T>
extends HadoopFileFormat<T>
implements StripedDataFormat<T>, TableInfo.Provider

An abstract implementation of HadoopFileFormat for Parquet.

Since:

0.7.0

Nested Class Summary

Nested classes/interfaces inherited from interface com.asakusafw.runtime.directio.hadoop.StripedDataFormat

StripedDataFormat.InputContext

Constructor Summary

Constructors

Constructor and Description
AbstractParquetFileFormat()

Method Summary

Modifier and Type	Method and Description
List<DirectInputFragment>	computeInputFragments(StripedDataFormat.InputContext context) Compute DirectInputFragment for the input.
ModelInput<T>	createInput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, long offset, long fragmentSize, Counter counter) Creates a new ModelInput for the specified properties.
ModelOutput<T>	createOutput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, Counter counter) Creates a new ModelOutput for the specified properties.
abstract DataModelDescriptor	getDataModelDescriptor() Returns the target data model descriptor.
abstract ParquetFormatConfiguration	getFormatConfiguration() Returns the format configuration.
TableInfo	getSchema() Returns the TableInfo of this object.
Class<T>	getSupportedType() Returns the supported data model type.
abstract String	getTableName() Returns the table name.

Methods inherited from class com.asakusafw.runtime.directio.hadoop.HadoopFileFormat

createInput

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.FragmentableDataFormat

getMinimumFragmentSize, getPreferredFragmentSize

Constructor Detail

AbstractParquetFileFormat

public AbstractParquetFileFormat()

Method Detail

getFormatConfiguration

public abstract ParquetFormatConfiguration getFormatConfiguration()

Returns the format configuration.

Returns:

the format configuration

getDataModelDescriptor

public abstract DataModelDescriptor getDataModelDescriptor()

Returns the target data model descriptor.

Returns:

the target data model descriptor

getTableName

public abstract String getTableName()

Returns the table name.

Returns:

the table name

getSchema

public TableInfo getSchema()

Description copied from interface: TableInfo.Provider

Returns the TableInfo of this object.

Specified by:

getSchema in interface TableInfo.Provider

Returns:

the table schema

getSupportedType

public Class<T> getSupportedType()

Description copied from interface: DataFormat

Returns the supported data model type.

Specified by:

getSupportedType in interface DataFormat<T>

Returns:

the supported type

computeInputFragments

public List<DirectInputFragment> computeInputFragments(StripedDataFormat.InputContext context)
                                                throws IOException,
                                                       InterruptedException

Description copied from interface: StripedDataFormat

Compute DirectInputFragment for the input.

Specified by:

computeInputFragments in interface StripedDataFormat<T>

Parameters:

context - the current input context

Returns:

the computed result

Throws:

IOException - if failed to compute fragments by I/O error

InterruptedException - if interrupted while computing fragments

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 org.apache.hadoop.fs.FileSystem fileSystem,
                                 org.apache.hadoop.fs.Path path,
                                 long offset,
                                 long fragmentSize,
                                 Counter counter)
                          throws IOException,
                                 InterruptedException

Description copied from class: HadoopFileFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class HadoopFileFormat<T>

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

counter - the current counter

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   org.apache.hadoop.fs.FileSystem fileSystem,
                                   org.apache.hadoop.fs.Path path,
                                   Counter counter)
                            throws IOException,
                                   InterruptedException

Description copied from class: HadoopFileFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class HadoopFileFormat<T>

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

counter - the current counter

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted