com.asakusafw.directio.hive.parquet.v2

Class CompatibilityV2

java.lang.Object

com.asakusafw.directio.hive.parquet.Compatibility

com.asakusafw.directio.hive.parquet.v2.CompatibilityV2

public class CompatibilityV2
extends Compatibility

compatibility layer for Direct I/O Parquet support.

Since:

0.10.3

Constructor Summary

Constructors

Constructor and Description
CompatibilityV2()

Method Summary

Modifier and Type	Method and Description
List<DirectInputFragment>	computeInputFragments(AbstractParquetFileFormat<?> format, StripedDataFormat.InputContext context) delegate from StripedDataFormat.computeInputFragments(InputContext).
<T> ModelInput<T>	createInput(AbstractParquetFileFormat<T> format, Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, long offset, long fragmentSize, Counter counter) delegate from AbstractParquetFileFormat.createInput(Class, FileSystem, Path, long, long, Counter).
<T> ModelOutput<T>	createOutput(AbstractParquetFileFormat<T> format, Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, Counter counter) delegate from AbstractParquetFileFormat.createOutput(Class, FileSystem, Path, Counter).
Optional<ParquetValueDriver>	findValueDriver(org.apache.hadoop.hive.serde2.typeinfo.TypeInfo typeInfo, Class<?> valueClass) delegate to ParquetValueDrivers.find().
Optional<org.apache.parquet.column.ParquetProperties.WriterVersion>	findVersionId(String name) returns the version symbol if the name.
Class<org.apache.parquet.hadoop.metadata.CompressionCodecName>	getCompressionCodecNameClass() returns the compression kind class.
protected int	getPriority() returns the priority of this compatibility layer.

Methods inherited from class com.asakusafw.directio.hive.parquet.Compatibility

getInstance

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompatibilityV2

public CompatibilityV2()

Method Detail

getPriority

protected int getPriority()

Description copied from class: Compatibility

returns the priority of this compatibility layer.

Specified by:

getPriority in class Compatibility

Returns:

the priority (higher value is higher priority), or negative value if it is not available

computeInputFragments

public List<DirectInputFragment> computeInputFragments(AbstractParquetFileFormat<?> format,
                                                       StripedDataFormat.InputContext context)
                                                throws IOException,
                                                       InterruptedException

Description copied from class: Compatibility

delegate from StripedDataFormat.computeInputFragments(InputContext).

Specified by:

computeInputFragments in class Compatibility

Parameters:

format - the source format

context - the current input context

Returns:

the computed result

Throws:

IOException - if failed to compute fragments by I/O error

InterruptedException - if interrupted while computing fragments

createInput

public <T> ModelInput<T> createInput(AbstractParquetFileFormat<T> format,
                                     Class<? extends T> dataType,
                                     org.apache.hadoop.fs.FileSystem fileSystem,
                                     org.apache.hadoop.fs.Path path,
                                     long offset,
                                     long fragmentSize,
                                     Counter counter)
                              throws IOException,
                                     InterruptedException

Description copied from class: Compatibility

delegate from AbstractParquetFileFormat.createInput(Class, FileSystem, Path, long, long, Counter).

Specified by:

createInput in class Compatibility

Type Parameters:

T - the data type

Parameters:

format - the source format

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

counter - the current counter

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public <T> ModelOutput<T> createOutput(AbstractParquetFileFormat<T> format,
                                       Class<? extends T> dataType,
                                       org.apache.hadoop.fs.FileSystem fileSystem,
                                       org.apache.hadoop.fs.Path path,
                                       Counter counter)
                                throws IOException,
                                       InterruptedException

Description copied from class: Compatibility

delegate from AbstractParquetFileFormat.createOutput(Class, FileSystem, Path, Counter).

Specified by:

createOutput in class Compatibility

Type Parameters:

T - the data type

Parameters:

format - the source format

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

counter - the current counter

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted

findVersionId

public Optional<org.apache.parquet.column.ParquetProperties.WriterVersion> findVersionId(String name)

Description copied from class: Compatibility

returns the version symbol if the name.

Specified by:

findVersionId in class Compatibility

Parameters:

name - the version name

Returns:

the related symbol, or empty if it is not defined

getCompressionCodecNameClass

public Class<org.apache.parquet.hadoop.metadata.CompressionCodecName> getCompressionCodecNameClass()

Description copied from class: Compatibility

returns the compression kind class.

Specified by:

getCompressionCodecNameClass in class Compatibility

Returns:

the compression kind class

findValueDriver

public Optional<ParquetValueDriver> findValueDriver(org.apache.hadoop.hive.serde2.typeinfo.TypeInfo typeInfo,
                                                    Class<?> valueClass)

Description copied from class: Compatibility

delegate to ParquetValueDrivers.find().

Specified by:

findValueDriver in class Compatibility

Parameters:

typeInfo - the Hive type info

valueClass - the ValueOption type

Returns:

the corresponded ParquetValueDriver, or empty if it is not found