com.asakusafw.runtime.directio.hadoop

Class SequenceFileFormat<K,V,T>

java.lang.Object

org.apache.hadoop.conf.Configured

com.asakusafw.runtime.directio.hadoop.HadoopFileFormat<T>

com.asakusafw.runtime.directio.hadoop.SequenceFileFormat<K,V,T>

Type Parameters:

K - the type of raw sequence file key

V - the type of raw sequence file value

T - the type of target data model

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

public abstract class SequenceFileFormat<K,V,T>
extends HadoopFileFormat<T>

Data model format of InputStream / OutputStream . This implementation class must have a public constructor without any parameters.

Since:

0.2.6

Version:

0.4.0

Constructor Summary

Constructors

Constructor and Description
SequenceFileFormat()

Method Summary

Modifier and Type	Method and Description
protected abstract void	copyFromModel(T model, K key, V value) Copy the data model into the key and value.
protected abstract void	copyToModel(K key, V value, T model) Copy key and value into the target data model.
ModelInput<T>	createInput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, long offset, long fragmentSize, Counter counter) Creates a new ModelInput for the specified properties.
protected abstract K	createKeyObject() Returns a key object.
ModelOutput<T>	createOutput(Class<? extends T> dataType, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, Counter counter) Creates a new ModelOutput for the specified properties.
protected abstract V	createValueObject() Returns a value object.
org.apache.hadoop.io.compress.CompressionCodec	getCompressionCodec(org.apache.hadoop.fs.Path path) Returns a compression codec for output sequence files.
long	getMinimumFragmentSize() Returns the lower limit of fragment size (in bytes).

Methods inherited from class com.asakusafw.runtime.directio.hadoop.HadoopFileFormat

createInput

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.FragmentableDataFormat

getPreferredFragmentSize

Methods inherited from interface com.asakusafw.runtime.directio.DataFormat

getSupportedType

Constructor Detail

SequenceFileFormat

public SequenceFileFormat()

Method Detail

getMinimumFragmentSize

public long getMinimumFragmentSize()
                            throws IOException,
                                   InterruptedException

Description copied from interface: FragmentableDataFormat

Returns the lower limit of fragment size (in bytes). Each fragment must not smaller than this size.

Returns:

the lower limit of fragment size, or -1 as infinite

Throws:

IOException - if failed to compute bytes count

InterruptedException - if interrupted

createKeyObject

protected abstract K createKeyObject()

Returns a key object.

Returns:

a key object

createValueObject

protected abstract V createValueObject()

Returns a value object.

Returns:

a value object

copyToModel

protected abstract void copyToModel(K key,
                                    V value,
                                    T model)
                             throws IOException

Copy key and value into the target data model.

Parameters:

key - the source key object

value - the source value object

model - the target data model

Throws:

IOException - if failed to copy

copyFromModel

protected abstract void copyFromModel(T model,
                                      K key,
                                      V value)
                               throws IOException

Copy the data model into the key and value.

Parameters:

model - the source data model

key - the target key object

value - the target value object

Throws:

IOException - if failed to copy

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 org.apache.hadoop.fs.FileSystem fileSystem,
                                 org.apache.hadoop.fs.Path path,
                                 long offset,
                                 long fragmentSize,
                                 Counter counter)
                          throws IOException,
                                 InterruptedException

Description copied from class: HadoopFileFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class HadoopFileFormat<T>

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

counter - the current counter

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   org.apache.hadoop.fs.FileSystem fileSystem,
                                   org.apache.hadoop.fs.Path path,
                                   Counter counter)
                            throws IOException,
                                   InterruptedException

Description copied from class: HadoopFileFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class HadoopFileFormat<T>

Parameters:

dataType - the target data type

fileSystem - the file system to open the target path

path - the path to the target file

counter - the current counter

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted

getCompressionCodec

public org.apache.hadoop.io.compress.CompressionCodec getCompressionCodec(org.apache.hadoop.fs.Path path)
                                                                   throws IOException,
                                                                          InterruptedException

Returns a compression codec for output sequence files. Clients can override this method in subclasses, and return the suitable CompressionCodec object.

Parameters:

path - target path

Returns:

a compression codec used to output, or null if output will not be compressed

Throws:

IOException - if failed to create a compression codec

InterruptedException - if interrupted