com.asakusafw.runtime.io.text.directio

Class AbstractTextStreamFormat<T>

java.lang.Object

com.asakusafw.runtime.directio.BinaryStreamFormat<T>

com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat<T>

com.asakusafw.runtime.io.text.directio.AbstractTextStreamFormat<T>

Type Parameters:

T - the data type

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

public abstract class AbstractTextStreamFormat<T>
extends ConfigurableBinaryStreamFormat<T>

An abstract implementation of Direct I/O data format for formatted text files.

Since:

0.9.1

Constructor Summary

Constructors

Constructor and Description
AbstractTextStreamFormat()

Method Summary

Modifier and Type	Method and Description
protected void	afterInput(T model, StringOption path, TextInput<T> input) Processes the data object, which has filled the current record data.
protected void	beforeOutput(T model, StringOption path, TextOutput<T> output) Process the data object.
ModelInput<T>	createInput(Class<? extends T> dataType, String path, InputStream stream, long offset, long fragmentSize) Creates a new ModelInput for the specified properties.
ModelOutput<T>	createOutput(Class<? extends T> dataType, String path, OutputStream stream) Creates a new ModelOutput for the specified properties.
protected abstract RecordDefinition<T>	createRecordDefinition() Returns the RecordDefinition.
protected abstract TextFormat	createTextFormat() Returns the TextFormat.
protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec>	getCompressionCodecClass() Returns the compression codec class.
protected InputSplitter	getInputSplitter() Returns the input splitter of this format.
long	getMinimumFragmentSize() Returns the lower limit of fragment size (in bytes).
long	getPreferredFragmentSize() Returns the preferred fragment size (in bytes).
RecordDefinition<T>	getRecordDefinition() Returns the RecordDefinition.
TextFormat	getTextFormat() Returns the TextFormat.

Methods inherited from class com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat

getConf, setConf

Methods inherited from class com.asakusafw.runtime.directio.BinaryStreamFormat

createInput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.DataFormat

getSupportedType

Constructor Detail

AbstractTextStreamFormat

public AbstractTextStreamFormat()

Method Detail

createTextFormat

protected abstract TextFormat createTextFormat()

Returns the TextFormat.

Returns:

the text format

createRecordDefinition

protected abstract RecordDefinition<T> createRecordDefinition()

Returns the RecordDefinition.

Returns:

the record structure definition

getInputSplitter

protected InputSplitter getInputSplitter()

Returns the input splitter of this format.

Returns:

the input splitter, or null is input split is disabled

getCompressionCodecClass

protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodecClass()

Returns the compression codec class.

Returns:

the compression codec class, or null if compression is disabled

afterInput

protected void afterInput(T model,
                          StringOption path,
                          TextInput<T> input)

Processes the data object, which has filled the current record data.

Parameters:

model - the data object

path - the current input path

input - the current input

beforeOutput

protected void beforeOutput(T model,
                            StringOption path,
                            TextOutput<T> output)

Process the data object.

Parameters:

model - the data object

path - the current output path

output - the current output

getTextFormat

public final TextFormat getTextFormat()

Returns the TextFormat.

Returns:

the text format

getRecordDefinition

public final RecordDefinition<T> getRecordDefinition()

Returns the RecordDefinition.

Returns:

the record structure definition

getPreferredFragmentSize

public final long getPreferredFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the preferred fragment size (in bytes).

Returns:

the preferred fragment size, or -1 as infinite

getMinimumFragmentSize

public final long getMinimumFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the lower limit of fragment size (in bytes). Each fragment must not smaller than this size.

Returns:

the lower limit of fragment size, or -1 as infinite (never split into fragments)

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 String path,
                                 InputStream stream,
                                 long offset,
                                 long fragmentSize)
                          throws IOException,
                                 InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   String path,
                                   OutputStream stream)
                            throws IOException,
                                   InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted