com.asakusafw.runtime.io.line.directio

Class AbstractLineStreamFormat<T>

java.lang.Object

com.asakusafw.runtime.directio.BinaryStreamFormat<T>

com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat<T>

com.asakusafw.runtime.io.line.directio.AbstractLineStreamFormat<T>

Type Parameters:

T - the data type

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

public abstract class AbstractLineStreamFormat<T>
extends ConfigurableBinaryStreamFormat<T>

An abstract implementation of Direct I/O data format for line read/write.

Since:

0.10.3

Constructor Summary

Constructors

Constructor and Description
AbstractLineStreamFormat()

Method Summary

Modifier and Type	Method and Description
ModelInput<T>	createInput(Class<? extends T> dataType, String path, InputStream stream, long offset, long fragmentSize) Creates a new ModelInput for the specified properties.
ModelOutput<T>	createOutput(Class<? extends T> dataType, String path, OutputStream stream) Creates a new ModelOutput for the specified properties.
protected abstract void	emit(LineOutput output, T model, StringOption path) Emits the next line from the data object into the target output.
protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec>	getCompressionCodecClass() Returns the compression codec class.
protected abstract LineConfiguration	getConfiguration() Returns the line based text format configuration.
protected InputSplitter	getInputSplitter() Returns the input splitter of this format.
long	getMinimumFragmentSize() Returns the lower limit of fragment size (in bytes).
long	getPreferredFragmentSize() Returns the preferred fragment size (in bytes).
protected abstract boolean	parse(LineInput input, T model, StringOption path) Parses next line and fill into the target data object.

Methods inherited from class com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat

getConf, setConf

Methods inherited from class com.asakusafw.runtime.directio.BinaryStreamFormat

createInput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.DataFormat

getSupportedType

Constructor Detail

AbstractLineStreamFormat

public AbstractLineStreamFormat()

Method Detail

getConfiguration

protected abstract LineConfiguration getConfiguration()

Returns the line based text format configuration.

Returns:

the line based text format configuration

parse

protected abstract boolean parse(LineInput input,
                                 T model,
                                 StringOption path)
                          throws IOException

Parses next line and fill into the target data object.

Parameters:

input - the current input

model - the target data object

path - the current input path

Returns:

true if it was succeeded, otherwise false

Throws:

IOException - if I/O error was occurred

emit

protected abstract void emit(LineOutput output,
                             T model,
                             StringOption path)
                      throws IOException

Emits the next line from the data object into the target output.

Parameters:

output - the current output

model - the data object

path - the current output path

Throws:

IOException - if I/O error was occurred

getInputSplitter

protected InputSplitter getInputSplitter()

Returns the input splitter of this format.

Returns:

the input splitter, or null is input split is disabled

getCompressionCodecClass

protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodecClass()

Returns the compression codec class.

Returns:

the compression codec class, or null if compression is disabled

getPreferredFragmentSize

public final long getPreferredFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the preferred fragment size (in bytes).

Returns:

the preferred fragment size, or -1 as infinite

getMinimumFragmentSize

public final long getMinimumFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the lower limit of fragment size (in bytes). Each fragment must not smaller than this size.

Returns:

the lower limit of fragment size, or -1 as infinite (never split into fragments)

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 String path,
                                 InputStream stream,
                                 long offset,
                                 long fragmentSize)
                          throws IOException,
                                 InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   String path,
                                   OutputStream stream)
                            throws IOException,
                                   InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted