com.asakusafw.runtime.io.csv.directio

Class AbstractCsvStreamFormat<T>

java.lang.Object

com.asakusafw.runtime.directio.BinaryStreamFormat<T>

com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat<T>

com.asakusafw.runtime.io.csv.directio.AbstractCsvStreamFormat<T>

Type Parameters:

T - the data type

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

public abstract class AbstractCsvStreamFormat<T>
extends ConfigurableBinaryStreamFormat<T>

An abstract implementation of Direct I/O data format for CSV files.

Since:

0.10.3

Constructor Summary

Constructors

Constructor and Description
AbstractCsvStreamFormat()

Method Summary

Modifier and Type	Method and Description
ModelInput<T>	createInput(Class<? extends T> dataType, String path, InputStream stream, long offset, long fragmentSize) Creates a new ModelInput for the specified properties.
ModelOutput<T>	createOutput(Class<? extends T> dataType, String path, OutputStream stream) Creates a new ModelOutput for the specified properties.
protected abstract void	emit(CsvEmitter output, T model, StringOption path) Emits the next record body from the data object into the target output.
protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec>	getCompressionCodecClass() Returns the compression codec class.
protected abstract CsvConfiguration	getConfiguration(boolean head) Returns the CSV format configuration.
protected InputSplitter	getInputSplitter() Returns the input splitter of this format.
long	getMinimumFragmentSize() Returns the lower limit of fragment size (in bytes).
long	getPreferredFragmentSize() Returns the preferred fragment size (in bytes).
protected abstract void	parse(CsvParser input, T model, StringOption path) Parses next record body and fill the contents into the target model.

Methods inherited from class com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat

getConf, setConf

Methods inherited from class com.asakusafw.runtime.directio.BinaryStreamFormat

createInput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.DataFormat

getSupportedType

Constructor Detail

AbstractCsvStreamFormat

public AbstractCsvStreamFormat()

Method Detail

getConfiguration

protected abstract CsvConfiguration getConfiguration(boolean head)

Returns the CSV format configuration.

Parameters:

head - true to returns a configuration for the head of file, otherwise false

Returns:

the CSV format configuration

parse

protected abstract void parse(CsvParser input,
                              T model,
                              StringOption path)
                       throws IOException

Parses next record body and fill the contents into the target model.

Parameters:

input - the current input

model - the target data object

path - the current input path

Throws:

IOException - if I/O error was occurred

emit

protected abstract void emit(CsvEmitter output,
                             T model,
                             StringOption path)
                      throws IOException

Emits the next record body from the data object into the target output.

Parameters:

output - the current output

model - the data object

path - the current output path

Throws:

IOException - if I/O error was occurred

getInputSplitter

protected InputSplitter getInputSplitter()

Returns the input splitter of this format.

Returns:

the input splitter, or null is input split is disabled

getCompressionCodecClass

protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodecClass()

Returns the compression codec class.

Returns:

the compression codec class, or null if compression is disabled

getPreferredFragmentSize

public final long getPreferredFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the preferred fragment size (in bytes).

Returns:

the preferred fragment size, or -1 as infinite

getMinimumFragmentSize

public final long getMinimumFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the lower limit of fragment size (in bytes). Each fragment must not smaller than this size.

Returns:

the lower limit of fragment size, or -1 as infinite (never split into fragments)

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 String path,
                                 InputStream stream,
                                 long offset,
                                 long fragmentSize)
                          throws IOException,
                                 InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   String path,
                                   OutputStream stream)
                            throws IOException,
                                   InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted