com.asakusafw.runtime.io.json.directio

Class AbstractJsonStreamFormat<T>

java.lang.Object

com.asakusafw.runtime.directio.BinaryStreamFormat<T>

com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat<T>

com.asakusafw.runtime.io.json.directio.AbstractJsonStreamFormat<T>

Type Parameters:

T - the data type

All Implemented Interfaces:

DataFormat<T>, FragmentableDataFormat<T>, org.apache.hadoop.conf.Configurable

public abstract class AbstractJsonStreamFormat<T>
extends ConfigurableBinaryStreamFormat<T>

An abstract implementation of Direct I/O data format for JSON files.

Since:

0.10.3

Constructor Summary

Constructors

Constructor and Description
AbstractJsonStreamFormat()

Method Summary

Modifier and Type	Method and Description
protected void	afterInput(T model, StringOption path, JsonInput<T> input) Processes the data object, which has filled the current record data.
protected void	beforeOutput(T model, StringOption path, JsonOutput<T> output) Process the data object.
protected abstract void	configureJsonFormat(JsonFormat.Builder<T> builder) Configures JsonFormat builder.
ModelInput<T>	createInput(Class<? extends T> dataType, String path, InputStream stream, long offset, long fragmentSize) Creates a new ModelInput for the specified properties.
ModelOutput<T>	createOutput(Class<? extends T> dataType, String path, OutputStream stream) Creates a new ModelOutput for the specified properties.
protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec>	getCompressionCodecClass() Returns the compression codec class.
protected InputSplitter	getInputSplitter() Returns the input splitter of this format.
JsonFormat<T>	getJsonFormat() Returns the JsonFormat.
long	getMinimumFragmentSize() Returns the lower limit of fragment size (in bytes).
long	getPreferredFragmentSize() Returns the preferred fragment size (in bytes).

Methods inherited from class com.asakusafw.runtime.directio.hadoop.ConfigurableBinaryStreamFormat

getConf, setConf

Methods inherited from class com.asakusafw.runtime.directio.BinaryStreamFormat

createInput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.asakusafw.runtime.directio.DataFormat

getSupportedType

Constructor Detail

AbstractJsonStreamFormat

public AbstractJsonStreamFormat()

Method Detail

configureJsonFormat

protected abstract void configureJsonFormat(JsonFormat.Builder<T> builder)

Configures JsonFormat builder.

Parameters:

builder - the target builder

getInputSplitter

protected InputSplitter getInputSplitter()

Returns the input splitter of this format.

Returns:

the input splitter, or null is input split is disabled

getCompressionCodecClass

protected Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodecClass()

Returns the compression codec class.

Returns:

the compression codec class, or null if compression is disabled

afterInput

protected void afterInput(T model,
                          StringOption path,
                          JsonInput<T> input)

Processes the data object, which has filled the current record data.

Parameters:

model - the data object

path - the current input path

input - the current input

beforeOutput

protected void beforeOutput(T model,
                            StringOption path,
                            JsonOutput<T> output)

Process the data object.

Parameters:

model - the data object

path - the current output path

output - the current output

getJsonFormat

public final JsonFormat<T> getJsonFormat()

Returns the JsonFormat.

Returns:

the JSON format

getPreferredFragmentSize

public final long getPreferredFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the preferred fragment size (in bytes).

Returns:

the preferred fragment size, or -1 as infinite

getMinimumFragmentSize

public final long getMinimumFragmentSize()

Description copied from interface: FragmentableDataFormat

Returns the lower limit of fragment size (in bytes). Each fragment must not smaller than this size.

Returns:

the lower limit of fragment size, or -1 as infinite (never split into fragments)

createInput

public ModelInput<T> createInput(Class<? extends T> dataType,
                                 String path,
                                 InputStream stream,
                                 long offset,
                                 long fragmentSize)
                          throws IOException,
                                 InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelInput for the specified properties.

Specified by:

createInput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

offset - starting stream offset

fragmentSize - suggested fragment bytes count, or -1 as infinite

Returns:

the created reader

Throws:

IOException - if failed to create reader

InterruptedException - if interrupted

createOutput

public ModelOutput<T> createOutput(Class<? extends T> dataType,
                                   String path,
                                   OutputStream stream)
                            throws IOException,
                                   InterruptedException

Description copied from class: BinaryStreamFormat

Creates a new ModelOutput for the specified properties.

Specified by:

createOutput in class BinaryStreamFormat<T>

Parameters:

dataType - the target data type

path - the path about the target stream (for label)

stream - the target stream

Returns:

the created writer

Throws:

IOException - if failed to create writer

InterruptedException - if interrupted