com.asakusafw.runtime.directio.hadoop

Class HadoopDataSourceUtil

java.lang.Object

com.asakusafw.runtime.directio.hadoop.HadoopDataSourceUtil

public final class HadoopDataSourceUtil
extends Object

Utilities for Direct data access facilities on Hadoop.

Since:

0.2.5

Version:

0.10.0

Field Summary

Fields

Modifier and Type	Field and Description
static Charset	COMMENT_CHARSET Charset for commit mark file comments.
static String	DEFAULT_SYSTEM_DIR The default system directory name.
static String	KEY_LOCAL_TEMPDIR The attribute key name of local tempdir.
static String	KEY_PATH The key name of path.
static String	KEY_SYSTEM_DIR The key name of system directory for this format.
static String	PREFIX The key prefix of data sources.
static String	PREFIX_BEGIN_MARK The file name prefix of transaction began mark.
static String	PREFIX_COMMIT_MARK The file name prefix of transaction committed mark.
static String	TRANSACTION_INFO_DIR The transaction directory name.

Method Summary

Modifier and Type	Method and Description
static boolean	contains(org.apache.hadoop.fs.Path parent, org.apache.hadoop.fs.Path child) Returns whether the parent path contains the child path, or not.
static OutputTransactionContext	createContext(org.apache.hadoop.mapreduce.JobContext context, String datasourceId) Deprecated. Use BridgeOutputFormat.createContext(JobContext, String) instead
static OutputTransactionContext	createContext(String executionId, String datasourceId) Creates output context from execution ID and datasource ID.
static OutputAttemptContext	createContext(org.apache.hadoop.mapreduce.TaskAttemptContext context, String datasourceId) Deprecated. Use BridgeOutputFormat.createContext(TaskAttemptContext, String) instead
static Collection<org.apache.hadoop.fs.FileStatus>	findAllTransactionInfoFiles(org.apache.hadoop.conf.Configuration conf) Returns the all transaction info files.
static org.apache.hadoop.fs.Path	getCommitMarkPath(org.apache.hadoop.conf.Configuration conf, String executionId) Returns the commit mark path.
static org.apache.hadoop.fs.Path	getLocalTemporaryDirectory(org.apache.hadoop.fs.LocalFileSystem localFileSystem) Returns the local temporary directory.
static org.apache.hadoop.fs.Path	getSystemDir(org.apache.hadoop.conf.Configuration conf) Returns the system directory.
static org.apache.hadoop.fs.Path	getSystemDir(org.apache.hadoop.conf.Configuration conf, boolean resolve) Returns the system directory.
static String	getTransactionInfoExecutionId(org.apache.hadoop.fs.Path transactionInfoPath) Extracts an execution ID from the transaction info.
static org.apache.hadoop.fs.Path	getTransactionInfoPath(org.apache.hadoop.conf.Configuration conf, String executionId) Returns the transaction info path.
static boolean	isLocalAttemptOutputDefined(org.apache.hadoop.fs.LocalFileSystem localFileSystem) Returns whether the local attempt output directory is defined.
static List<DirectDataSourceProfile>	loadProfiles(org.apache.hadoop.conf.Configuration conf) Loads a profile list from the configuration.
static DirectDataSourceRepository	loadRepository(org.apache.hadoop.conf.Configuration conf) Loads DirectDataSourceRepository from Configuration.
static void	move(Counter counter, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path from, org.apache.hadoop.fs.Path to) Moves all files in source directory into target directory.
static void	move(Counter counter, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path from, org.apache.hadoop.fs.Path to, int threads) Moves all files in source directory into target directory.
static void	moveFromLocal(Counter counter, org.apache.hadoop.fs.LocalFileSystem localFs, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path from, org.apache.hadoop.fs.Path to) Moves all files in source directory into target directory.
static List<org.apache.hadoop.fs.FileStatus>	onlyMinimalCovered(List<org.apache.hadoop.fs.FileStatus> statList) Returns only minimal covered files.
static List<org.apache.hadoop.fs.FileStatus>	search(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path base, FilePattern pattern) Searches file/directories by pattern.
static <T> HadoopFileFormat<T>	toHadoopFileFormat(org.apache.hadoop.conf.Configuration configuration, DataFormat<T> format) Converts DataFormat into an equivalent HadoopFileFormat.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PREFIX

public static final String PREFIX

The key prefix of data sources.

See Also:

Constant Field Values

KEY_PATH

public static final String KEY_PATH

The key name of path.

See Also:

Constant Field Values

KEY_SYSTEM_DIR

public static final String KEY_SYSTEM_DIR

The key name of system directory for this format.

See Also:

Constant Field Values

KEY_LOCAL_TEMPDIR

public static final String KEY_LOCAL_TEMPDIR

The attribute key name of local tempdir.

See Also:

Constant Field Values

PREFIX_BEGIN_MARK

public static final String PREFIX_BEGIN_MARK

The file name prefix of transaction began mark.

See Also:

Constant Field Values

PREFIX_COMMIT_MARK

public static final String PREFIX_COMMIT_MARK

The file name prefix of transaction committed mark.

Since:

0.10.0

See Also:

Constant Field Values

DEFAULT_SYSTEM_DIR

public static final String DEFAULT_SYSTEM_DIR

The default system directory name.

Since:

0.10.0

See Also:

Constant Field Values

TRANSACTION_INFO_DIR

public static final String TRANSACTION_INFO_DIR

The transaction directory name.

Since:

0.10.0

See Also:

Constant Field Values

COMMENT_CHARSET

public static final Charset COMMENT_CHARSET

Charset for commit mark file comments.

Method Detail

loadProfiles

public static List<DirectDataSourceProfile> loadProfiles(org.apache.hadoop.conf.Configuration conf)

Loads a profile list from the configuration.

Parameters:

conf - target configuration

Returns:

the restored profile list

Throws:

IllegalArgumentException - if some parameters were null

loadRepository

public static DirectDataSourceRepository loadRepository(org.apache.hadoop.conf.Configuration conf)

Loads DirectDataSourceRepository from Configuration.

Parameters:

conf - configuration object

Returns:

the created repository

Throws:

IllegalArgumentException - if some parameters were null

isLocalAttemptOutputDefined

public static boolean isLocalAttemptOutputDefined(org.apache.hadoop.fs.LocalFileSystem localFileSystem)

Returns whether the local attempt output directory is defined.

Parameters:

localFileSystem - current local file system

Returns:

true to defined, otherwise false

Throws:

IllegalArgumentException - if some parameters were null

getLocalTemporaryDirectory

public static org.apache.hadoop.fs.Path getLocalTemporaryDirectory(org.apache.hadoop.fs.LocalFileSystem localFileSystem)
                                                            throws IOException

Returns the local temporary directory.

Parameters:

localFileSystem - the local file system

Returns:

the output path (must be on local fs), or null if not defined

Throws:

IOException - if failed to compute the path

IllegalArgumentException - if some parameters were null

createContext

public static OutputTransactionContext createContext(String executionId,
                                                     String datasourceId)

Creates output context from execution ID and datasource ID.

Parameters:

executionId - current execution ID

datasourceId - target datasource ID

Returns:

output context

Throws:

IllegalArgumentException - if some parameters were null

createContext

@Deprecated
public static OutputTransactionContext createContext(org.apache.hadoop.mapreduce.JobContext context,
                                                                 String datasourceId)

Deprecated. Use BridgeOutputFormat.createContext(JobContext, String) instead

Creates output context from Hadoop context.

Parameters:

context - current context in Hadoop

datasourceId - datasource ID

Returns:

the created context

Throws:

IllegalArgumentException - if some parameters were null

createContext

@Deprecated
public static OutputAttemptContext createContext(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                             String datasourceId)

Deprecated. Use BridgeOutputFormat.createContext(TaskAttemptContext, String) instead

Creates output context from Hadoop context.

Parameters:

context - current context in Hadoop

datasourceId - datasource ID

Returns:

the created context

Throws:

IllegalArgumentException - if some parameters were null

getTransactionInfoExecutionId

public static String getTransactionInfoExecutionId(org.apache.hadoop.fs.Path transactionInfoPath)

Extracts an execution ID from the transaction info.

Parameters:

transactionInfoPath - target path

Returns:

execution ID, or null if is not a valid transaction info

Throws:

IllegalArgumentException - if some parameters were null

See Also:

getCommitMarkPath(Configuration, String)

getTransactionInfoPath

public static org.apache.hadoop.fs.Path getTransactionInfoPath(org.apache.hadoop.conf.Configuration conf,
                                                               String executionId)
                                                        throws IOException

Returns the transaction info path.

Parameters:

conf - the current configuration

executionId - target transaction ID

Returns:

target path

Throws:

IOException - if failed to compute the path by I/O exception

IllegalArgumentException - if some parameters were null

getCommitMarkPath

public static org.apache.hadoop.fs.Path getCommitMarkPath(org.apache.hadoop.conf.Configuration conf,
                                                          String executionId)
                                                   throws IOException

Returns the commit mark path.

Parameters:

conf - the current configuration

executionId - target transaction ID

Returns:

target path

Throws:

IOException - if failed to compute the path by I/O exception

IllegalArgumentException - if some parameters were null

findAllTransactionInfoFiles

public static Collection<org.apache.hadoop.fs.FileStatus> findAllTransactionInfoFiles(org.apache.hadoop.conf.Configuration conf)
                                                                               throws IOException

Returns the all transaction info files.

Parameters:

conf - the current configuration

Returns:

target path

Throws:

IOException - if failed to find files by I/O error

IllegalArgumentException - if some parameters were null

getSystemDir

public static org.apache.hadoop.fs.Path getSystemDir(org.apache.hadoop.conf.Configuration conf)
                                              throws IOException

Returns the system directory.

Parameters:

conf - the current configuration

Returns:

the system directory

Throws:

IOException - if I/O error was occurred

getSystemDir

public static org.apache.hadoop.fs.Path getSystemDir(org.apache.hadoop.conf.Configuration conf,
                                                     boolean resolve)
                                              throws IOException

Returns the system directory.

Parameters:

conf - the current configuration

resolve - true to resolve the result path, otherwise false

Returns:

the system directory

Throws:

IOException - if I/O error was occurred

Since:

0.10.0

search

public static List<org.apache.hadoop.fs.FileStatus> search(org.apache.hadoop.fs.FileSystem fs,
                                                           org.apache.hadoop.fs.Path base,
                                                           FilePattern pattern)
                                                    throws IOException

Searches file/directories by pattern.

Parameters:

fs - target file system

base - base path

pattern - search pattern

Returns:

found files, or an empty list if not found

Throws:

IOException - if failed to search by I/O error

IllegalArgumentException - if some parameters were null

onlyMinimalCovered

public static List<org.apache.hadoop.fs.FileStatus> onlyMinimalCovered(List<org.apache.hadoop.fs.FileStatus> statList)

Returns only minimal covered files. If the parameter contains both directory and its children, this result includes only the directory.

Parameters:

statList - target files

Returns:

minimal covered

contains

public static boolean contains(org.apache.hadoop.fs.Path parent,
                               org.apache.hadoop.fs.Path child)

Returns whether the parent path contains the child path, or not. If the parent and child is same, this returns false.

Parameters:

parent - the parent path

child - the child path

Returns:

true if parent path strictly contains the child, otherwise false

Throws:

IllegalArgumentException - if some parameters were null

move

public static void move(Counter counter,
                        org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path from,
                        org.apache.hadoop.fs.Path to)
                 throws IOException

Moves all files in source directory into target directory.

Parameters:

counter - counter which accepts operations count

fs - file system

from - path to source directory

to - path to target directory

Throws:

IOException - if failed to move files

IllegalArgumentException - if some parameters were null

moveFromLocal

public static void moveFromLocal(Counter counter,
                                 org.apache.hadoop.fs.LocalFileSystem localFs,
                                 org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path from,
                                 org.apache.hadoop.fs.Path to)
                          throws IOException

Moves all files in source directory into target directory.

Parameters:

counter - counter which accepts operations count

localFs - the local file system

fs - the target file system

from - path to source directory (must be on local file system)

to - path to target directory

Throws:

IOException - if failed to move files

IllegalArgumentException - if some parameters were null

move

public static void move(Counter counter,
                        org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path from,
                        org.apache.hadoop.fs.Path to,
                        int threads)
                 throws IOException,
                        InterruptedException

Moves all files in source directory into target directory.

Parameters:

counter - counter which accepts operations count

fs - file system

from - path to source directory

to - path to target directory

threads - the number of threads for moving each file

Throws:

IOException - if failed to move files

InterruptedException - if interrupted while moving files

IllegalArgumentException - if some parameters were null

Since:

0.9.0

toHadoopFileFormat

public static <T> HadoopFileFormat<T> toHadoopFileFormat(org.apache.hadoop.conf.Configuration configuration,
                                                         DataFormat<T> format)
                                                  throws IOException

Converts DataFormat into an equivalent HadoopFileFormat.

Type Parameters:

T - the data type

Parameters:

configuration - the current configuration

format - the target data format

Returns:

the related format

Throws:

IOException - if the given DataFormat is not supported

Since:

0.9.1