com.datasalt.pangool.tuplemr.mapred.lib.output
Class ProxyOutputFormat

java.lang.Object
  extended by org.apache.hadoop.mapreduce.OutputFormat<K,V>
      extended by org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
          extended by com.datasalt.pangool.tuplemr.mapred.lib.output.ProxyOutputFormat
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable

public class ProxyOutputFormat
extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
implements org.apache.hadoop.conf.Configurable

This special implementation of FileOutputFormat is used as a proxy for being able to support any type of OutputFormat at the same time that we support Multiple Output Formats (also with any type of OutputFormat).

The idea is to use the "_temporary" folder for storing everything (including multiple sub/folders) so that we have the full control over the commit / fail process.

The wrapped (proxied) output format can be of any type. It is configured through PROXIED_OUTPUT_FORMAT_CONF.


Nested Class Summary
 class ProxyOutputFormat.ProxyOutputCommitter
           
 
Field Summary
protected  String baseDir
           
protected  org.apache.hadoop.conf.Configuration conf
           
protected  String originalDir
           
protected  org.apache.hadoop.mapreduce.OutputFormat outputFormat
           
static String PROXIED_OUTPUT_FORMAT_CONF
           
 
Constructor Summary
ProxyOutputFormat()
           
 
Method Summary
 void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
           
 org.apache.hadoop.conf.Configuration getConf()
           
 org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
           
 org.apache.hadoop.mapreduce.RecordWriter getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
           
 void setConf(org.apache.hadoop.conf.Configuration conf)
           
 
Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
getCompressOutput, getDefaultWorkFile, getOutputCompressorClass, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

PROXIED_OUTPUT_FORMAT_CONF

public static final String PROXIED_OUTPUT_FORMAT_CONF

conf

protected org.apache.hadoop.conf.Configuration conf

outputFormat

protected org.apache.hadoop.mapreduce.OutputFormat outputFormat

originalDir

protected String originalDir

baseDir

protected String baseDir
Constructor Detail

ProxyOutputFormat

public ProxyOutputFormat()
Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)
Specified by:
setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()
Specified by:
getConf in interface org.apache.hadoop.conf.Configurable

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                         throws IOException,
                                                                InterruptedException
Specified by:
getRecordWriter in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
Throws:
IOException
InterruptedException

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
                      throws IOException
Overrides:
checkOutputSpecs in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
Throws:
IOException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                               throws IOException
Overrides:
getOutputCommitter in class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
Throws:
IOException


Copyright © –2014 Datasalt Systems S.L.. All rights reserved.