com.datasalt.pangool.utils
Class HadoopUtils

java.lang.Object
  extended by com.datasalt.pangool.utils.HadoopUtils

public class HadoopUtils
extends Object

Put here usefull methods for manipulating things in the HDFS, etc.


Constructor Summary
HadoopUtils()
           
 
Method Summary
static void deleteIfExists(org.apache.hadoop.fs.FileSystem dFs, org.apache.hadoop.fs.Path path)
           
static String fileToString(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
          Reads the content of a file into a String.
static void incCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext ctx, String groupName, Enum counter)
          Utility for doing ctx.getCounter(groupName, counter.toString()).increment(1);
static HashMap<Integer,Double> readIntDoubleMap(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FileSystem fs)
          Reads maps of integer -> double
static HashMap<Integer,Double> readIntDoubleMapFromGlob(org.apache.hadoop.fs.Path glob, org.apache.hadoop.fs.FileSystem fs)
          Reads maps of integer -> double from glob paths like "folder/part-r*"
static HashMap<Integer,Integer> readIntIntMap(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FileSystem fs)
          Reads maps of integer -> integer
static HashMap<Integer,Integer> readIntIntMapFromGlob(org.apache.hadoop.fs.Path glob, org.apache.hadoop.fs.FileSystem fs)
          Reads maps of integer -> integer from glob paths like "folder/part-r*"
static void stringToFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, String string)
          Creates a file with the given string, overwritting if needed.
static void synchronize(org.apache.hadoop.fs.FileSystem fS1, org.apache.hadoop.fs.Path p1, org.apache.hadoop.fs.FileSystem fS2, org.apache.hadoop.fs.Path p2)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

HadoopUtils

public HadoopUtils()
Method Detail

deleteIfExists

public static void deleteIfExists(org.apache.hadoop.fs.FileSystem dFs,
                                  org.apache.hadoop.fs.Path path)
                           throws IOException
Throws:
IOException

synchronize

public static void synchronize(org.apache.hadoop.fs.FileSystem fS1,
                               org.apache.hadoop.fs.Path p1,
                               org.apache.hadoop.fs.FileSystem fS2,
                               org.apache.hadoop.fs.Path p2)
                        throws IOException
Throws:
IOException

stringToFile

public static void stringToFile(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path path,
                                String string)
                         throws IOException
Creates a file with the given string, overwritting if needed.

Throws:
IOException

fileToString

public static String fileToString(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path path)
                           throws IOException
Reads the content of a file into a String. Return null if the file does not exist.

Throws:
IOException

readIntDoubleMap

public static HashMap<Integer,Double> readIntDoubleMap(org.apache.hadoop.fs.Path path,
                                                       org.apache.hadoop.fs.FileSystem fs)
                                                throws IOException
Reads maps of integer -> double

Throws:
IOException

readIntDoubleMapFromGlob

public static HashMap<Integer,Double> readIntDoubleMapFromGlob(org.apache.hadoop.fs.Path glob,
                                                               org.apache.hadoop.fs.FileSystem fs)
                                                        throws IOException
Reads maps of integer -> double from glob paths like "folder/part-r*"

Throws:
IOException

readIntIntMap

public static HashMap<Integer,Integer> readIntIntMap(org.apache.hadoop.fs.Path path,
                                                     org.apache.hadoop.fs.FileSystem fs)
                                              throws IOException
Reads maps of integer -> integer

Throws:
IOException

readIntIntMapFromGlob

public static HashMap<Integer,Integer> readIntIntMapFromGlob(org.apache.hadoop.fs.Path glob,
                                                             org.apache.hadoop.fs.FileSystem fs)
                                                      throws IOException
Reads maps of integer -> integer from glob paths like "folder/part-r*"

Throws:
IOException

incCounter

public static void incCounter(org.apache.hadoop.mapreduce.TaskInputOutputContext ctx,
                              String groupName,
                              Enum counter)
Utility for doing ctx.getCounter(groupName, counter.toString()).increment(1);



Copyright © –2014 Datasalt Systems S.L.. All rights reserved.