com.datasalt.pangool.utils
Class InstancesDistributor

java.lang.Object
  extended by com.datasalt.pangool.utils.InstancesDistributor

public class InstancesDistributor
extends Object

This class contains useful methods for serializing/deserializing instances that implement Serializable. This class is used in Pangool to distribute the instances around the cluster

You can do things like saving a Java Serializable instance and recovering it afterwards. Check methods distribute(Object, String, Configuration) and loadInstance(Configuration, Class, String, boolean) for this purpose.


Field Summary
static String DEFAULT_HDFS_TMP_FOLDER_CONF_VALUE
           
static String HDFS_TMP_FOLDER_CONF
           
 
Constructor Summary
InstancesDistributor()
           
 
Method Summary
static void distribute(Object obj, String fileName, org.apache.hadoop.conf.Configuration conf)
          Utility method for serializing an object and saving it in a way that later can be recovered anywhere in the cluster.
static
<T> T
loadInstance(org.apache.hadoop.conf.Configuration conf, Class<T> objClass, String fileName, boolean callSetConf)
          Given a Hadoop Configuration property and an Class, this method can re-instantiate an Object instance that was previously distributed using * distribute(Object, String, Configuration).
static void removeFromCache(org.apache.hadoop.conf.Configuration conf, String filename)
          Delete a file that has been distributed using distribute(Object, String, Configuration).
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

HDFS_TMP_FOLDER_CONF

public static final String HDFS_TMP_FOLDER_CONF

DEFAULT_HDFS_TMP_FOLDER_CONF_VALUE

public static final String DEFAULT_HDFS_TMP_FOLDER_CONF_VALUE
See Also:
Constant Field Values
Constructor Detail

InstancesDistributor

public InstancesDistributor()
Method Detail

distribute

public static void distribute(Object obj,
                              String fileName,
                              org.apache.hadoop.conf.Configuration conf)
                       throws FileNotFoundException,
                              IOException,
                              URISyntaxException
Utility method for serializing an object and saving it in a way that later can be recovered anywhere in the cluster.

The file where it has been serialized will be saved into a Hadoop Configuration property so that you can call loadInstance(Configuration, Class, String, boolean) to re-instantiate the serialized instance.

Parameters:
obj - The obj instance to serialize using Java serialization.
fileName - The file name where the instance will be serialized.
conf - The Hadoop Configuration.
Throws:
FileNotFoundException
IOException
URISyntaxException

loadInstance

public static <T> T loadInstance(org.apache.hadoop.conf.Configuration conf,
                                 Class<T> objClass,
                                 String fileName,
                                 boolean callSetConf)
                      throws IOException
Given a Hadoop Configuration property and an Class, this method can re-instantiate an Object instance that was previously distributed using * distribute(Object, String, Configuration).

Type Parameters:
T - The object type.
Parameters:
conf - The Hadoop Configuration.
objClass - The object type class.
fileName - The file name to locate the instance
callSetConf - If true, will call setConf() if deserialized instance is Configurable
Throws:
IOException

removeFromCache

public static void removeFromCache(org.apache.hadoop.conf.Configuration conf,
                                   String filename)
                            throws IOException
Delete a file that has been distributed using distribute(Object, String, Configuration).

Throws:
IOException


Copyright © –2014 Datasalt Systems S.L.. All rights reserved.