algebraic e.g. SUM. In these cases, you can apply some initital function f_init on subsets of X to get partial results. You can then combine partial results from different subsets of X using an intermediate function f_intermed. To get the final answers, several partial results can be combined by invoking a final f_final function. For the function SUM, f_init, f_intermed, and f_final are all SUM. See the code for builtin AVG to get a better idea of how algebraic works. When eval functions implement this interface, Pig will attempt to use MapReduce's combiner. The initial funciton will be called in the map phase and be passed a single tuple. The intermediate function will be called 0 or more times in the combiner phase. And the final function will be called once in the reduce phase. It is important that the results be the same whether the intermediate function is called 0, 1, or more times. Hadoop makes no guarantees about how many times the combiner will be called in a job.]]>
One key thing to note is that if a subclass of AlgebraicEvalFunc wishes to use any constructor arguments, it MUST call super(args).

IMPORTANT: the implementation of the Accumulator interface that this class provides is good, but it is simulated. For maximum efficiency, it is important to manually implement the accumulator interface. See {@link Accumulator} for more information on how to do so.]]> The default implementation interprets the {@link OutputSchema} annotation, if one is present. Otherwise, it returns null (no known output schema). @param input Schema of the input @return Schema of the output]]>
  • FuncSpec(this.getClass().getName(), new Schema(new Schema.FieldSchema(null, DataType.DOUBLE)))
  • FuncSpec(IntSum.getClass().getName(), new Schema(new Schema.FieldSchema(null, DataType.INTEGER))) This would indicate that the main implementation is used for doubles, and the special implementation IntSum is used for ints.]]> The argument type for the function. @param The return type for the function. @param The exception type that the function throws.]]> The argument type for the function. @param The return type for the function.]]>
  • {@link LoadFunc#setUDFContextSignature(String)}
  • {@link IndexableLoadFunc#initialize(Configuration)}
  • {@link LoadFunc#setLocation(String, org.apache.hadoop.mapreduce.Job)}
  • {@link IndexableLoadFunc#seekNear(Tuple)}
  • {@link LoadFunc#getNext} called multiple times to retrieve data and perform the join
  • {@link IndexableLoadFunc#close} @since Pig 0.6]]> For example, you could express IsEmpty as follows: 
    public class IsEmpty extends IteratingAccumulatorEvalFunc {
     public Boolean exec(Iterator iter) throws IOException {
         return !iter.hashNext();
     }
 }
    Count could be implemented as follows: 
    public class Count extends IteratingAccumulatorEvalFunc {
     public Long exec(Iterator iter) throws IOException {
         long ct = 0;
         for (; iter.hasNext(); iter.next()) {
             ct++;
         }
         return ct;
     }
 }
    ]]>     { private static int numInvocations = 0; private static Reporter reporter; static { // Register this class for static data cleanup JVMReuseManager.getInstance().registerForStaticDataCleanup(MyUDF.class); } // Write a public static method that performs the cleanup // and annotate it with @StaticDataCleanup @StaticDataCleanup public static void staticDataCleanup() { numInvocations = 0; reporter = null; } #### UDF Code goes here ###### } @since Pig 0.14]]> /user/ @return the absolute location based on the arguments passed @throws IOException if the conversion is not possible]]> :/{code} in Hadoop MapReduce mode, or of the form {code}file:///{code} in Hadoop local mode. @param location the location string specified in the load statement @param curDir the current file system directory @return the absolute path of file in the file system @throws FrontendException if the scheme of the location is incompatible with the scheme of the file system]]> LoadFunc contains the basic features needed by the majority of load functions. For more advanced functionality there are separate interfaces that a load function can implement. See {@link LoadCaster}, {@link LoadMetadata}, {@link LoadPushDown}, {@link OrderedLoadFunc}, {@link CollectableLoadFunc}, and {@link IndexableLoadFunc}.]]> Apache Pig version 0.11.0-SNAPSHOT (r1202387) compiled Nov 15 2011, 15:22:09 ]]>
  • map
    In each map, bloom filters are computed on the join keys partitioned by the hashcode of the key with {@link #PIG_BLOOMJOIN_NUM_FILTERS} number of partitions. Bloom filters from different maps are then combined in the reducer producing one bloom filter per partition. This is efficient and fast if there are smaller number of maps (<10) and the number of distinct keys are not too high. It can be faster with larger number of maps and even with bigger bloom vector sizes, but the amount of data shuffled to the reducer for aggregation becomes huge making it inefficient.
  • reduce
    Join keys are sent from the map to the reducer partitioned by hashcode of the key with {@link #PIG_BLOOMJOIN_NUM_FILTERS} number of reducers. One bloom filter is then created per partition. This is efficient for larger datasets with lot of maps or very large {@link #PIG_BLOOMJOIN_VECTORSIZE_BYTES}. In this case size of keys sent to the reducer is smaller than sending bloom filters to reducer for aggregation making it efficient.
    • Default value is map.]]>     Throwable.getMessage() method) shown to the user]]> Throwable.getMessage() method) shown to the user @param location - The SourceLocation that generated this error.]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param location - The SourceLocation that generated this error.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown. @param location - The SourceLocation that generated this error.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param location - The SourceLocation that generated this error.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown. @param location - The SourceLocation that generated this error.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown. @param location - The SourceLocation that generated this error.]]>
  • the {@linkplain Class#getName() name} of the class of this object
  • ": " (a colon and a space)
  • "ERROR " (the string ERROR followed by a a space)
  • the result of invoking this object's {@link #getErrorCode} method
  • ": " (a colon and a space)
  • the result of invoking {@link Throwable#getLocalizedMessage() getLocalizedMessage} method If getLocalizedMessage returns null, then just the class name is returned. @return a string representation of this throwable.]]> PigException class. Details such as the source of the error, error message, error code, etc. are contained in this class. The default values for the attributes are: errorSource = BUG errorCode = 0 retriable = false detailedMessage = null location = null]]> The pig script can be from local file, then you can use FileInputStream. Or pig script can be in memory which you build it dynamically, the you can use ByteArrayInputStream Pig script can even be in remote machine, which you get wrap it as SocketInputStream.
    The parameters in the pig script will be substituted with the values in the map and the parameter files. The values in params Map will override the value in parameter file if they have the same parameter @param in @param params the key is the parameter name, and the value is the parameter value @param paramsFiles files which have the parameter setting @throws IOException]]>     PigServer server = new PigServer(); server.registerQuery("A = load 'foo';"); server.registerQuery("B = filter A by $0 > 0;"); server.registerQuery("C = order B by $1;"); Then 
     server.openIterator("B");
    filtered but unsorted data will be returned. If instead a user does 
     server.openIterator("C");
    filtered and sorted data will be returned. @param id Alias to open iterator for @return iterator of tuples returned from the script @throws IOException]]>     PigServer server = new PigServer(); server.registerQuery("A = load 'foo';"); server.registerQuery("B = filter A by $0 > 0;"); server.registerQuery("C = order B by $1;"); Then 
     server.store("B", "bar");
    filtered but unsorted data will be stored to the file bar. If instead a user does 
     server.store("C", "bar");
    filtered and sorted data will be stored to the file bar. Equivalent to calling {@link #store(String, String, String)} with org.apache.pig.PigStorage as the store function. @param id The alias to store @param filename The file to which to store to @return {@link ExecJob} containing information about this job @throws IOException]]>     PigServer server = new PigServer(); server.registerQuery("A = load 'foo';"); server.registerQuery("B = filter A by $0 > 0;"); server.registerQuery("C = order B by $1;"); Then 
     server.store("B", "bar", "mystorefunc");
    filtered but unsorted data will be stored to the file bar using mystorefunc. If instead a user does 
     server.store("C", "bar", "mystorefunc");
    filtered and sorted data will be stored to the file bar using mystorefunc.

    @param id The alias to store @param filename The file to which to store to @param func store function to use @return {@link ExecJob} containing information about this job @throws IOException]]> DOT format. @param verbose Controls the amount of information printed @param markAsExecute When set will treat the explain like a call to execute in the respoect that all the pending stores are marked as complete. @param lps Stream to print the logical tree @param eps Stream to print the ExecutionEngine trees. If null, then will print to files @param dir Directory to print ExecutionEngine trees. If null, will use eps @param suffix Suffix of file names @throws IOException if the requested alias cannot be found.]]> OUT exec(IN input) method when writting a UDF that operates on only the first input (of expected type IN) from the Tuple.]]> /user/ @return the absolute location based on the arguments passed @throws IOException if the conversion is not possible]]> /user/ @return the absolute location based on the arguments passed @throws IOException if the conversion is not possible]]>

    Extend this class and implement the 
    OUT exec(Tuple input)
    method when writting a UDF that operates on multiple inputs from the Tuple.]]>     Public interfaces and classes for Pig.

    {@link org.apache.pig.PigServer} is the interface to Pig for use by other java programs. Other interfaces and classes in this package are for use by users when implementing user defined functions.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> o is a DateTimeWritable with the same value.]]> o is a DoubleWritable with the same value.]]> Kerberos is a authentication system that uses tickets with a limited valitity time.
    As a consequence running a pig script on a kerberos secured hadoop cluster limits the running time to at most the remaining validity time of these kerberos tickets. When doing really complex analytics this may become a problem as the job may need to run for a longer time than these ticket times allow.
    A kerberos keytab file is essentially a Kerberos specific form of the password of a user.
    It is possible to enable a Hadoop job to request new tickets when they expire by creating a keytab file and make it part of the job that is running in the cluster. This will extend the maximum job duration beyond the maximum renew time of the kerberos tickets.

    Usage:

    1. Create a keytab file for the required principal.

      Using the ktutil tool you can create a keytab using roughly these commands:
      addent -password -p niels@EXAMPLE.NL -k 1 -e rc4-hmac
      addent -password -p niels@EXAMPLE.NL -k 1 -e aes256-cts
      wkt niels.keytab

    2. Set the following properties (either via the .pigrc file or on the command line via -P file)
      • java.security.krb5.conf
        The path to the local krb5.conf file.
        Usually this is "/etc/krb5.conf"
      • hadoop.security.krb5.principal
        The pricipal you want to login with.
        Usually this would look like this "niels@EXAMPLE.NL"
      • hadoop.security.krb5.keytab
        The path to the local keytab file that must be used to authenticate with.
        Usually this would look like this "/home/niels/.krb/niels.keytab"
    NOTE: All paths in these variables are local to the client system starting the actual pig script. This can be run without any special access to the cluster nodes.]]>     contextSignature.]]> When writing data, the first entry in the {@link Tuple} is treated as the row in the Accumulo key, while subsequent entries in the tuple are handled as columns in that row. {@link Map}s are expanded, placing the map key in the column family and the map value in the Accumulo value. Scalars are placed directly into the value with an empty column qualifier. If the columns argument on the constructor is omitted, null or the empty String, no column family is provided on the Keys created for Accumulo

    When reading data, if aggregateColfams is true, elements in the same row and column family are aggregated into a single {@link Map}. This will result in a {@link Tuple} of length (unique_column_families + 1) for the given row. If aggregateColfams is false, column family and column qualifier are concatenated (separated by a colon), and placed into a {@link Map}. This will result in a {@link Tuple} with two entries, where the latter element has a number of elements equal to the number of columns in the given row.

    ]]>     Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]>
  • pig.exec.reducers.bytes.per.reducer - how many bytes of input per reducer (default is 1000*1000*1000)
  • pig.exec.reducers.max - constrain the maximum number of reducer task (default is 999)
    • If using a loader that implements LoadMetadata the reported input size is used, otherwise attempt to determine size from the filesystem.

    e.g. the following is your pig script 

     a = load '/data/a';
 b = load '/data/b';
 c = join a by $0, b by $0;
 store c into '/tmp';
    and the size of /data/a is 1000*1000*1000, and the size of /data/b is 2*1000*1000*1000 then the estimated number of reducer to use will be (1000*1000*1000+2*1000*1000*1000)/(1000*1000*1000)=3]]>     Comparator Design

    A few words on how comparators are chosen. In almost all cases we use raw comparators (the one exception being when the user provides a comparison function for order by). For order by queries the PigTYPERawComparator functions are used, where TYPE is Int, Long, etc. These comparators are null aware and asc/desc aware. The first byte of each of the NullableTYPEWritable classes contains info on whether the value is null. Asc/desc is written as an array into the JobConf with the key pig.sortOrder so that it can be read by each of the comparators as part of their setConf call.

    For non-order by queries, PigTYPEWritableComparator classes are used. These are all just type specific instances of WritableComparator.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> UdfContext.getUdfContext().getJobConf()]]> (combine)? -> (reduce)? inner operators. The inner operators contain the physical plan of the execution phase.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Implementation of physical operators that use hadoop as the execution engine and data storage.

    Design

    Physical operators use the operator, plan, visitor, and optimizer framework provided by the {@link org.apache.pig.impl.plan} package.

    As with {@link org.apache.pig.impl.logicalLayer}, physical operators consist of {@link org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators} and {@link org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators}. In many data processing systems relational operators and expression operators are modeled as different entities because they behave differently. Pig blurs, though does not entirely remove, this distinction because of its support for nested operations.

    Conceptually, relational operators work on an entire relation (in Pig's case, a bag). In terms of implementation, they operate on one record (tuple) at a time. This avoids needing to load the entire relation into memory before operating on it.

    Expression operators, on the other hand, operate on the assumption that they are provided their entire input at invocation time and provide their entire output when they are finished.

    Pig's hadoop implementation implements a pull based model, where each operator calls getNext() on the operator before it in the plan. getNext() is implemented for each of the different data types, so that operators can request the data type they expect. Relational operators will always expect a tuple. Expression operators can request any data type.

    As with the logical plan, physical relational operators often have embedded physical plans. When a relational operator calls getNext() on its predecessor and receives a tuple, it will attach that tuple to its embedded physical plan(s) and then call getNext() on the root node(s) of those plan(s) in order to get the output. For example, the Pig Latin filter A by $0 != 5 will produce a POFilter object, with an embedded physical plan that consists of POProject(0), POConst(5), both attached to PONotEqual. Each time POFilter.getNext() is called, it will call its predecessors getNext() method, and then attach the input to POProject and POConst. It will then call PONotEqual.getNext(). PONotEqual will in turn call POProject.getNext() and POConst.getNext(), and then evaluate and return the results. If the result is true, POFilter will return its input tuple. If the answer is false, it will call it's predecessor's getNext() method and try again.

    Given Pig's nested data and execution models, there are places it is necessary to move between relational and expression operators. Consider the following Pig Latin script: A = load 'myfile'; B = group A by $0; C = foreach B { C1 = filter $1 by $0 > 0; C2 = distinct C1; generate group, COUNT(C2), SUM(C1.$0); } In particular, the foreach section presents some interesting challenges.

    First, foreach has three separate outputs, all of which require separate but parallel executions. To address this, each element of the foreach is described by a separate embedded plan. This can cause duplication of operations, as in this plan. In this case splitting the plans for COUNT and SUM cause a double execution of the C1 = filter section of the script. But it avoids needing to place a split operator between filter -> distinct and filter -> SUM.

    The second issue presented by the nested logic is that the foreach operator is going to receive a tuple with the format ($0, bag), where bag is a collection of all the tuples with a given value for $0. It will then attach that to the filter. But filter does not expect a bag. It expects to get tuples. On the other end, distinct will be outputing tuples. But COUNT() expects C2 to be a bag that can be processed by COUNT as a whole.

    To address this issue, some operators have been modified to provide "bookend" functionality. That is, the ability to translate between relational and expression operators. The embedded plan for calculating the COUNT in the foreach will look like: POProject(1) -> PODistinct -> POProject(*) -> COUNT(). The first POProject(1) will have a bag attached as its input by POForeach. But POFilter will call getNext(Tuple). In this case, POProject will know to open the bag and provide the tuples one at a time, until the bag is empty, at which point it will return STATUS_EOP. The PODistinct will be expecting to return tuples, but POProject(*) will call getNext(bag). In this case all relational operators will be able to accumulate all of the tuples by calling getNext(tuple) on themselves until they see STATUS_EOP, packaging those tuples into a bag, and then returning that bag.

    And third, project is being subtly overloaded here. In cases where the script says C = foreach B generate $1, this type of projection means take the second element from the tuple and project it. But in cases like C = foreach B generate SUM($1.$0) and $1 is a bag, this type of projection expects to receive a bag ($1) and output a modified bag ($1 with only the first field, $0, remaining in all the tuples in the bag). To handle this issue, POProject will, when it sees that its predecessor is a POProject and its successors is an expression operator it will perform a projection on the bag (that is, perform the specified project on each tuple in the bag) rather than on a tuple.]]> The output is consumed by for the demultiplexer operator (PODemux) in the format (key, {bag of tuples}) where key is an indexed WritableComparable, not the wrapped value as a pig type.]]> This operator is used when merging multiple Map-Reduce splittees into a Map-only splitter during multi-query optimization. The package operators of the reduce plans of the splittees form an indexed package list inside this operator. When this operator receives an input, it extracts the index from the key and calls the corresponding package to get the output data.

    Due to the recursive nature of multi-query optimization, this operator may be contained in another multi-query packager.

    The successor of this operator must be a PODemux operator which knows how to consume the output of this operator.]]> This operator is used when merging multiple Map-Reduce splittees into a Map-only splitter during multi-query optimization. The reduce physical plans of the splittees become the inner plans of this operator.

    Due to the recursive nature of multi-query optimization, this operator may be contained in another demux operator.

    The predecessor of this operator must be a POMultiQueryPackage operator which passes the index (indicating which inner reduce plan to run) along with other data to this operator.]]> Ex. join A by ($0,$1), B by ($1,$2);]]> The assumption here is that the logical to physical translation will create this dummy operator with just the filename using which the input branch will be stored and used for loading Also the translation should make sure that appropriate filter operators are configured as outputs of this operator using the conditions specified in the LOSplit. So LOSplit will be converted into: | | | Filter1 Filter2 ... Filter3 | | ... | | | ... | ---- POSplit -... ---- This is different than the existing implementation where the POSplit writes to sidefiles after filtering and then loads the appropriate file.

    The approach followed here is as good as the old approach if not better in many cases because of the availability of attachinInputs. An optimization that can ensue is if there are multiple loads that load the same file, they can be merged into one and then the operators that take input from the load can be stored. This can be used when the mapPlan executes to read the file only once and attach the resulting tuple as inputs to all the operators that take input from this load. In some cases where the conditions are exclusive and some outputs are ignored, this approach can be worse. But this leads to easier management of the Split and also allows to reuse this data stored from the split job whenever necessary.]]> StreamSpec. @return the {@link StreamingCommand} for this StreamSpec]]>

    In 2) key1 key2 equal? null null N foo null N null foo N foo foo Y (1,1) (1,1) Y (1,) (1,) N (1,2) (1,2) Y @param o @return]]> . Abuses the interface a bit in that there is no input RRD to convert in this case. Instead input is the source path of the POLoad.]]> The condition we look for is POFilters with a constant boolean (true) expression as it's plan.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Vertex 3 (Union + Store) will be optimized to Vertex 1 (Load + Store), Vertex 2 (Load + Store). Both the vertices will be writing output to same store location directly which is supported by Tez. 2) Union followed by groupby Vertex 1 (Load), Vertex 2 (Load) -> Vertex 3 (Union + POLocalRearrange) -> Vertex 4 (Group by) will be optimized to Vertex 1 (Load + POLR), Vertex 2 (Load + POLR) -> Vertex 4 (Group by)]]> Foo, specify a column as either Foo: or Foo:*. To fetch only columns in the CF that start with bar, specify Foo:bar*. The resulting tuple will always be the size of the number of tokens in columnList. Items in the tuple will be scalar values when a full column descriptor is specified, or a map of column descriptors to values when a column family is specified. @throws ParseException when unable to parse arguments @throws IOException]]>

  • -loadKey=(true|false) Load the row key as the first column
  • -gt=minKeyVal
  • -lt=maxKeyVal
  • -gte=minKeyVal
  • -lte=maxKeyVal
  • -regex=match regex on KeyVal
  • -limit=numRowsPerRegion max number of rows to retrieve per region
  • -maxResultsPerColumnFamily= Limit the maximum number of values returned per row per column family
  • -delim=char delimiter to use when parsing column names (default is space or comma)
  • -ignoreWhitespace=(true|false) ignore spaces when parsing column names (default true)
  • -cacheBlocks=(true|false) Set whether blocks should be cached for the scan (default false).
  • -caching=numRows number of rows to cache (faster scans, more memory).
  • -noWAL=(true|false) Sets the write ahead to false for faster loading.
  • -minTimestamp= Scan's timestamp for min timeRange
  • -maxTimestamp= Scan's timestamp for max timeRange
  • -timestamp= Scan's specified timestamp
  • -includeTimestamp= Record will include the timestamp after the rowkey on store (rowkey, timestamp, ...)
  • -includeTombstone= Record will include a tombstone marker on store after the rowKey and timestamp (if included) (rowkey, [timestamp,] tombstone, ...)
  • -caster=(HBaseBinaryConverter|Utf8StorageConverter) Utf8StorageConverter is the default To be used with extreme caution, since this could result in data loss (see http://hbase.apache.org/book.html#perf.hbase.client.putwal). @throws ParseException @throws IOException]]> Below is an example showing how to load data from HBase: 
    {@code
 raw = LOAD 'hbase://SampleTable'
       USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
       'info:first_name info:last_name friends:* info:*', '-loadKey true -limit 5')
       AS (id:bytearray, first_name:chararray, last_name:chararray, friends_map:map[], info_map:map[]);
 }
    This example loads data redundantly from the info column family just to illustrate usage. Note that the row key is inserted first in the result schema. To load only column names that start with a given prefix, specify the column name with a trailing '*'. For example passing friends:bob_* to the constructor in the above example would cause only columns that start with bob_ to be loaded.

    Note that when using a prefix like friends:bob_*, explicit HBase filters are set for all columns and prefixes specified. Querying HBase with many filters can cause performance degredation. This is typically seen when mixing one or more prefixed descriptors with a large list of columns. In that case better perfomance will be seen by either loading the entire family via friends:* or by specifying explicit column descriptor names.

    Below is an example showing how to store data into HBase: 

    {@code
 copy = STORE raw INTO 'hbase://SampleTableCopy'
       USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
       'info:first_name info:last_name friends:* info:*');
 }
    Note that STORE will expect the first value in the tuple to be the row key. Scalars values need to map to an explicit column descriptor and maps need to map to a column family name. In the above examples, the friends column family data from SampleTable will be written to a buddies column family in the SampleTableCopy table.]]>     true if stderr data of task should be persisted on HDFS, false otherwise]]> HadoopExecutableManager provides support for secondary outputs of the managed process and also persists the logs of the tasks on HDFS.]]> AddDuration returns the result of a DateTime object plus a Duration object

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Duration Format: http://en.wikipedia.org/wiki/ISO_8601#Durations

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime, dr:chararray);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dr: chararray}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,PT1S)
 (2008-02-06T02:06:02.000Z,PT1M)
 (2007-03-05T03:05:03.000Z,P1D)
 ...

 dtadd = FOREACH ISOin GENERATE AddDuration(dt, dr) AS dt1;

 DESCRIBE dtadd;
 dtadd: {dt1: datetime}

 DUMP dtadd;

 (2009-01-07T01:07:02.000Z)
 (2008-02-06T02:07:02.000Z)
 (2007-03-06T03:05:03.000Z)
    ]]>     AVG can operate on any numeric type. It can also operate on bytearrays, which it will cast to doubles. It expects a bag of tuples of one record each. If Pig knows from the schema that this function will be passed a bag of integers or longs, it will use a specially adapted version of AVG that uses integer arithmetic for summing the data. The return type of AVG will always be double, regardless of the input type.

    AVG implements the {@link org.apache.pig.Accumulator} interface as well. While this will never be the preferred method of usage it is available in case the combiner can not be used for a given calculation]]> -namespace Namespace for an automatically generated output schema.

  • -schemafile Specifies URL for avro schema file from which to read the input schema (can be local file, hdfs, url, etc).
  • -schemaclass Specifies fully qualified class name for avro class in your classpath which implements GenericContainer.
  • -examplefile Specifies URL for avro data file from which to copy the input schema (can be local file, hdfs, url, etc).
  • -allowrecursive Option to allow recursive schema definitions (default is false).
  • -doublecolons Option to translate Pig schema names with double colons to names with double underscores (default is false).
    • ]]>     'a_b_c' BagToString(bag, '+') -> 'a+b+c' bag = {(a,b), (c,d), (e,f)} BagToString(bag) --> 'a_b_c_d_e_f' If input bag is null, this UTF will return null;]]> (a,b,c) {(a,b), (c,d), (e,f)} --> (a,b,c,d,e,f); If input bag is null, this UDF will return null;]]> this filter. @param nbHash The number of hash functions to consider.]]> B = group A all;
    D = foreach B generate group,COR(A.$0,A.$1,A.$2);
    ]]>     This class implements {@link org.apache.pig.Algebraic}, so if possible the execution will performed in a distributed fashion.

    There are no restrictions as to the data types inside the bag to be counted.

    COUNT implements the {@link org.apache.pig.Accumulator} interface as well. While this will never be the preferred method of usage it is available in case the combiner can not be used for a given calculation.]]> This class implements {@link org.apache.pig.Algebraic}, so if possible the execution will performed in a distributed fashion.

    There are no restrictions as to the data types inside the bag to be counted.

    COUNT_STAR implements the {@link org.apache.pig.Accumulator} interface as well. While this will never be the preferred method of usage it is available in case the combiner can not be used for a given calculation.]]> B = group A all;
    D = foreach B generate group,COV(A.$0,A.$1,A.$2);
    ]]>     { (a, b, c), (null, null, null), (a, b, null), (a, null, c), (a, null, null), (null, b, c), (null, null, c), (null, b, null) }

    The "all" marker is null by default, but can be set to an arbitrary string by invoking a constructor (via a DEFINE). The constructor takes a single argument, the string you want to represent "all".

    Usage goes something like this: 

    {@code
 events = load '/logs/events' using EventLoader() as (lang, event, app_id);
 cubed = foreach x generate
   FLATTEN(piggybank.CubeDimensions(lang, event, app_id))
     as (lang, event, app_id),
   measure;
 cube = foreach (group cubed
                 by (lang, event, app_id) parallel $P)
        generate
   flatten(group) as (lang, event, app_id),
   COUNT_STAR(cubed),
   SUM(measure);
 store cube into 'event_cube';
 }

    Note: doing this with non-algebraic aggregations on large data can result in very slow reducers, since one of the groups is going to get all the records in your relation.]]> DaysBetween returns the number of days between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     The implementation assumes that both bags being passed to this function will fit entirely into memory simultaneously. If that is not the case the UDF will still function, but it will be very slow.]]> tuple.get(0) against tuple.get(1) to determine if the first argument ends with the string in the second.]]> The Function must have a default no-arg constructor, which will be used. For Functions that take args in the constructor, initialize the function in a subclass of this one and call super(function).

    Example: DEFINE myUdf FunctionWrapperEvalFunc('MyFunction')]]> required, if seemingly nonsensical, for Pig to do its work.

    The Invoker family of udfs understand the following class names (all case-independent):

  • String
  • Long
  • Float
  • Double
  • Int

    Invokers can also work with array arguments, represented in Pig as DataBags of single-tuple elements. Simply refer to string[], for example.

    This UDF allows one to dynamically invoke Java methods that return a T

    Usage of the Invoker family of UDFs (adjust as appropriate): 

     {@code
 -- invoking a static method
 DEFINE StringToLong InvokeForLong('java.lang.Long.valueOf', 'String')
 longs = FOREACH strings GENERATE StringToLong(some_chararray);

 -- invoking a method on an object
 DEFINE StringConcat InvokeForString('java.lang.String.concat', 'String String', 'false')
 concatenations = FOREACH strings GENERATE StringConcat(str1, str2); 
 }

    The first argument to the constructor is the full path to desired method.
    The second argument is a list of classes of the method parameters.
    If the method is not static, the first element in this list is the object to invoke the method on.
    The second argument is optional (a no-argument static method is assumed if it is not supplied).
    The third argument is the keyword "static" (or "true") to signify that the method is static.
    The third argument is optional, and true by default.

    @param ]]>

  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>
  • Jodatime: http://joda-time.sourceforge.net/
  • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

    •  Example usage:
 
 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.071Z)
 (2008-02-06T02:06:02.062Z)
 (2007-03-05T03:05:03.053Z)
 ...
 
 truncated = FOREACH ISO in GENERATE GetYear(dt) AS year,
     GetMonth(dt) as month,
     GetWeek(dt) as week,
     GetWeekYear(dt) as weekyear,
     GetDay(dt) AS day,
     GetHour(dt) AS hour,
     GetMinute(dt) AS min,
     GetSecond(dt) AS sec;
     GetMillSecond(dt) AS milli;

 DESCRIBE truncated;
 truncated: {year: int,month: int,week: int,weekyear: int,day: int,hour: int,min: int,sec: int,milli: int}

 DUMP truncated;
 (2009,1,2,2009,7,1,7,1,71)
 (2008,2,6,2008,6,2,6,2,62)
 (2007,3,10,2007,5,3,5,3,53)
    ]]>     HoursBetween returns the number of hours between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     UDF swaps keys with values and returns the new inverse Map.
    Note in case original values are non-unique, the resulting Map would
    contain String Key -> DataBag of values. Here the bag of values is composed
    of the original keys having the same value. 
     Note: 1. UDF accepts Map with Values of primitive data type
       2. UDF returns Map
 
 grunt> cat 1data
 [open#1,1#2,11#2]
 [apache#2,3#4,12#24]

 
    
 grunt> a = load 'data' as (M:[int]);
 grunt> b = foreach a generate INVERSEMAP($0);

 grunt> dump b;
 ([2#{(1),(11)},apache#{(open)}])
 ([hadoop#{(apache),(12)},4#{(3)}])
    ]]>     For each object represented by the path (either directly, or via a glob): If object is a directory, and path/metaname exists, use that as the metadata file. Else if parentPath/metaname exists, use that as the metadata file.

    Resolving conflicts, merging the metadata, etc, is not handled by this method and should be taken care of by downstream code.

    @param path Path, as passed in to a LoadFunc (may be a Hadoop glob) @param metaname Metadata file designation, such as .pig_schema or .pig_stats @param conf configuration object @return Set of element descriptors for all metadata files associated with the files on the path.]]> 

     
 grunt> cat data
 [open#apache,1#2,11#2]
 [apache#hadoop,3#4,12#hadoop]

 grunt> a = load 'data' as (M:[]);
 grunt> b = foreach a generate KEYSET($0);
 grunt> dump b;
 ({(open),(1),(11)})
 ({(3),(apache),(12)})
    ]]>
  • the string to process
  • the substring to find @exception java.io.IOException @return last location of substring, or null in case of processing errors.]]> A = load 'mydata' as (name); B = foreach A generate LASTINDEXOF(name, ","); ]]> MAX can operate on any numeric type and on chararrays. It can also operate on bytearrays, which it will cast to doubles. It expects a bag of tuples of one record each. If Pig knows from the schema that this function will be passed a bag of integers or longs, it will use a specially adapted version of MAX that uses integer arithmetic for comparing the data. The return type of MAX will match the input type.

    MAX implements the {@link org.apache.pig.Accumulator} interface as well. While this will never be the preferred method of usage it is available in case the combiner can not be used for a given calculation.]]> MilliSecondsBetween returns the number of milliseconds between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     MIN can operate on any numeric type and on chararrays. It can also operate on bytearrays, which it will cast to doubles. It expects a bag of tuples of one record each. If Pig knows from the schema that this function will be passed a bag of integers or longs, it will use a specially adapted version of MIN that uses integer arithmetic for comparing the data. The return type of MIN will match the input type.

    MIN implements the {@link org.apache.pig.Accumulator} interface as well. While this will never be the preferred method of usage it is available in case the combiner can not be used for a given calculation.]]> MinutesBetween returns the number of minutes between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     NOTE: does not work with UDFs that implement the Accumulator interface

    Setting a default value will cause it to be used instead of null when the UDF times out. The appropriate value among in, long, string, etc, is used. The default fields of these annotations are arrays for Java reasons I won't bore you with.

    Set them as if they were primitives: @MonitoredUDF( intDefault=5 )

    There is currently no way to provide a default ByteArray, Tuple, Map, or Bag. Null will always be used for those.

    Currently this annotation is ignored when the Accumulator interface is used.]]> MonthsBetween returns the number of months between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     A non-deterministic UDF is one that can produce different results when invoked on the same input. Examples of non-deterministic behavior might be, for example, getCurrentTime() or random().

    Certain Pig optimizations depend on UDFs being deterministic. It is therefore very important for correctness that non-deterministic UDFs be annotated as such.

    ]]>
  • -s, --stripeSize Set the stripe size for the file
  • -r, --rowIndexStride Set the distance between entries in the row index
  • -b, --bufferSize The size of the memory buffers used for compressing and storing the stripe in memory
  • -p, --blockPadding Sets whether the HDFS blocks are padded to prevent stripes from straddling blocks
  • -c, --compress Sets the generic compression that is used to compress the data. Valid codecs are: NONE, ZLIB, SNAPPY, LZO
  • -v, --version Sets the version of the file that will be written ]]> OutputSchema to tell Pig what the expected output is. This can be used in place of {@link EvalFunc#outputSchema(Schema)}

    The default implementation of {@link EvalFunc#outputSchema(Schema)} will look at this annotation and return an interpreted schema, if the annotation is present.

    Implementing a custom {@link EvalFunc#outputSchema(Schema)} will override the annotation (unless you deal with it explicitly, or by calling super.outputSchema(schema)).

    Here's an example of a complex schema declared in an annotation: @OutputSchema("y:bag{t:tuple(len:int,word:chararray)}")]]> Understands the following options, which can be specified in the second paramter:

    • -schema Loads / Stores the schema of the relation using a hidden JSON file.
    • -noschema Ignores a stored schema during loading.
    • -tagFile Appends input source file name to beginning of each tuple.
    • -tagPath Appends input source file path to beginning of each tuple.
    @param delimiter the single byte character that is used to separate fields. @param options a list of options that can be used to modify PigStorage behavior @throws ParseException]]>     An optional second constructor argument is provided that allows one to customize advanced behaviors. A list of available options is below:
    • -schema Reads/Stores the schema of the relation using a hidden JSON file.
    • -noschema Ignores a stored schema during loading.
    • -tagFile Appends input source file name to beginning of each tuple.
    • -tagPath Appends input source file path to beginning of each tuple.

    Schemas

    If -schema is specified, a hidden ".pig_schema" file is created in the output directory when storing data. It is used by PigStorage (with or without -schema) during loading to determine the field names and types of the data without the need for a user to explicitly provide the schema in an as clause, unless -noschema is specified. No attempt to merge conflicting schemas is made during loading. The first schema encountered during a file system scan is used. If the schema file is not present while '-schema' option is used during loading, it results in an error.

    In addition, using -schema drops a ".pig_headers" file in the output directory. This file simply lists the delimited aliases. This is intended to make export to tools that can read files with header lines easier (just cat the header to your data).

    Source tagging

    If-tagFile is specified, PigStorage will prepend input split name to each Tuple/row. Usage: A = LOAD 'input' using PigStorage(',','-tagFile'); B = foreach A generate $0; The first field (0th index) in each Tuple will contain input file name. If-tagPath is specified, PigStorage will prepend input split path to each Tuple/row. Usage: A = LOAD 'input' using PigStorage(',','-tagPath'); B = foreach A generate $0; The first field (0th index) in each Tuple will contain input file path

    Note that regardless of whether or not you store the schema, you always need to specify the correct delimiter to read your data. If you store reading delimiter "#" and then load using the default delimiter, your data will not be parsed correctly.

    Compression

    Storing to a directory whose name ends in ".bz2" or ".gz" or ".lzo" (if you have installed support for LZO compression in Hadoop) will automatically use the corresponding compression codec.
    output.compression.enabled and output.compression.codec job properties also work.

    Loading from directories ending in .bz2 or .bz works automatically; other compression formats are not auto-detected on loading.]]> String specifying the field delimiter.]]> fieldDel separated lines and fieldDel separated lines into tuples.]]>

    Syntax:
    String RegexExtract(String expression, String regex, int match_index).
    Input:
    expression-source string.
    regex-regular expression.
    match_index-index of the group to extract.
    Output:
    extracted group, if fail, return null.
    Matching strategy:
    Try to only match the first sequence by using {@link Matcher#find()} instead of {@link Matcher#matches()} (default useMatches=false).
    DEFINE NON_GREEDY_EXTRACT REGEX_EXTRACT('true');
    ]]>
    Syntax:
    String RegexExtractAll(String expression, String regex).
    Input:
    expression-source string.
    regex-regular expression.
    Output:
    A tuple of matched strings.
    Matching strategy:
    Trying to match the entire input by using {@link Matcher#matches()} instead of {@link Matcher#find()} (default useMatches=true).
    DEFINE GREEDY_EXTRACT REGEX_EXTRACT_ALL('false');
    ]]>     A = load 'mydata' as (name); B = foreach A generate REPLACE(name, 'blabla', 'bla'); The first argument is a string on which to perform the operation. The second argument is treated as a regular expression. The third argument is the replacement string. This is a wrapper around Java's String.replaceAll(String, String);]]> { (a, b, c), (a, b, null), (a, null, null), (null, null, null) } ]]> SecondsBetween returns the number of seconds between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     tuple.get(0) against tuple.get(1) to determine if the first argument starts with the string in the second.]]> if not provided, it's assumed to be '\s' (space) the optional third column may provide a limit to the number of results.
    If limit is not provided, 0 is assumed, as per Java's split(). @exception java.io.IOException]]>     input tuple: first column is assumed to have a string to split;
    the optional second column is assumed to have the delimiter or regex to split on;
    if not provided, it's assumed to be '\s' (space)
    the optional third column may provide a limit to the number of results.
    If limit is not provided, 0 is assumed, as per Java's split().]]>     if not provided, it's assumed to be '\s' (space) the optional third column may provide a limit to the number of results.
    If limit is not provided, 0 is assumed, as per Java's split(). @throws java.io.IOException]]>     input tuple: first column is assumed to have a string to split;
    the optional second column is assumed to have the delimiter or regex to split on;
    if not provided, it's assumed to be '\s' (space)
    the optional third column may provide a limit to the number of results.
    If limit is not provided, 0 is assumed, as per Java's split().]]>     A = load 'mydata' as (name); B = foreach A generate SUBSTRING(name, 10, 12); First argument is the string to take a substring of.
    Second argument is the index of the first character of substring.
    Third argument is the index of the last character of substring.
    if the last argument is past the end of the string, substring of (beginIndex, length(str)) is returned.]]>     If null, bag arguments are replaced by empty bags.

    The implementation assumes that both bags being passed to this function will fit entirely into memory simultaneously.
    If that is not the case the UDF will still function, but it will be very slow.]]> SubtractDuration returns the result of a DateTime object plus a Duration object

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Duration Format: http://en.wikipedia.org/wiki/ISO_8601#Durations

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime, dr:chararray);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dr: chararray}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,PT1S)
 (2008-02-06T02:06:02.000Z,PT1M)
 (2007-03-05T03:05:03.000Z,P1D)
 ...

 dtsubtract = FOREACH ISOin GENERATE SubtractDuration(dt, dr) AS dt1;

 DESCRIBE dtsubtract;
 dtsubtract: {dt1: datetime}

 DUMP dtsubtract;

 (2009-01-07T01:07:00.000Z)
 (2008-02-06T02:05:02.000Z)
 (2007-03-04T03:05:03.000Z)
    ]]>     SUM can operate on any numeric type. It can also operate on bytearrays, which it will cast to doubles. It expects a bag of tuples of one record each. If Pig knows from the schema that this function will be passed a bag of integers or longs, it will use a specially adapted version of SUM that uses integer arithmetic for summing the data. The return type of SUM is double for float, double, or bytearray arguments and long for int or long arguments.

    SUM implements the {@link org.apache.pig.Accumulator} interface as well. While this will never be the preferred method of usage it is available in case the combiner can not be used for a given calculation.]]> describe a; a: {a0: int,a1: int} grunt> b = foreach a generate TOBAG(a0,a1); grunt> describe b; b: {{int}} example 2 grunt> describe a; a: {a0: (x: int),a1: (x: int)} grunt> b = foreach a generate TOBAG(a0,a1); grunt> describe b; b: {{(x: int)}} example 3 grunt> describe a; a: {a0: (x: int),a1: (y: int)} -- note that the inner schemas have matching types but different field aliases. -- the aliases of the first argument (a0) will be used in output schema: grunt> b = foreach a generate TOBAG(a0,a1); grunt> describe b; b: {{(x: int)}} example 4 grunt> describe a; a: {a0: (x: int),a1: (x: chararray)} -- here the inner schemas do not match, so output schema is not well defined: grunt> b = foreach a generate TOBAG(a0,a1); grunt> describe b; b: {{NULL}}]]> ToDate converts the ISO or the customized string or the Unix timestamp to the DateTime object.

    ToDate is overloaded.

    Syntax:
    DateTime ToDate(Long millis).
    Input:
    the milliseconds.
    Output:
    the DateTime object.
    Syntax:
    DateTime ToDate(String dtStr).
    Input:
    the ISO format date time string.
    Output:
    the DateTime object.
    Syntax:
    DateTime ToDate(String dtStr, String format).
    Input:
    dtStr: the string that represents a date time.
    format: the format string.
    Output:
    the DateTime object.
    Syntax:
    DateTime ToDate(String dtStr, String format, String timezone).
    Input:
    dtStr: the string that represents a date time.
    format: the format string.
    timezone: the timezone string.
    Output:
    the DateTime object.
    ]]>     1, $2->$3 This UDF also accepts a bag with 'pair' tuples (i.e. tuples with a 'key' and a 'value').]]> ToMilliSeconds converts the DateTime to the number of milliseconds that have passed since January 1, 1970 00:00:00.000 GMT.

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601
    • Unix Time: http://en.wikipedia.org/wiki/Unix_time

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 toMilliSeconds = FOREACH ISOin GENERATE ToMilliSeconds(dt) AS unixTime:long;

 DESCRIBE toMilliSeconds;
 toMilliSeconds: {unixTime: long}

 DUMP toMilliSeconds;

 (1231290421000L)
 (1202263562000L)
 (1173063903000L)
 ...
    ]]>     ToString converts the DateTime object of the ISO or the customized string.

    ]]>     ToUnixTime converts the DateTime to the Unix Time Long

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601
    • Unix Time: http://en.wikipedia.org/wiki/Unix_time

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (dt:datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 toUnix = FOREACH ISOin GENERATE ToUnixTime(dt) AS unixTime:long;

 DESCRIBE toUnix;
 toUnix: {unixTime: long}

 DUMP toUnix;

 (1231290421L)
 (1202263562L)
 (1173063903L)
 ...
    ]]>     -namespace Namespace for an automatically generated output schema.
  • -schemafile Specifies URL for avro schema file from which to read the input schema (can be local file, hdfs, url, etc).
  • -examplefile Specifies URL for avro data file from which to copy the input schema (can be local file, hdfs, url, etc).
  • -allowrecursive Option to allow recursive schema definitions (default is false).
    • ]]>     Note that output tuple contains all values, not just unique ones.
    For obtaining unique values from map, use VALUESET instead. 
     
 grunt> cat data
 [open#apache,1#2,11#2]
 [apache#hadoop,3#4,12#hadoop]

 grunt> a = load 'data' as (M:[]);
 grunt> b = foreach a generate VALUELIST($0);
 grunt> dump b;
 ({(apache),(2),(2)})
 ({(4),(hadoop),(hadoop)})
    ]]>     Note, this UDF returns only unique values. For all values, use
    VALUELIST instead. 
     
 grunt> cat data
 [open#apache,1#2,11#2]
 [apache#hadoop,3#4,12#hadoop]

 grunt> a = load 'data' as (M:[]);
 grunt> b = foreach a generate VALUELIST($0);
 ({(apache),(2)})
 ({(4),(hadoop)})
    ]]>     WeeksBetween returns the number of weeks between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     YearsBetween returns the number of years between two DateTime objects

    • Jodatime: http://joda-time.sourceforge.net/
    • ISO8601 Date Format: http://en.wikipedia.org/wiki/ISO_8601

     Example usage:

 ISOin = LOAD 'test.tsv' USING PigStorage('\t') AS (datetime, dt2:datetime);

 DESCRIBE ISOin;
 ISOin: {dt: datetime,dt2: datetime}

 DUMP ISOin;

 (2009-01-07T01:07:01.000Z,2008-02-01T00:00:00.000Z)
 (2008-02-06T02:06:02.000Z,2008-02-01T00:00:00.000Z)
 (2007-03-05T03:05:03.000Z,2008-02-01T00:00:00.000Z)
 ...

 diffs = FOREACH ISOin GENERATE YearsBetween(dt, dt2) AS years,
 MonthsBetween(dt, dt2) AS months,
 WeeksBetween(dt, dt2) AS weeks,
 DaysBetween(dt, dt2) AS days,
 HoursBetween(dt, dt2) AS hours,
 MinutesBetween(dt, dt2) AS mins,
 SecondsBetween(dt, dt2) AS secs;
 MilliSecondsBetween(dt, dt2) AS millis;

 DESCRIBE diffs;
 diffs: {years: long,months: long,weeks: long,days: long,hours: long,mins: long,secs: long,millis: long}

 DUMP diffs;

 (0L,11L,48L,341L,8185L,491107L,29466421L,29466421000L)
 (0L,0L,0L,5L,122L,7326L,439562L,439562000L)
 (0L,-10L,-47L,-332L,-7988L,-479334L,-28760097L,-28760097000L)
    ]]>     This package contains builtin Pig UDFs. This includes {@link org.apache.pig.EvalFunc}s, {@link org.apache.pig.LoadFunc}s and {@link org.apache.pig.StoreFunc}s.]]>     B, "C"->D @return a map containing the provided objects]]> PigServer pigServer = new PigServer(ExecType.LOCAL); Data data = resetData(pigServer); data.set("foo", tuple("a"), tuple("b"), tuple("c"), tuple(map("d","e", "f","g")), tuple(bag(tuple("h"),tuple("i"))) ); pigServer.registerQuery("A = LOAD 'foo' USING mock.Storage();"); pigServer.registerQuery("STORE A INTO 'bar' USING mock.Storage();"); List out = data.get("bar"); assertEquals(tuple("a"), out.get(0)); assertEquals(tuple("b"), out.get(1)); assertEquals(tuple("c"), out.get(2)); assertEquals(tuple(map("f", "g", "d", "e" )), out.get(3)); assertEquals(tuple(bag(tuple("h"),tuple("i"))), out.get(4)); With Schema: 
      PigServer pigServer = new PigServer(ExecType.LOCAL);
  Data data = resetData(pigServer);

  data.set("foo", "blah:chararray",
      tuple("a"),
      tuple("b"),
      tuple("c")
      );

  pigServer.registerQuery("A = LOAD 'foo' USING mock.Storage();");
  pigServer.registerQuery("B = FOREACH A GENERATE blah as a, blah as b;");
  pigServer.registerQuery("STORE B INTO 'bar' USING mock.Storage();");

  assertEquals(schema("a:chararray,b:chararray"), data.getSchema("bar"));

  List out = data.get("bar");
  assertEquals(tuple("a", "a"), out.get(0));
  assertEquals(tuple("b", "b"), out.get(1));
  assertEquals(tuple("c", "c"), out.get(2));
    ]]>     @see org.apache.pig.data.InterSedes#readDatum(java.io.DataInput, byte)]]> DataBag provides an Iterator interface, that allows callers to read through the contents. The iterators are aware of the data spilling. They have to be able to handle reading from files, as well as the fact that data they were reading from memory may have been spilled to disk underneath them.

    The DataBag interface assumes that all data is written before any is read. That is, a DataBag cannot be used as a queue. If data is written after data is read, the results are undefined. This condition is not checked on each add or read, for reasons of speed. Caveat emptor.

    Since spills are asynchronous (the memory manager requesting a spill runs in a separate thread), all operations dealing with the mContents Collection (which is the collection of tuples contained in the bag) have to be synchronized. This means that reading from a DataBag is currently serialized. This is ok for the moment because pig execution is currently single threaded. A ReadWriteLock was experimented with, but it was found to be about 10x slower than using the synchronize keyword. If pig changes its execution model to be multithreaded, we may need to return to this issue, as synchronizing reads will most likely defeat the purpose of multi-threading execution.

    DataBags come in several types, default, sorted, and distinct. The type must be chosen up front, there is no way to convert a bag on the fly. Default data bags do not guarantee any particular order of retrieval for the tuples and may contain duplicate tuples. Sorted data bags guarantee that tuples will be retrieved in order, where "in order" is defined either by the default comparator for Tuple or the comparator provided by the caller when the bag was created. Sorted bags may contain duplicates. Distinct bags do not guarantee any particular order of retrieval, but do guarantee that they will not contain duplicate tuples.]]> know that the object you have is a Boolean you should just cast it. @param o object to cast @param type of the object you are casting @return The object as a Boolean. @throws ExecException if the type can't be forced to a Boolean.]]> know that the object you have is an Integer you should just cast it. @param o object to cast @param type of the object you are casting @return The object as an Integer. @throws ExecException if the type can't be forced to an Integer.]]> know that the object you have is an Integer you should just cast it. Unlike {@link #toInteger(Object, byte)} this method will first determine the type of o and then do the cast. Use {@link #toInteger(Object, byte)} if you already know the type. @param o object to cast @return The object as an Integer. @throws ExecException if the type can't be forced to an Integer.]]> know that the object you have is a Long you should just cast it. @param o object to cast @param type of the object you are casting @return The object as a Long. @throws ExecException if the type can't be forced to a Long.]]> know that the object you have is a Long you should just cast it. Unlike {@link #toLong(Object, byte)} this method will first determine the type of o and then do the cast. Use {@link #toLong(Object, byte)} if you already know the type. @param o object to cast @return The object as a Long. @throws ExecException if the type can't be forced to an Long.]]> know that the object you have is a Float you should just cast it. @param o object to cast @param type of the object you are casting @return The object as a Float. @throws ExecException if the type can't be forced to a Float.]]> know that the object you have is a Float you should just cast it. Unlike {@link #toFloat(Object, byte)} this method will first determine the type of o and then do the cast. Use {@link #toFloat(Object, byte)} if you already know the type. @param o object to cast @return The object as a Float. @throws ExecException if the type can't be forced to an Float.]]> know that the object you have is a Double you should just cast it. @param o object to cast @param type of the object you are casting @return The object as a Double. @throws ExecException if the type can't be forced to a Double.]]> know that the object you have is a DateTime you should just cast it. @param o object to cast @param type of the object you are casting @return The object as a Boolean. @throws ExecException if the type can't be forced to a Boolean.]]> know that the object you have is a Double you should just cast it. Unlike {@link #toDouble(Object, byte)} this method will first determine the type of o and then do the cast. Use {@link #toDouble(Object, byte)} if you already know the type. @param o object to cast @return The object as a Double. @throws ExecException if the type can't be forced to an Double.]]> know that the object you have is a String you should just cast it. @param o object to cast @param type of the object you are casting @return The object as a String. @throws ExecException if the type can't be forced to a String.]]> know that the object you have is a String you should just cast it. Unlike {@link #toString(Object, byte)} this method will first determine the type of o and then do the cast. Use {@link #toString(Object, byte)} if you already know the type. @param o object to cast @return The object as a String. @throws ExecException if the type can't be forced to a String.]]> know that the object you have is a Map you should just cast it. @param o object to cast @return The object as a Map. @throws ExecException if the type can't be forced to a Double.]]> know that the object you have is a Tuple you should just cast it. @param o object to cast @return The object as a Double. @throws ExecException if the type can't be forced to a Double.]]> know that the object you have is a bag you should just cast it. @param o object to cast @return The object as a Double. @throws ExecException if the type can't be forced to a Double.]]> containing the tuples]]> Time complexity: O(1), after allocation]]> to add a finalize() that calls delete on the files . This helps in getting rid of the finalize() in the classes such as DefaultAbstractBag, and they can be freed up without waiting for finalize to be called. Only if those classes have spilled to disk, there will be a (this) class that needs to be finalized. CAUTION: if you assign a new value for a variable of this type, the files (if any) in the old object it pointed to will be scheduled for deletion. To avoid that call .clear() before assigning a new value.]]> . It only exists so that DataType.findType() can distinguish an internal map type that maps object to object from an external map type that is string to object.]]> containing the tuples]]> @param key Object]]> Tuples are constructed only by a {@link TupleFactory}. A {@link DefaultTupleFactory} is provided by the system. If users wish to use their own type of Tuple, they should also provide an implementation of {@link TupleFactory} to construct their types of Tuples.]]> This saves memory over creating a new byte[] and ByteArrayOutputStream each time data is written.

    Typical usage is something like the following:

    
 WritableByteArray buffer = new WritableByteArray();
 while (... loop condition ...) {
   buffer.reset();
   ... write to buffer ...
   byte[] data = buffer.getData();
   int dataLength = buffer.getLength();
   ... write data to its ultimate destination ...
 }
    ]]>     This package contains implementations of Pig specific data types as well as support functions for reading, writing, and using all Pig data types.

    Whenever possible, Pig utilizes Java provided data types. These include Integer, Long, Float, Double, Boolean, String, and Map. Tuple, Bag, and DataByteArray are implemented in this package.

    Design

    The choice was made to utilize Java provided types for two main reasons. One, it minimizes the burden on UDF developers, as they will have full access to these types with no need to convert to and from Pig specific types. Two, maintenance costs will be lower as there is no need to implement and maintain Pig specific data classes. The drawback is that the only common parent of all these types is Object. Thus Pig is often required to treat its data objects as Objects and then implement static methods to manipulate these Objects, rather than being able to define a PigDatum class with common funcitons.

    Three data types were implemented as Pig specific classes: {@link org.apache.pig.data.DataByteArray}, {@link org.apache.pig.data.Tuple}, and {@link org.apache.pig.data.DataBag}.

    DataByteArray represents an array of bytes, with no interpretation of those bytes provided or assumed. This could have been represented as byte[], but a separate class was constructed to provide common functions needed to manipulate these objects.

    Tuple represents an ordered collection of data elements. Every field in a tuple can contain any Pig data type. Tuple is presented as an interface to allow differing implementations in cases where users have unique representations of their data that they wish to preserve in their in memory representations. The {@link org.apache.pig.data.TupleFactory} is an abstract class, to enable a user who has defined his own tuples to provide a factory that creates those tuples. Default implementations of Tuple and TupleFactory are provided and used by default.

    DataBag represents a collection of Tuples. DataBags can be of default type (no extra features), sorted (tuples are sorted according to a provided comparator function), or distinct (no duplicate tuples). As with Tuple, DataBag is presented as an interface, and {@link org.apache.pig.data.BagFactory} is an abstract class. Default implementations of DataBag, BagFactory, and all three types of bags are provided.]]> Google Protocol Buffers. It uses zig-zag encoding to efficiently encode signed values. If values are known to be nonnegative, {@link #writeUnsignedVarLong(long, DataOutput)} should be used. @param value value to encode @param out to write bytes to @throws IOException if {@link DataOutput} throws {@link IOException}]]> Google Protocol Buffers. Zig-zag is not used, so input must not be negative. If values can be negative, use {@link #writeSignedVarLong(long, DataOutput)} instead. This method treats negative input as like a large unsigned value. @param value value to encode @param out to write bytes to @throws IOException if {@link DataOutput} throws {@link IOException}]]> Encodes signed and unsigned values using a common variable-length scheme, found for example in Google's Protocol Buffers. It uses fewer bytes to encode smaller values, but will use slightly more bytes to encode large values.

    Signed values are further encoded using so-called zig-zag encoding in order to make them "compatible" with variable-length encoding.

    This is taken from mahout-core, and is included to avoid having to pull in the entirety of Mahout.

    ]]>     This can be called for jars that are pre-installed on the Hadoop cluster to reduce the size of the job jar.]]> -Dpig.notification.listener=MyClass -Dpig.notification.listener.arg=myConstructorStringArg This method will properly initialize the class with the args, if they exist. @param conf @param classParamKey the property used to identify the class @param argParamKey the property used to identify the class args @param clazz The class that is expected @return T]]> StreamingCommand @return StreamingCommand for the alias]]> "totalreducers": the value is an integer wich indicates the number of total reducers for this join job
  • "partition.list": the value is a bag which contains a list of tuples with each tuple representing partitions for a skewed key. The tuple has format of <join key>,<min index of reducer>, <max index of reducer>
    • For example, a join job configures 10 reducers, and the sampling process finds out 2 skewed keys, "swpv" needs 4 reducers and "swps" needs 2 reducers. The output file would be like following: {totalreducers=10, partition.list={(swpv,0,3), (swps,4,5)}} The name of this file is set into next MR job which does the actual join. That job uses this information to partition skewed keys properly]]>     returned by the underlying InputFormat's getSplits() method). So if the supplied split index is 0, this loader will read the entire file. If it is non zero it will read the partial file beginning from that split to the last split. The call sequence to use this is: 1) construct an object using the constructor 2) Call getNext() in a loop till it returns null]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param location - The location of the exception.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown. @param location - The location of the exception.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> The logical operators that represent a pig script and tools for manipulating those operators. The logical layer contains the logical operators themselves, as well as validators that check the logical plan, an optimizer, and a general visitor utility for working with the logical plans.

    Design

    Logical operators use the operator, plan, visitor, and optimizer framework provided by the {@link org.apache.pig.impl.plan} package.

    Logical operators consist of both relational and expression operators. Relational operators work on an entire bag. Expression operators work on an element of a tuple (which may also be a bag). Due to Pig's nested data and execution model the distinction between relational and expression operators is not always clear. And some operators such as LOProject function as both.

    In a traditional data base system, a query execution plan is constructed from relational operators, such as project, filter, sort, aggregate, join. Each of these may contain an expression tree, made up of expression operators. For example, consider a SQL query select a from T where a = 5;. The where clause would be represented by a filter operator with an expression tree for a=5.

    Pig takes a similar approach, except that the operators contained inside of a relational operator may also be relational. For example, a foreach statement that has a nested script, such as foreach B { C = distinct $1; generate group, COUNT(C);}. This foreach needs to contain not just an expression tree but the distinct relational operator. For this reason, Pig's relational operators do not contain expression trees. Instead they contain one or more LogicalPlans themselves. This allows Pig to arbitrarily nest the logical plan. In this sense Pig is more similar to a traditional procedural language where certain statements (e.g. if, while) can contain any other statement in the language rather than being like SQL where the statement execution tends to be more linear.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> 3, 2->3, and 3->4, this walker guarnatees that 1 and 2 will be visited before 3 and 3 before 4, but it does not guarantee whether 1 or 2 will be visited first.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> 3, 2->3, and 3->4, this walker guarantees that 4 will be visited before 3 and 3 before 1 and 2, but it does not guarantee whether 1 or 2 will be visited first.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user]]> Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer]]> Throwable.getMessage() method) shown to the user @param errCode - The error code shown to the user @param errSrc - The error source @param retry - If the exception is retriable or not @param detailedMsg - The detailed message shown to the developer @param cause - The cause (which is saved for later retrieval by the Throwable.getCause() method) indicating the source of this exception. A null value is permitted, and indicates that the cause is nonexistent or unknown.]]> stdin.]]> stdout.]]> stderr of the managed process. @throws IOException]]> ExecuatbleManager just sends out the received error message to the stderr of itself. @param error error message from the managed process.]]> ExecutableManager is responsible for startup/teardown of the external process and also for managing it. It feeds input records to the executable via it's stdin, collects the output records from the stdout and also diagnostic information from the stdout.]]> InputHandler for the given input specification of the StreamingCommand. @param command StreamingCommand @return InputHandler for the given input specification @throws ExecException]]> OutputHandler for the given output specification of the StreamingCommand. @param command StreamingCommand @return OutputHandler for the given output specification @throws ExecException]]> InputType @return the handled InputType]]> Tuple to the managed executable. @param t input Tuple @throws IOException]]> InputHandler since there is no more input to be sent to the managed process. @param process the managed process - this could be null in some cases like when input is through files. In that case, the process would not have been exec'ed yet - if this method if overridden it is the responsibility of the implementer to check that the process is usable. The managed process object is supplied by the ExecutableManager to this call so that this method can check if the process is alive if it needs to know. @throws IOException]]> InputHandler to the OutputStream from which it reads input and sends it to the managed process. @param os OutputStream from which to read input data for the managed process @throws IOException]]> stdin or in an {@link InputType#ASYNCHRONOUS} manner via an external file which is subsequently read by the executable.]]> OutputType. @return the handled OutputType]]> OutputHandler to the InputStream from which to read the output data of the managed process. @param is InputStream from which to read the output data of the managed process @throws IOException]]> Tuple of the managed process. @return the next output Tuple of the managed process @throws IOException]]> OutputHandler. @throws IOException]]> stdout or in an {@link OutputType#ASYNCHRONOUS} manner via an external file to which the process wrote its output.]]> StreamingCommand with the given command. @param pigContext PigContext structure @param argv parsed arguments of the command]]> StreamingCommand. @param executable the executable for the StreamingCommand]]> StreamingCommand. @param argv the command line arguments for the StreamingCommand]]> String[]]]> Handle to which the specification is to be attached. @param handleSpec HandleSpec for the given handle.]]> StreamingCommand. @param spec input specification]]> StreamingCommand. @return input specification of the StreamingCommand]]> StreamingCommand. @param spec specification for the primary output of the StreamingCommand]]> StreamingCommand. @return specification of the primary output of the StreamingCommand]]> Handle. @param handle Handle of the stream @return specification for the given Handle]]> true if the stderr of the managed process should be persisted, false otherwise.]]> true if the stderr of the managed process should be persisted, else false]]> true if files of this command should be shipped, false otherwise]]> true if files of this command should be shipped, false otherwise]]> StreamingCommand encapsulates all relevant details of the command specified by the user either directly via the STREAM operator or indirectly via a DEFINE operator. It includes details such as input/output/error specifications and also files to be shipped to the cluster and files to be cached.]]> stdin, stdout or a file-path)]]> stdin, stdout or a file-path) @param spec serializer/deserializer spec]]> name of the HandleSpec. @return the name of the HandleSpec (one of stdin, stdout or a file-path)]]> name of the HandleSpec. @param name name of the HandleSpec (one of stdin, stdout or a file-path)]]> HandleSpec. @return the serializer/deserializer spec of the HandleSpec]]> HandleSpec. @param spec the serializer/deserializer spec of the HandleSpec]]> stdin/ stdout/stderr or a named file and also the serializer/deserializer specification to be used to read/write data to/from the stream.]]> Low memory is defined as more than 50% of the tenured pool being allocated. Spillable objects are tracked using WeakReferences so that the objects can be GCed even though this class has a reference to them.]]> String into a byte representing the field delimiter. @param delimiter a string that may be in single-quoted form @return the field delimiter in byte form]]> Text to a Tuple @param val a line of text @param fieldDel the field delimiter @return tuple constructed from the text]]> Tuple @param buf the byte array @param length number of bytes to consume from the byte array @param fieldDel the field delimiter @return tuple constructed from the bytes]]> without any enclosing curly-braces.Not for use with Schema#toString @return Schema instance @throws ParserException]]> NOTE: use this call when you need to generate a Schema object from the representation generated by Schema#toString. This call strips the enclosing outer curly braces from the toString representation, which are placed there because the actual representation of the schema data is as a Bag-type relation. @param schemaString a String representation of the Schema to instantiate, in the form generated by Schema.toString() @return Schema instance @throws ParserException]]> Type of objects in Avro array]]> The type of the Avro object]]> bag({(int)}) @param sch @return modified schema @throws FrontendException]]> FuncSpec that performs the casting functionality @param spec the FuncSpec that does the casting]]> FuncSpec that performs the casting functionality @return the FuncSpec that does the casting]]> a:int - dereference of single column in a tuple gives the field Dereference ( 0,2 ) --> Tuple(a#2:int, c#6:int) #7 - dereference of more than one column gives a tuple Dereference ( 1 ) --> Dereference ( 1 ) --> b:bag{b_b#5:float}#8 - dereference of a bag gives a bag]]> ]]> Cube operator syntax 

     {@code alias = CUBE rel BY { CUBE | ROLLUP }(col_ref) [, { CUBE | ROLLUP }(col_ref) ...];}
 alias - output alias 
 CUBE - operator
 rel - input relation
 BY - operator
 CUBE | ROLLUP - cube or rollup operation
 col_ref - column references or * or range in the schema referred by rel

    The cube computation and rollup computation using UDFs {@link org.apache.pig.builtin.CubeDimensions} and {@link org.apache.pig.builtin.RollupDimensions} can be represented like below 

     {@code
 events = LOAD '/logs/events' USING EventLoader() AS (lang, event, app_id, event_id, total);
 eventcube = CUBE events BY CUBE(lang, event), ROLLUP(app_id, event_id);
 result = FOREACH eventcube GENERATE FLATTEN(group) as (lang, event),
          COUNT_STAR(cube), SUM(cube.total);
 STORE result INTO 'cuberesult';
 }
    In the above example, CUBE(lang, event) will generate all combinations of aggregations {(lang, event), (lang, ), ( , event), ( , )}. For n dimensions, 2^n combinations of aggregations will be generated. Similarly, ROLLUP(app_id, event_id) will generate aggregations from the most detailed to the most general (grandtotal) level in the hierarchical order like {(app_id, event_id), (app_id, ), ( , )}. For n dimensions, n+1 combinations of aggregations will be generated. The output of the above example query will have the following combinations of aggregations {(lang, event, app_id, event_id), (lang, , app_id, event_id), ( , event, app_id, event_id), ( , , app_id, event_id), (lang, event, app_id, ), (lang, , app_id, ), ( , event, app_id, ), ( , , app_id, ), (lang, event, , ), (lang, , , ), ( , event, , ), ( , , , )} Total number of combinations will be ( 2^n * (n+1) ) Since cube and rollup clause use null to represent "all" values of a dimension, if the dimension values contain null values it will be converted to "unknown" before computing cube or rollup.

    ]]>     In case of a RANK operation (row number mode), are used two steps: 1.- Each tuple is counted sequentially on each mapper, and are produced global counters 2.- Global counters are gathered and summed, each tuple calls to the respective counter value in order to calculate the corresponding rank value. or not: 
     In case of a RANK BY operation, then are necessary five steps:
   1.- Group by the fields involved on the rank operation: POPackage
   2.- In case of multi-fields, the key (group field) is flatten: POForEach
   3.- Sort operation by the fields available after flattening: POSort
   4.- Each group is sequentially counted on each mapper through a global counter: POCounter
   5.- Global counters are summed and passed to the rank operation: PORank
    @param loRank describe if the rank operation is on a row number mode or is rank by (dense or not)]]>     {@code alias = RANK rel ( BY (col_ref) (ASC|DESC)? ( DENSE )? )?;} alias - output alias RANK - operator rel - input relation BY - operator col_ref - STAR or Column References or a range in the schema of rel DENSE - dense rank means a sequential value without gasp among different tuple values. ]]>     Map> where Integer is the column number of the field and Set is the set of Keys in this field ( field is a map field only ). It does this for only the top level schema in load. Algorithm: Traverse the Plan in ReverseDependency order ( ie. Sink to Source ) For LogicalRelationalOperators having MapLookupExpression in their expressionPlan collect uid and keys related to it. This is retained in the visitor For ForEach having nested LogicalPlan use the same visitor hence there is no distinction required At Sources find all the uids provided by this source and annotate this LogicalRelationalOperator ( load ) with Map> containing only the column numbers that this LogicalRelationalOperator generates NOTE: This is a simple Map Pruner. If a map key is mentioned in the script then this pruner assumes you need the key. This pruner is not as optimized as column pruner ( which removes a column if it is mentioned but never used )]]> project(0), project(1), ... project(n-2), project(n-1) If input schema is null, project(*) is not expanded. It also expands project range ( eg $1 .. $5). It won't expand project-range-to-end (eg $3 ..) if the input schema is null.]]> all subsequent Pig pipelines in this script. If you wish to register it for only a single Pig pipeline, use register within that definition. @param jarfile Path of jar to include. @throws IOException if the indicated jarfile cannot be found.]]> all subsequent Pig pipelines in this script. If you wish to register UDFS for only a single Pig pipeline, use register within that definition. @param udffile Path of the script UDF file @param namespace namespace of the UDFs @throws IOException]]> all subsequent Pig pipelines defined in this script. If you wish to define it for only a single Pig pipeline, use define within that definition. @param alias name of the defined alias @param definition string this alias is defined as]]> all subsequent Pig pipelines defined in this script. If you wish to set it for only a single Pig pipeline, use set within that definition. @param var variable to set @param value to set it to]]> p = Pig.compile("A = load '$input';"); and then calls this function it will look for a variable called input in the host language. Scoping rules of the host language will be followed in selecting which variable to bind. The variable bound must contain a string value. This method is optional because not all host languages may support searching for in scope variables. @throws IOException if host language variables are not found to resolve all Pig Latin parameters or if they contain unsupported types.]]> The RubyDataBag class uses JRuby's API for the defintion Ruby class using Java code. The comments in this class will more extensively explain the annotations for those not familiar with JRuby.

    In JRuby, the annotations are provided for convenience, and are detected and used by the "defineAnnotatedMethods" method. The JRubyClass annotation sets the class name as it will be seen in the Ruby runtime, and alows you to include any modules. In the case of the RubyDataBag, within Ruby we just want it to be called DataBag, and we want it to be enumerable.]]> new RubyDataByteArray that is the concatentation of the encapsulated DataByteArray and arg]]> A RubyString, which will have {@link Utils#getSchemaFromString} called on it, and it will be added.

    A RubySchema, which will be added directly. IMPORTANT NOTE: since this API abstracts away from the distinction between Schema/FieldSchema, its important to understand how a Schema is added to another. In this case, the FieldSchema is pulled directly out of the given Schema. Thus, where in Pig a Schema.FieldSchema might be passed around, internally to this class, generally a Schema will be passed around encapsulating it.

    A list will create the Schema for a Tuple whose elements will be the elements of the list. Each element will be subjected to the same rules applied here.

    A hash in the form of:
    {"name:tuple"=>["x:int","y:int,z:int"], "name2:bag"=>["a:chararray"]}
    The keys must be a tuple, bag, or map, and the value must be an array. @param arg an object (generally an IRubyObject or String) to convert. See above for the rules on valid arguments @return the Schema constructed for the given argument]]> ]]>