Pages

Showing posts with label Hadoop. Show all posts
Showing posts with label Hadoop. Show all posts

Tuesday, 23 December 2014

Top Sea Life observed!

One of my favorite hobbies is scuba diving. Diving frees my mind and makes me feel completely at ease: it's like being in another world.
All divers are used to having a logbook in which they write down their dives in order to remember them later or even certificate they dived all those times. Also, you can do this in Diveboard, a social network where divers share their dives. You can get this dataset (and many others) from Global Biodiversity Information Facility.
So, I decided to use it as a functional dataset in my next example with Hadoop.

In this post, we'll see how programming a MapReduce job to rank the species viewed reported in Diveboard using Apache Avro. Avro is a data serialization framework offering features such as rich data structures, a compact binary format and integration with many languages. It needs a schema to serialize and deserialize data and provides splittable files for HDFS and schema evolution.

As usual, Maven dependencies (just Avro included):
<dependency>
 <groupId>org.apache.avro</groupId>
 <artifactId>avro</artifactId>
 <version>${avro.version}</version>
</dependency>
<dependency>
 <groupId>org.apache.avro</groupId>
 <artifactId>avro-mapred</artifactId>
 <version>${avro.version}</version>
 <classifier>hadoop2</classifier>
</dependency>
<dependency>
 <groupId>org.apache.avro</groupId>
 <artifactId>avro-maven-plugin</artifactId>
 <version>${avro.version}</version>
</dependency>
<dependency>
 <groupId>org.apache.avro</groupId>
 <artifactId>avro-compiler</artifactId>
 <version>${avro.version}</version>
</dependency>
<dependency>
 <groupId>org.apache.avro</groupId>
 <artifactId>avro-ipc</artifactId>
 <version>${avro.version}</version>
</dependency>

Also, for performing code generation you can include the Avro Maven plugin in your pom.xml :
<plugin>
 <groupId>org.apache.avro</groupId>
 <artifactId>avro-maven-plugin</artifactId>
 <version>${avro.version}</version>
 <executions>
  <execution>
   <phase>generate-sources</phase>
   <goals>
    <goal>schema</goal>
   </goals>
   <configuration>
    <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
    <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
    <testSourceDirectory>${project.basedir}/src/test/avro/</testSourceDirectory>
    <testOutputDirectory>${project.basedir}/src/test/java/</testOutputDirectory>
    <fieldVisibility>PRIVATE</fieldVisibility>
    <includes>
     <include>**/*.avsc</include>
    </includes>
    <testIncludes>
     <testInclude>**/*.test</testInclude>
    </testIncludes>
   </configuration>
  </execution>
 </executions>
</plugin>

I want to get the most viewed species for each country. For this, in the Avro schema I'm going to group Diveboard records by country and scientific name (ResultRecord). Additionally, all related info about each record is represented in a list of type DiveBoardRecord.

{
    "type":"record",
    "name":"ResultRecord",
    "fields":[
  {
         "name":"counter",
         "type":"int",
         "order":"descending"
     },
  {
         "name":"scientificName",
         "type":"string"
     },
     {
         "name":"country",
         "type":"string"
     },
     { 
      "name":"dbRecordList",
      "type":{
       "type":"array",
       "items":{
        "type":"record",
        "name":"DiveBoardRecord",
        "fields":[
            {
                "name":"id",
                "type":"string",
                "order": "ignore"
            },
         {
             "name":"diverProfileUrl",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"diverName",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"eventDate",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"waterBody",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"scientificName",
             "type":"string"
         },
         {
             "name":"country",
             "type":"string"
         },
         {
             "name":"locality",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"verbatimLocality",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"latitude",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"longitude",
             "type":"string",
             "order": "ignore"
         },
         {
             "name":"url",
             "type":"string",
             "order": "ignore"
         }
        ]
    }
   }
  } 
    ]
}
In line 8, the value of "order" for the field "counter" is "descending". It'll sort records by this field (most viewed species at the beginning of the final output result).

Then, it's time to write de driver! (before, I generated the sources of the schema). The driver consists of two chained jobs:
  • The first one ("Sea Life Aggregator") does the parsing -the downloaded dataset is in CSV format so we can parse each line easily- in the map phase and afterwards, the reduce phase generates an intermediate output with the ResultRecord (including the list of all records and the counter of them).
  • The second one ("Sea Life Sorter") just sorts the previous intermediate result. It uses the original Mapper and Reducer classes which they'll sort the results in the shuffle and sort phase (remember the "counter" in the schema).
Once these jobs are configured, the ControlledJob encapsulates them with a dependency and... execute the job!
import java.io.File;
import java.io.FilenameFilter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import org.apache.avro.file.DataFileReader;
import org.apache.avro.io.DatumReader;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapred.AvroValue;
import org.apache.avro.mapreduce.AvroJob;
import org.apache.avro.mapreduce.AvroKeyInputFormat;
import org.apache.avro.mapreduce.AvroKeyOutputFormat;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob;
import org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;

public class DiveBoardDriver extends Configured implements Tool {

 private static final Logger LOG = Logger.getLogger(DiveBoardDriver.class);

 public static class DiveBoardMapper
   extends
   Mapper<LongWritable, Text, AvroKey<DiveBoardRecord>, AvroValue<DiveBoardRecord>> {

  private DiveBoardRecord parse(String line) {
   String[] columns = line.split("\t");
   if (columns == null || columns.length != 49) {
    return null;
   }
   try {
    DiveBoardRecord record = new DiveBoardRecord();

    record.setId(columns[0]);
    record.setDiverProfileUrl(columns[6]);
    record.setDiverName(columns[14]);
    record.setEventDate(columns[18]);
    record.setWaterBody(columns[22]);
    record.setCountry(columns[23]);
    record.setLocality(columns[25]);
    record.setVerbatimLocality(columns[26]);
    record.setLatitude(columns[32]);
    record.setLongitude(columns[33]);
    record.setUrl(columns[39]);
    record.setScientificName(columns[40]);

    return record;
   } catch (Exception e) {
    return null;
   }
  }

  @Override
  public void map(LongWritable key, Text value, Context context)
    throws IOException, InterruptedException {
   DiveBoardRecord record = parse(value.toString());
   if (record != null) {
    context.write(new AvroKey<DiveBoardRecord>(record),
      new AvroValue<DiveBoardRecord>(record));
   }
  }
 }

 public static class DiveBoardReducer
   extends
   Reducer<AvroKey<DiveBoardRecord>, AvroValue<DiveBoardRecord>, AvroKey<ResultRecord>, NullWritable> {

  public void reduce(AvroKey<DiveBoardRecord> key,
    Iterable<AvroValue<DiveBoardRecord>> records, Context context)
    throws IOException, InterruptedException {
   List<DiveBoardRecord> dbRecordList = new ArrayList<DiveBoardRecord>();
   for (AvroValue<DiveBoardRecord> avroValue : records) {
    dbRecordList.add(avroValue.datum());
   }

   ResultRecord result = ResultRecord.newBuilder()
     .setCounter(dbRecordList.size())
     .setCountry(dbRecordList.get(0).getCountry())
     .setScientificName(dbRecordList.get(0).getScientificName())
     .setDbRecordList(dbRecordList).build();

   context.write(new AvroKey<ResultRecord>(result), NullWritable.get());
  }
 }

 public int run(String[] args) throws Exception {
  Configuration conf = this.getConf();
  Job job1 = Job.getInstance(conf, "Sea Life Aggregator");
  job1.setJarByClass(getClass());

  FileInputFormat.addInputPath(job1, new Path(args[0]));
  FileOutputFormat.setOutputPath(job1, new Path(args[1]));

  job1.setMapperClass(DiveBoardMapper.class);
  job1.setReducerClass(DiveBoardReducer.class);

  AvroJob.setMapOutputKeySchema(job1, DiveBoardRecord.SCHEMA$);
  AvroJob.setMapOutputValueSchema(job1, DiveBoardRecord.SCHEMA$);

  job1.setOutputFormatClass(AvroKeyOutputFormat.class);
  AvroJob.setOutputKeySchema(job1, ResultRecord.SCHEMA$);

  Configuration conf2 = new Configuration(conf);
  Job job2 = Job.getInstance(conf2, "Sea Life Sorter");
  job2.setJarByClass(getClass());

  FileInputFormat.addInputPath(job2, new Path(args[1]));
  FileOutputFormat.setOutputPath(job2, new Path(args[2]));

  job2.setInputFormatClass(AvroKeyInputFormat.class);
  job2.setMapperClass(Mapper.class);
  AvroJob.setInputKeySchema(job2, ResultRecord.getClassSchema());

  AvroJob.setMapOutputKeySchema(job2, ResultRecord.SCHEMA$);
  job2.setMapOutputValueClass(NullWritable.class);

  job2.setOutputFormatClass(AvroKeyOutputFormat.class);
  job2.setReducerClass(Reducer.class);
  AvroJob.setOutputKeySchema(job2, ResultRecord.getClassSchema());

  ControlledJob cJob1 = new ControlledJob(conf);
  cJob1.setJob(job1);

  ControlledJob cJob2 = new ControlledJob(job2, Arrays.asList(cJob1));

  JobControl jobCtrl = new JobControl("Top Sea Life Observed");
  jobCtrl.addJob(cJob1);
  jobCtrl.addJob(cJob2);

  LOG.info("Starting chained jobs");
  Thread thread = new Thread(jobCtrl);
  thread.setDaemon(true);
  thread.start();
  while (!jobCtrl.allFinished()) {
   Thread.sleep(500);
  }
  if (jobCtrl.getFailedJobList().size() > 0) {
   for (ControlledJob cj : jobCtrl.getFailedJobList()) {
    LOG.error("Error executing job: " + cj.getJobName());
   }
   return 0;
  } else {
   LOG.info("Jobs successfully completed");
   return 1;
  }
 }

 public static void main(String[] args) throws Exception {
  if (arguments.length != 3) {
   System.err.printf("Usage: %s [generic options] <input> <intermediate result> <output>\n",
       DiveBoardDriver.class.getSimpleName());
   ToolRunner.printGenericCommandUsage(System.err);
   System.exit(-1);
  }
  LOG.info("Running DiveBoardDriver program...");
  int status = ToolRunner.run(new DiveBoardDriver(), args);

  DatumReader<ResultRecord> datumReader = new SpecificDatumReader<ResultRecord>(
    ResultRecord.SCHEMA$);
  File[] outputFiles = new File(args[2])
    .listFiles(new FilenameFilter() {
     @Override
     public boolean accept(File dir, String name) {
      return name.toLowerCase().endsWith(".avro");
     }
    });

  for (File file : outputFiles) {
   DataFileReader<ResultRecord> fileReader = new DataFileReader<ResultRecord>(
     file, datumReader);
   for (ResultRecord resultRecord : fileReader) {
    System.out.println(resultRecord.getCounter() + "\t"
      + resultRecord.getScientificName() + "\t"
      + resultRecord.getCountry());
   }
  }
  System.exit(status);
 }
}


And the winner is... the "Pterois volitans" (aka Red Lionfish) in Egypt!!!

Monday, 8 December 2014

Testing Hadoop: MRUnit + Mockito + Reflection

I'm going to start these post series with one of the main principles in TDD: writing tests first.

As we know, applications based on distributed computing models like MapReduce are tricky to debug. There a lot of cases in which our code can throw an exception or behaves in a unexpected way because of a small bug in our code or problably due to the input data received. So whatever the reason is, it's better to detect and fix these bugs in early stages of our development.

For testing our MapReduce jobs, there is a well-known framework called MRUnit, in fact, it was a few years ago a TLP (Top Level Project) from the Apache Incubator. It helps us out to test and debug our pieces of code in isolation and in an easy way: you can write unit tests for your mappers and reducers and integration tests as well.
There are a lot of examples out there talking about how to use this library, so I prefer focusing in a mix of concepts to able to test our code using MRUnit, Mockito and Java Reflection API. The idea is as follows:
  • Unit testing of mappers and reducers via MRUnit.
  • Mocking classes with Mockito to emulate behaviour of some variables in the mapper and/or the reducer.
  • Java Reflection API to modify some private variables defined in these classes.
Firts of all, defining our dependencies in Maven (take into account the "classifier" tag, I'm using Hadoop 2 version):

 <dependencies>
  <dependency>
   <groupid>junit</groupid>
   <artifactid>junit</artifactid>
   <version>4.12</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupid>org.apache.mrunit</groupid>
   <artifactid>mrunit</artifactid>
   <version>1.1.0</version>
   <classifier>hadoop2</classifier>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupid>org.mockito</groupid>
   <artifactid>mockito-all</artifactid>
   <version>1.10.19</version>
   <scope>test</scope>
  </dependency>

  <dependency>
   <groupid>org.apache.hadoop</groupid>
   <artifactid>hadoop-hdfs</artifactid>
   <version>${hadoop.version}</version>
  </dependency>
  <dependency>
   <groupid>org.apache.hadoop</groupid>
   <artifactid>hadoop-auth</artifactid>
   <version>${hadoop.version}</version>
  </dependency>
  <dependency>
   <groupid>org.apache.hadoop</groupid>
   <artifactid>hadoop-common</artifactid>
   <version>${hadoop.version}</version>
  </dependency>
  <dependency>
   <groupid>org.apache.hadoop</groupid>
   <artifactid>hadoop-client</artifactid>
   <version>${hadoop.version}</version>
  </dependency>

  <dependency>
   <groupid>redis.clients</groupid>
   <artifactid>jedis</artifactid>
   <version>2.6.2</version>
  </dependency>
 </dependencies>

You can see an unexpected dependency: Jesis, a Java client to interact with Redis (a key-value cache and store system with good performance).
Why is this? The reason is pretty simple: Redis will work as a cache engine for sending values to the mapper for its operations (in this case we'll use it as a dictionary, this is, joining two datasets).

This is the driver class with its inner classes (mapper and reducer):

import java.io.IOException;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

import redis.clients.jedis.Jedis;

public class Driver extends Configured implements Tool {

 public static class ExampleMapper extends
   Mapper {
  Jedis jedisClient = new Jedis("localhost");

  @Override
  public void setup(Context context) {
   jedisClient.select(0);
  }

  @Override
  public void map(LongWritable key, Text value, Context context)
    throws IOException, InterruptedException {
   String[] columns = value.toString().split(";");
   if (columns.length < 14) {
    return;
   }
   String mappedValue = jedisClient.get(columns[10]);
   if (mappedValue != null && !mappedValue.equals("")) {
    columns[10] = mappedValue;
   }
   value.set(StringUtils.join(columns, ";"));
   context.write(value, NullWritable.get());
  }
 }

 @Override
 public int run(String[] args) throws Exception {
  if (args.length != 2) {
   System.err.printf("Usage: %s [generic options] <input> <output>\n",
     getClass().getSimpleName());
   ToolRunner.printGenericCommandUsage(System.err);
   return -1;
  }
  Job job = Job.getInstance(getConf(), "MapReduce Test Example");
  job.setJarByClass(getClass());
  
  FileInputFormat.addInputPath(job, new Path(args[0]));
  FileOutputFormat.setOutputPath(job, new Path(args[1]));

  job.setMapperClass(ExampleMapper.class);
  job.setMapOutputKeyClass(Text.class);
  job.setMapOutputValueClass(NullWritable.class);
  
  job.setNumReduceTasks(0);

  return job.waitForCompletion(true) ? 0 : 1;
 }

 public static void main(String[] args) throws Exception {
  int status = ToolRunner.run(new Driver(), args);
  System.exit(status);
 }
}
This job has no mystery at all, actually it's a Map-Only job and consists of replacing a value from a column with its mapped value got from Redis (kinda map-side join). Just for the shake of illustration.
The datasets merged contain projects funded under seventh framework programme for research and technological development (FP7) from 2007 to 2013, from European Union Open Portal. The job maps activity codes (cached in Redis) in the projects extract.

On the other side, the test cases:

import java.io.IOException;
import java.lang.reflect.Field;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;

import redis.clients.jedis.Jedis;

import org.junit.Before;
import org.junit.Test;

import static org.mockito.Mockito.*;


public class TestDriver {

 MapDriver mapDriver;

 @Before
 public void setUp() throws NoSuchFieldException, SecurityException,
   IllegalArgumentException, IllegalAccessException {
  Driver.ExampleMapper mapper = new Driver.ExampleMapper();
  mapDriver = MapDriver.newMapDriver(mapper);

  Jedis mockedJedis = mock(Jedis.class);
  when(mockedJedis.get(any(String.class))).thenReturn("Dummy value");
  
  Field privateField = mapper.getClass().getDeclaredField("jedisClient");
  privateField.setAccessible(true);
  privateField.set(mapper, mockedJedis);
 }

 @Test
 public void testMapper() throws IOException, NoSuchFieldException,
   SecurityException, IllegalArgumentException, IllegalAccessException {
  mapDriver.withInput(new LongWritable(),
      new Text(
        "86250;217246;BONUS+;Multilateral call for research projects within the Joint Baltic Sea Research Programme BONUS+;BONUS EEIG - representing altogether 10 RTD organisations in the Baltic Sea states ;2007-05-10;2012-05-09;22512219;7266762;FP7-GA;ERANET;;FP7-2007-ERANET-4.2.;relatedContact:Dr Kaisa KONONEN"));
  mapDriver.withOutput(
      new Text(
        "86250;217246;BONUS+;Multilateral call for research projects within the Joint Baltic Sea Research Programme BONUS+;BONUS EEIG - representing altogether 10 RTD organisations in the Baltic Sea states ;2007-05-10;2012-05-09;22512219;7266762;FP7-GA;Dummy value;;FP7-2007-ERANET-4.2.;relatedContact:Dr Kaisa KONONEN"),
      NullWritable.get());

  mapDriver.runTest();
 }
}
We don't want to connect to Redis for testing purposes (or maybe we couldn't reach it); we must emulate it. Because of that, in the first highlighted lines we create a mock object and set the returning value of its get method. But that's not enough, the mapper has a private varible, jedisClient, and we have to "mocketizer" it. To do this, we need the Reflection API to modify this variable in runtime setting this mocked object.

And that's it! MRUnit takes care of its part and test whatever we like in our MapReduce jobs! :-P

Sunday, 30 November 2014

Some tips for Hadoop devs

Being a Hadoop beginner can be a bit difficult. There are a lot of concepts you need to know before you can understand how this works, but keep working on it and you'll see the results very soon! As Larry Page said: "Always work hard on something uncomfortably exciting". ;-)

Probably you want to get your hands on too quickly without knowing the basics and that's not a good thing. Although what I'm going to include here don't cover the basics, I think some practical tips can be useful when you're going to implement a Hadoop job.

Extends from Configured and implements Tool in your driver
You can run a job without doing this and it'll work fine. However, if you want to change some config params (Hadoop is highly configurable) you'll have to change your code or even hardcode it.
An useful utility class is "org.apache.hadoop.util.GenericOptionsParser" which helps us out to parse and sets generic Hadoop arguments. It's not a good style of coding using this class directly so instead, we must use "org.apache.hadoop.util.ToolRunner" class that use it internally. Also, ToolRuner class needs a "org.apache.hadoop.util.Tool" object to exec its run() method. For that, extending "org.apache.hadoop.conf.Configured" and implementing "org.apache.hadoop.util.Tool" interface (all this in the driver class) will help to get the "org.apache.hadoop.conf.Configurable" object and it'll require to implement the run() method. Finally, implementing this run() method to configure our job, we just have to call ToolRunner.run().
public class MyDriver extends Configured implements Tool {

 public int run(String[] args) throws Exception {
  Configuration conf = this.getConf();
  Job job = Job.getInstance(conf, "My job");

  //...

  return job.waitForCompletion(true) ? 0 : 1;
 }

 public static void main(String[] args) throws Exception {
  int status = ToolRunner.run(new DiveBoardDriver(), arguments);
  System.exit(status);
 }
}

Set Jar file to use in the job
Hadoop needs to know where the jar with Map and Reduce tasks is to send it to nodes to perform these tasks. You can do this using setJarByClass() method, which will find out the jar by the specified class or using setJar() method, which will set the jar file location.
job.setJarByClass(getClass());
job.setJar("/path/to/my/jar");

Number of reducer tasks to use
If your job is a Map-only job, you must to set it to zero. If not, it'll depend on the number of reducer slots available in your cluster (if you're working on local, just 1 is ok) and the files that the job is processing too. There is a formula in which you multiply a factor (with values between 0.95 to 1.75) by the number of reducer slosts available.
job.setNumReduceTasks(N);

Adding third-party libraries to a job
If you have some dependencies in your job, this is, third-party libraries, you'll need these libraries are installed in all nodes of your cluster.
Using the CLI (Command Line Interface), you'll be able to include these libraries by the "-libjars" param and it'll work but, what if you're not using the CLI? One possible option would be to build a "fat jar" including all dependencies in it, containing your classes and third-party classes too, but the file could be too big.
The distributed cache can help us: you'll have to upload our third-party libraries to HDFS and then, add these files to the distributed cache. Prior versions to Hadoop 2.2.0 had to use "org.apache.hadoop.filecache.DistributedCache" class, for later versions this functionality is included in "org.apache.hadoop.mapreduce.Job" class.
job.setCacheFiles(URI[] files);
job.addCacheFile(new URI("/path/to/file"));

Not always Hadoop is a good option
Before thinking in "how to solve this problem with Hadoop", you should ask yourself what are the requirements: it could be files you have to process aren't that huge and you can use another tool for it, or maybe you need a quick response in the results (real-time o near real-time), or time to develop and deploy your application, etc.
Hadoop is quite interesting and you can do so many things with the tools of its ecosystem but this has nothing to do with the functionality you want to solve.


Saturday, 15 November 2014

The Hadoop Ecosystem Table

If you start learning about Big Data and start googling with this concept one of most keywords collected is Hadoop: an open-source software for processing incredibly large datasets, etc...,  but when digging deeper into it you get a lot more, and so on and so forth.

You get confused and start reading some books to organize your thoughts about these new concepts. You find a lot of technologies. The more you find  the more you get, they don't stop emerging continuously!

Isn't there an index with these technologies categorized and a brief summary of them?
Yes! there is a project on GitHub which keeps the track of Hadoop related projects, hahah: The Hadoop Ecosystem Table