Hadoop 101

Whether you are using Hadoop yourself or planning to have someone in your team using it, you should know how it works. Take this exercise to run your very first MapReduce job.

What is Hadoop?
First, let's look at the basics. Hadoop is an open-source framework that allows the processing of datasets using a simple programming model for deploying and processing the data. The key is that it lets you process this data in a clustered environment while using commodity hardware. You can start with a single machine during development and add machines without increasing complexity. This is referred to the ability to scale out. Hadoop uses concepts from functional programming (another hot phrase) to handle large datasets. Finally, it has a built-in model to handle failures within the cluster to ensure the data is processed even if machine failure occurs.

Why Hadoop?
The simple answer is it provides a method to process large amounts of data using a clustered and scalable architecture. It provides a means to process data when the amount, complexity, or unstructured nature of the data overwhelms conventional methods. This is because it was built from the ground up to work in a clustered environment; as the amount of data grows, computing power can be added simply through configuration.

Finally, though other models address these same challenges, Hadoop has become a standard with commercial backing, a growing knowledge base, good documentation, and a growing talent pool. Put together, these reasons make Hadoop a viable alternative, and they are why it is being used with ever increasing frequency.

Components of Hadoop
To understand Hadoop, you need to understand the basic building blocks of the framework. The first component, Commons, contains (as the name implies) common functionality. In addition, Hadoop has its own file system, HDFS, which is made to be fault tolerant and supplies the cornerstone to let it run on commodity hardware.

The final component of the Hadoop system is MapReduce. It implements the model that allows data to be processed in a parallel manner. In the future, we will implement our own code in Java for this layer. Note that the upcoming 2.x release contains a v2 implementation of MapReduce called YARN (but I won't go into that here, as it is still an Alpha version at the time of this writing).

Try it out
Now that we have laid the scene for using Hadoop and we have quickly covered a few introductory concepts, it is time to get the software. The Hadoop stable release can be found here. At the time of this writing, this will download version 1.0.4.

Once it is extracted, you will need to go to the conf directory and set up JAVA_HOME in hadoop-env.sh. From the base directory where it is installed, create a directory called input, and copy some text files into the directory. (I used a text version of my resume.) Then run the following.

bin/hadoop jar hadoop-examples-1.0.4.jar wordcount input output

This will run the MapReduce job and create the output directory containing an output file with the result. Running in this mode, we are not using HDFS, but rather the local file system. You will get a chance to explore HDFS soon enough.

Congratulations. You just ran your first MapReduce job in standalone mode.

Related posts:

• Hadoop: Executives & Managers Need to Read the Warning Labels
• Stick to the Facts
• Big Data Tackles Financial Fraud
• The Beauty of Big Data Models

— Brian Enochson, Software Development Consultant