In big data or generation analytics strategies discussion, it is certain that the buzzword “Hadoop” is always present. Still, there is a reasonable amount of confusion as to what Hadoop really is and what workloads it could best supply. As Hadoop is based on Google’s MapReduce, people who are most concerned about Hadoop are aware that it has been used in massive scale by large web companies for applications like search engines. Still, there’s a lot more to Hadoop than this.

Hadoop can be used for a multitude of data-processing workloads. There are some cases where Hadoop is being pushed other than its natural batch-processing sweet spot. It is mostly used across industries that experience tremendous growth in unstructured data volumes. There is also an increase of commercial software suppliers that sell items to make Hadoop simpler for mainstream clients. Hadoop is also being utilized by an increasing number of companies across different industries and individuals belonging to Fortune 100. Cloudera is deemed to be the most famous as it delivers its own enterprise-hardened distribution of the core Apache Hadoop project and it provides support, services and advanced management tools.


Despite these advancements and the widespread tools created to simplify processes using Hadoop, it still has a long way to go. The biggest concern for unskilled developers is creating Hadoop applications and workflows. It becomes difficult for organizations to hire qualified personnel for their Hadoop application deployments.

This 2011, Hadoop continues its mainstream adoption. In the coming months or years, companies will advance or start their big data efforts and many will reap the benefits from some hand-holding and technological assistance into this brave new world.

Introduction – Apache Hadoop

Hadoop is definitely one of the biggest innovations in data management. Created by former Yahoo employee Doug Cutting in 2005, Hadoop is now a full-pledged Apache Software Foundation project. It had several commercial vendors, software evolving in a fast pace and is inevitably the main topic for the term “big data.”

Big data is not limited to huge data storage, but also about evaluating insights from that data. Curt Monash, data industry analyst, explained that advanced analytics allows organizations to create more-informed decisions, plan for future action and monitor activity to find out when corrective action is necessary.  The ability of Hadoop to analyze large organizations’ data provides companies with a competitive advantage – from serving their clients better and optimally tuning their internal systems.

Aside from handling storage and data analysis, Hadoop is available as open-source software made to run on top of commodity servers. It indicates that users can process their huge amounts of data for a reasonable expense.

What Hadoop Is (and Is Not)

Hadoop is a mix of two separate sub-projects – the Hadoop MapReduce (HMR) parallel-processing framework and the Hadoop Distributed File System (HDFS).


HMR is a by-product of Google’s MapReduce software designed to speed its search engine results. It allows Hadoop applications to turn petabytes of data across the entire cluster. HDFS, which is based on Google File System,  is a distributed file system enhanced to keep unstructured data and feed it to Hadoop applications. Hadoop has also created its own ecosystem of related projects which are built to make Hadoop more functional.

Hadoop is open source and runs on clusters of commodity servers. Many organizations have to scale up to petabytes of data across hundreds or thousands of machines; and every free, open-source Hadoop distributions allow them to achieve this without the need for intense investment in software licenses or specialized hardware.  Early adopters such as Yahoo, Facebook and Twitter have built huge Hadoop implementations on top of commodity servers.

Some end-users and software vendors have modified Hadoop to handle streaming data. Still, Hadoop is most effective for batch-processing applications against existing datasets. Hadoop workloads generally are those that require results hourly, daily or weekly analysis. Though some organizations like Yahoo use Hadoop to process data regularly and at several occasions an hour. This creates rush results based on cluster size and dataset.

The Hadoop Ecosystem

There had been a significant improvement in the Hadoop ecosystem since becoming an official Apache Software Foundation project. Yahoo’s leadership claims to have led roughly 70 percent of the code. Similar to the Linux operating system, Hadoop had numerous distributions all stemming from the core one, Apache Hadoop. There is also the broadening support from third-party applications such as data warehouses, business intelligence software and data integration products. Furthermore, Hadoop has built a family of sub-projects designed to enhance the Hadoop experience or simply to utilize the core Hadoop technologies.