The term “Big Data” has been flinging around quite a lot lately. It is in the news all the time. We hear about how much it has pushed us into the future and into the internet of things. These things all will produce useful data that will need to be analyzed and stored. One technology that we hear more and more about is Hadoop.
Hadoop was birthed as an open source project from the Google filesystem (GFS), and Map Reduce white-papers; the creator is Doug Cutting and the open source community. Map reduce is the core of Hadoop, and allows the user to write very simple programs to distribute workload across a complex amount of data. The Google filesystem inspired the majority of the work for the open source Hadoop filesystem (HDFS). HDFS is a redundant filesystem written in Java that distributes data across multiple machines that can be analyzed using Map reduce programming. That is just a brief dive into what Hadoop is, and if you want to learn more I highly recommend you take a gander at the Yahoo Hadoop tutorial.
Here is an ecosystem filled with projects that make managing this complex monster easier on administrator’s and developer’s. One of these projects that I really enjoy is Hue, the Hadoop User Experience. It gives a web interface for the user to query their data using some of these projects that live in this big data ecosystem like:
Each of these tools sits in front of a plethora of data that the user is analyzing. This data can be anything from a company’s customer generated data that tells a music service what song to play next, to another company trying to figure out which ads to serve you based on your browsing history. My point being — Hue has access to some seriously valuable information.
As with most technologies, security is often an after-thought. It is important we test the security of these applications so that we can protect my data and your data from the evil-doers who will sell the same information or use it for awful things. Perhaps a criminal can use pilfered data about you to create malware that you will more easily fall prey to.