We are fascinated here at ReadWriteWeb about Hadoop. It can be used in so many ways. It gives you that sense of excitement that shows how big data can open up all kinds of possibilities.
So we got a tad excited tonight when we ran across a post by Mike Pearce about "10 Hadoopable Problems: or in other words, 10 things you can do with Hadoop. But excitement turned to disappointment when it reminded us of how limiting we can be when thinking about big data in standard terms.
We won't go into detail about each of the 10 ways Hadoop can be used. You can go check out the post yourself. Instead, we'll highlight a few and provide our own little view about big data, the failings of geek culture and the role information plays in our interface culture.
Hadoop is a transforming technology that through its analytic capabilities, can change the way we interface with the world. We use the term interface in deference to Interface Culture, the book by Steven Johnson that explored the Web's interactive elements and technology interfaces. He looked at buttons, links and metaphors such as the desktop and traced them back to medieval planning, Victorian novels, early cinema and the rise of our modern culture.
The interface culture we develop out of big data will spawn new works that help guide us into unfamiliar spaces as much as novels helped the Victorian era make sense of the new, industrial world.
Hadoop is a tool increasingly used to make sense of a new world that automatically creates data in overwhelming amounts. We manually create our own data through gestures on Facebook, from the images we post to Flickr and the tweets we post religiously. But data is also created automatically by intelligent agents who do the work on our behalf, sending information from machine-to-machine, analyzing itself along the way, increasing in intelligence through APIs or forking into new realms as its manipulated and turned into apps, recommendation engines and the rest.
Transforming data helps us make sense of an information universe, By analyzing it we create our own interface culture and in the process, better understand our world. New art, new intellectual movements and new societies will emerge from the data we are just starting to learn how to chisel into new shapes, new scuptures if you will that tells stories about who we are.
Unfortunately, the 10 examples (from a Cloudera presentation) don't draw us into a new world of possibilities. Sure, fraud detection (number seven) is important. Goodness knows how often we hear about it. I am sure there are lots of surveillance geeks out there who love the idea of monitoring trade with Hadoop as pointed out in number 8. Ad targeting comes in the four spot. That's a familiar topic. Search quality is ninth. More yawns. You get the picture.
All of these examples explore what we have become accustomed to in geek culture. Possibilities for how big data can be used in a strictly commercial sense or as a way to optimize processes or the technologies we have already developed.
It's implausible to believe that we will see any kind of diversity in geek culture if we continue grinding down this technically oriented view of the data around us. Focusing on incremental improvements in processes has been done for generations. It will make people a lot of money but its impact is minimal in the world we live. It will create jobs. We will without a doubt see a new generation of data analysts but there is more to this big data, right?
Perhaps it is too early to expect a renaissance. It's like we are medieval artists who are struggling to move beyond the concept of flat images. We are too consumed in the technological marvels of what we have created to fully understand the implications of what we have discovered and with it what we can create.
We will admit it is getting simpler to develop technologies and easier for people to use. More people are making apps. We have a new generation of developers who have taught themselves by following the principles of the view-source culture. More women are making inroads. We can thank open standards for that.
It's the software that mixes and cooks up that data which will truly transform our world. When that data is as accessible as flour is for baking or clay for sculpting then Hadoop and other analytics technologies like it will have real meaning.
And perhaps it is the ability to discover data and perform tricks with it that opens up this marvelous world. A world made from the big data we shape into images that help us realize an interface culture of a new modern era.