A web of infinite information: does that sound like a scary problem of “just too much”? Venerable blogger Om Malik and Twitter co-founder Evan Williams agreed in an interview today that it is. Williams, as a founder first of Blogger.com and now of Twitter, is probably more responsible for the explosion of social data online than any other single person. Luckily, not everyone feels the same way about this historical moment.
As the quantity of data produced and available thanks not just to blogging and social networks, but sensors, surveys and machine observation hockey sticks exponentially skyward – a growing number of people and institutions are embracing this change as an opportunity that could forever change the way we learn, communicate and understand the world. Hopefully the bloggers and social network creators of the world will participate with enthusiasm.
To be fair, Williams himself recognizes the explosion of data on Twitter an an opportunity to deliver new and greater forms of value than his company does today. And Malik’s own company organizes a conference on big data called Structure. The language of “too much” and “scary problem” exists within a context, though.
That context includes the Wall St. Journal’s fear-mongering series on social network data and advertising (“What They Know About You”), the stuggles between privacy and innovation (even in the US Federal government, where innovation is increasingly defended in the face of privacy concerns), the decline of sophisticated online tools like Delicious and RSS readers and the rise in polished, closed, proprietary systems of communication like Facebook. (“Grunt if you Like something or someone.”)
“Revealing the hidden laws and processes underlying societies constitutes the most pressing scientific grand challenge of our century.” – Dr Helbing, of the Swiss Federal Institute of Technology
Twitter is our open social graph.
It’s our machine-accessible mega-river of links and social information. It’s very worrisome to hear the man in charge of the product say he “totally agrees” with the statement that there’s “just too much” information on the internet.
He might recognize that his engineers need to process that information to make pretty packages for the millions of would-be users still living in the 20th century and intimidated by a fat stream of variably relevant information – but his engineers alone will never be able to bring as much to that challenge as a world of hackers can, as long as they have access to the data. For the leader to orient himself in this day and age away from showing outside parties the data is cause for concern. (This is something we’ve expressed concern about for at least the last 18 months, see How Twitter’s Staff Uses Twitter (And Why It Could Cause Problems).)
What’s the other side of the coin? Free, open, available data as an incredible opportunity for improving the human experience probably saw its best articulation to date with the broadcast of the BBC’s new hour-long special The Joy of Stats, which you can view in full below.
The BBC also
a Swiss project dubbed the Living Earth Simulator (LES), a massive data project aiming to simulate as much natural and social activity on earth as possible. Those simulations, to be carried out on a scale inspired by the Large Hadron Collider, would aim to discover all kinds of patterns hidden in the mass of human and ecological data, including social network data.
“Many problems we have today – including social and economic instabilities, wars, disease spreading – are related to human behaviour, but there is apparently a serious lack of understanding regarding how society and the economy work,” says to the BBC a Dr Helbing, of the Swiss Federal Institute of Technology, who chairs the FuturICT project which aims to create the simulator. “Revealing the hidden laws and processes underlying societies constitutes the most pressing scientific grand challenge of our century.”
Do “too much data” warblers want to sit that grand challenge out? Do they want to impede it by stoking a culture of fear, rather than of support for investigation?
Skeptics, even data-loving skeptics, note that data enough is not alone. As ReadWriteWeb contributor and data scientist Pete Warden told the BBC about the Living Earth Simulator, “It’s not that we don’t know enough about a lot of the problems the world faces, from climate change to extreme poverty, it’s that we don’t take any action on the information we do have.”
That may be true, but data driven knowledge will help illuminate problems, their finer points, possible solutions and opportunities to galvanize public support for taking action.
Don’t take my word for it, check out the fabulous BBC special with Dr. Hans Rosling below. Statistics, Rosling says, is the sexiest field in the world right now. More than just glitz and glamour, the ability to analyze an explosion of data promises new opportunities to really make a difference in the world.
Hopefully people who have helped build the instruments that enabled that explosion, through the inclusion of new voices from throughout society, will be excited about that and will participate whole heartedly.
Title image by nasa1fan/MSFC
