Home Twitter by the Petabyte: Using Big Data to Define Market Sentiment

Twitter by the Petabyte: Using Big Data to Define Market Sentiment

Millions of tweets run through Twitter. It’s the poster child for big data on the Web.

To get data out of Twitter and use it to track sentiment requires tools with considerable processing and computational capabilities.

BigSheets is a tool created by IBM that takes terabytes or even petabytes of Web data and turns it into information that provides business intelligence. The tool can be used for structured or unstructured data from internal or external sources. For instance, it can run streams of Twitter data for days, weeks or even months on particular keywords. That data can then be mashed up with internal information.

To demonstrate its capabilties, IBM evangeist David Barnes created an excellent demonstration video that shows how Twitter can be mined for sentiment about the iPhone, BlackBerry or Android mobile smartphones.

We took some segments from the video to demonstrate a process that is not nearly as technical as we thought it would be.

Barnes shows how the tool mined Twitter for tweets that mentioned the smartphone terms: iPhone, Android and BlackBerry. He then tracked the tweets for sentiment. Do people like, love or hate the products? Do they want to buy the products?

The tool can also be used to track multiple websites for hours, days, weeks or months.

During the video, Barnes mined the Twitter stream for three minutes using the smartphone keywords. He had previously ran a query for 36 hours to demonstrate how the data can be used. It pulled 305,000 tweets into the application.

BigSheets classified it by the name of the user, the time of the tweet, what the tweet said and other associated metadata.

Making a Tag Cloud

Barnes then applied the sentiment language to the analysis. About 50,000 tweets showed sentiment. He then sought out tweets that expressed a sentiment to buy something.

Barnes then applied visualizaton software to create a tag cloud of the analyzed tweets.

Analyzing the British Parliament

In his last example, Barnes showed how BigSheets can scrape data from websites to do analysis. He did a query for people, places and things. That data resulted in more data about crime and other information. He cross referenced that data with the members of pariiament voting records.

What Would Facebook Look Like?

Like any good demo, Barnes leaves us with more questions about big data than we ever had before.

Take Facebook, for example. What would Mark Zuckerberg have created if he had this tool when creating the now famous social network? What will future developers create with tools like this?

BigSheets is one of several data mining tools we are seeing that use MapReduce and other techniques to mine data. What’s amazing is what the tools an do but also the accessibility of the tools themselves.

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.