Last night, Twitter made official what many members of the developer community had long suspected - there would be no more whitelisting for data-hungry apps. For some developers, this either means that they have to come up with creative work-arounds or, for others, that their projects are dead in the water.
So why did Twitter kill the exception to the rule and what does it mean for the future of Twitter apps and the developers who create them?
What Is Whitelisting?
First, a quick primer on whitelisting. Whenever you use a third-party service that integrates with Twitter, it likely uses the API (application programming interface) to get data - to get your @replies, direct messages, tweets and so on. To make sure that these apps don't overrun the servers, however, Twitter imposes a rate limit, which means they can only make so many requests per hour. For more data intensive services - think Klout, Hootsuite or Twitalyzer, for example - these limits are far too low for them to operate. That's where the whitelist comes in. The "whitelist" is simply a list of services that are allowed to make a much higher number of requests per hour. By much higher, we mean 20,000 compared to 350.
So, What Has Changed?
Yesterday, Twitter platform lead Ryan Sarver announced on the Twitter Development Talk group that "Twitter will no longer grant whitelisting requests." Sarver describes the feature as a holdout of a time before Twitter offered other methods to access its data.
Twitter whitelisting was originally created as a way to allow developers to request large amounts of data through the REST API. It provided developers with an increase from 150 to 20,000 requests per hour, at a time when the API had few bulk request options and the Streaming API was not yet available.
Since then, we've added new, more efficient tools for developers, including lookups, ID lists, authentication and the Streaming API. Instead of whitelisting, developers can use these tools to create applications and integrate with the Twitter platform.
The real change, however, comes a bit further down into Sarver's announcement when he notes that "there are going to be some things that developers want to do that just aren't supported by the platform." Instead of granting whitelisting to make advanced research and analytics possible, writes Sarver, developers will need to contact Gnip, a reseller of Twitter data.
Why Get Rid of Whitelisting?
According to Abraham Williams, a "Hacker Advocate" and developer at Q&A startup Answerly, Twitter is getting rid of whitelisting to push developers onto its streaming APIs. That, and keeping Twitter up an running, of course.
"They're essentially saying if you can't get your data in real time or in less than 350 requests in an hour, find something else to do," said Williams. "Every API has some kind of rate limit and a lot of them don't have whitelisting."
"Is this a reasonable thing for Twitter to do? It is. This is about their growth and stability as a platform. Developers are secondary," said Peterson. "It's probably just a natural part of their growth. They must have hit the wall from a technology perspective."
What Does This Mean for the Twitter Ecosystem?
When Twitter started out in 2006, it was a simple, SMS-based service that allowed a small group of users to communicate with each other and put that content on the Web. A short six months after the service was born, the Twitter API was introduced. Ever since, developers have been creating apps around the microblogging system, using the data to analyze individuals social influence online, communications habits, and even worldwide trends. Those ideas that needed access to more data were able to requisition the company for whitelisted access. Now, whitelisted companies will be grandfathered in, but the doors are closed to everyone else.
So who will this move affect, first and foremost? According to Twitalyzer founder Eric Peterson, the most affected will be "everyone who for the last three months has been building stuff on the API" and waiting to hear back on their whitelist application.
"You can't really do a ton without that whitelist," said Peterson. "We would have run into rate limits in the first couple months and probably have said forget it."
One developer notes in response to the announcement that he needs around 800 requests per hour and that "This and a few other ideas I had just died."
Williams explained that the changes affect some developers (and ideas) more than others. He said that sites could use the streaming API and collect data moving forward, but gathering historical tweets will be a problem.
"The people who are up in arms about it are the people who need access to historical data," said Williams. "If I wanted to build an app that would search the tweets of people you follow and you followed more than 20 people, I wouldn't be able to pull your entire Tweet history."
That Leaves Us With...What's Next?
Of course, what's a limit without finding a clever way around it? The party line from Twitter on this one is that, if a developer needs more data than allowed, they'll need to pony up the bucks to Twitter data-reseller Gnip. As Ed Borasky, a data scientist who does work in text mining, journalism and social media, notes on the Google Group discussion, this squeezes out the little guy.
[Gnip is] a non-starter for small consultants, startups or non-profits. We simply will find problems to solve that don't require subscriptions to a high-volume and high-priced source of *raw*, *unstructured* data. There are plenty of large corporations who can sell "business intelligence" to other large corporations, but that's not a game that a small business can play. I am not a qualified prospect for Gnip's service and you shouldn't be pitching it to me. ;-)
From here, two are two distinct possibilities. First, developers will find a way over the wall and create alternative methods to get data, through nets of fake accounts or other methods. The second possibility - one requested in the conversation by Twitter API consultant and trainer Adam Green - is that an open market open up for Twitter data.
Now the next step in opening up this marketplace is to create multiple resellers of Twitter API data, and let them compete on price. Giving Gnip a monopoly over this market makes no sense. Twitter's biggest problem is the huge volume of requests. By blocking whitelisting you are forcing some developers to cheat by creating multiple accounts and distributing their requests across them. That can never be stopped. What you have to do is make it inefficient, by letting multiple resellers complete and drive the price of Twitter data down. Then the strongest reseller will take the load off of you and offer enough value added that developers will be willing to pay for data. That will never happen when only one reseller sets the price.
Twitter has benefited over the years by all of the additional value added by its developer community and hopefully the end of the whitelist won't aversely affect this same community. Green ends his note with a well-wishing to Twitter and we'll end with the same.
"You are riding a tiger. Good luck, and try to stay open and honest. This is a good step on that path," writes Green. "As long as you tell the truth, you will succeed."