The idea behind Amazon’s Mechanical Turk is pretty simple – break programming work down into bite-sized chunks, and put it in front of a large workforce that can do the work quickly and cheaply. Part of the challenge of that is making it easy for requesters to create the bites that workers are chewing on. The new categorization app from Amazon removes some of the hurdles of creating HITs (Human Intelligence Tasks) that ask workers to pick the best category for items. The result could make the crowdsource coding marketplace even more usable and popular.
Creating a request for Mechanical Turk isn’t overly difficult, but it takes a bit of time editing HTML and answering a lot of questions for which first-time users don’t have a lot of context – for example, deciding the qualifications for workers, or whether they need to be “masters” to take on a task.
Amazon is streamlining all that with the Categorization App by making assumptions about what its users would want, providing suggested pricing and helping to create the form that workers will see.
Right now, Amazon is providing an app only for categorization questions. Mechanical Turk offers templates for many more, including image filtering, image tagging, data extraction, data collection and several others. Expect that Amazon will add apps for most of these types of questions in the near future.
Boosting MT Revenues
Obviously, Amazon is trying to streamline the process for creating MT tasks in order to boost its revenue. More tasks means more money. But there’s a bit more to it than that.
With the Categorization App, Amazon’s assumptions all lead to more money per HIT. Users no longer have to decide what kind of worker to farm tasks out to; rather, Amazon makes that decision, automatically choosing the most expensive (master) workers. To increase accuracy, each HIT will be shown to two users by default, doubling the revenue.
Amazon will also suggest a pay level, to ensure that HITs are “priced attractively” to workers. Requesters can change this, but if a significant number of requesters accept the default pricing, that will probably drive HIT prices up over time.
Better Requests
Another reason that Amazon would want to create an app for MT requests is that a lot of requests are not very good. Amazon is in a position to evaluate the existing requests, and has found them wanting.
One problem? Too many categories. Amazon says that the maximum number of categories that workers can keep in mind is seven to 10. Yet they see requests with more than 150 categories!
The app limits requesters to 10 items. It requires that requesters provide instructions. It’s not foolproof, but it provides better odds that requesters will generate reasonable HITs. If requesters want more categories, they need to start from scratch with a custom template.
If you’re using Mechanical Turk now, what do you think of the Categorization App? Would you prefer to see Amazon simplify the service further, or is it already well-suited for the kind of crowdsourcing that you’re doing?
Illustration titled “Blogging Au Plein Air, Jean-Baptiste-Camille Corot” by Flickr user Mike Licht.
