ProPublica

Journalism in the Public Interest

Cancel

Reporting Recipe: Using Amazon’s Mechanical Turk for Data Projects

Announcing ProPublica’s guide to using Mechanical Turk for data-journalism projects.

.

Of all of journalism’s recent evolutions, data-driven reporting is one of the most celebrated. But as much as we should toast data’s powers, we must acknowledge its cost: Assembling even a small dataset can require hours of tedious work, deterring even the most disciplined of journalists and their editors.

Fortunately, there’s an affordable -- and amazing -- tool that can make the impossible easy: Amazon’s Mechanical Turk (mTurk).

For those unfamiliar with Mechanical Turk, it’s an online marketplace, set up by the online shopping site Amazon, where anyone can hire workers to complete short, simple tasks like quickly transcribing interviews, copying data from thousands of charts, and even sorting through satellite images in hopes of locating missing individuals. Amazon originally developed it as an in-house tool, and commercialized it in 2005. The mTurk workforce now numbers more than 100,000 workers in 200 countries.

At the urging of Panos Ipeirotis, a professor at New York University’s Stern School of Business, we began experimenting with mTurk last spring to clean, de-duplicate and reformat data. We’ve since used the tool to collect or proof more than 28,000 data points, from the names of companies that received stimulus money to the categorization of answers to our home loan modification questionnaire. We’re impressed with the speed and accuracy of its results. For example, a project we estimated would take a full-time staffer almost three days to finish was completed on mTurk overnight for $37, with 99 percent accuracy.

Mechanical Turk has proven to be more than a shortcut. It has freed up staff time for more complicated work. We’ve also used it to retrieve data from government databases that prohibit scraping.

We’ve summed up our knowledge of the tool and lessons learned in this guide, “ProPublica’s Guide to Mechanical Turk.” A lot of credit also goes to Professor Ipeirotis, who has answered many of our questions and reviewed our initial mTurk projects.

Got questions? Send them our way or post them below. Using mTurk in your data-driven journalism projects, or have some mTurk expertise to share? Compare notes in the comments below.

Read our guide to using Mechanical Turk.

Assuming you get good data back, this obviously a great business decision. But am I the only one who finds it a little demeaning that actual human labor—a real person—is reduced to the online representation of a robot?

Annie Shreffler

Oct. 15, 2010, 1:21 p.m.

As I read, I hadn’t thought of the work described as demeaning because it’s robotic, but I did wonder about the people actually accepting these tasks. If the staffer had done the work for 3 days, the cost would obviously have been much more that $37. So is this a living wage for those performing the tasks somewhere else, or is this more like side work for someone interested in extra cash?

I see that you only know your workers by ID numbers and email addresses, and you use “only approved workers… (just use Amazon’s built-in qualification tests to screen workers based on accuracy rate, location and experience).”  How did PP arrive at your specific hiring decisions and did you have any reservations about whether or not the rate is beneficial to the worker, other than a “some is better than none” reasoning? I didn’t see any evaluation of this kind on Prof. Ipeirotis’s blog, during my quick browse there, either.

Perhaps this value just didn’t figure into your thinking about data cruching at all, which is your perogative, but if that was a consideration, I would be interested in hearing more about it.

Thanks!
Annie

To be precise, I was not referring to the work itself as robotic. I was referring to the image of Mechnical Turk, which was, literally, a robot.

http://en.wikipedia.org/wiki/The_Turk

My emotional reaction to mturk is that knowing another human as only an ID number on the other end of some mysterious online labor market is a bit weird. I haven’t made up my mind about it, but it’s all a bit unsettling. I’d like to think that an organization as generously endowed as ProPublica could afford to pay minimum wage to someone looking to leverage data grinding experience into a career in journalism (that’s how I started my career, FWIW), but if they can’t, or if my experience is out of date, well okay. And I know nothing about the labor conditions for the workers on the other end. Maybe it’s an ideal setup for them, I really don’t know.

Amanda Michel

Oct. 15, 2010, 4:35 p.m.

To answer your questions… 

Ben, we don’t think of the work as demeaning; it’s work that we here at ProPublica do all the time. And the original Mechanical Turk was actually a hoax! It turned out that what people believed to be a machine was controlled by a hidden chess master. We’re using this tool because it makes it possible for us to do more reporting using data.

Annie (hello, btw!): Amazon doesn’t release many details about its workforce. Prof. Ipeirotis has run some worker surveys on mTurk that are opt-in and paid, so the results aren’t definitive. Many of the workers who responded to his survey were Americans who were more highly educated and using the tool to earn discretionary income. About forty percent of the people who responded were Indians, and they were far more likely to use the tool for basic income. We’ve had hundreds of people perform the tasks, and haven’t received any complaints about the size of payment. Overpaying also leads to bad results (high rates attract spammers). We’re aware that the pay per task is low, but we decided to go ahead with it. The work involved is piece work, which isn’t subject to the same wage standards as regular employment, even in the U.S. We’re learning a lot from breaking up our data-sets and may be able to integrate the piece-meal approach for our own work. That said, we’re still experimenting with this relatively new approach to cleaning and accessing data, and may modify our practices as we learn more. 

As far as hiring decisions, we look for workers with higher degrees of accuracy and have layered on qualification tests more recently to try and weed out spammers.

Annie Shrefler

Oct. 15, 2010, 4:52 p.m.

I appreciate your thoughtful response, Amanda. The curiosity you sparked in me about fair wages is satisfied by your explanation, and I’m reassured to know Propublica considered the issue. Most people do, after all, have the option to move on and look for other work offline. It’s also reassuring that you’re working with someone who has his eye trained on this kind of employment practice. I expect your news organization would be the first group to report any exploitation if it comes to that.

Good luck fighting off the spammers!

At the risk of being silly, I’d like underscore that I do not think of data cleaning as demeaning. And I never said that here, and argued against it in my previous comment.

As I said above, this sort of work was crucial in my career development, and it makes me uneasy to see it so casually sent overseas—especially when I’m unsure about what conditions the workers are laboring under (all I do know is that I’m encouraged to think of another person’s labor as little more than an anonymous eBay listing), and especially when it might come at the expense of hiring an intern who could use the work in ProPublica’s newsroom as a entry into journalism.

All that said, it sounds like a fabulous way to save money and a bright sign pointing toward the increased overshoring of American information jobs.

Panos Ipeirotis

Oct. 16, 2010, 5:39 p.m.

Ben, as you may observe, it is Srinivas (an intern) that works on this project. I would argue it is more beneficial for the intern to learn how to coordinate input from thousands of workers, rather than performing the task itself in their entirety.

Personally, I think that the major advantage of crowdsourcing is not the cost, but the speed. Cleaning 5-10 entries is a task easily handled by anyone. But clean 10,000 entries are you are tired, bored, and feeling completely unmotivated. So take the 10,000 entries task, give it to 1000 people, and everyone gets to done a small piece of the overall work. The task is done faster and the workers are not overburdened by huge, tedious tasks.

In addition to building databases (MTurk is great for compiling data from difficult-to-use sources, such as 990 form PDFS), MTurk is completely underused in other time-consuming areas of journalism. MTurk could potentially be used for reasonably priced comment moderation on high-volume news sites. I’m happy to see other journalists experimenting with this service.

Amanda Michel

Oct. 18, 2010, 9:02 a.m.

Ben, we’re open to ideas. Now that we’ve gotten down the basics we want to start experimenting with more advanced uses.

For an informed, thought-provoking, and literary treatment on mechanical turks—including social, political, and micro- and macro-economic aspects of this developing global labor force—read Cory Doctorow’s book FTW. Random House publisher I believe, in hardcover at any major booksellers now or available electronically for free: the author is a big proponent of Creative Commons licensing.  Lol and no I’m not in any way connected to the author or publisher.

Katharine Mieszkowski

Oct. 20, 2010, 7:04 p.m.

Back in 2006, I wrote a feature story for Salon about Amazon Mechanical Turk that addressed some of the issues that comment writers are raising here.

The story was called “I Make $1.45 a week and I love it!” Sub-head: “On Amazon Mechanical Turk, thousands of people are happily being paid pennies to do mind-numbing work. Is it a boon for the bored or a virtual sweatshop?”

Here’s a link:
http://www.salon.com/technology/feature/2006/07/24/turks

The story also appeared in the anthology “The Best of Technology Writing 2007”

Please review worker comments from the
workers in their forum for REAL insight
into the MTurk machine and how Amazon
treats it’s workers…
You can find it under Turker ProBoards, check
out the “suspension” threads to see how workers
are forced out with no explanation and the money
they earned is taken away from them.

I have to echo Dana R.‘s comment.  Although I have no doubt that Pro Publica is an upstanding employer, many requesters are not. 

Much more importantly, Amazon has an ineffectual non-interference policy, wherein a worker can be blocked or not paid at the sole discretion of the employer.  That’s fine, and all part of the contract; however, 3 of these blocks result in a permanent suspension from the site, fully governed by Amazon’s bureaucratic policy muck. 

You won’t believe the stories of the *best* MTurk workers, who wake up one morning to find their accounts frozen, funds forfeited, boilerplate customer services responses, and weeks of frustrating pleading ahead of them. 

You can read a collection of criticisms at Broken Turk http://brokenturk.blogspot.com.  Many of them are pulled from the top MTurk forum, http://turkernation.com .

Although I fully support the crowdsourcing movement, just like I support free markets in general, some focus from Pro Publica on what Amazon is making difficult, as well as what they are empowering, would be all in due course for an organization such as yours.

In other words, while you leverage the TurkForce, would you try to throw your weight in Amazon’s direction to help improve their MTurk policies.  Here’s a sampling.

*You can be banned for any reason
*Amazon won’t interfere with employers
*The is no employer rating system
*Automatic ban after three blocks
*Mysterious Terms of Service violations result in forfeiture of earnings
*Workers have no way to easily see when, by whom, or for what reason they were blocked.
*There is no differentiation between extremely experienced and high-reputation workers and anyone else; 3 blocks results in a ban no matter what.

These issues are very real, and as much attention which is paid to the uses of the site, a fraction of that put towards them would be very beneficial.

brokenturk @ yahoo.com for questions

Amanda Michel

Nov. 2, 2010, 9:48 a.m.

Thanks to everyone for all of your comments. We’ve dug into the issues you’ve raised. For whatever reasons, mTurk hasn’t developed a requester rating system or transparent appeals process. Unfortunately, Amazon doesn’t seem to penalize bad requesters either. 

But ProPublica does good work, and we use mTurk in a responsible way. We’ve crafted a mTurk policy which we’re sharing publicly. There will be a link to the policy within our instructions for specific projects on mTurk. We will also undertake an appeals process, ultimately overseen by our general manager.

Let us know if the policy could be further improved. 

best,
Amanda


ProPublica mTurk Policy

* We limit our work to mTurkers with HIT approval rates greater than 95 percent. 

* We block spammers. 

* Everyone makes mistakes. We don’t block mTurkers for errors. We only reject work when someone has made so many mistakes that we have to reissue the task. 

* We issue payment within 24-hours of a batch’s completion, except when the batch is posted on a Friday. So long as the HIT has completed, we issue payment on Monday. 

* You can appeal our decisions to reject work or block mTurkers. You’ll receive a message from us through mTurk, to which you can respond. You can also e-mail .(JavaScript must be enabled to view this email address) directly. All appeals will be overseen by ProPublica’s General Manager, Dick Tofel. 

* At the end of our tasks we ask mTurkers for their e-mail addresses. Going forward, we’ll let mTurkers know when we upload tasks. We’re committed to building a trusted network of mTurkers, and will begin rewarding those who consistently help us with bonuses. 

There’s an interesting evolution happening.  Namely, the good companies—ProPublica, CrowdFlower, CastingWords are developing their own policies and internal control systems; they also have established a reputation among independent forums for being reliable and fair.  However, none of those changes are accumulating to change the broader mechanical Turk interface.  It would be really, really helpful if companies and organizations which used the site responsibly started to group and recommend that their best practices were adopted for the site as a whole.  As happens, your responsible action is great, but it is self-contained unless you advocate for bigger changes and/or group up with those who share your priorities.  What do you think about trying to formulate a Turker’s bill of rights among some of the main companies.  We could develop a wiki, maybe, solicit ideas, and then present it to Amazon?  (-BrokenTurk, http://brokenturk.blogspot.com)

Brenda Jordan

Nov. 10, 2010, 6:10 p.m.

I have worked on MTurk since late last year. I understand I won’t get rich or anything, but I have made a good little bag of change, and it helps with lots of little things. I don’t mind the work and I can pick and choose tasks. I am good with words, so I mainly do word tasks and such. I do have my limit as to the amount of money I will do tasks for, and no matter where I am I can work at this. So, you see, when it’s all up front and you read the fine print, you should understand what it’s about or at least the basics..so don’t feel sorry for us lil’ Turkers who don’t mind adding to our finances with the small things of life..it’s all good!

Amanda Michel

Nov. 10, 2010, 7:34 p.m.

Brenda, thanks for chiming in. Have you worked on any ProPublica projects?

This article is produced with the help of readers:
ProPublica Reporting Network

ProPublica Reporting Network

Our collaborative reporting initiative that connects readers with our reporters to produce stories in the public interest.

Reporting Network

Sign up to be notified of reporting opportunities.

Get Updates

Stay on top of what we’re working on by subscribing to our email digest.

optional

Our Hottest Stories

  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •