{"id":2267,"date":"2017-02-09T06:15:37","date_gmt":"2017-02-09T12:15:37","guid":{"rendered":"http:\/\/www.mooreds.com\/wordpress\/?p=2267"},"modified":"2021-11-21T12:39:24","modified_gmt":"2021-11-21T18:39:24","slug":"using-amazon-mechanical-turk","status":"publish","type":"post","link":"https:\/\/www.mooreds.com\/wordpress\/archives\/2267","title":{"rendered":"Using Amazon Mechanical Turk"},"content":{"rendered":"

\"chess-1215079_640\"So, after over a decade, I finally found a use case where I had the clout and the need to use mechanical turk<\/a>. I wanted to write about my experiences.<\/p>\n

What I used it for: <\/strong>We were looking for some data on businesses.\u00a0 We had business name, city and state, and wanted full contact information.\u00a0 We paid a dime for each listing, and asked for email address and physical address.\u00a0 We asked about each listing twice so that we’d have some kind of double check.<\/p>\n

How effective was it? <\/strong>This varied.\u00a0 If you were using the master workers, it was very effective, but slower.\u00a0 If you open it up to all workers, you have to review their work more closely.\u00a0 The few times I rejected someone’s task, they wrote back and asked why and tried to make it right, which was a testament to the power of the system (it records rejections).\u00a0 Make sure you break the work into a couple of smaller groups so you can iterate on your instruction set (when workers asked questions on the first set, the answers went into the instructions for the second set).\u00a0 We still had to review all the listings and double check any that didn’t match between both task answers, but that was a lot quicker than googling for each business and doing the research ourselves.<\/p>\n

How much did it cost? <\/strong>On the order of a couple hundred bucks to process around fifteen hundred listings.<\/p>\n

What kind of time savings did we see?<\/strong> Assume we had 1500 business names, and it took us 90 seconds to google the business name and find the information.\u00a0 That is 1500 listings * 1.5 minutes == 37.5 hours, and this is on the low end.\u00a0 Instead, it took about 2-3 hours of setup, and then 36 hours of calendar time (when I was able to do other things like sleep and work on other problems), and we were done.\u00a0 Then I would say it was about 7-10 hours of review. So you are trading a couple hundred bucks for at least 20 hours of saved time.<\/p>\n

Would I do it again? <\/strong>I think mturk is perfect if your problem has the following three attributes: more money than time, a task that is extremely simple, and time to review the finished product.<\/p>\n

Other tips?<\/strong> You have to build it some kind of sampling for correctness. I have no idea what the quality is if you pay more than a dime per task.\u00a0 Make sure you think about edge cases.\u00a0 Provide tips to your workers (“check whois records as well as google”).<\/p>\n","protected":false},"excerpt":{"rendered":"

So, after over a decade, I finally found a use case where I had the clout and the need to use mechanical turk. I wanted to write about my experiences. […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,81,33],"tags":[],"class_list":["post-2267","post","type-post","status-publish","format-standard","hentry","category-technology-and-society","category-the-food-corridor","category-useful-tools"],"_links":{"self":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/2267","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2267"}],"version-history":[{"count":4,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/2267\/revisions"}],"predecessor-version":[{"id":3496,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/2267\/revisions\/3496"}],"wp:attachment":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2267"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2267"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2267"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}