{"id":1576,"date":"2014-08-21T09:22:31","date_gmt":"2014-08-21T15:22:31","guid":{"rendered":"http:\/\/www.mooreds.com\/wordpress\/?p=1576"},"modified":"2014-08-19T09:23:07","modified_gmt":"2014-08-19T15:23:07","slug":"building-a-system-with-ironmq-and-python","status":"publish","type":"post","link":"https:\/\/www.mooreds.com\/wordpress\/archives\/1576","title":{"rendered":"Building a System with IronMQ and Python"},"content":{"rendered":"
\"messages
Photo by andrewrennie<\/a> \"\"<\/a><\/small><\/figcaption><\/figure>\n

One of my most recent projects was writing a system to deliver real estate listing data to a content management system. The CMS was not in my control. Since the listing data source was bursty and I wasn’t sure how the CMS would handle the load, I decided to use a message queue, where the messages would have a JSON payload. Message queues are great at decoupling components of a system.<\/p>\n

For the queue, I used IronMQ<\/a>. The company already was using it, it has a free tier<\/a> (up to 24 messages a second), the service has been stable and reliable, it has great language SDKs<\/a>, and setting up a durable message queue is something I’d rather outsource. (I do wish Zapier supported it.) In other situations (when posting messages from mobile apps), we ran Varnish<\/a> in front of IronMQ so that it could be replaced easily. In this case, we didn’t because there were fewer moving pieces (it was server to server communication and it would be easier to swap out IronMQ should that be required).<\/p>\n

I wrote the bridge code from the listing database to the message queue in python. The shop was mostly Java and some python, and python seemed a better fit for a small ‘pull from here, push to there’ application. I used pytest<\/a> for unit testing, jenkins to run the unit tests in a CI environment, and autopep8<\/a> for formatting. My colleague was a more experienced python programmer, so I was able to lean on him for questions. I didn’t find python hard to pick back up (I’d scripted in python a little years ago), and it was a fun language to code in. Reminded me of perl w\/r\/t packages and quick developer feedback. I did miss Java’s dependency management, though (my college recommended virtualenv<\/a> as a possible solution).<\/p>\n

The JSON payload would allow developers writing the message consumer to use almost any language they wanted–any language if they used the IronMQ REST API<\/a> rather than an SDK.<\/p>\n

I can’t share the code, but for this kind of problem, python was a great solution. And I’ll reach for IronMQ any time I need a message queue. This pair of technologies was quick to implement, easy to deploy, and high performance wasn’t really a requirement, since the frequency of the listing delivery was the real bottleneck.<\/p>\n","protected":false},"excerpt":{"rendered":"

One of my most recent projects was writing a system to deliver real estate listing data to a content management system. The CMS was not in my control. Since the […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[62,10,33],"tags":[],"class_list":["post-1576","post","type-post","status-publish","format-standard","hentry","category-apis","category-dynamic-languages","category-useful-tools"],"_links":{"self":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/1576","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1576"}],"version-history":[{"count":3,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/1576\/revisions"}],"predecessor-version":[{"id":1580,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/posts\/1576\/revisions\/1580"}],"wp:attachment":[{"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1576"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1576"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mooreds.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1576"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}