Skip to content

Building a System with IronMQ and Python

messages photo
Photo by andrewrennie

One of my most recent projects was writing a system to deliver real estate listing data to a content management system. The CMS was not in my control. Since the listing data source was bursty and I wasn’t sure how the CMS would handle the load, I decided to use a message queue, where the messages would have a JSON payload. Message queues are great at decoupling components of a system.

For the queue, I used IronMQ. The company already was using it, it has a free tier (up to 24 messages a second), the service has been stable and reliable, it has great language SDKs, and setting up a durable message queue is something I’d rather outsource. (I do wish Zapier supported it.) In other situations (when posting messages from mobile apps), we ran Varnish in front of IronMQ so that it could be replaced easily. In this case, we didn’t because there were fewer moving pieces (it was server to server communication and it would be easier to swap out IronMQ should that be required).

I wrote the bridge code from the listing database to the message queue in python. The shop was mostly Java and some python, and python seemed a better fit for a small ‘pull from here, push to there’ application. I used pytest for unit testing, jenkins to run the unit tests in a CI environment, and autopep8 for formatting. My colleague was a more experienced python programmer, so I was able to lean on him for questions. I didn’t find python hard to pick back up (I’d scripted in python a little years ago), and it was a fun language to code in. Reminded me of perl w/r/t packages and quick developer feedback. I did miss Java’s dependency management, though (my college recommended virtualenv as a possible solution).

The JSON payload would allow developers writing the message consumer to use almost any language they wanted–any language if they used the IronMQ REST API rather than an SDK.

I can’t share the code, but for this kind of problem, python was a great solution. And I’ll reach for IronMQ any time I need a message queue. This pair of technologies was quick to implement, easy to deploy, and high performance wasn’t really a requirement, since the frequency of the listing delivery was the real bottleneck.

APIs Dynamic Languages Useful Tools | moore |

The limits of Google forms

paperwork photo
Photo by Harald Groven

I spent a fair amount of time in the spring working on Google forms. If you have Google apps for your domain and are doing any kind of data entry at all, you should use Google forms. With this, you can create a web and mobile friendly data entry form easily, with validation, for free.

And by you, I mean ‘you, the non technical user’–a big win.  I’m a big big fan of anything that removes developers from the loop because they are too often the bottleneck.  Of course, as you’ll see below, Google forms is not entirely the land of milk and honey.

Any data entered typically goes into a Google spreadsheet for easy batch processing. You can limit use of the form to your Google apps domain, too.

Here are some ways that I’ve seen Google forms used to eliminate manual re-keying of data:

  • Event RSVPs
  • Contact forms
  • Order forms for collateral
  • Transaction reporting
  • Timesheet or vacation tracking

Google forms are extremely flexible, but do have some significant limitations (which is why other form building companies like Wufoo haven’t had all the air sucked out of them).

Some limitations and issues are:

  • If you post to a Google spreadsheet, the typical use case, you are bound by spreadsheet limits: 400k rows, 256 columns.
  • You can’t host the form on your domain. The best solution to this issue I found was to do a URL redirect of a domain name. For example, from someform.example.com to the Google form URL. This lets you replace the form while still sending people to the same URL.
  • You can’t have a form email an arbitrary address on submission without custom code.
  • Multi page forms are possible, but clunky.
  • Validation is limited, though using regular expressions gives you a fair bit of power (but then takes form creation/maintenance out of the realm of the non developer).
  • UI customization is limited. A Google form will always look like a Google form (short of serious server side gymnastics). It will always have the ‘powered by Google forms’ link, the same crappy ‘response received’ page, and the same horrible handling of closed forms (an unmodifiable message from Google, with no way to customize it).
  • Option lists are static (though if you use formRanger, you can alleviate this issue).
  • File upload is not supported. Seriously. Even though Google wants you to use Google Drive.

As alluded to above, you can use Google Apps Script to alleviate some of the issues with Google forms. However, doing so pushes the maintenance of that form into developer land (or at least ‘power user’ land).

Even with all the warts, Google forms is a powerful tool. And did I mention it is free? If you or anyone in your business is currently doing manual data re-entry, and the limits above haven’t scared you away, I’d take a long hard serious look at Google forms.

BPA Google Apps | moore |

“Where there’s muck, there’s brass”

hydra photo
Photo by Andrew Jian

I was reading this Ask HN post about a consulting arrangement that seemed a Sisyphean task. Here’s an excerpt:

I have been asked to consult a company in how they should speed up their development process.

 

Today the application which in the end of it all is a web application, consists of a lot of old ASP classic code, a COM+ bridge for being able to function within a mix of a lot of different .NET libraries written in .NET 3.5 (CLR 2). COM+ acts as a bridge between the two technologies.

 

The teams cannot compile code inside the development tools, it can’t debug unless they do it with some obscure hacks and workarounds and it seems that no one is really in control of what is going on in the core code base and no one really want to touch the original code base to clean it up and refactor/re-write it.
…

Seems pretty hopeless, right?  “Teams can’t compile code”?!  Unfortunately, this type of task is more typical of consulting engagements than not. After all, if there was a simple solution, why would the company have engaged a high priced consultant?  (If you are consulting and not ‘high priced’, well, that’s a problem, but a different one better left for another post.)

The comments on the post are interesting and worth reading. I left one, but wanted to expand on it. Now, I’m not familiar with the tech stack at all, but I am familiar with a large codebase (where large is relative to the size of the team supporting it–100K LOC can be large to two person team) with a lot of technical debt that was crucial to the business.  I have also consulted for years.

Whenever you are consulting, the first task is always to ascertain the real problem.  Hint–it’s often not what you were explicitly hired to do.  In this case, I’m guessing the real issue is that the web application needs to change to meet business needs, and that it can’t do so fast enough because of the accretion of complexity.  But a guess isn’t good enough, you need to find out what you are being hired to do–it could be you are being hired to provide cover to spend money to rewrite the app or to be blamed when a development team misses dates or to actually speed up compilation.

Then, you need to learn who wants the task done, and who is writing the checks.  They are sometimes the same person, but not always.  You also need to learn how these folks want to be communicated with, including method, verbosity and frequency.

Finally, you can start to dig into the (software) problem.  (This process assumes you are doing time and materials billing.)  Do a preliminary investigation and look at some of the following:

  • given the end goal, what are intermediate steps that can get you there?  How long would it take to get one or two of these steps?
  • are there third party solutions that can get you 90% of the way to solving the business problem–this can include framework upgrades?
  • are there subsystems of the current hand coded solution that are isolated and can be reworked with minimal impact on the system?
  • are there one or two huge issues that would be a relatively easy win (version control, big bugs, moving configuration from code to a database, etc)?

Prepare ballpark estimates on the level of effort to accomplish some of these.  After that, you need to sit down with whoever wants the task done and whoever is paying for it.  Present your options, making it clear that any time estimates are truly SWAGs.

Let them decide.

If they ask for a recommendation, be prepared to make one, but the decision must be theirs.  They have the business context to know how much to invest in this system.

After that, either withdraw or start executing against the plan you and the decision makers have decided on.

Simple, right?

Consulting | moore |

Useful blogs for software developers in real estate

As I transition out of a full time role in real estate software development, I’m doing some cleaning up of my RSS feeds. (Yes, Virginia, people still use RSS readers. I’m a big fan of NewsBlur.)

If you are a software developer in the real estate space, here are some useful blogs to follow to understand the industry.

  • 1000 Watt is a consultancy that focuses on brokerages and realtors. If you like inside baseball, this is the blog to read. If you want to know about politics between portals and realtors, or high level strategies that 1000 Watt thinks will work for local brokers, or brokerage branding ideas, you should read this blog. They also do a good job of featuring the latest software startup to rock the real estate world. This blog is written by several members of the consultancy, but they all get bylines.
  • Calculated Risk is all about the big picture, the numbers and graphs. Bill McBride monitors the economy so you don’t have to. His real estate focus tends to be either nationwide or California, so he’s probably not going to write a post that is useful to your specific market, but his overall analysis of economic trends, whether positive or negative, is required reading. For instance, check out his job chart, which shows just how bad the great recession was, or the list of troubled banks (done by a contributor) or his graphs of other economic indicators like restaurant or trucking indices. If you are interested as a person or a developer in the mortgage market, read all of Tanta’s articles about mortgages–there’s a lot of money and lot of of complexity in that business (she passed away, but was a contributor for years).
  • The Geek Estate Blog is a unique fusion of technology and real restate. Drew Meyers was with Zillow and has done few startups outside of the real estate space. Because of this experience, he has a unique perspective on the real estate space and startups therein, and shares his perspective. Drew also takes guest posts (I’ve written a couple) and while sometimes they seem like thinly veiled sales pitches, other times they can clue you in to other technologies, startups or trends in the real estate world. I haven’t found a better source of real estate news that is focused on technology.
  • Your local excellent real estate blogger. In Boulder, that’s Osman Parvez of House Einstein, and no one else (I really like 8z’s market updates, but that only comes out once a month). I can’t tell you how to find this blog beyond searching for ‘[your location] real estate blogger’. They will most likely be a realtor (but they could be someone else–Dave Fratello wasn’t a realtor for a long time). They are using blogging as a bizdev tool and can give you a great view of your local market (for instance, check out this post about poptops in south Boulder, and this sample of his market stats).

Bonus read if you are a software developer looking to get into real estate: Game Plan is a book, not a blog, but I found it tremendously useful because it covers the recent history of the real estate business and will serve as a primer to get you up to speed on the various players and their concerns.

Finally, I also subscribed to some of the big tech giant tech blogs (Redfin, Zillow) but didn’t find much value in their sporadic posts.

If you are a software developer who is working in the real estate space, I’d love to know what you read.

Real Estate Technology | moore |

Video of the Week: Sales Safari

Amy Hoy discusses finding users’ pain points in this 50 minute presentation from 2013.

I think developers, myself included, are much more interested in the how (redis, nodejs, bright shiny things!) than in the why. Amy’s presentation is a good reminder that it is important to start with the why–what is your customer trying to achieve and why are they trying to achieve that. She calls this process a “sales safari”. Lots more information about this and other bootstrapping topics at her site.

Business Video | moore |

On the benefits of a private, internal API

polygon photo
Photos by Double–M

A few years ago, the company for which I worked went through the monumental task of defining neighborhoods for a number of cities in the area where they had real estate agents.  Neighborhood data is hard to get, and this task required a lot of back and forth between the person responsible for the mapping and the people who knew the neighborhoods.  The maps were captured in Google’s My Maps feature, and exported as KML to a vendor who would then build neighborhood pages and maps with the data.  Much of the neighborhood page would be driven off data entered in an admin back end system (it was a custom CMS, essentially).

Almost as an afterthought, I asked the vendor to provide an API for the neighborhoods, including the polygon data.  I wrote up an API spec, had it reviewed by my team, and obtained approval for the vendor to build it. If I recall, it was in the neighborhood of a couple thousand dollars, and the vendor had never been asked to build something like this before.

This one API allowed the company to apply dearly won neighborhood information in so many ways:

  • generate statistics by neighborhood against any lat/lng coded data
  • tag any geocoded content with neighborhood meta data
  • find new and sold listings by neighborhood
  • understand who were top listing agents in each neighborhood
  • create internal BI tools
  • write internal recruiting tools
  • pull other geocoded data by neighborhood
  • tag transactions with neighborhood meta data

Many of these were accomplished with a plugin to the data processing tool (Pentaho Kettle) that used the Java Topology Suite. Creating JTS geometries is expensive, so the plugin caches them with a simple hashmap cache. The plugin java code is garbage collected fully on each data load run, so this simple solution is appropriate, rather than a more complex LRU cache.

However, this solution isn’t perfect. Often, if a property was on the boundary, the JTS code would often put it in the wrong neighborhood. Boundaries of neighborhoods are incorrect or overlap. Points are incorrect because geocoding isn’t perfect. Human review is still required.

But, the very fact that the neighborhood data was so accessible meant that the company could ask questions (how many homes are in each neighborhood, what are the three newest listings in this neighborhood) that simply couldn’t have been asked if there was no API. Having an internal API that exposed hard won business knowledge within the company was beneficial, even if it will never be exposed or monetized outside the company.

8z APIs | moore |

WP-Inject Rocks

inject photo
Photo by MattysFlicks

Inspired by Drew Meyers over at the Geek Estate Blog, I’ve made an effort to start putting images into my blog posts. I manually searched Flickr for commercially licenseable images, upload them to my server and put them into my post. After I did this a couple of times, I thought–surely there’s a plugin for this. And if not, I should write one!

A quick search turned up WP-Inject. I installed it and never looked back. It takes care of searching Flickr (and another site that I’ve never heard of called Pixabay). It takes care of the attribution link. It uploads the file to my server. It puts the image into my blog post. Well worth the install if you want to add any images to your posts at all.

My only wish is that it handled captions a bit better, but this could be a config option I’m overlooking. And that’s a small flaw for such an awesome plugin!

Blog Useful Tools | moore |

Giving books as gifts

book photo
Photo by shutterhacks

A few years ago one of my contracting clients gave me a book for a Christmas gift. It was a thoughtful gesture. It mattered all the more that I received it from a client who I’d spent a lot of time working for and who I know had spent a lot of time thinking about some of the issues the book raised. I read the book, and applied its lessons to my life, both business and personal.

Since then, I’ve given and received books to business colleagues and clients. Books are a unique gift because they convey a message in a way that few other gifts do. A book is a more committing gift than a visa gift card or a piece of electronic equipment (both gifts I’ve also received and was thankful for). You are asking for the receiver’s time and focus, and making a statement that the gift is worthy of both of those. You’re also making the statement that the recipient will benefit from the book. It’s a gift that actually has some demands (a bit like giving a gym membership, without as much judgement).

When you are giving a book, you’re giving a new idea. You are saying you were touched or changed by this book, and you hope that the recipient will find the same idea. You also need to understand the receiver–will they find the same benefit? Have they already tuned into the ideas so it will deepen their understanding? Is this a new perspective for them?

Bonus! If you set giving worthy books as a goal, it will make you read more books. It also has the beneficial side effect of forcing you to evaluate books you read from the perspective of ‘would I want to give this as a gift?’. Not every book has to meet that criteria, just as not every meal should be broccoli and yogurt, but it is a useful filter if you have limited reading time.

I hope I’ve convinced you to try giving books to your clients, colleagues, bosses and employees. The easiest way to start is to find a solid business book that you’ve enjoyed reading (‘Getting to Yes’ and ‘The Personal MBA’ are two candidates) and sending it to someone you have worked with for a while. Then, start a list and add any other books that resonate with you. When the holidays come around, you’ll have a list to choose from. Start small because of this higher bar this sets–you’re not just asking for a ‘Thank You’ and a place on a shelf, you’re asking for time and focus.  And those are precious.

Books Business | moore |

Content generation for employee acquisition

interview photo
Photo by MattHurst

I was brainstorming the other day and thought of an add on service for recruiters. My explication of this service is focused on tech companies hiring engineers, but could easily be modified for any organization that is trying to find high value employees that are difficult to hire (high performing real estate agents, sales people, financial advisers, etc).

If your company is doing interesting things–solving tough problems or using interesting technologies–potential employees are very likely interested in your activities. How, though, will they find out about the cool problems your company is solving? Well, there’s your company website, back channel communications through professional networks, or presentations at conferences or meetups. But the traditional website serves many masters (including converting ‘customers who will pay you’), and content may be hard to generate or place. Professional networks often scale poorly, depending on where you are and what sector you are in. And presentations or meetups are rather scattershot and time consuming.

Enter the company tech blog. Write about solving interesting problems. Because the blog is tech focused, you avoid the issues with the traditional company website. (Of course, don’t write about any trade secrets or unprotected intellectual property, but in my experience a large chunk of the engineering problems solved at any company are scaffolding, not core knowledge.) It also scales, because the content is write once, recruit for years (not forever, because eventually the problems you discuss won’t be interesting). Basically, any argument in favor of content generation for customer acquisition can be applied to content generation for employee acquisition. So, set up that company blog and have your engineers start writing blog posts about interesting work they are doing. Soon, you’ll rank in Google and/or have Hacker News fame (see 42 floors).

…

Wait, what? You say that most engineers who are competent to write these articles either have no time, no interest, no ability or some combination of all three? I’ve seen many many company tech blogs that start off strong and interesting and then slowly fade away. This is more prevalent since other information dissemination platforms (your twitters, your facebooks, what have you) have proliferated, but for whatever reason, the key to a content generation strategy is to keep at it.

And that is where my idea shines. As a value add service for a recruiter, hire reporters to interview engineers. Have the reporters write the article, with engineer review to make sure it is correct, and have both on the byline (or use ‘as told to’). An interview about an interesting tech problem will probably take about an hour, and you can expect an hour for review, so you still have to carve out two hours from your engineer. If you have a team of ten engineers, and half are willing to be interviewed, that is less than two hours a month for a weekly blog post. Of course, you have to pay the reporter for more than two hours, but reporters are less expensive than engineers. Sure, this is an extra cost, but the article will be published. And the next one will get done. And eventually, the company will have a recruiting site working for you. Hard problems aren’t everything for engineers, but they count for a lot.

I mentioned this business plan to a friend and his feedback was–“seems like a great idea, but couldn’t an intern and a junior marketing person do this”? I think so, so I’d love to see more companies doing this! Hire that intern and that marketing person and start blogging about the hard problems you have faced and solved! However, maybe you outsource your recruiting efforts. If so, ask your recruiter about their content generation strategy.

If you are a recruiter, consider offering this as a value add service. (Eventually you may work yourself out of a job if your only value add is finding people, but good recruiters I’ve talked to offer more than just finding people–they screen them, make sure they are a culture fit, help the candidate through the process, and more.)

Do you know any companies or recruiters that are doing this? Do I have a fatal flaw in my idea? Let me know.

Business Recruiting | moore |

Saying Goodbye to 8z

Today is my last day as an FTE. After four years as an employee, and five years as an on and off contractor, I’m moving on from 8z. Partially–it looks like I’ll be doing some contracting.  8z is a great company that I think will be fixing what is broken in real estate for the foreseeable future.

I learned a lot in my time.  Here’s a top N list:

  1. Hiring is hard–I had the luxury of hiring a few employees and ICs, and it was a grind to find the right person.  I don’t know if that was because my budget was tight (so I was really only looking at entry or junior level candidates) or the team was small (so I needed someone willing to wear at least three hats–ops, developer, QA) or my location (Boulder) or the fact that developers are in high demand, but it was a grind. However, when I did find the right person (and I think I had a pretty good batting average) it was great to see the team gel and people advance in their skills and careers.
  2. Budgets are fiction, but a necessary one.  The planning required for a budget enforces rigor, but the actual spend almost always changed. Like Eisenhower said.
  3. Blogging is harder to do when you’re an employee. It’s just harder to find the time.
  4. Zapier is awesome.
  5. Managing while being an individual contributor is hard.  In a small team (we were never more than four), I (and everyone else) wore many hats, including PM, tech lead, developer, QA, DBA and business analyst.  From my reading–looking at you, Rands–some of the difficulty was intrinsic to management, not just management of a small team while wearing many hats.
  6. Having monthly one on ones with your direct reports is a bare minimum.  If you aren’t having these meetings with your reports or your superior, start doing so. The open lines of communication will pay dividends long term by making sure you aren’t surprised.
  7. Using the database as a bus lets you have a number of smaller isolated more easily understandable software processes.
  8. Being a software developer in a small team at a real estate company made it hard to find peers around Boulder. I didn’t fit into the startup culture nor did I fit into the bigCo culture.
  9. Default to using innodb.
  10. Core values can actually matter. If the CEO makes a regular effort to help them live.
  11. You can ship a lot of software with a small team.  But code rot will eventually bit you–here Jenkins and automated tests (junit, dbunit, qunit, phpunit, py.test, pentaho kettle testing, etc) are your friends. Jeff, thanks for pushing hudson!
  12. Gmail is an eminently usable web client.
  13. Having two equal co-founders is a tough row to hoe.
  14. Some people can tell the difference between Budweiser and Coors with their eyes closed.
  15. Google Apps is going to severely wound MS Office, and LibreOffice might do the rest.
  16. There is no substitute (none!) for asking the person who knows.  Many times I’d be discussing requirements or a bug, and come up with a question.  After batting it around a few times, the best course was always “well, let’s just ask XXX”, where XXX was the person who knew.  The nice thing about a small company is that the right person to ask is usually obvious, and if the first person you ask isn’t the right person, they know who is. Don’t speculate–find out!
  17. MLSes … if you are in real estate software and you want listing data, and you do, be prepared to spend a significant amount of time understanding your local MLS compliance rules.
  18. Some of my biggest wins in the last four years were finding the right tech partner, not building the software from scratch.
  19. APIs make sense, even in the smallest company.  Even if you are never going to expose them to the outside world for gain (and more so if you will).  They enforce a rigor and provide interfaces for polyglot development.  And you can be surprised where they’ll be useful.
  20. Java has a tremendous ecosystem.  This includes third party jars (like this one for querying polygons), frameworks like spring, and tools like maven.  I know lots of folks aren’t impressed with maven, but I found once you get your head around it, it can make your life easier. If there’s a build feature you want, maven already has it.  Plus, dependency management kicks ass–I hope that npm learns more from maven than just avoiding XML.
  21. A small team working primarily in Java seems to be an anomaly in this day and age.
  22. Having a consistent tech stack vision is good–it leads to reuse and fewer operational issues.
  23. It is far more satisfying to ship a 90% solution than not to ship at all.
  24. Hackfests and brown bags aren’t just for software companies.  With the support of the CEO, I introduced both of these to the wider company (see my post about monthly brown bags and semi annual hackfests) and they resulted in cross pollination, public speaking skills for employees, and some great ideas, including one that became a core 8z initiative.
  25. Having software in maintenance mode is tough on a team.  One of the biggest tasks for my team for about half of my time at 8z was to maintain an old complex website (10+ years old) just enough to keep it running. It was and is crucial to the business, but the business was trying to migrate away from it for a number of reasons (gobs of technical debt, among others).  This required painful choices, because often the right technical choice (fix the root cause) was the wrong business choice, and technical band-aids were the right business choice. That wears on me.
  26. Maps in Apache Cordova are hard to get right–the single threaded javascript execution environment really stinks when it comes to complex maps. (ht @intalex)
  27. A five character domain name gets a lot of purchase offers. You can also tell, to a first approximation, how technical someone is by how they react to your short domain name.
  28. Beer Friday (aka Friday afternoon club) is a great way to get people to interact cross department, but doesn’t work for everyone–not every coworker wants to hang out and have a beer.  Having other ways to cross pollinate makes sure everyone can participate.
  29. It is addictive to hear from consumers or other users that your software made their lives better.
  30. I learned as much or more from my failures as I did from my successes.  To name some particular personal failures, I made a hire that didn’t work out, a mobile app of which I was owner of that didn’t get to production (or even to QA) and a vendor who I evaluated, recommended and implemented that ended up not being a fit.  In each case it felt really crappy when it happened.  After I had some time and perspective, I was able to identify some lessons that hopefully will keep me from making the mistake again. You might hear more about these mistakes in the future.
  31. Finally, I enjoyed the the opportunity to be the primary technical voice at the executive table (while I never had the title, I was the effective CTO for a time). Bringing software engineering practices into a real estate company was fun and I think made 8z a better place to work. Of course, I could not have done any of that without the support of the owner/CEO, but it was fun to see what stuck and what didn’t.

The primary reason I joined 8z was that I felt after seven years of contracting and consulting that I wanted to be part of a team again. I also thought being an employee would let me grow in ways that you can’t as a contractor. Both of these turned out to be true.

Thanks for four great years!

8z | moore |