Fun stuff I’ve done

amusement park photo

Photo by Loozrboy

One of the companies I’ve met with wanted a bit more context around work I’ve done, so I wrote up some of the ‘greatest hits’ of tech work I’ve done in the past couple of years. It was so much fun, I thought I’d post it here so I’ll remember it in a few years.  This is just some of the tech highlights, and doesn’t include other things I’ve learned (managing folks, softer development skills, etc).

  • Colorado CSAs.info is a directory of Colorado CSAs. This site received 28k visits in 2013, and 26k visits so far in 2014. You can view the source. The source has some quick and dirty aspects, since this is a side project.
  • Home valuation processing software output. This was a fairly simple algorithm (linear regression among geographic datasets) but involved some interesting processing because of the number of records involved (1M+) and data sources. The graph shown here was generated by code I wrote. The content is human generated, however. I worked directly with the CEO on requirements for this v1 project.
  • I wrote an ebook about an aspect of mobile app development last year:
  • I built large chunks of COhomefinder (which has not been touched in 2 years, because the business has decided to move forward with an outsourced home search solution). The last code I touched was replacing google maps with mapquest. I also built a widget for featured home listings that could be dropped into other websites. You can see both on the search results page.
  • I wrote a hybrid mobile app that displays news and info about neighborhoods in the front range. This never really got much traction, unfortunately.

It’s fun to look back.


On the benefits of a private, internal API

polygon photo

Photos by Double–M

A few years ago, the company for which I worked went through the monumental task of defining neighborhoods for a number of cities in the area where they had real estate agents.  Neighborhood data is hard to get, and this task required a lot of back and forth between the person responsible for the mapping and the people who knew the neighborhoods.  The maps were captured in Google’s My Maps feature, and exported as KML to a vendor who would then build neighborhood pages and maps with the data.  Much of the neighborhood page would be driven off data entered in an admin back end system (it was a custom CMS, essentially).

Almost as an afterthought, I asked the vendor to provide an API for the neighborhoods, including the polygon data.  I wrote up an API spec, had it reviewed by my team, and obtained approval for the vendor to build it. If I recall, it was in the neighborhood of a couple thousand dollars, and the vendor had never been asked to build something like this before.

This one API allowed the company to apply dearly won neighborhood information in so many ways:

  • generate statistics by neighborhood against any lat/lng coded data
  • tag any geocoded content with neighborhood meta data
  • find new and sold listings by neighborhood
  • understand who were top listing agents in each neighborhood
  • create internal BI tools
  • write internal recruiting tools
  • pull other geocoded data by neighborhood
  • tag transactions with neighborhood meta data

Many of these were accomplished with a plugin to the data processing tool (Pentaho Kettle) that used the Java Topology Suite. Creating JTS geometries is expensive, so the plugin caches them with a simple hashmap cache. The plugin java code is garbage collected fully on each data load run, so this simple solution is appropriate, rather than a more complex LRU cache.

However, this solution isn’t perfect. Often, if a property was on the boundary, the JTS code would often put it in the wrong neighborhood. Boundaries of neighborhoods are incorrect or overlap. Points are incorrect because geocoding isn’t perfect. Human review is still required.

But, the very fact that the neighborhood data was so accessible meant that the company could ask questions (how many homes are in each neighborhood, what are the three newest listings in this neighborhood) that simply couldn’t have been asked if there was no API. Having an internal API that exposed hard won business knowledge within the company was beneficial, even if it will never be exposed or monetized outside the company.


Saying Goodbye to 8z

Today is my last day as an FTE. After four years as an employee, and five years as an on and off contractor, I’m moving on from 8z. Partially–it looks like I’ll be doing some contracting.  8z is a great company that I think will be fixing what is broken in real estate for the foreseeable future.

I learned a lot in my time.  Here’s a top N list:

  1. Hiring is hard–I had the luxury of hiring a few employees and ICs, and it was a grind to find the right person.  I don’t know if that was because my budget was tight (so I was really only looking at entry or junior level candidates) or the team was small (so I needed someone willing to wear at least three hats–ops, developer, QA) or my location (Boulder) or the fact that developers are in high demand, but it was a grind. However, when I did find the right person (and I think I had a pretty good batting average) it was great to see the team gel and people advance in their skills and careers.
  2. Budgets are fiction, but a necessary one.  The planning required for a budget enforces rigor, but the actual spend almost always changed. Like Eisenhower said.
  3. Blogging is harder to do when you’re an employee. It’s just harder to find the time.
  4. Zapier is awesome.
  5. Managing while being an individual contributor is hard.  In a small team (we were never more than four), I (and everyone else) wore many hats, including PM, tech lead, developer, QA, DBA and business analyst.  From my reading–looking at you, Rands–some of the difficulty was intrinsic to management, not just management of a small team while wearing many hats.
  6. Having monthly one on ones with your direct reports is a bare minimum.  If you aren’t having these meetings with your reports or your superior, start doing so. The open lines of communication will pay dividends long term by making sure you aren’t surprised.
  7. Using the database as a bus lets you have a number of smaller isolated more easily understandable software processes.
  8. Being a software developer in a small team at a real estate company made it hard to find peers around Boulder. I didn’t fit into the startup culture nor did I fit into the bigCo culture.
  9. Default to using innodb.
  10. Core values can actually matter. If the CEO makes a regular effort to help them live.
  11. You can ship a lot of software with a small team.  But code rot will eventually bit you–here Jenkins and automated tests (junit, dbunit, qunit, phpunit, py.test, pentaho kettle testing, etc) are your friends. Jeff, thanks for pushing hudson!
  12. Gmail is an eminently usable web client.
  13. Having two equal co-founders is a tough row to hoe.
  14. Some people can tell the difference between Budweiser and Coors with their eyes closed.
  15. Google Apps is going to severely wound MS Office, and LibreOffice might do the rest.
  16. There is no substitute (none!) for asking the person who knows.  Many times I’d be discussing requirements or a bug, and come up with a question.  After batting it around a few times, the best course was always “well, let’s just ask XXX”, where XXX was the person who knew.  The nice thing about a small company is that the right person to ask is usually obvious, and if the first person you ask isn’t the right person, they know who is. Don’t speculate–find out!
  17. MLSes … if you are in real estate software and you want listing data, and you do, be prepared to spend a significant amount of time understanding your local MLS compliance rules.
  18. Some of my biggest wins in the last four years were finding the right tech partner, not building the software from scratch.
  19. APIs make sense, even in the smallest company.  Even if you are never going to expose them to the outside world for gain (and more so if you will).  They enforce a rigor and provide interfaces for polyglot development.  And you can be surprised where they’ll be useful.
  20. Java has a tremendous ecosystem.  This includes third party jars (like this one for querying polygons), frameworks like spring, and tools like maven.  I know lots of folks aren’t impressed with maven, but I found once you get your head around it, it can make your life easier. If there’s a build feature you want, maven already has it.  Plus, dependency management kicks ass–I hope that npm learns more from maven than just avoiding XML.
  21. A small team working primarily in Java seems to be an anomaly in this day and age.
  22. Having a consistent tech stack vision is good–it leads to reuse and fewer operational issues.
  23. It is far more satisfying to ship a 90% solution than not to ship at all.
  24. Hackfests and brown bags aren’t just for software companies.  With the support of the CEO, I introduced both of these to the wider company (see my post about monthly brown bags and semi annual hackfests) and they resulted in cross pollination, public speaking skills for employees, and some great ideas, including one that became a core 8z initiative.
  25. Having software in maintenance mode is tough on a team.  One of the biggest tasks for my team for about half of my time at 8z was to maintain an old complex website (10+ years old) just enough to keep it running. It was and is crucial to the business, but the business was trying to migrate away from it for a number of reasons (gobs of technical debt, among others).  This required painful choices, because often the right technical choice (fix the root cause) was the wrong business choice, and technical band-aids were the right business choice. That wears on me.
  26. Maps in Apache Cordova are hard to get right–the single threaded javascript execution environment really stinks when it comes to complex maps. (ht @intalex)
  27. A five character domain name gets a lot of purchase offers. You can also tell, to a first approximation, how technical someone is by how they react to your short domain name.
  28. Beer Friday (aka Friday afternoon club) is a great way to get people to interact cross department, but doesn’t work for everyone–not every coworker wants to hang out and have a beer.  Having other ways to cross pollinate makes sure everyone can participate.
  29. It is addictive to hear from consumers or other users that your software made their lives better.
  30. I learned as much or more from my failures as I did from my successes.  To name some particular personal failures, I made a hire that didn’t work out, a mobile app of which I was owner of that didn’t get to production (or even to QA) and a vendor who I evaluated, recommended and implemented that ended up not being a fit.  In each case it felt really crappy when it happened.  After I had some time and perspective, I was able to identify some lessons that hopefully will keep me from making the mistake again. You might hear more about these mistakes in the future.
  31. Finally, I enjoyed the the opportunity to be the primary technical voice at the executive table (while I never had the title, I was the effective CTO for a time). Bringing software engineering practices into a real estate company was fun and I think made 8z a better place to work. Of course, I could not have done any of that without the support of the owner/CEO, but it was fun to see what stuck and what didn’t.

The primary reason I joined 8z was that I felt after seven years of contracting and consulting that I wanted to be part of a team again. I also thought being an employee would let me grow in ways that you can’t as a contractor. Both of these turned out to be true.

Thanks for four great years!


Running a brown bag lunch series in your office

Courtesy of smoothfluid

Courtesy of maxually

Brown bag lunches are great opportunities for employees to share their knowledge, learn new skills, and bring a small company together.  By ‘brown bag lunch’, I mean an internal presentation lasting about an hour, made by an employee on an interesting topic of their choice.  The name comes from everyone bringing their lunch to work on that day, rather than eating out.

8z has been doing them for over two years, and here are some lessons.

  • Schedule them monthly, and one mont at a time.  Don’t try to schedule out the whole year.
  • Have presenters spend as little time as possible building a powerpoint.  It’s hard to get away from them as a structural crutch, but they don’t really add a lot of value.
  • Bring in real business situations.  One of the most memorable presentations occurred when presenters analyzed a recorded call during the presenation.
  • Have someone be point and recruit people individually.  Don’t count on volunteers, especially at the beginning.
  • It’s OK to miss a month or two if other stuff is going on.  Hello December.
  • Record them if you can.  All you need is an ipad and a youtube account.
  • Technical presentations (like application architecture) are appreciated by the business folks.
  • Everyone has something to say.
  • You can have people repeat every six months or so.
  • Some people won’t want to speak.
  • Presenting in pairs can work.
  • Make sure the presenter leaves plenty of time for Q&A.  8z budgets an hour for the talk and Q&A.
  • Schedule it so founders/executives attend.  This makes a powerful statement and exposes them to direct ideas.
  • Be prepared to capture changes/feedback from the presentation.
  • The departmental cross pollination is a major benefit.
  • Consider themed potlucks (mexican, breakfast for lunch, etc) instead of brown bag lunches.

How do you spread knowledge within your small company?


Observations on a Writing a Custom Report with Java, Quickbooks, Jasper Reports, Google Spreadsheets and Google Drive

A recently released project is using java and spring to pull data from quickbooks, a mysql database and google spreadsheets, munging the data in various ways, and using jasper reports and jfreechart to generate a good looking report and a CSV of transactions that will give our brokers weekly updates on how they are doing compared to their goals for the year. I then upload it to Google Drive and send an email notifying each realtor that they have a new file.  It’s always nice to release useful software–feels like a new day dawns.

A few observations from this project:

  • The tech was interesting, but it was actually more interesting to see how the needs of tech drove the business to ‘tighten up’ their processes. Whether that was making sure there was one place to look for user type data, or defining exactly what constituted achieving a goal, or making sure that any new realtors who joined created business goals (and committed them to writing), the binary nature of software forced the business (or, more accurately, people in the business) to make decisions. Just another example of business process crystallization. This also meant deferring some software development. Where the business couldn’t answer a question definitively, rather than force them to do so, we chose to defer development.
  • I’m glad that jasper reports makes it so easy to generate PDFs–you basically create an XML file (I was unable to find a spec, but there were plentiful examples) and then put tokens for dynamic content. Then you compile the XML file, give it a map of said tokens and values (which can be text, numbers, dates or images), and then export the object to PDF. Note that I was not using Jasper in a typical way–reporting from large amounts of similar data via a data connection–because I wanted different PDFs for each user. Perhaps there was a way to do this elegantly, but I was just trying to get stuff done. Creating a grid in jasper was interesting, though.
  • JFreechart had a very weird issue where on stage the graph labels were bolded and italicized, but not on production. Since we make every effort to keep these two environments in sync and they were running exactly the same code, this was a mystery. Finally solved it when we discovered java was different (same version, different vendors: openjdk vs sun java). Had been running this way for years. Oops.
  • Interacting with google spreadsheets is great for the business but a pain in the butt for developers. It’s great for our business because it is extremely easy for someone who is not a programmer to create a ‘database’ that is versioned, backed up, almost always accessible and helps them ‘get stuff done’ in a measured way. It’s a pain for developers because it is a ‘database’ and not a database–no referential integrity or data typing. Also, google provides cell based access and row based access, forcing you to choose. And the java libraries are old. Beats excel though–at least it is accessible from a server. We ended up writing a library to wrap Google’s Java SDK to make some common operations easier.
  • Pushing to google drive is interesting–I alluded to this in my last post but you have to be ready for failure. I ended up using a BlockingQueue for this–I throw files (a data structure defining the file, actually) to be uploaded on the queue, then consumers executing in a different thread each take one off, try to upload it, and if it fails, put it back on. I considered using a third party durable queue like IronMQ, but thought it was overkill.
  • Using the Quickbooks SDK, with all the accounting data exposed, lets you build some pretty powerful graphs that are useful to the business. But the docs are jumbled, with a lot of them aimed at developers who are building integrations to sell to Quickbooks users. Support is OK for standard operations, but for things like renewing your token, you have to drop down to the REST API (see my SO question) This article does a good job of outlining the various projects but as a dev you’ll have to ignore certain sets of information–never fun when getting up to speed.
  • We do a lot of backend processing and spring and maven and a custom assembler that generates a tarball when using ‘maven install’ have been great. I also finally figured out how to work with maven to use ‘release:prepare’ and ‘release:perform’ for releasing libraries, as opposed to going my own way, and that has made things much much easier. Learn your tools, folks!
  • I’m once again astounded by the richness of the java library ecosystem. There doesn’t seem to be very much that I can think of doing that doesn’t have at least one, and probably three, java implementations.

Want your next software purchase to ‘talk’ to your existing systems?

I wrote a post, over at the Geek Estate Blog (one of the best real estate tech blogs, imho) about APIs and why if you are a realtor or a broker, you should look for software with an API

Well, first, let’s talk about what an API is. API stands for “Applications Programming Interface”–not very illuminating. What that means is that the software providing the API has a way to allow other software (applications) to interact (interface) with it, with no human involvement. That is, the two software systems can ‘talk’ to each other.

Full post here.


How to collect usage statistics in your phonegap/cordova application

My company recently wrote a couple of mobile applications. Since one is for consumer use (search for ‘8z neighborhood’ in the App Store/Google Play if you live in Colorado or the Bay area and want to check it out), I wanted to know what type of users were downloading and installing it, what was being used, and other general usage statistics.

I asked a mobile app vendor we’ve worked with what they used for usage stats, and they said Flurry. I also looked at Google Analytics, Mobile–this is a nice q&a explaining the major players in the mobile analytics market. We didn’t want anything complicated, just basic stats with the possibility of collecting further information in the future, so I went with the vendor recommendation. Flurry also works with the two platforms we were targeting: IOS and Android, as well as many others.

Flurry is zero cost, but nothing’s free–they want your data and you grant them a “right, for any purpose, to collect, retain, use, and publish in an aggregate manner” all data collected from your application. I’ve seen similar TOS from most of the free analytics vendors, so this was no surprise.

To use Flurry with cordova/phonegap, and with plugman, I was ready to plugmanify an existing Flurry phonegap plugin. Luckily, someone else had already done it. All I had to do was update the plugin to work with Cordova 2.9, and to use the latest IOS7 compatible Flurry library.

After you install the plugin (I recommend doing so in an after_platform_add hook script), you simply add this to your code after the deviceready event fires: window.plugins.flurry.startSession(sessionkey). I inject the session key using a hook script, because I wanted a different key for stage and production builds, and also for each device platform (Flurry requires the latter). Because hooks only get the root directory as context (although this is supposed to change) I had to put some logic in the javascript to call startSession with the appropriate key:

 App.config.flurryid = "";
    if (window.device && window.device.platform) {
        if (window.device.platform == "Android") {
            App.config.flurryid = /*REP*/ 'notreallyanandroidflurryid' /*REP*/ ;
        }
        if (window.device.platform == "iOS") {
            App.config.flurryid = /*REP*/ 'notreallyaniosflurryidxxx' /*REP*/ ;
        }
    }

Although I have not used any of the more specific event tracking, the basic statistics are a great start (including new users, retained users, device versions, etc).

Don’t fly blind when you release your phonegap/cordova mobile app–use Flurry or something simliar.


Hackfest tips for companies with few developers

Last year, my company ran a hackfest.  This year, we are doing it again.  The company I work for, 8z Real Estate, is about 20% real software developers, though everyone at the company is familiar with software and technology.

How do we run a successful hackfest when only a few employees can build software?

  • Include everyone.  It will be a richer, more fun experience with more people.  Get executive buy in–I found the original ‘fed ex day’ post helpful in explaining the idea.
  • Set goals and expectations.  At a typical hackfest (or hackathon), running code is the goal.  For us, autonomy and exploration is more important.  In the announcement email we state: “the idea is to give everyone a chance to do something work related that they want to do, or try, or explore, but don’t have time to because of the hustle and bustle of work life.”
  • Reset deliverable expectations.  Rather than running code, the deliverable at a typical hackfest,  at an event with many non technical attendees other deliverables should be embraced.  Spreadsheets, powerpoints, text documents, mockups, link gallerys, images–these are all artifacts that capture an exploration.  They can also be referred to in the future.  (I don’t think a presentation without an artifact is a good idea, because of the lack of permanence, although I guess you could videotape it.)
  • Encourage developers to work on different teams.  Spreading the developer viewpoint and code writing ability across as many groups as possible is a good thing, as it will allow the groups to push their ideas further. That said, if a developer really wants to work on his or her own idea, don’t force them to join a team.
  • Make sure contractors feel welcome.  Because this isn’t a typical workday, it can be difficult to justify paying contractors to attend.  But a hackfest reinforces company culture and can make contractors feel part of the team.  We compromise by inviting contractors and paying them a mutually agreed upon reduced hourly rate.  If they are technical, this also adds to the pool of developers as well.
  • Have the hackfest onsite, preferably in one conference room. Especially for the first one, the hum of people working will be motivating and exciting.
  • Have the hackfest happen on one day.  Pick one that is slow–for real estate, that means closer to the year end holidays.
  • Plan for some ‘normal’ work to be done on the day of the hackfest.  We need to provide daily customer support, so on hackfest day we try to compress a full day’s work into a few hours, then shut off the phones.

Then, some general hackfest principles that I believe are true no matter what the attendee’s skillset.

  • Start with a timeboxed ideation whiteboard session.  This lets everyone see all the great ideas, and find what interests them.
  • Use the ideation session to head off any ‘typical work’ tasks that people suggest (‘I just have to verify 4 more bugs on the foobar widget’).
  • Let teams self organize, but encourage cross pollination between departments and teams.  A hackfest is a great way to build trust between people who don’t normally work together.  On the other hand, if someone is very passionate about an idea that no one else cares enough about team up on, let that person pursue their passion.
  • Handling managers at a hackfest is a sticky subject.  On the one hand, there is benefit to treating them as another employee–they get the benefits of working with different people and ideas.  On the other hand, because of their job, they may (unintentionally!) warp the autonomy of the team.  Last year, the CEO worked alone, but all the other managers were treated as employees.  Discuss this issue, especially with higher level executives, before the day of the hackfest.
  • Order in lunch, which keeps the momentum going.

Any tips for a good hackfest, especially one attended by fewer developers than non technical people?



Java address parsing gets an upgrade

I wrote last year about address parsing solutions in java, and how an open source project call JGeocoder had worked out well for us.  I wanted to announce that my company, 8z, has significantly improved the address parsing capability of JGeocoder based on data from a number of property listings.

My colleague, Karamjeet Khalsa, added this functionality as well as more than fifty unit tests.  An address that previously would have failed but now is parsed correctly is: 25266 Road 38.1 Dolores CO. Or try this one: 10 Black Bear Gypsum CO.  This code focuses on not just parsing address, city and state, but also breaking apart the address into components like street number, unit number, etc (for US style addresses).

Working with the current maintainer of the project, Karamjeet uploaded the new jar yesterday, and it is now ready to download.  This is the first release in four years, so if you need address parsing, go take a look!



© Moore Consulting, 2003-2017 +