Skip to content

Java address parsing gets an upgrade

I wrote last year about address parsing solutions in java, and how an open source project call JGeocoder had worked out well for us.  I wanted to announce that my company, 8z, has significantly improved the address parsing capability of JGeocoder based on data from a number of property listings.

My colleague, Karamjeet Khalsa, added this functionality as well as more than fifty unit tests.  An address that previously would have failed but now is parsed correctly is: 25266 Road 38.1 Dolores CO. Or try this one: 10 Black Bear Gypsum CO.  This code focuses on not just parsing address, city and state, but also breaking apart the address into components like street number, unit number, etc (for US style addresses).

Working with the current maintainer of the project, Karamjeet uploaded the new jar yesterday, and it is now ready to download.  This is the first release in four years, so if you need address parsing, go take a look!

A Tale of Two Mobile App Development Program Applications

I just can’t believe the difference in the application process between Google (for Google Play) and Apple (for the App Store).

For Google, I had to

  • sign in to a Google account
  • give them an email address and a phone number
  • agree to terms and conditions
  • pay $25 (but make sure you have a valid Google Wallet account!)
  • wait for the registration to be processed
  • invite other developers in my company

For Apple (where I need to register as a corporation) I had to

  • Upgrade a mac to Lion
  • Start the signup process
  • Realize that the company doesn’t have an updated Dun and Bradstreet profile
  • Call Dun and Bradstreet
  • Be sent to http://iupdate.dnb.com/ to update the profile
  • Find out who the executive officers of the company were
  • Get them to sign up to update the profile
  • Update the profile
  • Submit it
  • Wait for over 5 business days
  • Resubmit my application to Apple
  • Get rejected again because they didn’t recognize the company as a legal entity
  • Exchange a couple of emails with Apple employees
  • Determine what the rejection is possibly because I didn’t want for D&B to update the Apple database (up to 14 business days!)

And all that is before I can see (and agree to) the terms of use and pay the $99/year, and get my application processed.

Whew.

How do you decide to pull the plug on a project?

Over the past six months, I’ve been involved with two projects that, to put it politely, didn’t go exactly as planned.

Project A was cancelled, after a significant amount of preparatory work had been done, and project B was just released, after a much more significant amount of effort was expended.

What caused one to be cancelled and the other to be continued?

  • Timing: A had a tight schedule, with a deadline that made sense from the business’ perspective.  When it became clear that the deadline would be missed, cancellation was a logical option.  B had goals but no deadline, which made it harder to make a go/no go decision
  • Risk: A required some server rejiggering that would have made it extremely difficult to roll back.  B, on the other hand, would have been a simple software re-release to roll back to previous, somewhat working, software.
  • Business impact: A was a ‘nice to have’ project, whereas B was addressing something that had caused employees and users pain for years.
  • Sunk costs: A had fewer sunk costs than B, which paradoxically made it easier to cancel (paradoxically, because you should not consider sunk costs when looking at an investment).

What is the difference between pushing through to finish a difficult project and polishing a turd that you really should abandon?  It’s a fine balance, and as project B dragged on, I wasn’t sure at times which path we were headed down.  As someone who loves to ship, it’s hard for me to give up on a project that I had a hand in building, but no business writes software in a vacuum, and those business needs can and do serve as valuable checkpoints on the software process.  Because of the huge business value, it made sense to push through on project B.

Sometimes business priorities change, or, as in the case of project A, deadlines impose a different set of priorities (beyond the purely technical).

Regardless, I think that the more I consider any projects I have dragging on through the lenses of timing, risk, return and sunk costs, the easier it will be to make a go/no go decision.

List of online real estate solutions

I recently was involved with a extensive survey of online solutions for my employer, a real estate brokerage.  Some were open source, some commercial, some inexpensive and some expensive.  Basically, any full solution that might have worked was added to the list.  Some of these are aimed only at real estate agents (as opposed to a brokerage), but others definitely scale up to brokerages.  (I say full solutions, so you won’t see anything on the list like Onboard Informatics or Altos Research.)

I just wanted to list all of them here for posterity and to help anyone else looking for a real estate solution.

More details on moving away from Google Maps

Well, our move from Google Maps to MapQuest is pretty much done (the start was mentioned here), so I thought I’d give a more exhaustive overview, including some more insight into our thinking.  For context, we, in this post, refers to 8z Real Estate, a real estate brokerage with a great website that drives a huge chunk of business.  (Of course, errors and opinions expressed here are all mine.)

First, if you are considering switching from Google Maps, definitely think about not leaving!  Depending on your customizations, it can be a real pain.  You can also think about it in terms of spending dollars with Google or spending dollars with your developers.

A good place to start is to make sure you know how many map views you are using.  I found three good ways to do this.

  • estimate using your web analytics tools (page A uses 1 map, and is viewed X times a day, page B uses two maps and a streetview and is viewed Y times a day).  This has the benefit of being free and in your control, and not requiring you to make any code changes.  It has the detriment of being imprecise, especially if you are displaying maps via javascript that might not be tracked by your analytics tool.  I used six months of our highest traffic days to get a feel for maximum map views per day.
  • ask a Google sales rep.  This has the benefit of being very precise–they can tell you how many map views they recorded for you on a given day, and, again, doesn’t require any code changes.  It has a number of downsides, though.  First, you have to get ahold of Google sales (using the ‘contact sales’ button here).  I had to submit this form at least twice before I got a call from them.  Second, you have to interact with a live person.  And third, they only have information for a single day.
  • install the new Google Maps API key and look at the API console.  The upside to this is that reviewing your usage doesn’t have any external dependencies and the console  is very precise in tracking your usage.  The downside is that you are now on Google’s radar, and it may require a bit of work (depending on when you added Google Maps to your site).

We actually ended up using all three of these methods at different times, but found the API console to be the best option long term because it is the most accurate.

Once you’ve found your map view usage, check out the usage limits, which will inform your decision.  Two other considerations before you decide to move off the free Google Maps.

  • If you have used the free Google geocoding service, you should re-geocode all your points, as it is outside the terms of service to display these points anywhere other than on a Google map.
  • It is unclear to me when they will start the clock on the new usage limits (as announced here).  I read the new terms carefully, and you have a grace period of three months (you have to be above the free limit for three months straight) before charges start.  Note however, that once you are over for the grace period, you never can get back into it (100 days of over 25K usage followed by 3 years of under it followed by 1 day over it will still result in charges for that 1 day overage).  Additionally, Google will give a months notice of the usage limits enforcement on the Google Geo Developer blog.  That means that you’ll have at least four months of lead time.  And note that they’ve moved the enforcement deadline at least once.

Ok, you’ve decided that it simply isn’t worth the risk, and that you need to take action to avoid being charged for Google Maps lickety split.  If you haven’t, godspeed.

The options for avoiding being charged are outlined very clearly:

  • Decrease your usage
  • Pay for a license
  • Pay for daily overages

Armed with your daily usage information (don’t forget to factor in traffic increases), you can see which of these options makes the most sense.  Contact Google enterprise sales if you feel a license purchase makes sense.  Be aware that the 25K daily limit works out to over 9M map views a year.  So if you are buying a license just to deal with daily traffic (as opposed to wanting the application to live on an intranet or some other non publicly available location, or a seasonal traffic spike), you’ll want to price out at least a 10M view license.

Once I got to talking to the Google representative, it became clear that the map license was pretty expensive.  The price range for a 30M license ranged from the mid 5 figures to the low 6 figures, per year.  We also looked at daily overages as an option.  Unfortunately this turned out to be hundreds of dollars a day.  Neither of these solutions made sense economically, so we decided to spend developer hours instead of dollars to decrease our Google Maps usage.

There are two ways to decrease Google Map usage, of course.  You can decrease use of maps all around (for example, by not having maps show up unless a user takes some action or removing extra maps), or you can migrate some or all of your current Google Maps to a different provider.

We tried both paths forward.  We removed some extra maps from the site (again, determined by our usage data), but that didn’t reduce usage enough, and also impacted users (as determined by email complaints!).

Therefore, we determined it was better to show the javascript maps every time.  We also had a technical wrinkle.  We use GWT  and are locked into Google Maps v2 (due to dependence on this library).  Google Maps v2 will no longer be supported in 2013.  I don’t believe there is a supported GWT binding for Google Maps v3, although that may get written later (here are two open source, non Google supported bindings that I found: http://code.google.com/p/gwt-google-maps-v3/ and http://code.google.com/p/gwt-maps3/ that may solve the problem for others).  Since v2 was deprecated on May 19, 2010, we knew we’d have to revisit Google Maps in the next year (by May 19, 2013, at worst), so that meant a hybrid solution (part Google Maps, part some other provider) was less appealing than otherwise.

We looked at a few different contenders to replace Google Maps.  Our criteria included:

  • quick to set up
  • accurate display
  • free for our usage leverl
  • close in features to Google Maps, at least the features we used
  • looked good

In the end, I prototyped three solutions that met the first four criteria above, and asked the team for feedback on the fourth.

The solutions included:

Some options that might be worth looking at for some, but didn’t work for us were:

  • Bing Maps, which we ignored due to license limits
  • Google Maps v3.  This version has significantly cheaper overages, so if you are right around the 25K usage, that might be worthwhile, but we were way over 25K mapviews, and also, as mentioned above, had GWT ties to v2.
  • Rolling our own maps (ht briantimoney).  I think this wasn’t the right solution for us two reasons.  One, we were looking for something that was a relatively easy ‘drop in’ replacement for Google Maps, and crunching a bunch of map tiles didn’t seem to fit that requirement.  Two, I’m not sure there’s a lot of value in unique maps for our real estate web application (though there is in having points of interest and accurate mapping of properties).

We ended up choosing the Mapquest Javascript API for the best look and feel.

The next step was to update the pages where maps were used.  There were two main types of pages that used maps.  One type of page showed a single property on a map.  The other showed multiple markers on a map, as results of a search.

The single property page was trivial to replace.  There were a couple of interesting wrinkles.  The first was replacing the default marker icon.  I had to tweak the iconoffset and shadowoffset properties of the icon object for correct display.  Additionally, the new map had to be displayable from GWT.  This required a JSNI method run in a deferred command, in order to make sure that the MapQuest javascript has loaded.

Based on map usage data, I was hoping we could get by just replacing the single property maps, but this didn’t decrease our map usage enough.  On to search results! (If you aren’t interested in GWT, you can skip to the bottom of this post, as the next section is all about GWT gyrations.)

Replacing the search results was slightly more complex.  I reviewed all the current code and categorize our existing GWT/Google Maps based functionality into three groups.

  • functionality that the MapQuest API could exactly replicate (putting a marker on a map)
  • functionality that the MapQuest API had an equivalent for (displaying information on click of a marker)
  • functionality that we had previously built but, in the name of simplification, weren’t going to port to the MapQuest based solution

After this categorization was done, it was fairly simple to port the needed functionality from GWT to javascript (jQuery/MapQuest).

One additional twist is formatting code shared between GWT and java (displaytag decorators, among other code).  The MapQuest javascript needed to be able to call that formatting code, as well as other GWT functionality (for example, creating a lightbox).  No sweat, right?  Here’s an example of calling GWT code from javascript.  The javascript that was calling the GWT code was running in a windows.load jQuery anonymous function.

Well, that worked fine in FF, and often in Chrome, but I was getting an error message from Chrome occasionally and IE all the time.  The Chrome error was pretty clear: ReferenceError: gwtListingUtilgetMLSIDLogoandOfficeName is not defined (gwtListingUtilgetMLSIDLogoandOfficeName is the name of one of the GWT methods that was exported for javascript to use).  The issue was that the map javascript was executing before the GWT code was fully loaded, and not finding methods it was expecting.

The solution was to chain the javascript, so that the map javascript was run via a GWT JSNI call after GWT was fully loaded, and all needed GWT functions were exported.  To provide for other GWT on load chaining, I used the onload pattern outlined here and created an addGWTOnLoadEvent.

Then I needed to create an anonymous function that GWT could call after it was loaded .  So, to recap the path of the map creation code (because it is kind of confusing to me):

  • on page load, addGWTOnLoadEvent(startupSearchMap()); where startupSearchMap returns an anonymous function
  • in GWT code, we export needed GWT functions to javascript: $wnd.gwtListingUtilgetMLSIDLogoandOfficeName = function(mlsId, source Id, baseImageURL, officeName, listingStatus, withBreaks) { return @com.cohomefinder.util.sharedwithgwt.ListingUtil::getMLSIDLogoandOfficeName(Ljava/lang/String;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;Z)(mlsId, sourceId, baseImageURL, officeName,listingStatus,withBreaks);};
  • at the end of the export JSNI function, we call gwtonload: if ($wnd.gwtonload) { $wnd.gwtonload(); }
  • which executes the anonymous function returned by startupSearchMap which then binds to the DOM element and creates the map.  Needed GWT functions are guaranteed to be present.

This would all be a lot simpler if we didn’t have valuable code written in GWT, but that is a sunk cost that I don’t ever see rewriting.

After that, it was all just testing and making a few UI tweaks.  However, the depth and maturity of the MapQuest API and community scared me a bit, with forums that weren’t as deep as the Google Maps groups I had been accustomed too, and with posts like “IE9 not supported” by the latest MapQuest API.  The API documentation was also underwhelming, with plenty of methods undocumented.  But it all worked out.  If I was doing some aggressive, bounds pushing map functionality, I don’t know if these concerns would have been more of a hindrance.

So, to recap, if you are thinking about moving off of Google Maps:

  • find out your map views per day
  • consider usage solutions that allow you to remain with Google Maps
  • if you must move off of Google Maps, consider all other solution providers and find the one that fit
  • make sure you replicate your current functionality with the map provider you choose

In general, I would rather not have moved off Google Maps (it felt like running just to keep in place, and Google Maps is a great product), but the economics dictated this choice.  I hope this post helps you decide if moving off Google Maps is right for you.

8z Real Estate Hackfest

Recently, at the company I work for, 8z Real Estate, we had a hackfest.  A hackfest, for those not in the know, is a chance for employees to spend time working on whatever they want to do but don’t have time to do during the normal business day.  It’s also known as ‘FedEx day’ because you build something to ship in one day.

The idea is to give everyone a chance to do something work related that they want to do, or try, or explore, but don’t have time to because of the hustle and bustle of work life.

From the post at Atlassian (as far as I know, the originators of FedEx day).

[The] task must be something “out of the ordinary”. This is hard to define – but basically the spirit is that you can’t do something you would normally do. It’s a chance to attack all those “I wonder if XYZ would work… “, “It would be nice if we could… ” small … tasks that always get pushed off in the heat of battle.”

We had about 11 employees and contractors gather at the office.  Our schedule:

  • 8:00 Employees arrives, normal work begins
  • 10:00 Employees cease normal work
  • 10:00 – 10:30 brainstorm session
  • 10:30 – 10:45 pick projects
  • 12:00-1:00 eat lunch (Snarfs!)
  • 10:45-4:15 continue work on project
  • 4:30-5:30 present projects (8 min presentation) and drink a beer

We had an 2 hour block of work at the beginning of the day because we needed to, but after that, almost no one did everyday work.  Phones were off, email was closed.

The type of projects selected varied.  Most folks weren’t developers, so we didn’t have a ton of shipped software.  But we had some really interesting ideas, ranging from investigation of interesting technologies (what’s coming down the pike with our e-newsletter sender, infographics) to outlines of business ideas, to refactoring of business processes.

The excitement of all working together, in one room, on different projects, for a fixed amount of time, with no interruptions was one highlight.  I also really enjoyed people’s varied takes on aspects of the business.  It was also impressive to see the skills that I didn’t know some people had (powerpoint, for one).  It was awesome how many good ideas we had, even though some of them would have taken a hackfest week to implement.

All in all, it was a worthy experiment and something every business should consider doing.

On becoming an employee

I made the choice a few months ago to become an full time employee.  After a number of years of contracting, this was not an easy choice.  However, the company for which I work, 8z Real Estate, was not a black box to me.  I had contracted for them off and on since 2005.  Still, this was a big step for me, and I just wanted to blog about some of my thoughts around it.

Key reasons I became an employee:

  • An opportunity to really focus on the technical side.  No more being worried about the next gig.  Becoming an employee makes it easier for an employer to offer more freedom (for example, to explore a different skill set) to an employee as well.
  • The chance to be part of a kickass team with good teammates and a great manager.
  • It is fun to be part of a growing company.  We have Beer Fridays and other social events.
  • Less stress about work in general, although, as Bob Lewis said, the only difference between an employee and a contractor is that the employee has the illusion of job security.
  • 8z is doing some pretty cool stuff with some pretty interesting data, and offers a valuable service to many many people.  I can’t tell you how many folks have told me they love COhomefinder, the flagship webapp that I work on.

What I miss about being a contractor:

  • Business development–it was fun and challenging to think about how I could help everyone I met, or how they could help me
  • Extreme flexibility of hours, including when and how much I worked.
  • Greenfield development is always nice.  I am constantly learning and so sometimes cringe when I see code I wrote years ago.
  • Being able to say ‘no’/fire a client.

What I don’t miss about being a contractor:

  • Waiting for invoices to roll in and/or reminding clients about outstanding invoices.
  • Thinking about the next gig before the current one is done.
  • Regular overcommitment–making hay while the sun shines is what I had to do, but I did over commit at times.  Work life balance is easier to achieve.
  • Paperwork.

I’ll be blogging about my work at 8z from time to time going forward, so I added an 8z category to this blog.