JFreeChart for the win


I wanted to make some charts showing linear regressions, but they needed to be images (so libraries like d3.js were out).  My first inclination was to use Google’s Image Charts API, but it has been deprecated.  I looked around and couldn’t find anything like it.

So, it looked like I’d need to code something up.  Some searching around turned up some great open source projects.  Using JFreeChart and Apache Commons Math, and some blog posts (this one about regression, this one about drawing dashed lines) and a post on StackOverflow about scatter plot basics, I was able to put together the above regression line chart in about half a day.

Hackfest tips for companies with few developers

Last year, my company ran a hackfest.  This year, we are doing it again.  The company I work for, 8z Real Estate, is about 20% real software developers, though everyone at the company is familiar with software and technology.

How do we run a successful hackfest when only a few employees can build software?

  • Include everyone.  It will be a richer, more fun experience with more people.  Get executive buy in–I found the original ‘fed ex day’ post helpful in explaining the idea.
  • Set goals and expectations.  At a typical hackfest (or hackathon), running code is the goal.  For us, autonomy and exploration is more important.  In the announcement email we state: “the idea is to give everyone a chance to do something work related that they want to do, or try, or explore, but don’t have time to because of the hustle and bustle of work life.”
  • Reset deliverable expectations.  Rather than running code, the deliverable at a typical hackfest,  at an event with many non technical attendees other deliverables should be embraced.  Spreadsheets, powerpoints, text documents, mockups, link gallerys, images–these are all artifacts that capture an exploration.  They can also be referred to in the future.  (I don’t think a presentation without an artifact is a good idea, because of the lack of permanence, although I guess you could videotape it.)
  • Encourage developers to work on different teams.  Spreading the developer viewpoint and code writing ability across as many groups as possible is a good thing, as it will allow the groups to push their ideas further. That said, if a developer really wants to work on his or her own idea, don’t force them to join a team.
  • Make sure contractors feel welcome.  Because this isn’t a typical workday, it can be difficult to justify paying contractors to attend.  But a hackfest reinforces company culture and can make contractors feel part of the team.  We compromise by inviting contractors and paying them a mutually agreed upon reduced hourly rate.  If they are technical, this also adds to the pool of developers as well.
  • Have the hackfest onsite, preferably in one conference room. Especially for the first one, the hum of people working will be motivating and exciting.
  • Have the hackfest happen on one day.  Pick one that is slow–for real estate, that means closer to the year end holidays.
  • Plan for some ‘normal’ work to be done on the day of the hackfest.  We need to provide daily customer support, so on hackfest day we try to compress a full day’s work into a few hours, then shut off the phones.

Then, some general hackfest principles that I believe are true no matter what the attendee’s skillset.

  • Start with a timeboxed ideation whiteboard session.  This lets everyone see all the great ideas, and find what interests them.
  • Use the ideation session to head off any ‘typical work’ tasks that people suggest (‘I just have to verify 4 more bugs on the foobar widget’).
  • Let teams self organize, but encourage cross pollination between departments and teams.  A hackfest is a great way to build trust between people who don’t normally work together.  On the other hand, if someone is very passionate about an idea that no one else cares enough about team up on, let that person pursue their passion.
  • Handling managers at a hackfest is a sticky subject.  On the one hand, there is benefit to treating them as another employee–they get the benefits of working with different people and ideas.  On the other hand, because of their job, they may (unintentionally!) warp the autonomy of the team.  Last year, the CEO worked alone, but all the other managers were treated as employees.  Discuss this issue, especially with higher level executives, before the day of the hackfest.
  • Order in lunch, which keeps the momentum going.

Any tips for a good hackfest, especially one attended by fewer developers than non technical people?

Run through the finish

Make that last code change.

Write that last test.

Look in that document, rather than trying to remember what color the icon is supposed to be.

Write that documentation.

Look at your work with the eyes of a user.

I work in a small development department (2 developers plus a number of contractors) and I need to constantly remind myself to run through the finish.

I ran in high school and college (cross country and track) and at the very end, it is easy to let your guard down and coast.  After all, you’ve done almost all of the hard work.  And no one is really behind you.  And it hurts (oh yes, it hurts).  So, why not ease off a bit?

The problem is, there is someone behind you, and they have you in their cross hairs.  They have the incentive and the vision of their competition, but you have the lead.  Why give up any advantage?

Development isn’t painful, but it can be a slog.  Yes, yes, I’m sure there are shops that never have a slog.  But, for most mortals, there are requirements that are changed or were forgotten, tasks that are less fun than others, tweaks to the UI for the Nth time, vendor rigidity that never surfaced during the sales process, and other sundry annoyances.  But you as the developer own the final product.  You can choose to coast, since you did 99% of the work (and did it well), or you can choose to run through the finish.

Run through the finish.

Small scale data migrations

So, I’ve recently been involved in another data migration, the second one in three years.  These are small migrations, with thousands records.  One person to take care of this size of data migration with some effort, but the amount of data is still large enough that manual data re-entry isn’t really an option–the error rate and the cost and the management difficulty mean that software is the better option.

Here are some lessons I learned from these data migrations.

Learn as much as you can about the data models–both the old and the new–as you can.  This includes, in preferred order, talkingto any people familiar with the old system, talking to any people familiar with the new system, looking at the databases via a sql client, reading documentation (if any is written), and looking at code.  I spent some time thrashing around in old system code for a while.  Then I asked the developer for a tour, and learned more in that hour than I had in the previous day of looking at code.

Map entities and concepts as early as you can.  Take special note of any that are in the old and not in the new (and what you are planning to do with them).  Those that are in the new and not in the old aren’t as big of an issue.  Also, attributes of entities are as important as entities, so note discrepancies there.  Early on I noticed that one of the two primary entities in the old system did not exist in the new system.  This led to some interesting conversations with the business users that saved me work.

As above, talk to people who are going to be using the new system, and who use the old system, throughout the migration process.  An entity or attribute that will be a royal pain to migrate may not be used anymore!  Or, the business person might have some good ideas on how to map something in the old system into the new system.  Someone who uses the software you are migrating has more domain expertise than you.  Let them try the new system with migrated data as soon as some data is moved. Make sure to guide their experience so they don’t spin their wheels looking in corners of the system that not yet migrated.

Start a spreadsheet of tasks, so that everytime you uncover something that needs to be done while you are in the process of doing something else, you can note it and keep on your original task.  My spreadsheets are simple; three columns are enough: task name, completed (with an X for completion, blank for still open) and notes (for possible implementation solutions, people to talk to, relevant urls, or any other text that will help me complete the task).

Document all the migration steps, preferably to the point you can cut and paste commands.  Include any discrepancies discovered, special commands to run, access to all needed systems, names of relevant people, areas that need further investigation, and basically anything else you would want handed to you if you were starting on this project.  This helps immensely if you need to pass off the project, or come back to it later (even just a few days), and provides documentation of entities on the old and new system.

Write scripts wherever possible, but don’t try to script the whole process–access to different servers can be hard to automate.  Use whatever language you feel is best for these scripts.  I’ve used bash, sql, perl, and awk/sed, but I don’t shy away from a compiled language like java, especially if a library exists that can save me time.  Make sure to put these scripts into version control, and document the purpose with comments at the top and a good name.  I wouldn’t worry too much about unit testing or refactoring this software, because chances are it will be seldom used once the migration occurs.

Get familiar with the concatenate function of your database.  Using queries to write DDL for the new system based on data from the old system can save you writing a script in an imperative language.  When migrating from Expression Engine to WordPress, I used a statement like <code>select concat(‘update wp_comments set comment_author_email = ”’,email,”’ where comment_author = ”’,name,”’;’) from exp_comments where name in (select distinct(name) from exp_comments);</code> to generate an update statement for WordPress for each comment author in the EE database.

Think about data types and representations.  Especially if you are moving from one database to another.  When I was moving from MSSQL to MySQL, date fields were particularly thorny.

Realize that these types of projects are typically difficult slogs.  There were moments where I despaired of ever getting through the migration in a timely fashion.  To do it right, you need a fantastic attention to detail, an understanding of the business needs, and an ability to drive things through to the finish.  All of this can be pretty draining–I find it far more draining than bug fixing or building new features.

Control the old and new systems–try to not have new capabilities added during the migration.  If you can’t guarantee that, can the migration wait until the new and old systems stabilize?  If not, checkpoint the migration against the new capabilities during the process, and realize that you are introducing a lot of extra work and complexity into an already complex process.

Have a staging system where you can practice your migrations without affecting anyone.  Plan to go through at least two or three of these new staging systems so that you can get the migration steps solid before you touch production.  Start from a clean slate each time so no time is spent chasing phantom bugs from a previous migration that didn’t finish or wasn’t entirely correct.  This is what makes the migration documentation you write so important.  Be aware that the new stage system and the new production system will not necessarily be the same.

Lastly, avoid committing to a schedule if at all possible.  And if you must, pad it and only commit after you’ve done a thorough analysis.  Because there are so many hidey holes and areas of the old system that you won’t understand, there is a high probability that you’ll be discovering new issues and data you need to migrate halfway through the project.  (This is a special case of the requirements nightmare known as ‘build system B that acts exactly like system A’.)  Communicate progress to the business.

While this is not my favorite type of project, when done well it can have tremendous business value.  Combining newer, more flexible systems with rich older data, without re keying the data, can make system users much happier.  In some cases, if there is no migration, the newer system simply can’t be used.

Why isn’t everyone a developer?

I was at a party recently kvetching with some fellow developers and managers of developers about how hard it is to find good talent here in Colorado.  One fellow said he feared a return of the last days of the 1990s, when the hiring bar, and thus the quality of code produced, was drastically lowered.

On the way home, my SO asked why, in a time where many are having difficulty finding jobs, why everyone wasn’t a software developer.  At first, I thought, why not ask “why isn’t everyone a doctor; we have a shortage of those”, but then I realized that it is not applicable.  Especially if you are willing to be a junior developer, the low educational and licensure hurdles to becoming a developer, the remuneration, and the flexibility and intellectual challenges combine to make it a very attractive job.  (Given I am a developer, I may be biased.)

I think there are five attributes you need to have, that progressively fewer people meet.  (I’m purposely ignoring experience programming, because everyone was junior at one time.)

  1. Willingness to work in an office–it’s not everyone who wants to drive (or otherwise commute to an office), sit on their butt every day, deal with office politics, and miss out on natural light.  Even work in a home office or a coffee shop, while alleviating a some of the above issues, is a lot of sitting.
  2. Computer literacy–a developer has to have a basic level of comfort with a computer.  This is a filter that will probably become less relevant for middle class Americans as younger people who grew up with a computer reach the workforce, but for a substantial number of workers, computer interaction is tedious, confusing and irritating.
  3. A logical mind.  Computers aren’t very squishy (fuzzy logic aside) and while logic can be learned by anyone, it takes work to think of the world in the kind of systems that can be modeled in software.
  4. Initiative.  You’ll notice that throughout this, I talk about a developer, not a programmer.  That’s because a programmer, someone who takes a set of specifications and codes them, is a job that is rapidly disappearing.  Developers, who have a broader skill set, including, if need be, the entire stack from business analysis to coding to testing to deployment to support, need to have initiative.  Initiative is, incidentally, a precious attribute for any position.
  5. Lastly, I come to the actual skills of the job–coding, business analysis, testing, etc.  This is the last hurdle, and one of the smaller ones.

So, there are many good reasons why everyone isn’t a software developer, even though there’s plenty of software to be written.

What is the difference between a developer and a programmer?

In my mind, developers have skills that are a superset of the skills of a programmer.


  • understand how software works
  • think about the best way to solve problems they are presented with
  • write good code
  • learn new techniques, languages and manners of creating software
  • write documentation
  • write tests


  • do all of the above
  • work with the business to define requirements
  • test beyond basic unit tests, up to and including user acceptance testing
  • understand basic system administration
  • do basic database tuning
  • deploy software and solutions
  • can do some UX design
  • can manipulate graphics
  • support and maintain what they’ve written

In other words, developers can span the gamut of needs–they are jack of all trades.  Not all positions are fit for developers because they can be a mile wide and an inch deep, but in general having someone who can handle the entire software lifecycle is more useful in more situations than having someone who is handed a spec and can crank out code.

(Note that I’ve never worked in a huge company for longer than a few months, so perhaps that is where programmers are more useful.)

On Being Disrupted

This is a tough post to write.  I’m at the tail end of an evaluation process for my company that ended up with us deciding to go with a third party vendor for software which powers key area of our business.  It is augmentation rather than replacement, but still.

It was hard for me, because this particular key functionality was previously provided by custom software that I helped build over years.

I like to build things!  Like most other software developers, I get excited about building stuff.  One of the unmentioned frustrations of many software developers is building something and then seeing it shelved.

However, it was clear from the survey of solutions that there were three choices:

  1. buy something off the shelf
  2. get better as software developers and really, really accelerate our development
  3. have the business be negatively affected by this piece of software

Now, #3 is obviously not a good solution.  #2 is a great solution, but is hard to put into practice, especially in a short time frame with a large code base (though we are trying to use some of the agile methodologies to make our software development more productive).

#1 it is.

Was this a wise choice?  Talk to me at the end of the implementation, but I am hopeful.  We did take several steps to protect ourself (after all outsourcing core business functionality can be deadly), including:

  • a long, laborious evaluation
  • engagement with multiple vendors, and
  • building a set of criteria to help us determine if this outsourcing is meeting our needs

One thing that helped me take this decision a little less personally is to redefine in my mind the value of software to the company.  It’s not the particular implementation of the software that provides the value.

Unlike a software company, my company doesn’t exist to write software (see Five Worlds for more on different types of sofware development).  Instead, software exists to serve the company.  Having something off the shelf provides the similar functionality for much cheaper.  It also allows me and other members of the software team to write software that is unique.

Having been a contractor and having worked for a startup and consultancy, I’m used to being the disruptor.  In this scenario, I was the disrupted (ht, David Skinner).  It’s a humbling place to be, even if I wasn’t disrupted out of a job.

Using Google Apps to generate a simple site from a form

I was blue skying with a friend the other day and came up with checklist to build a simple canned website from a form submission, using Google Apps and Google Apps Scripting.

  1. User goes to a form.  (More on creating a form)
  2. User submits images, video links, text, email, background color, etc.
  3. That all goes into a google spreadsheet, which triggers the building of a site.  Here’s trigger info.
  4. You can automatically build a google site.  Here’s the api: site creation, page creation.  If you want your own domain name (example.com) I think you can do that, but you’d need to integrate with a DNS service to set that up–this part bears a little more investigating.
  5. If desired, you can generate the QR code for the site.
  6. If you need to tell the form submitter that the site is created, you can email them a link to the QR code so they can print it out.
  7. You can even remove content from the site via cron (see triggers link up above), though you can’t automatically delete the site.
  8. It’s conceivable you could even use google checkout to charge users beforehand (though this is a bit less supported).

What’s great is that you don’t need to have any tool other than a browser to build this entire stack.  Google is truly creating a toolkit/macro language in the cloud.

How to get the GWT developer browser plugin

The Google Web Toolkit developer plugin, which integrates with a variety of browsers and lets you debug GWT code from within the Eclipse IDE, is astonishingly easy to get, but I haven’t found any great instructions, so I wrote this brief post.

You just need to start your IDE, start up any application in Development mode, and then visit the address the development mode server is running on in your browser (it will look something like this: “http://localhost:8080/host.html&gwt.codesvr=” if you are running off an external server–not sure what it looks like for folks using embedded jetty). It will send you to the ‘Missing Plugin’ page (or you can go there directly).

Some browsers are not supported (Safari on Windows is one I know of). The best list of supported browsers I was able to find was on the ‘Missing Plugin’ page; click the ‘plugins for other systems’ link.

BTW, this is different than the GWT plugin for Eclipse.

Useful Tools: StatsMix makes it easy to build a dashboard

I haven’t been to a BDNT lately, but still get their email announcements.  In August, all the 2010 TechStars folks presented, and were listed in the email.  I took a look at each company, and signed up when the company seemed to be doing something cool.  I always want to capture my preferred username, mooreds!

One that was very interesting to me was StatsMix; I signed up for their beta.  On Nov 1, I got invited to sign up.  Wahoo!

Statsmix lets users build custom dashboards.  I am developing an interest in web analytics (aside: if you are interested in this topic, I highly recommended Web Analytics 2.0, by Avinahsh Kaushik).  I’ve been playing with Piwik, an open source analytics toolkit, but Statsmix offers a slicker solution.

They have made it dead simple to create a custom dashboard for users.  They offer integration with, at this time, 29 services (twitter, mailchimp, youtube, Google Analytics, etc).  I could not find an up to date list of integration services outside of their webapplication!  The best I could find was this list from September.  While the integration interface is slick, the data integration is rudimentary.  For example, they will let you monitor the number of rows in a Google Spreadsheet, but nothing more (like rows in different columns, or the value in a particular cell–would be nice to see them integrate with Google Apps Scripting); you can track the number of likes on Facebook, but not the number of comments.

The real power of StatsMix comes from the ease of integration with your own custom stats.  They offer an API which is accessible via REST.  This means that you can push information from your database to a beautiful looking dashboard with shell scripts and a cron job.  Very cool!  It would be nice to see a plugin for Magento or other ecommerce vendors; I recently had a client, The Game Frame, that would have been a great fit for this type of dashboard, since it aggregates beyond what the ecommerce software provides.

Other cool features:

  • The whole UI is beautiful and farily intuitive.
  • The dashboard supports custom date ranges.
  • They will send you an email of stats every day, and apparently have some kind of limited version you can pass onto clients.  I didn’t play with the email feature at all, though it is extremely useful.

However, all is not perfect.  Some issues with StatsMix include:

  • As mentioned above, the integration with third party services leaves something to be desired.  What they offer is a nice start, but it’d be great to see them create some kind of marketplace where developers could build solutions.  For example, the twitter widget only tracks the number of followers.  From the TWitter API, it appears to be pretty easy to track the number of mentions, which could be a useful metric.
  • It wasn’t clear how to share a dashboard, though that may be an upcoming feature.
  • The terms of use are, as always, pretty punishing.
  • Once you develop a number of custom metrics, you are tied to their platform.  That wouldn’t be so bad, except…
  • They are planning to charge for the service, but give no insight into what to expect.  There is a tab called ‘Billing’ but all it says is: “During our beta, StatsMix is free to use. After the beta, you’ll be able to manage your billing preferences on this page.”  If I was considering using this as part of my business, I would want much more insight into possible costs before I committed much time to custom metric buildouts.  I’m fine with them making money, just want more insight into this key aspect of their web app.

All in all, it is well worth a try.  If you to, let me know by posting a comment.  I have 5 invites to give out.

© Moore Consulting, 2003-2017 +