#TBT: Precision and Accuracy in Software

I originally wrote this in Dec of 2004. I still think that having someone who can answer engineers’ questions authoritatively increases productivity (of the engineer). However, now I’d emphasize that developers need to spend some time learning their domain to gain some intuition, and truly great business software engineers will learn when to bump a question up to a business person and when their intuition can be trusted.

————————————————————-

Back in college, when I took first year physics lab, there was a section of the course that focused on teaching the difference between precision and accuracy in measurement. This distinction was crucial in experimental physics, since measurement is the bedrock of such experimentation. Basically, precision is how many digits of a measurement actually mean something. If I’m measuring the length of a room with my stride (and found it to be 30 feet long), the precision is less than if I were to measure the length of the room with a tape measure (and found it to be 33 feet, 6 and ¾ inches long). However, it’s possible that the stride measurement is more accurate than the length found with the tape measure, that is, it reflects how long the room actually is. (Perhaps there’s clothing on the floor which adds tape measurement, but which I stride over.)

These concepts aren’t just valid in physics; I think they’re also useful in software. When building a piece of software, I am precise if I build what I say I am going to build, and I am accurate if what I build actually meets the client’s business needs, that is, it solves the business problem. Almost every development tool either makes development more precise or more accurate.

The concept of precision lends itself easily to automation. For example, unit testing is rapidly gaining credence as a useful software technique. With unit testing, a developer writes test cases for each part of their code (often at the method level). The running of these tests ensures that code is actually doing what the developer thinks it is doing. I like writing unit tests; it gives me comfort to know that corner cases are taken care of and that changes to code can be fairly easily regression tested. Other techniques besides unit testing that help ensure precision include:

Round tripping: using a tool like TogetherJ, I can ensure that the model (often described in UML) and the code are in sync. This makes it easier for me to verify my mental model against the code.

Specification writing: The more precise a spec is, the easier it is to translate into code.

Compilers: the checking that occurs at compilation time can be very helpful in ensuring that the code is doing what I think it is doing–at a very low level. Obviously, this technique depends on the language used.

Now, precision is needed, because if I am not confident that I understand what the code is doing, then I’m in real trouble. However, accuracy is much more important. Having a customer onsite is a great example of a technique to ensure accuracy: you have a business domain expert available all the time for developers’ questions. In this situation, when a developer stumbles across a part of the business problem that they don’t quite understand, the don’t do what developers normally do (in order of decreasing accuracy):

  1. Ask another developer, which works great if the target audience is developers, but not so well otherwise.
  2. 2Best approximation (read: guess at the correct answer).
  3. Ignore the issue. (‘I’ve got a lot more code to write before I can go home today, and we’re shipping in two weeks. We’ll just let the customer discover it and deal with it as a bug.’)

Instead, they have a real live business person, to whom this software really matters (hopefully), who they can ask. Doing this makes it much more likely that the final solution will actually solve the business problem. Other techniques to help improve accuracy include:

Issue tracking software (I use Bugzilla): Having a place where questions and conversations are recorded is truly helpful in making sure the mental model of the business user and the programmer are in sync. Using a web based tool means that non-technical users can participate and contribute.

Specification writing: A well written spec allows both the business user and developer to have a sense of what is being built, which means that the business user can correct invalid notions at an early stage. However, if a spec is too detailed, it can be used to justify precision at the cost of accuracy (‘hey, the code does exactly what’s specified’ is the excuse you’ll hear).

Spring and other dependency injection tools, as well as IDEs: These tools help accuracy by decreasing the costs of changing code.

Precision and accuracy are both important in software engineering. Perhaps the best way to characterize the two concepts is that precision is the mapping of the programmer’s model of the problem to the computer’s model, whereas accuracy is the mapping of the business’ needs to the programmer’s model. However, though both are needed, accuracy is much harder to obtain. Knowing that I’m building precisely what I think I’m building is beneficial only insofar as what I think I’m building is actually what the customer needs.


Get Developers Going with Vagrant

computer photo

Photo by jurvetson

I just ran across the Data Science at the Command Line, a new book, website and VM, which gives you many, many unix tools to manipulate large data sets.

I have played around with the tools, but what really stood out to me was how easy it was to get them installed. While the author does provide install instructions, the VM is ready to go in only five steps (two if you already have vagrant and virtual box installed).

I have used Vagrant in the past and it makes managing VMs very easy. If you are distributing a developer tool, make it easy to get up and running by letting the user download a virtual environment that is correctly configured. Then trying out your software becomes a 30 second test, not a 10 minute test.

Of course, the author of Data Science at the Command Line is not alone.  I noticed Ionic is providing a VM for android development too.  It looks like Vagrantcloud lets you share virtual instances with the world as well.

Frankly, I used to download and install cygwin first thing on every Windows computer I bought. But VMs are so easy now it’s easier to just install VirtualBox, Vagrant and a version of Linux (CentOS or Ubuntu).


Gardening and software development

It’s the end of spring/early summer in the northern hemisphere, so it’s time to plan the vegetable garden again. I was putting some tomatoes in the other day and musing about the similarities between gardening and software development. To wit:

  • I have a lot of hesitancy about planting–especially perennials.  It feels so permanent, and I might screw things up, and maybe I should go back to the drawing board, or maybe just do it next weekend….  But just starting makes the problem so much easier–it loses its weight.  Your garden will never be perfect, but an imperfect garden is 100% better than no garden.  Similarly, when confronted with a new project or feature, half the battle is just starting.
  • You will have ample opportunity to make mistakes in both gardening and software development, so feel free to learn from them.  I don’t know where I heard it, but “it’s fine to make mistakes, just try not to make the same ones.”
  • Automate, automate, automate.  The more you can rely on machinery to free you from the drudge of gardening, the more you can rest assured that you will have a great crop.  Similarly, the more you can rely on automated testing and scripts, the more complex you can make systems, and the more freely you can change them.
  • Trying something different is fun.  I planted artichokes this year.  I also played around with easyrec.  I can’t speak for the artichokes yet, but exploring a new tool was interesting and fun.  Look up from your code once and a while and visit hackernews (thanks to Jeff Beard for turning me on to that resource) to find something new to learn about.

I think that many software developers are obsessed with passive income, but I think that gardening is the original passive income stream–food grown for you while you are doing something else!


Testing with Pentaho Kettle – adding new logic

We finally have a working test suite, so let’s break some code.  We have a new requirement that we greet users who are under the age of 30 with ‘howdy’ because that’s how the kids are saying ‘hello’ nowadays.

You just jumped into a series of blog posts on testing ETL transformations written with Pentaho Data Integration. Previous posts have covered:

The first thing we should do is write a test that exercises the logic we are trying to write.  We make a directory with a name descriptive of the behavior we are trying to test, and add a row to the tests.csv driver file pointing to the files in that directory. Here’s how the additional line will look:

agebasedgreeting,agebasedgreeting/input.txt,agebasedgreeting/expected.txt

And we will copy over the data files from the first test case we had (simplerun) and modify them to exhibit the expected behavior (a new greeting for users under 30). We don’t have to modify my input file, since it has people both under 30 and over 30 in it, but just to catch any crazy boundary conditions, we will add someone who is 30 and someone who is 31 (we already have Jane Doe, who is 29).

Then we need to modify the expected output file to reflect the howdyification of the greeting. You can check out both source files on github.

Then we run the tests.

pentaho-failed-test-75

You can see the failure in the log file that kettle generates and in the build/results directory.  You can also see that we added a job entry to clean up the build directory so that when we run tests each time, we have a clean directory into which to write our output file.

pentaho-failed-test-75

Now that we have a failing test, we can modify the core logic to make the test pass. Writing the logic is an exercise left to the reader. (Or you could look at the github project :).

We re-run the tests to see if they pass, but it looks like simplerun fails before we can even test agebasedgreeting:

pentaho-failed-test-2-75

We can do a diff of the expected and output files and see that, whoops, the simplerun testcase had some users that were under 30 and affected by the logic change.

This points out two features of this method of testing:

  1. Regression testing is built in.
  2. Because of the way we are abort tests, TestSuiteRunner only runs until our first failure.

The easiest way to fix this issue is to inspect output.txt and verify that it is as expected for the simplerun test.  If so, we can simply copy it over to simplerun/expected.txt and use that file as the new golden table.

We also realize that we are passing in the hello field to the output.txt file and that doing so is no longer required.  So we can update the expected.txt in both directories to reflect that.  Running the tests again gives us success.

pentaho-success-75

Now that we’ve added code, I’ll look at some next steps you can take if you are interested in further testing your ETL processes.

Signup for my infrequent emails about pentaho testing.


Testing with Pentaho Kettle – business logic

So, the first step in building the test harness is to create a skeleton of the transformations we will need to run.  These transforms contain the business logic of your ETL process.

Pssssst. This article is part of a series.  Previous posts covered:

Typically, I find that my processing jobs break down into 4 parts:

  • setup (typically job entries)
  • loading data to a stream (extract)
  • processing that data (transform)
  • saving that data to a persistent datastore (load)

Often, I combine the last two steps into a single transformation.

So, for this sample project (final code is here), we will create a couple of transformations containing business logic.  (All transformations are built using Spoon on Windows with Pentaho Data Integration version 4.4.0.)

The business needs to greet people appropriately, so our job will take a list of names and output that same list with a greeting customized for each person.  This is the logic we are going to be testing.

First, the skeleton of the code that takes our input data and adds a greeting.  This transformation is called ‘Greet The World’.

pentaho-basic-logic-75

I also created a ‘Load People to Greet’ transformation that is just a text file input step and a copy rows to results step.pentaho-basic-logic-load-75

The last piece you can see in this is the ‘GreetFolks’ job which merely strings together these two transformations.  This would be the real job that would be run regularly to serve the business’ needs.

pentaho-basic-logic-job-75

This logic is not complicated, but could grow to be quite complex.  Depending on the data we are being passed in, we could grow the logic in the ‘Greet The World’ transformation to be quite complex–the variety of greetings could depend on the time of year, any special holidays happening, the gender or age or occupation of the person, etc, etc.

Astute observers may note that I didn’t write a test first.  The reason for this is that getting the test harness right before you write these skeletons is hard.  It’s easier to write the simplest skeleton, add a test to it, and then for all future development, right a failing test first.

As a reminder, I’ll be publishing another installment of this tutorial in a couple of days.  But if you can’t wait, the full code is on github.

Signup for my infrequent emails about pentaho testing.


Testing with Pentaho Kettle/PDI

Pentaho Kettle (more formally called Pentaho Data Integration) is an ETL tool for working with large amounts of data.  I have found it a great solution for building data loaders to integrate external data sources into .  I’ve used it to pull data from remote databases, flat files and web services, and munge that data, and then push it into a local data store (typically a SQL database).

However, transforming data can be complex.  This is especially true when the transformation process builds up over time–new business rules come into play, exceptions are made, and the data transformation process slowly becomes more and more complex.  This is true of all software, but data transformation has built in complexity and a time component that other software processes can minimize.

This complexity in turn leads to a fear of change–the data transformation becomes fragile and brittle.  You have to spend a lot of time thinking about changes.  Testing such changes becomes a larger and larger effort, because you need to cover all the edge cases.  In some situations, you may want to let your transform run for a couple of days in a staging environment (you have one of those, right?) to see what effect changes to the processing have on the data.

What is the antidote for that fear?  Automated testing!

While automated testing for Pentaho Kettle is not as easy as using Junit or Ruby on Rails, it can be done.  There are four major components.

  • First, the logic you are testing.  This is encapsulated in a job or transform.  It should take data in, transform it and then output it.  If you depend on databases, files or other external resources for input, mock them up using text files and the Get rows from result step.  Depending on your output, you may want to mock up some test tables or other resources.
  • Second, the test case runner.  This is a job that is parameterized and sets up the environment in which your logic runs, including passing the needed input data.  It also should check that the output is expected, and succeed or fail based on that.
  • Third, the test suite runner.  This takes a list of tests, sets up parameters and passes them to the test case runner.  This is the job that is run using kitchen.sh or kitchen.bat.  You will need a separate test case and test suite runner for each type of logic you are testing.
  • Finally, some way to run the job from the command line so that it can be automated.  Integration into a CI environment like Hudson or Jenkins is highly recommended.

It can be time consuming and frustrating to set up these test harnesses, because you are essentially setting up a separate job to run your logic and therefore doing twice the work.  In addition, true unit testing, like the frameworks mentioned above, is impossible with Kettle due the way columns are treated–if you modify your data structure, you have to make those changes for all the tests.  However, setting up automated testing will save you time in the long run because:

  • the speed at which you can investigate “interesting” data (aka data that breaks the system) is greatly increased, as is your control (“I want to see what happens if I change this one field” becomes a question you can ask)
  • regression tests become far easier
  • if you run into weird data and need to add special case logic, you can also add a test to ensure that this logic hangs around
  • you can test edge cases (null values, large fields, letters instead of where numbers are) without running the entire job
  • you can mimic time lag without waiting by varying inputs

I hope I’ve convinced you to consider looking at testing for Pentaho Kettle.  In the next couple of posts I will examine various facets of testing with Pentaho Kettle, including building out code for a simple test.

Signup for my infrequent emails about pentaho testing.


JFreeChart for the win

test

I wanted to make some charts showing linear regressions, but they needed to be images (so libraries like d3.js were out).  My first inclination was to use Google’s Image Charts API, but it has been deprecated.  I looked around and couldn’t find anything like it.

So, it looked like I’d need to code something up.  Some searching around turned up some great open source projects.  Using JFreeChart and Apache Commons Math, and some blog posts (this one about regression, this one about drawing dashed lines) and a post on StackOverflow about scatter plot basics, I was able to put together the above regression line chart in about half a day.


Hackfest tips for companies with few developers

Last year, my company ran a hackfest.  This year, we are doing it again.  The company I work for, 8z Real Estate, is about 20% real software developers, though everyone at the company is familiar with software and technology.

How do we run a successful hackfest when only a few employees can build software?

  • Include everyone.  It will be a richer, more fun experience with more people.  Get executive buy in–I found the original ‘fed ex day’ post helpful in explaining the idea.
  • Set goals and expectations.  At a typical hackfest (or hackathon), running code is the goal.  For us, autonomy and exploration is more important.  In the announcement email we state: “the idea is to give everyone a chance to do something work related that they want to do, or try, or explore, but don’t have time to because of the hustle and bustle of work life.”
  • Reset deliverable expectations.  Rather than running code, the deliverable at a typical hackfest,  at an event with many non technical attendees other deliverables should be embraced.  Spreadsheets, powerpoints, text documents, mockups, link gallerys, images–these are all artifacts that capture an exploration.  They can also be referred to in the future.  (I don’t think a presentation without an artifact is a good idea, because of the lack of permanence, although I guess you could videotape it.)
  • Encourage developers to work on different teams.  Spreading the developer viewpoint and code writing ability across as many groups as possible is a good thing, as it will allow the groups to push their ideas further. That said, if a developer really wants to work on his or her own idea, don’t force them to join a team.
  • Make sure contractors feel welcome.  Because this isn’t a typical workday, it can be difficult to justify paying contractors to attend.  But a hackfest reinforces company culture and can make contractors feel part of the team.  We compromise by inviting contractors and paying them a mutually agreed upon reduced hourly rate.  If they are technical, this also adds to the pool of developers as well.
  • Have the hackfest onsite, preferably in one conference room. Especially for the first one, the hum of people working will be motivating and exciting.
  • Have the hackfest happen on one day.  Pick one that is slow–for real estate, that means closer to the year end holidays.
  • Plan for some ‘normal’ work to be done on the day of the hackfest.  We need to provide daily customer support, so on hackfest day we try to compress a full day’s work into a few hours, then shut off the phones.

Then, some general hackfest principles that I believe are true no matter what the attendee’s skillset.

  • Start with a timeboxed ideation whiteboard session.  This lets everyone see all the great ideas, and find what interests them.
  • Use the ideation session to head off any ‘typical work’ tasks that people suggest (‘I just have to verify 4 more bugs on the foobar widget’).
  • Let teams self organize, but encourage cross pollination between departments and teams.  A hackfest is a great way to build trust between people who don’t normally work together.  On the other hand, if someone is very passionate about an idea that no one else cares enough about team up on, let that person pursue their passion.
  • Handling managers at a hackfest is a sticky subject.  On the one hand, there is benefit to treating them as another employee–they get the benefits of working with different people and ideas.  On the other hand, because of their job, they may (unintentionally!) warp the autonomy of the team.  Last year, the CEO worked alone, but all the other managers were treated as employees.  Discuss this issue, especially with higher level executives, before the day of the hackfest.
  • Order in lunch, which keeps the momentum going.

Any tips for a good hackfest, especially one attended by fewer developers than non technical people?


Run through the finish

Make that last code change.

Write that last test.

Look in that document, rather than trying to remember what color the icon is supposed to be.

Write that documentation.

Look at your work with the eyes of a user.

I work in a small development department (2 developers plus a number of contractors) and I need to constantly remind myself to run through the finish.

I ran in high school and college (cross country and track) and at the very end, it is easy to let your guard down and coast.  After all, you’ve done almost all of the hard work.  And no one is really behind you.  And it hurts (oh yes, it hurts).  So, why not ease off a bit?

The problem is, there is someone behind you, and they have you in their cross hairs.  They have the incentive and the vision of their competition, but you have the lead.  Why give up any advantage?

Development isn’t painful, but it can be a slog.  Yes, yes, I’m sure there are shops that never have a slog.  But, for most mortals, there are requirements that are changed or were forgotten, tasks that are less fun than others, tweaks to the UI for the Nth time, vendor rigidity that never surfaced during the sales process, and other sundry annoyances.  But you as the developer own the final product.  You can choose to coast, since you did 99% of the work (and did it well), or you can choose to run through the finish.

Run through the finish.


Small scale data migrations

So, I’ve recently been involved in another data migration, the second one in three years.  These are small migrations, with thousands records.  One person to take care of this size of data migration with some effort, but the amount of data is still large enough that manual data re-entry isn’t really an option–the error rate and the cost and the management difficulty mean that software is the better option.

Here are some lessons I learned from these data migrations.

Learn as much as you can about the data models–both the old and the new–as you can.  This includes, in preferred order, talkingto any people familiar with the old system, talking to any people familiar with the new system, looking at the databases via a sql client, reading documentation (if any is written), and looking at code.  I spent some time thrashing around in old system code for a while.  Then I asked the developer for a tour, and learned more in that hour than I had in the previous day of looking at code.

Map entities and concepts as early as you can.  Take special note of any that are in the old and not in the new (and what you are planning to do with them).  Those that are in the new and not in the old aren’t as big of an issue.  Also, attributes of entities are as important as entities, so note discrepancies there.  Early on I noticed that one of the two primary entities in the old system did not exist in the new system.  This led to some interesting conversations with the business users that saved me work.

As above, talk to people who are going to be using the new system, and who use the old system, throughout the migration process.  An entity or attribute that will be a royal pain to migrate may not be used anymore!  Or, the business person might have some good ideas on how to map something in the old system into the new system.  Someone who uses the software you are migrating has more domain expertise than you.  Let them try the new system with migrated data as soon as some data is moved. Make sure to guide their experience so they don’t spin their wheels looking in corners of the system that not yet migrated.

Start a spreadsheet of tasks, so that everytime you uncover something that needs to be done while you are in the process of doing something else, you can note it and keep on your original task.  My spreadsheets are simple; three columns are enough: task name, completed (with an X for completion, blank for still open) and notes (for possible implementation solutions, people to talk to, relevant urls, or any other text that will help me complete the task).

Document all the migration steps, preferably to the point you can cut and paste commands.  Include any discrepancies discovered, special commands to run, access to all needed systems, names of relevant people, areas that need further investigation, and basically anything else you would want handed to you if you were starting on this project.  This helps immensely if you need to pass off the project, or come back to it later (even just a few days), and provides documentation of entities on the old and new system.

Write scripts wherever possible, but don’t try to script the whole process–access to different servers can be hard to automate.  Use whatever language you feel is best for these scripts.  I’ve used bash, sql, perl, and awk/sed, but I don’t shy away from a compiled language like java, especially if a library exists that can save me time.  Make sure to put these scripts into version control, and document the purpose with comments at the top and a good name.  I wouldn’t worry too much about unit testing or refactoring this software, because chances are it will be seldom used once the migration occurs.

Get familiar with the concatenate function of your database.  Using queries to write DDL for the new system based on data from the old system can save you writing a script in an imperative language.  When migrating from Expression Engine to WordPress, I used a statement like <code>select concat(‘update wp_comments set comment_author_email = ”’,email,”’ where comment_author = ”’,name,”’;’) from exp_comments where name in (select distinct(name) from exp_comments);</code> to generate an update statement for WordPress for each comment author in the EE database.

Think about data types and representations.  Especially if you are moving from one database to another.  When I was moving from MSSQL to MySQL, date fields were particularly thorny.

Realize that these types of projects are typically difficult slogs.  There were moments where I despaired of ever getting through the migration in a timely fashion.  To do it right, you need a fantastic attention to detail, an understanding of the business needs, and an ability to drive things through to the finish.  All of this can be pretty draining–I find it far more draining than bug fixing or building new features.

Control the old and new systems–try to not have new capabilities added during the migration.  If you can’t guarantee that, can the migration wait until the new and old systems stabilize?  If not, checkpoint the migration against the new capabilities during the process, and realize that you are introducing a lot of extra work and complexity into an already complex process.

Have a staging system where you can practice your migrations without affecting anyone.  Plan to go through at least two or three of these new staging systems so that you can get the migration steps solid before you touch production.  Start from a clean slate each time so no time is spent chasing phantom bugs from a previous migration that didn’t finish or wasn’t entirely correct.  This is what makes the migration documentation you write so important.  Be aware that the new stage system and the new production system will not necessarily be the same.

Lastly, avoid committing to a schedule if at all possible.  And if you must, pad it and only commit after you’ve done a thorough analysis.  Because there are so many hidey holes and areas of the old system that you won’t understand, there is a high probability that you’ll be discovering new issues and data you need to migrate halfway through the project.  (This is a special case of the requirements nightmare known as ‘build system B that acts exactly like system A’.)  Communicate progress to the business.

While this is not my favorite type of project, when done well it can have tremendous business value.  Combining newer, more flexible systems with rich older data, without re keying the data, can make system users much happier.  In some cases, if there is no migration, the newer system simply can’t be used.



© Moore Consulting, 2003-2017 +