Skip to content

Feature branch development

I remember when I had lunch with a friend, back when I was using SVN and he was using git.  He said “it’ll change your life”.  I had read about darcs years ago and had read Joel’s post about distributed version control systems.  I was still like “meh.  SVN does what I need it to do–tag releases, keep track of changes, and I can branch if I need to”.

Boy, was I wrong.

I’ve been using git since around 2014, and it’s great.  I don’t remember where I heard it, but someone said “branching in SVN is easy, it’s the merging that is difficult”.  Git takes the pain out of merging (mostly, of course you can still hose yourself pretty well).  I have to also reference this classic XKCD comic whenever I talk about git.

Regardless, one of the things I’m loving about working with git is that if you always use feature branches, use story numbers in your branch name (like story-update-123), and you set up your prepare-commit-msg script, you can track back every change to a story.  Here’s my prepare-commit-msg script:

#!/bin/sh

# from http://stackoverflow.com/a/16061192/203619

if story_id=`git branch |grep '*'|sed 's/.*-//'`
then
    echo "[#$story_id]" >> "$1"
fi

Another nice feature of the branch handling is that you can roll forward and backward branches. I use bitbucket, so I use the GUI revert command, which creates a nice revert PR. (I then immediately revert the revert PR so that I can apply the intended changes in the future by merging the second PR.) This makes it possible to push a feature to staging, realize it isn’t fully baked, and revert it so you can get out another feature release. Perhaps you could do this with SVN or another centralized VCS, but I was never comfortable enough with branches to do it.

All in all, git has been life changing as a developer. Thanks Russ, you were right!

Tips for working with offshore teams

Remember “The World Is Flat”? For software development, the world is definitely getting more planar. I’ve had the privilege of working with a few offshore software development teams directly, and have discussed the practice with other engineers.  I wanted to share some of my tips. My experience has been in web development, but I believe these tips apply to most general software development. (If you’re looking for a tiger team/extreme expertise in a specific area, I’d treat an offshore team just like another vendor.)

If you can, pick a team that you have worked with before. Just like onsite software development, knowing personalities, strengths and weaknesses of team members is crucial to delivering quality software. If you haven’t worked with someone before, then ask for referrals and check references.

Pick an easy first project. This could be something not on the critical path, a rebuild of outside functionality, or a rewrite of an existing piece of software. If you have a prototype that you can point to and say “make it work like that”, that can reduce complicated requirements discussions. One successful offshore project I know of took a Cordova mobile application and rewrote the app in objective c and android java.

In general, projects that have a lot of iteration and experimentation are tough to offshore, because of communication latency.  You lose out on feedback cycles.

Have a working agreement. This can be an informal document, but you want to specify roles and responsibilities (who is releasing? who signs off on stories that have been finished? how does planning happen?). Make this document a living one.

Having regular time overlap can speed up feedback, to some extent. If you are offshoring to a country in the same timezone, this is painless.  If you are working with a team from an offset timezone, you may need to adjust your sleep schedule.  It is best if both parties have to adjust their lifestyles somewhat–it is more equitable. Record these overlap hours in your working agreement. Shifting my schedule to have a significant overlap doubled the number of feedback cycles (code reviews, questions) that can be made between myself and an offshore developer.

You need one person to own the relationship on each shore. If there are process questions, these folks sort it out. They can button up any lingering requirements uncertainties. They may pull other folks in to make decisions, but this pair owns the success of the offshoring relationship.

Leverage asynchronous tools. Make sure you use a progress tracker like pivotal or trello. slack is fantastic. So are the modern source code management SaaS applications like github and bitbucket. A live prototyping tool like Invision is useful. Record decisions in writing.

Finally, decide how closely you need to follow the code delivery. You may or may not want to see “how the sausage is made” This can be based on a number of factors:

* how much technical expertise does the onshore team have? Are they interested in acquiring more?
* how much time does the onshore team have? You don’t want the onshore team to be a roadblock.
* will the project be standalone (a marketing website)? Or integrated into the main codebase?
* is what the offshore team writing core functionality to your application and/or business?
* who is responsible for maintenance and changes after delivery?

Many of these tips are best practices that should be followed wherever your development team is.  But they are especially important when they aren’t in the next office.

Offshore development can extend your budget and pull in timelines.  It can help you build a better product, or allow you to access skillsets you might not be able to hire.  You just want to make sure it works.

Run through the finish line

I used to run cross country way back when. One of the differences between the good runners (like me) and the great runners (Dave F, Todd E) was talent. But drive was more important than talent. One of the key pieces of advice my coach gave us was to “run through the finish line”. On the face of it, it seems obvious–you are at the end of a 5k race, why wouldn’t you want to finish as fast as possible? Why waste that effort? But I can’t tell you how many times I passed (and was passed) in the last 50 feet of a race. I saw the finish line, I relaxed (as much as possible) and slacked. Or, I saw someone else doing that and knew I could move up one more place. If you’ve left it all on the course (easier to do in a 5k than some other races) it is hard to have anything left to run that last bit. But drive can provide that last bit of energy.

In a software project of any size, there’s also that “last 50 feet”. If you are developing, it’s the last 10% of the project that takes 50% of the time. If you are running a project, it’s the last few weeks when all the little things that were put off during the big build out phase come out of the woodwork and need to be dealt with. Or it’s when the project blows up 2 weeks before deadline. When you are dealing with a vendor, it’s the crunch time before you launch, when any organizational complexity ignored comes due.

If you want to get across the finish line successfully, sound of body and mind, and with your team intact, what can you do to finish strong?

Be realistic: I never ran a 15 minute 5k. So starting out at that pace was a sure way to make sure I had nothing left at the 2 mile marker, let alone the finish. Make sure you have realistic expectations at the start of your project. If you or your team don’t know how to scope the work, spend some time learning how to do so upfront.

Cultivate drive: remind the team regularly why this project is important. If there are features that you discover are harder or less useful than supposed at the beginning, don’t be afraid to shift effort. Also, refer back to previous successes or failures for motivation.

Confront complexity early: don’t save it for the finish. That’s like picking up a 40lb sandbag at the 4k mark. Derisk a project by doing the hardest parts first. In some situations you can’t (final integration of components or external system access), but try to do it as early as possible (via CI or mocking up external systems and loudly asking for access as soon as possible).

Hold a reserve: experienced runners know to save a bit for the end. Don’t ask your team to work crazy hours in the beginning or middle of a project, because they’ll have no reserve to push through the finish line. Every software developer I’ve ever talked to knows the last 2 weeks of a project will require extra effort–don’t waste that expectation by requiring more work early on.

Drive is important to finishing a 5k; it’s also important to complete a software project. Running all the way through the finish line can be the difference between a failed delivery and a successful project. Plan accordingly.

Automating one off deployment tasks

When I am deploying a rails application, there are sometimes one off tasks that I need done at release time, but not before. I’ve handled such tasks in the past by:

  • adding a calendar entry (if I know when the release is happening)
  • add a task or a story for the release and capture the tasks there
  • writing a ‘release checklist’ document that I have to remember to check on release.

Depending on release frequency, application complexity and team size, the checklist may be small or it may have many tasks on it. Some tasks on a checklist can include:

  • sending a communication (an email or slack message) to the team detailing released features
  • restarting an external service to pick up configuration or code changes
  • notifying a customer that a bug they reported has been fixed
  • kicking off an external process via an API call now that a required dependency has been released

What these all have in common is that they:

  • are not regular occurrences; they don’t happen every deploy
  • are affecting entities (users or software) beyond code and database
  • may or may not require human interaction

after_party is a gem helps with these tasks. This gem is similar in functionality to database migrations because each after_party task is run once per environment (this is done by checkpointing the task timestamp in the database). However, after_party tasks are arbitrary rake tasks, rather than the database manipulation DSL of migrations.

You add this call to your deployment scripts (the example is for heroku, feel free to replace heroku run with bundle exec):

heroku run rake after_party:run --app appname

This rake task will will run every time, but if an after_party task has already been run and successfully recorded in the database, it will not be run again.

You can generate these tasks files: rails generate after_party:task my_task --description="desc"

Here’s what an after_party task looks like:

namespace :after_party do
  desc 'Deployment task: notify customer about bug fix'
  task notify_issue_111: :environment do
    puts "Running deploy task 'notify_issue_111'"

    TfcAdminNotesMailer.send_release_update_notification("customer X", "issue 111").deliver_now

    AfterParty::TaskRecord.create version: '20180113164945'
  end  # task :notify_issue_111
end  # namespace :after_party

This particular task sends a release update notification email to the customer service user reminding them that we should notify customer X that issue #111 has been resolved. I couldn’t figure out how to make a rake task send email directly, hence the ActionMailer. In general you will want to push all of your logic from the rake task to POROs or other testable objects.

I could have sent the message directly to the customer rather than to customer service. However, I was worried that if a rollback happened, the customer might be informed incorrectly about the state of the application. I also thought it’d be a nice touchpoint, since issues are typically reported to customer service initially. The big win is that ten of these can be added over a period of weeks, and I could have gone on vacation during the release, and the customer update reminders would still be sent.

All is not puppies and rainbows, however. This gem doesn’t appear to be maintained. The last release was over two years ago, though there are forks that have been updated more recently. It works fine with my rails4 app. The main alternative that I’m aware of is hijacking a database migration and calling a rake task or ruby code in the ‘up’ migration clause. That seems non intuitive to me. I wouldn’t expect a database migration to touch anything outside of, well, the database.

Another thing to be aware of is that there is no rollback/roll forward functionality with after_party. Once a task is run, it won’t get run again (unless you run the rake task manually or modify the task_records database table).

This gem will help you automate one off deployment tasks, and I hope you enjoy it.

What can a senior developer do that a junior developer can’t do?

I saw this post about senior vs junior lawyers a while ago and it sparked some thoughts about the senior/junior divide.  I have also seen companies interested in hiring only senior developers. This is not a post hating on junior developers–the only path to a senior developer is starting as a junior developer.

Anyway, I think that senior developers will help you in a number of ways:

  • Senior developers have made their mistakes on someone else’s dime.  We all make mistakes.  One time I took 4 hours to figure out that the reason I couldn’t connect to an Amazon RDS instance was because the security group was too locked down.  But people integrate their mistakes into their experience and don’t make the same mistake twice.  When you hire a senior developer, you’re getting all their previous experience as well as their current effort.
  • Senior developers can own a larger portion of a project than junior can.  This can include non coding tasks like system setup, estimation, requirements definition and customer interaction.  This gives you leverage.
  • They are able to understand interactions between various pieces of your system.
  • They have an intuitive feel for performance impacts, again, probably because of previous mistakes.
  • They map your current solutions and problems into their existing knowledge: “Oh, this is how we handled caching at my <$JOB – 2>.”
  • They bring best practices from other companies (and can comment on worst practices).
  • They can get up to speed more quickly.
  • They can mentor other developers in areas of expertise.
  • They know the power of automation and knowledge sharing.
  • They have previously been confronted with difficult, complex problems and have an idea of how to attack new, difficult complex problems.
  • They understand that almost all problems are people problems, and that communication is a key part of any project’s success.

Junior developers, in my experience, bring value for the money as well.

  • They have more malleable viewpoints, especially with respect to tooling and technology.
  • They are typically very eager to learn.
  • When you are explaining systems to them, they can’t leverage existing knowledge.  This forces you to speak from first principles and may highlight erroneous assumptions.
  • Because they don’t have the knowledge of existing systems, they question why things are done a certain way.  Systems and processes evolve, but sometimes tasks are done only because they’ve “always been done that way” which isn’t necessarily the most efficient.  It’s hard to see that without the “eyes of a beginner”.
  • They are less cynical.

All good developers in my mind are:

  • Eager to learn.
  • Unwilling to say “that’s not my job”.
  • Accept ownership of the success of the project and the business.

If I had to sum up the differences between junior and senior developers in one word, it is would be “scope”.  A senior developer should be able to handle a wider scope of projects, responsibilities and problems than a junior developer.  How wide that scope is depends on the developer, their experience, and the problem space, but there are certain aspects of scope that are the same across domains (non coding tasks).

Seek The Golden Comment: “zeus exit status 1” fix

This bit me yesterday, so I wanted to get it written down.  zeus is a preloader that makes running tests faster.  It can be a bit finicky about the gems available, even when using rvm.

Yesterday, I tried to upgrade a rails 4.2 app to rails 5.  (I failed.)  When I checked out my source branch to work on a different issue, ran a bundle install, and then a zeus start, I saw this:


zeus rake
zeus runner (alias: r)
zeus console (alias: c)
zeus server (alias: s)
zeus generate (alias: g)
zeus destroy (alias: d)
zeus dbconsole
zeus test (alias: rspec, testrb)
exit status 1

And then zeus exited.  I did some google searches and turned up these two issues: 118 and 237. Lots of folks having similar issues.

After reading carefully through these issue, and trying some of the suggested fixes as I did so, I arrived at the golden comment. I reproduce it here in its beautiful entirety:

There’s a lot of “try this” in here, but no actual debugging steps. Here’s how you can find the actual issue:

zeus –log ZEUS.LOG start then cat ZEUS.LOG

Excellent!  Using this logfile I quickly determined that the root issue was a collision in my json gem versions and was able to get zeus running again.

But that’s not really the point. The point is that this user (thank you Steven!) didn’t just provide an answer, he provided the means for me to diagnose and find my own answer.

I wish github had some way of calling out highly recommended comments, as if I’d seen his comment first, it would have saved me some time.

Just goes to show, you should always read the comments fully and look for the golden one when you are troubleshooting.

Functional Core, OO Shell

This video is about 30 minutes long.  Mike Gehard mentioned it to me, and I enjoyed the heck out of it.  Many takeaways from this.  One is is how using test doubles make sense at the beginning of a software project, but how it will eventually come back to bite you as the method signatures of dependencies change, and your tests don’t fail.  Another is how you can write ‘fauxO’, a nice combination that has the strengths of functional programming and OO programming.  A third is how and where integration tests make sense.

Overall, what he proposes is that the boundary of any object that makes decisions have parameters that are simple values so that that it is always easy to unit test them.  Out of this proposal fall several nice features.

Well worth the time to watch.

The wonders of outsourcing devops

I have maintained a Jenkins server (actually, it was Hudson, the precursor). I’ve run my own database server.  I’ve installed a bug tracking system, and even extended it. I’ve set up web servers (apache and nginx).

And I’ll tell you what, if I never have to do any of these again, I’ll be happy as a clam. There are so many tools out there that let you outsource your infrastructure.  Often they start out free and end up charging when you reach a certain point.

By outsourcing the infrastructure to a service privder, you you let specialists focus on maintaining that infrastructure. They achieve scale that you’d be hard pressed to. They hire experts that you won’t be able to hire. They respond to vulnerabilities like it is their job (which it is).

Using one of these services also lets you punch above your weight. If you want, with AWS or GCP you can run your application in multiple data centers around the globe. With heroku, you can scale out during busy times, and you can scale in during slow times. With circleci or github or many of the other devtool offerings, you can have your ci/cd/source repository environment continually improved, without any effort on your part (besides paying the credit card bill).  Specialization wins.

What is the downside? You lose control–the ability to fine tune your infrastructure in ways that the service provider may not have thought of.  You have to conform to their view of the world.  You also may, depending on the service provider, have performance impacted.

At a certain scale, you may need that control and that performance.  But, you may or may not reach that scale.

It can be frustrating to have to workaround issues that, if you just had the appropriate level of access, you would be able to fix quickly.  It’s also frustrating having to come up to speed on the docs and environment that the service provider makes available.

That said, I try to remember all the other tasks that these services are taking off my plate, and the focus allowed on the unique business differentiators.

Leverage

As a software developer, and especially as a senior (expensive) software developer, you need leverage. Leverage makes you more productive.  I find it also makes the job more fun.

Some forms of leverage:

  • test suite. A suite provides leverage both by serving as a living form of documentation (allowing others to understand the code) and a regression suite so that changes to underlying code can be made with assurances that external code behavior don’t happen.
  • libraries and frameworks, like Rails. By solving common problems, libraries, open source or not, can accelerate the building of your product. Depending on the maturity of the library or framework, they may cover edge cases that you would have to discover via user feedback.
  • iaas solutions, like AWS EC2. By giving you IT infrastructure that you can manipulate via software, you can apply software engineering techniques to ensure validity of your infrastructure and make deployments replicable.
  • paas solutions, like Heroku. These may force your application to conform to certain limitations, but take a whole host of operations tasks off your plate (deployments, patching servers). When a new bug comes out affecting Nginx, you don’t have to spend time checking your servers–your provider does. When you reach a certain scale, thost limitations may come back to bite you, but in the early days of a project or company, having them off your plate allow you to focus on business logic.
  • saas solutions, like Google Apps or Delighted. You can have an entire business solution available for a monthly fee. These can be large in scope, like Google Apps, or small in scope, like Delighted, but either way they solve an entire business problem. You can trade time for money.
  • experience, aka the mistakes you’ve made on someone else’s dime. This allows you leverage by pruning the universe of possibilities for solving problems, based on what’s worked in the past. You don’t spend time doing exploration or spikes. Note that experience may guide you toward or away from any of the points of leverages mentioned above. And that experience needs to be tempered with learning, as the software world changes.
  • team. There’s only so much software you can write yourself. A team can help, both in terms of executing against a software design/architecture and improving it via their own experience.

Leverage allows you to be more productive and the more experienced you get, the more you should seek it.

Machine sympathy vs human constraint

I had beers with an work acquaintance recently. He’s a developer of a large system that helps contact management. Talk turned, as it so often does in these situations, to the automation of development work. We both were of the opinion that it was far far in the future. This was three whole decades of experience talking, right? And of course, we weren’t talking our book–ha ha. I’m sure that artisan weavers in the 1800s were positive that their bespoke designs and craftmanship would mean full employment no matter what kind of looms were developed.

But seriously, we each had an independent reason for thinking that software development would not be fully automated anytime soon.

My reason:

It’s very hard to fully think through all the edge cases of development. This includes failure states, exceptional conditions, and just plain human idiosyncrasies. Yes, this is what every system must do. That’s right. Anything you want handled by an automated system has two options: plan for every detail or bump exceptional cases up to human beings to make judgements. The former requires a lot of planning and exercising the system, while the latter slows the system down and introduces labor costs into the mix.

This system definition is hard to do and hard to automate. I’ve seen at least five new languages/IDEs/software platforms over the years that claimed to allow a normal human being to build such robust automatic systems, but they all seem to fail in the short term. I believe that is because normal human beings just don’t think through edge cases, but those edge cases are a key part of software.

His reason:

When systems reach a certain size, abstractions fail (I commented about this years and years ago). Different size, different failures. But just as an experienced car mechanic knows what kind of system failures are likely under what conditions, experienced software engineers, especially those who understand first principles, have insight into these failures. This intution (he called it “machine sympathy”) is something that can only be acquired by experience, and, by its very nature, can’t be automated. The systems are so complex and the layers so deep that every failure is likely to be unique in some manner.

So, which one is more likely to remain a relevant issue. It depends on the organization and system size. Moore’s law (and all the corollaries for other pieces of software systems) works both for and against machine sympathy. For, because, as hardware gets better, the chances of system breakdown decrease, and against, because as hardware gets better, larger and larger systems get more affordable. Whereas I believe the human constraint is ever present at all sizes of system (though less present in smaller ones where there is less concern about ‘bumping up’ issues to humans, or even just not handling edge cases at all).

What do you think?