Skip to content

Knowledge in software development: what’s your wood lot?

Gene Logsdon, who passed away in 2016, was a farmer and a wise man.  In a recent repost (originally written in 2008), he discusses why wood is more valuable than gold.  From the post:

You can’t eat gold like you can the bounty of trees in fruits, nuts, maple syrup, and various edible mushrooms and herbal treasures of the woodland. You can’t warm yourself with gold. You can’t bask in the shade of gold. You can’t make fence posts out of gold. A gold house would be mighty expensive. You can’t make a windbreak out of gold. You can’t make furniture, violins, guitars, wall paneling, picture frames, gun stocks, tomato stakes, flooring, barns, chicken coops, and hog houses out of gold. You can’t mulch a garden with gold leaf. Gold does not take in carbon dioxide and give off oxygen to preserve an environment we can live in. Gold does not provide habitat for millions of wild animals and zillions of insects necessary for a sustainable environment. And in fact, you can make methane out of wood much more efficiently than ethanol out of corn. All gold can do is go up and down in price and invariably it turns out to be a poor investment, as many panic buyers learn the hard way.

This resonated with me.  Of course when things go awry, as they did in 2008 and 2001 (in the USA), having a pile of cash feels good.  But money is not wealth (this article covers that, as well as some other key points about capitalism).  Money can come and go, but wealth, whether knowledge, relationships or wood lots, remains with you.  Some forms of wealth require more care and feeding than others, but they all have the attribute of providing value in the real world.

When you are in software development, knowledge is a common form of wealth.  I always tell teams I lead that almost every problem we’re facing is a new one, because if it wasn’t new, it most likely would have been automated or turned into a service or library that we could use.  So there are plenty of chances to gain new knowledge.

There are a few categories of software development knowledge.

  • Domain knowledge is understanding the real world problem domain.  So, if you’ve worked, as I have, in a real estate brokerage, you understand how the business works.  Who pays for what, how money and effort flow, who the main players are and what they are called.  All of these are valuable should you work in that domain again because you can come up to speed more quickly, and you understand how the real world maps to the software system.  This type of knowledge may change, but on the order of years or decades.  (I haven’t worked in real estate since 2014, but if I went back, there’d still be many of the same concepts.)
  • General technology is fundamental to software development.  I’d put abstractions like algorithms and data structures, specific technologies like distributed systems, database indices and HTTP, and best practices like automated testing and requirements gathering into this bag.  Having this type of knowledge will let you understand a typical system even if you haven’t seen the nuts and bolts of it.  Of course, the type of general technology that is most useful to you depends on your domain, to some extent.  The general technology of a chipset engineer is different than that of a UX designer.   Within the same domain, this knowledge is good for years.
  • Specific technology knowledge are the keywords that recruiters put on job postings, and what some consultants chase after.  Elm or Elixir, Rails or React Native, Kubernetes or kvm.  These technologies are very specific.  I’m of two minds about this knowledge.  On the one hand, if you pick the right tech, you can be in very high demand and get rates commensurate with that.  On the other hand, new specific technologies are constantly being born, and keeping up with all of them, let alone becoming an expert in them, requires large investments of time (perhaps the reason for the premium).  However, I have sympathy for folks looking for specific technology expertise.  Yes, any competent software developer can pick up any language in a few weeks, but the idiosyncracies, libraries and community practices take time.  Depending on your budget, you may find it cheaper to pay someone who has acquired that knowledge elsewhen.  This knowledge ages quickly.
  • Leadership knowledge is orthogonal to software knowledge, just like domain knowledge.  Since software is built by human teams, knowing how to lead such teams toward business goals, whether formally or informally, is a valuable skill. This can include specific techniques like agile development and ‘one on ones’ and more general areas of knowledge like how to connect or communicate people and how to navigate politics.  This knowledge has a long half life and can pay dividends throughout your career.

In addition to knowledge a software developer can accumulate other forms of wealth like books, videos, and relationships.  These can all serve as credentials for a new position or (occasionally) generate passive income.

As we head into 2018, think about how you can build longer term wealth in your career by picking the right kind of knowledge.

Last week in AWS

If you are interested in AWS, you have to subscribe to ”Last Week in AWS”, a newsletter which covers, well, much of what has happened in AWS recently.  Both posts and news from the AWS team and external sources are covered.  There is a lot of snarkiness, but a ton of knowledge as well.

The newsletter is out every Monday and is run by Cory Quinn, an experienced AWS consultant.

Saying Thank You

Sometimes it’s a good idea to stop and say thank you.

Thank you to my parents, who instilled a love of learning in me and trusted me to make my own decisions. Now that I have my own children, many of the choices you make seem even wiser now.

Thank you to my wife, my biggest cheerleader and BS detector. It’s a joy to walk through life by your side.

Thank you to Dave B, who took a chance on someone who didn’t have any real software development experience. I still remember troubleshooting winsock issues.

Thank you to Bryan B, who offered me an internship after college and then a job that I couldn’t believe I was getting paid for. The friends and connections I made at that job have reverberated throughout my adult life.

Thank you to all my consulting clients over the last decade plus, from Mike M and Pat M to Brian T to Corey S and Andy P to Josh G, Chris H and Matt K. I always have loved the rush of “I need to deliver but don’t have any idea how to do this” that happens two weeks into every consulting engagement, and I appreciated your patience.

Thank you to Anthony F and Lane H. I learned so much about software development, building a business and how to act from my years working for you.

Thank you to Ashley C. Building something from scratch is a wild ride.

I want to leave you with this comic, from the great Nathan Pyle.

By Nathan Pyle

Thank you.

Heroku pipelines and asset_paths and font downloads being cancelled

My font downloads were being cancelled when I was using a CDN in front of a rails app hosted on Heroku.  I was testing out using heroku pipelines, which lets you promote a slug built in one environment directly to another.  Rails was also serving the CSS/JS/font files, but was behind a CDN and config.action_controller.asset_host was set to the cloudfront URL, so most clients were pulling from the CDN.

The issue I ran into was that in order to get my fonts to show up correctly when I was compiling my slugs on production deploy (before I moved to heroku pipelines), I needed to use erb and the asset_path tag. So my font css file looked like this:

  src:url(<%= asset_path('fontawesome-webfont.eot?v=3.0.1') %>);
  src:url(<%= asset_path('fontawesome-webfont.eot?#iefix&v=3.0.1') %>) format('embedded-opentype'),
  url(<%= asset_path('fontawesome-webfont.woff?v=3.0.1') %>) format('woff'),
  url(<%= asset_path('fontawesome-webfont.ttf?v=3.0.1') %>) format('truetype');
  font-style:normal }

However, this caused the asset_path in this CSS file to be set to the full CDN hostname plus the asset path, something like (when the slug was compiled on staging). When the slug was compiled on production, the value was set to When I was using heroku pipelines, the slug was unchanged between staging and production, just as advertised. However, this meant that the CSS was being served off of the production CDN host (d256) and the fonts in that CSS file were referencing the staging CDN host (d123). Browsers didn’t like that. This question indicates that this is due to cross resource permissions not being set correctly.

What were my options?

I could return to deploying slugs by compiling them on staging and production. However, I like the concept of pipelines and the immutable artifact.

I could try to find some way to post process the slug and change that link in the font CSS. But that didn’t seem possible, based on documentation, and kinda defeats the purpose. I did discover the heroku release phase.

I tried some of the other solutions outlined on this question to tweak how the asset pipeline compiled the font references, but nothing else worked.

I could have tried to set the permissions correctly for the font download. One worry there would be having the production environment depend on the staging CDN. Never a good idea to mix environments.

I also could have tried relative paths to the font files in the CSS file. I didn’t do this because I thought of my final solution first.

The solution I finally went what was using a CDN that hosted the font files of the correct version and used those. I also chose this solution because we won’t be using these fonts (which we primarily use for icons) for long–in the near future we’ll be removing the icons from the application.

Evolution of admin features

Time is short in a startup and the edge cases are many.  I have noticed that when I am building features for The Food Corridor, especially admin or edge case features, there is a progression based on frequency of feature need.  An example of this is something like handling a refund, which is not a core use case.

  1. The first time, the issue is handled via a developer.
  2. If it happens more than once, the process is documented in a google doc.  It might be triggered by an email from another part of the system.  We have an entire google folder called ‘operations’ which is full of such documents.
  3. The developer automates the process via a rake task.  The rake task is a thin layer around a service class, which is tested.
  4. Non technical admins get access to the process via a web interface.  The web interface plugs into the service class as well.  This web interface may handle only part of the issue, and require access to other systems (in the case of a refund, access to the payment processor).
  5. Non admin users get access to the process via a web interface.  This web interface is fully fleshed out.

For core functionality, you obviously want to push the ability to self serve to the end user as soon as possible.  It simply wouldn’t make sense for us to build out the ability to schedule bookings, limit it to a developer or admin user.

But for a lot of functionality, this progression is helpful.  The developer has time to fully understand the issue by handling it him- or herself.  The documentation generated by handling the issue manually is a great start for the requirements document.  And if an issue only pops up once every quarter, or even more rarely, a minimum amount of developer time is spent on it.  If something happens often, it is far more likely to get automated.

The power of December challenges

I am in the final stretches of my December blogging challenge.  I’ve done this with other challenges in the past (exercise, meditation).

While some folks start their New Year’s Resolution’s in January, I find that December works better for me (maybe July will be better for you).  A challenge is different than a resolution, because a resolution has no end date, where a challenge stops.  Also, because of the holidays, this month is already choppy.  That means if I take some time each day to do something new, it’s not going to impact other obligations as much as it would in a less choppy time.  December is also the darkest month in Boulder; having a task to ‘check off’ every day feels like progress, and that feels good.

You can either pick a new thing or a task you’ve tried to do in the past.  All of my experience is with the latter.  New things are often exciting enough for me that I don’t need any kind of push to experience them.  But something I’ve tried lackadaisically that I intellectually know is worth exploring–that kind of challenge is perfect for a month.

A month is long enough to give a new habit a chance and to learn some of the benefits and warts of it. It’s also short enough that you can gut your way through it.  I read about people doing challenges for a year, but I am not sure I have the mental stamina to commit for that long.  After some December challenges, I was happy to drop the activity, because I didn’t get much value from it (or not enough for the effort).  Others I picked up later.

As far pure mechanics, I’ve found that a PDF calendar that you can ‘X’ out each day is helpful.  Post this someplace you will see it (by your bed or your desk).  Get a big sharpie and enjoy the ‘X’ing out process.

I also think it’s important to publicly state any kind of ‘I’m doing this for a month’ goals because that external pressure will help you when encountering a day where you really don’t want to do the task (like today, for me. haha). “Public” can mean announced on Facebook, Twitter or a blog, but it could be a conversation with your roommate or SO or family members.

I know that this is a bit late for December 2017, but hopefully it will inspire you in the future–there are plenty of months for a challenge in 2018.

Happy Holidays!

Just wanted to wish everyone a happy holiday!  The meaning of Christmas has changed for me over the years.

When I was a kid, it was about the presents (I remember you, bike with the red banana seat!).

When I was a teenager, it was about the time off from school, and nintendo video games, and poker with friends.

When I was a young adult, it was about the time off from work and family time and connecting with friends who were back.

When I was in my thirties, it was about family and friends.

Now that I have a family of my own, it is about family and connecting with the chain of ancestors.  It is also about creating our own traditions.

May your Christmas be whatever you want it to be!

Hackernews melted my server

My post about Founding Engineers caught the attention of some folks on Hacker News, which is kinda like a focused subreddit for tech folks.  There were some great comments.  My server melted down, though, and some folks had to read the google cached version.  I think the post peaked at around #25 on the front page.  Can’t imagine what a pounding the number one post gets.

I was able to restart my web server later in the day (the post unfortunately happened at the same time I was doing a release of The Food Corridor application), and saw from my web stats that I had as many people visit on that one day as I get in a typical week.  That was only the folks that my stats system was able to capture, so I’m guessing there were a lot more.

I have thought for a long time about making my wordpress site publish to s3, using a tool like simply static.  I haven’t done that yet, but was able to leverage WP Super Cache’s CDN integration to serve up the static assets of the blog from CloudFront, AWS’s CDN.  This post was very helpful.


If the images in your Rails image_tag calls don’t have a checksum…

This last week, I spent a lot of time learning about how Rails serves static files, how it interacts with a CDN like CloudFront, and how misconfigurations can really screw up your application.  I wanted to document this here so that if I run into these situations again, I can troubleshoot them more easily.

Problem #1: We did an large release, with a lot of moving pieces

We (The Food Corridor) recently engaged with a consulting company to do a refresh of the look and feel of our application.  They don’t just do UX and design, but also implementation using overseas developers and QA.  I was excited to let said developers focus on look and feel (not my strength) but made a mistake in not setting up an entire environment for them.  Instead, I let them use our staging environment.  Things took longer than predicted (as they always do, and some of it was due to my availability).  I was doing some tidying up, changing the deploy process, etc, and ended up merging a lot of changes into our codebase.  This meant that the first release had a lot more risk that previous releases–there was some of the new look and feel as well as all my changes.

That made troubleshooting any issues that came up with the release difficult, because it wasn’t clear what caused it–was it deployment changes, some of my code, some of the look and feel code?

Solution #1: Set up a ‘review’ environment for any long running branches.  I haven’t gone the full ‘review app’ path yet, but based on the docs, it doesn’t seem too hard to set up.  For now we just have one review environment that can be shared.

Problem #2: Certain images appeared on staging but not on production.

This was one of the issues that caused some heartache during the first release.  There were some images (svgs, to be exact) that were present on staging and not on production (from the browser, you’d get a 404).  But staging and production had the exact same codebase, including images.  We were doing a fresh deploy to production.  What was the issues?

I rolled back a lot of the changes I’d made to make sure it wasn’t a deployment issue (turning off pipelines, unsetting environment variables, etc).  No love.  We were still seeing 404s.  Looking at the HTML, on production the image had a different name, without the hash at the end.  From slack:

Interesting. on prod the envelope image is here: &ltlimg alt=”Message” src=””>

and on staging it is here: <img alt=”Message” src=””>
… the svg isn’t being compiled on production for some reason

That led me to learn about the heroku asset pipeline and in particular the way rails4 apps are treated.  I then dove into the compiled slug, and then saw that the message-c9189....svg image was present in the staging slug under public/assets but that there was no message.svg file in production. There was, however, a message-text-89ab....svg file.

Solution #2: The staging environment had some old copies of the image files. The files had been renamed, but the image_tag calls still referenced those old files. We had to update them. I also updated the build process to run a heroku repo purge-cache every time staging was built so that we wouldn’t have any of those old files lying around, using the heroku repo plugin/. (It’s fine to make a mistake, but try not to make the same mistake twice.)

Moving from a feature to a product to a company

This post from Seth Levine discusses the path of a startup along this continuum:

Features provide specific point value to users. Products stitch together related features into bundles that are essentially universal in their need across the problem set you are solving (put another way, if each of your users buys your “product” for a different reason you’ve probably just created a feature set, not a true product). Companies have product that is broad enough in its use and impact that a huge number of users gain value from it in a market that is both large and where the user need is similar enough to drive broad adoption of the same solution.

It’s an interesting perspective that is worth evaluating your current organization against.  Are you in the feature, product or company phase?  Rapid application development with libraries and modern frameworks makes it really easy to move from a feature to a product phase, even though some organizations that stay in the feature space–something like this store locator app or a data product like Mattermark where extra work is required to gain value.  We certainly found that when we started The Food Corridor that features alone weren’t going to be defensible or sustainable, so we moved quickly on to being a product.

It’s also worth noting that “Company” status for a VC like Seth is probably a lot larger than a sustainable SaaS company which can feed a small team and provide a lot of value to a niche market–companies like Small Farm Central or Fileboard.  You don’t have to be a “Company” to be a success, in other words.  You don’t even have to be a “Product”.  But knowing where you are in the continuum can help you define your focus.