Obstacles to building high availability software systems

Open sign

Is your system available?

I saw a discussion on a slack about obstacles to high availability systems and wanted to record the edited version for posterity (mostly for future me, as I blog for myself). Note that in any mention of high availability systems would be remiss if I didn’t mention the Google SRE book, which is slow reading but free and full of great information.

First, what is high availability? I like this definition from Digital Ocean:

In computing, the term availability is used to describe the period of time when a service is available, as well as the time required by a system to respond to a request made by a user. High availability is a quality of a system or component that assures a high level of operational performance for a given period of time.

Design considerations of a system that will hinder high availability fall into two categories.

The first category is actions that you don’t take, but could take:

  • single points of failure: if you have a piece of your system which is unique and it fails (and everything fails, all the time), the entire system’s availability will be affected.
  • missing or incomplete automation: if you need human beings to resurrect failed parts of your system, it will meaningful amounts of time and will be error prone.
  • failing to build in elasticity and scalability of resources: when usage increases, new resources should be automatically brought online. Failure to do so will impact system performance and that could impact system availability
  • missing or incomplete system instrumentation: if you don’t monitor your system, you won’t be able to even know its availability (until you hear from your users).
  • application statefulness (on the compute nodes): this impacts your ability to use elastic resources and to grow parts of your system that are under load. (If you aren’t designing a greenfield system, this may be an externally imposed requirement due to existing software.)

The second is in actions you can’t take because of external requirements on the system:

  • data sovereignty: if you are legally limited to certain data centers, you have fewer options for your system, this can hinder building the system.
  • tenancy: if you need to have single tenancy for security or legal reasons, you may have fewer options for elastic solutions.
  • data models and authority requirements: poorly performing data models can impact performance. If your application requires certain operations must be from the source of record (permissions checks, for example) then a poorly performing source data model can impact performance which can impact availability.
  • latency: if you have a highly latency sensitive system, then you may need to trade availability for decreased latency. Since availability often means geographic dispersion (to avoid disasters impacting multiple pieces of a system), it impacts latency requirements.
  • cost: high availability systems, because they have no single points of failure, cost more.

Again, this was a discussion from a slack of AWS instructors, but the commentary is mine, as are any mistakes. Thanks to Chad, Richard, Jon, Ryan and everyone else!


Book Review: Working With Coders

Woman with 1s and 0sSoftware is so integral to business processes and relatively inexpensive compared to labor that I believe every company is going to be a custom software company, in the same way that every company is an accounting company or every company uses paper. I happened on an interesting blog post and saw the author had written a book, “Working With Coders”. How non technical folks interact with coders is a topic of perennial interest to me, so I picked it up after reading the first few pages on Amazon. The book is written for clients, CEOs or project managers who are going to be working with developers to deliver applications that will provide business value.

Frankly, I couldn’t put it down.

The author, Patrick, is an engaging, opinionated writer. He breaks down complicated concepts into easily digestible pieces. Where there’s more to the story, there’s a footnote with a snarky comment or a link to more information. Patrick also provides nuts and bolts examples to show why something that seems simple to change is not (scaling text in a browser, for example). He also covers how big decisions like language, frameworks and library choices at the beginning of a project constrain freedom and choices further down.

Patrick covers what developers do, how they think, and why projects often fail. I thought his explanation of the benefits of agile development was darn good, and his explanation that even agile projects fail more often then they succeed was pretty depressing. He also discusses how the house construction metaphor for building software is just a big fat untruth.

I also enjoyed the section about testing in general, the various types of testing, and where they make sense. There’s also a section on finding coders, including a good explanation of why not to hire them as employees (you might be better off just hiring a development shop, depending on your needs). The chapter on how to deal with common issues (“the team hates each other”, “we’re behind schedule”) was worthwhile. His solutions won’t work for everyone. Maybe you’ll want to deal with these issues differently, but considering them before they happen will only help you prepare.

Of course, I also enjoyed the chapter on how to keep coders happy (continuous learning, quiet, a fast computer). In general the author is careful to avoid stereotypes, but does do a good job of covering common themes. I haven’t met too many developers who love working in bullpen environments.

I am definitely not the target audience. Neither is someone who is an experienced manager of developers. However, I am a subject of the book, so it resonated with me and I definitely found myself nodding along. There aren’t too many books I have wanted to distribute copies of (the two others are “The Hard Thing About Hard Things” and “Climate Wars”), but this is one.

If you work in a consulting practice with inexperienced clients or if you work in a product company with an owner or higher up that isn’t technical, reading this book will give you insights into their questions and thought processes. And if you can find a way to give them this book without being condescending (“hey, I found this book fascinating for helping facilitate conversation, maybe you will too”), both they and you will benefit.


Book Review: Beforelife

I really enjoyed Beforelife. It’s a whimsical stroll through a fantasy land that exists after death.  The main character, Ian, arrives in the afterlife with an unusual trait: perfect recollection of his life before death.  This leads him to be institutionalized, where he meets some madcap associates, including multiple people who think they are Napoleon and someone who thinks he was a dog.

The rest of the book is all about how this group helps Ian discover what’s special about him.

I enjoyed the plot twists, which I won’t outline because spoilers.  There are several other historical characters, some named (Socrates, Isaac Newton), others hinted at.  I also enjoyed the world construction.  The world after death is populated by immortals, with interesting ramifications for areas of study like science and social constructs like policing.

If you’re looking for a light hearted jaunt and you liked Douglas Adams or Terry Pratchett, you’ll enjoy Beforelife.


The UNIX and Linux System Administration Handbook

I’ve been reading the UNIX and Linux System Administration Handbook.  It’s a real tome, with about 1500 pages.  It’s got five authors and some great cartoons, and covers everything from shell scripts to disk to email to system management daemons (check out the table of contents).  No one should ever read this book cover to cover.  That would be just silly.

I’ve been really enjoying picking and choosing chapters to read, however.  The sheer breadth of this book means that anyone with an interest in modern software development can find something useful in it.

Given my interest in AWS, I read all the sections about cloud computing.  These were high level and not super interesting to me, but I think they’d be great if you were a novice about cloud computing, and they did have a great survey of the major public cloud providers and when it made sense to use each of them.

Then I moved on to the networking sections.  I honestly can say that I didn’t understand fundamental routing protocols before I read that section.  This is obviously closer to the heart of system administration, and the authors did a great job with concepts and hands on knowledge of networking.

After that I moved on to containers.  Did you know that Docker is the new hotness?  I had heard of it, but didn’t understand why.  Now I do.  It’s hot for much the same reason as the ‘fat jar’ deployment is preferred in java land.  Having one single artifact that rolls up code and dependencies is a way to simplify deployments of production code, including rollbacks.  The authors focus on the fundamentals of containers, primarily Docker, but they also cover various orchestration layers like Mesos and Kubernetes.

I’m now in the middle of a chapter about continuous integration and continuous deployment, where they are discussing the concepts as well as Jenkins, one of the key technologies (see, I told you everyone could find something in this book).  After that, I look forward to reading about configuration management.

If you work in software at all and are involved in production systems, you’ll be able to find something in the UNIX and Linux System Administration Handbook (and if you aren’t, I’d be interested in knowing who owns that responsibility).


Review of Modular Rails

I am currently working on modifying an existing large rails app.  I am customizing some of the look and feel and extending functionality.

The app is under current development and I wanted to be able to take advantage of bug fixes or improvements, without impacting my customizations.  Or at least minimizing that impact.

Being fairly new to Rails, I surveyed the landscape and thought that building my customizations as an engine would be a good way to go.  (I was wrong, because engines have a hard time reaching out and modifying the application that they are part of.  At least that seemed to be a non standard use of engines from what I can find.) The author of Modular Rails has some good blog posts about engines and modularity, so I bought his book.

Pluses:

  • Good overview of how to extend three major components of rails app, models, views and controllers
  • Easy reading style
  • Leverages existing gems like deface
  • Mentions testing
  • Starts from first principles and then later gives you a gem to speed up development
  • Not too long
  • Information on setting up your own gems server

Minuses:

  • Focus on ‘greenfield’ apps.  No mention of integration with existing monoliths.
  • Uses nested modules, unlike every other engine article out there
  • Assumes relatively advanced knowledge of rails
  • Fair bit of fluff–lots of ‘mv’ commands
  • Extra charge for source code

All in all I am glad I read this book.  It didn’t fit my needs, but it didn’t promise that either.  I found it a good overview of the engine concept, even if he did do some things in a non standard manner and was a bit verbose about unix commands.

If you have done more Rails development, it will be more useful, and it is a great way to think about building new freestanding applications.  I haven’t surveyed the entire rails book landscape but I haven’t found anything out there focusing on Rails engines that is better.


On failing to enter a partnership

I recently explored a business partnership opportunity that was quite exciting.  I had met the possible partner a few years ago.  He has technical chops (a software developer) and runs a company in a sector I’m very interested in.  He had a SaaS application that had real traction–users, revenue.  It wasn’t profitable, but looked like it could be shortly.  If we could find a way to work together, I could own the technical side of things and let him focus on selling and marketing.

However, it didn’t end up working out.  No blowups, thankfully, just a failure to find an arrangement that worked for both parties.

It was quite the emotional roller coaster ride for me.  A business partnership is like marriage without the sex, and so we were both cautious, but it was very easy to get excited about working together and building a big business.  It was all the more exciting to me because he’d done this before.

Here’s what went right:

  • We had open conversations about each of our financial needs.
  • He built a budget and business plan.
  • We used Skype for conversations so that non verbal cues were available.
  • We worked together for a month before hand, which gave us some context.
  • We checked references and had our spouses meet.
  • We planned to get together and work face to face.
  • We used Google docs, which was a great way to share spreadsheets and documents about the business.
  • I reached out to a few mentors to get advice.  Every single person said: “write your expectations down”.  Every one.
  • On the advice of one of the mentors, I read the Nolo books about LLCs, partnerships and business buyouts.  These were tremendous.  Not only did they lay out scenarios I had never considered (what if one of the members of the partnership is disabled?  what if you want to bring someone new in?  what if you want to give time rather than money in exchange for equity?  and many many more), they also give you sample agreements.
  • We set deadlines, both for documents and for coming to a final decision.  We stuck to those.
  • We parted amicably.

And here’s what went wrong:

  • We put off explicitly valuing the existing company until late in the game.  Then it became clear we had pretty different numbers in mind.  We should have done this as soon as we were both interested.
  • We didn’t really know each other, the month of working together notwithstanding.  Just as for investing, I am beginning to think that partnerships work best with lines, not points.
  • The budget and business plan were limited in scope (one year out, then some major assumptions about future years).  Hard to make more detailed predictions about the future, especially since the plan called for major new products.
  • I was in a different state than he was.  This would have made finding common service providers (CPA, mediator, etc) difficult.  No real fix for this, other than one of us moving.
  • Valuing an ongoing, non profitable bootstrapped business is really hard, because most of the value of the business is in the future.  I found some articles, but didn’t find much about this particular scenario.
  • We didn’t really nail down whether this was a partnership, a buy in to an existing business or a valued employee relationship.  Each of these have different equity implications.
  • I didn’t sell myself as well as I should have.

All in all a great experience.  I learned a ton.  Of course, I would have been happier if we could have reached agreement, but I understand why we ended up where we did.


Hard Numbers About My Book

numbers photo

Photo by MrHicks46

I self published a Developing Cross Platform Mobile Applications with Cordova CLI last fall. It’s about the Cordova command line tool (Cordova CLI). Cordova is a way for web developers to build mobile apps that are distributed via the various app stores and can access app functionality (your list of contacts, for instance). The CLI is the main method of interacting with Cordova–managing your files, building apps, etc.

You can see my take on writing the book and marketing it, but I wanted to give some hard numbers on the results.

I thought I’d share some lessons from this experience. I started on 8/29/2013, had my first purchase, for $4, on 9/3/2013 (Leanpub lets users buy in progress books and get any updates), and published the 100% complete version on 10/4/2013. The book is 54 pages long.

Sales numbers as of today:

  • 56 readers
  • 47 readers who paid for the book
  • 9 readers who received free copies (reviewers and members of the Cordova community, because I am grateful for the work they’ve done)
  • 2 refunds (Leanpub has a 45 day no questions asked “Happiness Guarantee” refund policy)
  • Price range: $4-$20
  • Current price range: $20
  • Total royalties: $572.73
  • Number of people who paid over list price: 2

Here’s a graph of when purchases took place:

Sales By Week for Developing With Cordova CLI

Visits/conversions from 8/29/2013 to 9/9/2014 to the leanpub page (note that these don’t match up with the above numbers because if someone disables Javascript or cookies from Google, Google Analytics won’t track the purchase, and also I didn’t enable conversion tracking until Sep 18, 2013):

  • 5897 sessions
  • 4731 visitors
  • 37 book purchases tracked
  • 0.63% conversion rate
  • 53.75% abandonment rate
  • 84.92% bounce rate
  • 79.67% new sessions

Traffic sources:

  • 59.4% of sessions are from organic search
  • 13.5% from referrals
  • 13.2% from social
  • 10.6% direct
  • 2.3% from “other”
  • 0.9% from paid search

Highest referral sites for visits:

  1. Stackoverflow
  2. mooreds.com
  3. devgirl.org (I wrote a guest post)
  4. twitter
  5. google groups
  6. cloudfour.com (I wrote a guest post)
  7. hacker news

Highest referral sites for conversions (out of 14 conversions):

  1. Direct
  2. devgirl.org
  3. Google (organic)
  4. mooreds.com (embed and referrals combined)
  5. twitter

Pricing matters.  I varied the price, depending on how close the book was to being complete. The vast majority of my royalties came when I priced the book at $20.  That felt fair to me for a fifty page ebook about a technical topic. If someone charges $60/hour and this book saves them an hour, they have a 3x ROI (yes, I probably should charge more).

Overall, writing a book has been a great learning experience (and the couple hundred dollars in my pocket feel good too).  I learned about writing, research, and marketing. I learned that very few people buy an ebook and return it. I learned that sometimes a single sentence in a book can take you an hour to verify. I learned that it can be fun to write and promote your book. In retrospect, the only thing I’d change would be to write on an evergreen topic.  Cordova is moving fast enough that, while there is still value in this book, it needs a refresh every six months or so to remain relevant.


My Book Marketing Process

bazaar photo

Photo by -Marlith-

Last year, I wrote a technical book, focused on a niche tool of a niche technology. I posted about my writing process, but wanted to write about my marketing process as well. As my father, who has published a few books, says: “once you finish the book, the hard work [marketing] begins”.  After all, the world, and even the Internet, is a big place, and no one was going to beat a path to my door (or yours).

I tried several different marketing channels. Here’s a list and some commentary on how each channel worked in terms of traffic and sales.  My main goal was to drive traffic to my leanpub book page, where purchase was only a click away.

  • I published a blog post about the book and put an embedded ‘buy now’ link on my blog sidebar. I also did some follow up posts about Cordova CLI, including one about accessing more info from hooks in later version of Cordova and running multiple version of Cordova. Hard to tell if the additional content helped, but I had quite a few sales referred from my domain.
  • I set up a Google alert for ‘cordova CLI’ to see if there was anyplace I could reply and help people with questions while linking to my book page.
  • Monitored and answered Stackoverflow questions about Cordova and Cordova CLI–these typically also figured prominently in the Google alert. SO was the third highest referrer to my leanpub book page.
  • Google adwords: this didn’t work so well. I had 7K impressions and a 0.52% CTR but no conversions. I ran the campaign for about two months, but most of the keywords were pretty obscure.
  • Guest posts at on topic blogs: this worked great. I posted two different posts–Three hooks your Cordova/Phonegap application needs and Phonegap makes mobile app development more accessible. The more aligned the blog post was with the audience of the blog, the better the conversion. I also engaged in the comments section as well.
  • Emails to tech influencers (often with a coupon for a free or reduced price book). This was both marketing and a ‘thank you’ to all the people who either help out with Cordova, or who I have read for years and just wanted to thank (ahem, Matt Raible). It did result in some tweets. Thanks to Jason Grigsby, Brian Leroux and Holly Schinsky
  • Posting on the phonegap google group about the book, with a coupon. This didn’t drive that many visits, but the ones who did converted very well (>25%).
  • Posting on the phonegap google group answering general questions with a link in my signature. This drove some traffic and lead to one conversion. It didn’t scale very well, as when I stopped answering questions, the traffic fell to zero quickly.
  • Emails to members of the phonegap developer directory. I ended up sending out about 50 of these (both myself and hiring someone from Odesk). Surprisingly, I didn’t see any sales from these emails, which were admittedly “cold calls”.
  • Posted to hackernews. No discussion, but a non trivial number of visits.

Here are some things I should have done, or should have done sooner:

  • Set up a custom domain, like this person did.
  • Build a useful email list.  I actually have one, but it’s hard to find out how to subscribe.  Oh, and post to said list.
  • Written and flogged a talk about Cordova. There are probably two or three meetups in the Denver or Boulder area that would have been happy to have me talk, and I expect I would have sold some books through that venue.
  • Added the book to Amazon and Barnes and Noble. I just did that (links to Amazon, Barnes & Noble). Leanpub makes it silly simple to do this, however the royalty hit you take for the distribution is high.
  • Written more articles and pitched them to online magazines. In fact, I should have, when writing the book, queued up 3-5 articles to pitch to publish on launch day.
  • Emailed my readers via leanpub and asked how I could help them (I did this, belatedly).
  • Kept the book up to date. (I still may review, expand and update the book, because it is still a topic that interests me (and there is interesting stuff happening with Cordova, like Ionic and Google Glass support.)

I missed some of these marketing opportunities because my life just got too busy. Others because we stopped using Cordova at my day job, which meant I had less interest in keeping up with Cordova in general. And that, in turn, made it hard to keep up with marketing effort.

The final piece of advice I would share about marketing an ebook is that it is a long game. I realized that even a book on a topic so short lived as a Cordova book–it covers 2.9, 3.0 and 3.1, and the current version is 3.5–there are still ‘long tail’ effects. I make sales now, almost a year after first release.


Priorities and Focus

ben horowitz photo

Photo by TCDisrupt

I enjoyed Ben Horowitz’s ‘The Hard Thing about Hard Things’.  It was an in depth view of the lonely life of a CEO going through hard hard times.  I was only recently out of school when the dotcom bust happened, but I still remember layoffs–the company where I worked was going to be huge! “offices in Singapore and London!” until it wasn’t–which happened the week before Thanksgiving.

I cannot imagine what it was like to be in upper management in those times.

Must have been pretty wretched.

My favorite part of the book, though, is this section:

One very hot day my father came over for a visit. We could not afford air-conditioning, and all three children were crying as my father and I sat there sweating in the 105-degree heat.

 

My father turned to me and said, “Son, do you know what’s cheap?”

 

Since I had absolutely no idea what he was talking about, I replied, “No, what?”

 

“Flowers. Flowers are really cheap. But do you know what’s expensive?” he asked.

 

Again, I replied, “No, what?”

 

He said, “Divorce.”

 

Something about that joke, which was not really a joke, made me realize that I had run out of time. Up until that point, I had not really made any serious choices. I felt like I had unlimited bandwidth and could do everything in life that I wanted to do simultaneously. But his joke made it suddenly clear that by continuing on the course I was on, I might lose my family. By doing everything, I would fail at the most important thing. It was the first time that I forced myself to look at the world through priorities that were not purely my own. I thought that I could pursue my career, all my interests, and build my family. More important, I always thought about myself first. When you are part of a family or part of a group, that kind of thinking can get you into trouble, and I was in deep trouble. In my mind, I was confident that I was a good person and not selfish, but my actions said otherwise. I had to stop being a boy and become a man. I had to put first things first. I had to consider the people who I cared about most before considering myself.

It’s a beautiful vignette.

As a father with a lot of interests, this really resonated with me. Kids (or, I imagine, other major challenges such as a sick relative) take so much time that they force you to re-evaluate everything and think about your priorities and life focus.


My Book Writing Process

writing photo

Photo by mrsdkrebs

I wrote an ebook last fall. It’s about the Cordova command line tool (Cordova CLI). Cordova is a way for web developers to build mobile apps that are distributed via the various app stores and can access app fucntionality (your list of contacts, for instance). The CLI is the main method of interacting with Cordova–managing your files, building apps, etc. It’s a niche, but one that I felt passionately about. I love the command line and I love webapps. I find the whole native app experience distasteful. (“Didn’t we already watch this movie?”)

I thought I’d share some lessons from this experience. This post will be about the writing, the next one about the marketing, and the final one about the results.

I wrote a series of blog posts, gathered here, to determine if I would enjoy writing a book, and to give me a head start on the content. Those blog posts were primarily written in July when my wife was away visiting family. I figured that if I didn’t want to write the book, the blog posts would have value anyway–writing clarifies issues for me. If they were useful and shared, then I would have some market validation.

On August 1, weeks after publishing my first Cordova blog post, I received an email from a reader who said the posts were useful, and asked a question. This was enough validation that I decided to start the book. It was five weeks from creating the project on leanpub to completing it (late August to early October).

During the book writing, I focused on expanding on the blog posts to make sure they were authoritative, adding new content as I came up with questions, and verifying all the claims using the Cordova software. I remember spending an hour trying to confirm one part of one sentence in the book. I also spent time answering and asking questions on the Cordova/Phonegap Google group. I also monitored the Cordova developer list, more for interesting topics than for dialog. And a new version of Cordova (3.1) was released just a week or so before I was done with the book, so I spent some time double checking how the new version of the CLI worked.

As far as time, I averaged an hour a day for that five week period–remember, a good chunk of the content was already written and I was simply revising it. I found time on the weekends and in the early morning.

I ended up finding a flickr image to use for the cover page–thanks Marc!

In the end, the writing was fun and a grind all at the same time. You just have to make the time.



© Moore Consulting, 2003-2017 +