Machine sympathy vs human constraint

I had beers with an work acquaintance recently. He’s a developer of a large system that helps contact management. Talk turned, as it so often does in these situations, to the automation of development work. We both were of the opinion that it was far far in the future. This was three whole decades of experience talking, right? And of course, we weren’t talking our book–ha ha. I’m sure that artisan weavers in the 1800s were positive that their bespoke designs and craftmanship would mean full employment no matter what kind of looms were developed.

But seriously, we each had an independent reason for thinking that software development would not be fully automated anytime soon.

My reason:

It’s very hard to fully think through all the edge cases of development. This includes failure states, exceptional conditions, and just plain human idiosyncrasies. Yes, this is what every system must do. That’s right. Anything you want handled by an automated system has two options: plan for every detail or bump exceptional cases up to human beings to make judgements. The former requires a lot of planning and exercising the system, while the latter slows the system down and introduces labor costs into the mix.

This system definition is hard to do and hard to automate. I’ve seen at least five new languages/IDEs/software platforms over the years that claimed to allow a normal human being to build such robust automatic systems, but they all seem to fail in the short term. I believe that is because normal human beings just don’t think through edge cases, but those edge cases are a key part of software.

His reason:

When systems reach a certain size, abstractions fail (I commented about this years and years ago). Different size, different failures. But just as an experienced car mechanic knows what kind of system failures are likely under what conditions, experienced software engineers, especially those who understand first principles, have insight into these failures. This intution (he called it “machine sympathy”) is something that can only be acquired by experience, and, by its very nature, can’t be automated. The systems are so complex and the layers so deep that every failure is likely to be unique in some manner.

So, which one is more likely to remain a relevant issue. It depends on the organization and system size. Moore’s law (and all the corollaries for other pieces of software systems) works both for and against machine sympathy. For, because, as hardware gets better, the chances of system breakdown decrease, and against, because as hardware gets better, larger and larger systems get more affordable. Whereas I believe the human constraint is ever present at all sizes of system (though less present in smaller ones where there is less concern about ‘bumping up’ issues to humans, or even just not handling edge cases at all).

What do you think?

Early Product Lessons

fence-238475_640I wanted to jot down some lessons I’ve learned being an early stage technical founder of an unfunded startup, from no product or revenue -> product and revenue. (Of The Food Corridor, if you’re interested in the startup.) I had the luxury of a co-founder who had spent years immersed in the problem space and months researching the niche. If you can find that, it really really helps in product development.

That said, here are some other lessons. For an idea of our timeline, we did a build or buy or both evaluation in March, started building in April, did beta testing in May and launched June 1.


Determine features through demand/pull, rather than push

Once you have a product that you can show users, show it to them!

It will be embarrassing.  Record all their feedback and note patterns (we did a month of beta testing, as noted above). Then, let the user requests pull features from you, rather than push features to them. This serves a couple of purposes:

  • people will know that you are hearing them, and will be more forgiving of inevitable issues
  • you will build features that people want to use
  • you’ll develop a sense of users needs
  • you’ll learn to politely say no to requests that are off base/only useful to one user

Everything is broken

Everything is borked, all the time. At an early stage startup you just don’t have time to do everything right (nor should you, because the wrong thing perfectly engineered is a waste). So there will be features that are half done, or edge cases unhandled, or undocumented build systems. Do the best you can, and realize that it gets better, but make your priority getting something out that users can give feedback from. “Usage is like oxygen for ideas.” – Matt Mullenweg

You have to walk a fine line between building something quickly and building something that you can build on later.  Get used to ambiguity and brokenness and apologizing to the customer.  (But not too used to the apologies!)

UX/UI polish is relative

Our app is a number of open source gems smashed together with some scaffolded ruby code.  The underlying framework had a decent look and feel, but there are definitely some UI and UX holes.  I thought I’d have to spend some time working on those, but our customers thought the product was beautiful and useful.  My standards were different than their standards.

That doesn’t mean that the app can look horrible, but a plain old bootstrap theme or one of the other common CSS themes is ok. You need to know your audience–many people are stuck using a mix of software and are used to navigating clunky user interfaces. If your interface is just decent, but still solves the problem, you’ll be OK.  Of course, you’ll want to solve gross UX issues eventually, but a startup is all about balance.  A friend of mine gave me the advice: “don’t allow your users to make any mistakes”.

Favor manual process for complex edge cases

There have been a couple of situations during the build where a lot of work was needed to handle an edge case. For example, prorating monthly plans. Once you start thinking about prorating in depth, it turns out to be a really interesting problem with a lot of edge cases. But guess what?  For your startup, edge cases can be a wild goose chase.

When an edge case rears its head, you should consider the following options (in preferential order).

  • can you outsource the complexity (Stripe handles proration, for example, and I guarantee you they handle edge cases you don’t).
  • can you make it a manual process? If it doesn’t happen that often and/or a real time response is unneeded, you can often get by with a manual solution. This may be partly automated, for example, an SQL query that generates an email to a human who can handle the exceptional situation.
  • if neither of the above apply, can you defer it? Maybe for a few months, maybe for just a few weeks. But sometimes requirements change and you learn things from users that may make this edge case less important.
  • if all of the above don’t apply, you may need to bite the bullet and write code.

Back end and front end development doesn’t have to be synchronized

Most users equate the front end with the complete product. Most developers know that, just like an iceberg, there’s a lot of back end processing hidden in any project. But guess what? When you are getting feedback from users, some of the backend processes need to work, but many don’t. For example, we had a billing system that handled monthly invoices. We didn’t need to build the billing system while we were getting feedback from users on what type of charges they needed to handle. We did, however, need to know that we could build it. So make sure you can build the backend system to support your front end system, perhaps by building one path through, but defer the full build-out until you have to.

What about you?  Any tips for early stage product engineering?

How to call shell scripts from java properly

shell photo

Photo by dd21207

This has caused me some grief over the past few weeks, so here’s some tips on calling bash scripts from java correctly.

Use ProcessBuilder. Please.

Make sure you set the redirectErrorStream to true, or make other provisions for handling stderr.

I’ve found that inheritIO is useful. Not calling this caused a failure in a bash script that called other java programs.

Make sure your shell scripts have are executable, or that you call them with ‘bash scriptname’. Tarballs preserve permissions, zipfiles do not.

Read the output of the shell script, even if you do nothing with it (but you probably want to log it, if nothing else). If you don’t, buffers fill up and the process will hang:

final InputStreamReader isr = new InputStreamReader(p.getInputStream());
final BufferedReader br = new BufferedReader(isr);
String line = null;
StringBuffer sb = new StringBuffer();
while ((line = br.readLine()) != null) {

Check your exit values, and wait for your shell script to finish: (If looking for speed, or more complex process flow, write it in java.)

int exitValue = p.waitFor();
if (exitValue != 0) {
// error log

How to engage with recruiters

recruiter photo

Photo by JimsFlicker

If you are a software engineer or developer right now, you are in demand.  There’s a dearth of software developers, and so you are probably getting calls from recruiters.  This is location dependent–my understanding is that the SF market is white hot, and the market for software developers in Yuma CO is softer.

When I first was approached by a recruiter, I was flattered.  Here was a stranger trying to find me a job.  Quickly I realized that recruiters were not working for me (they get paid to place a candidate, so they are working for the company) and that most recruiters didn’t have a technical background at all.  This made me feel less charitable toward recruiters, and I have found that common amongst developers, especially now that they have so many job options.

Older and wiser, now I have a more balanced approach.  Here’s my tips for dealing with tech recruiters:

  • Communicate what I want.  This means knowing what I want in a new job or contract, which can include availability (both hours and start date), minimum rate, location, work condition (remote), technology, sector.  Tell the recruiter all this up front.  I often state: “hey, at this point in time, I’m rather picky, and I don’t want to waste your time.  Here is what would interest me: …”.  This lets them self select out quickly, before the “get to know you” phone call.  I should probably put this on my Hire Me page, now that I think about it.  Just need to make sure I keep this up to date.
  • Realize I am in in their database forever.  I gave my resume to someone in 2004 who I lived with who worked at a recruiting firm.  I still get calls from that same firm.
  • Make sure I am treated like a professional, and don’t put up with non professional acts.  Do they respond when I ask them to, answer my questions, and treat me with respect.  I actually had to hang up on a recruiter recently who, after I had told her I was busy and couldn’t talk right then, in an email and earlier on the call, continued to ask “just a few questions” about my skill set.  It wasn’t fun, but she was not being respectful of my time.
  • The other side of the coin is that I need to treat them with equal respect. That means being honest with them about needs and skills and responding to emails in a timely, respectful fashion.  It also means if I hear of someone with a needed skillset, doing an intro (wearing my informal recruiter hat).
  • Make the exchange of information a two way street.  Whenever I get contacted by a recruiter, they always want “5-10 minutes to talk to see how I can help you in your career”.  When I get them on the phone, I make sure to ask them questions about the market, other skills they’ve seen in demand, what specific technologies they’ve seen needs for, what kind of companies are hiring.
  • Thank them for supporting various meetups, if they do.  The local Java Users’ Group has pizza and soda provided every month by a recruiting firm.  This actually makes me more likely to respond to recruiters from that firm, because, while it’s not caviar, it’s nice to have dinner paid for.
  • Realize the difference between being sought for who I am (that’s being headhunted, and is reserved for senior folks and niches) and being sought for my skills.  I’ve never been in the position of having a stranger seeking me for who I am (I imagine internet famous folks like Matt Raible probably have), but I have had former co-workers do so.  Incidentally, the former is by far a better way to join an organization.
  • I keep my linked in profile up to date.  This makes it very easy to send recruiters a resume, should they ask for one.
  • I realize the tables may turn.  Right now, and for the near future, development is in great demand.  But I’ve been through one ‘software recession’ and I expect there will be more.  Beyond the basic dignity I ascribe to every human being, recruiters can get me a job.  So, except in the cases where recruiters aren’t respectful, I treat them with respect.
  • I don’t take random recruiter calls or requests on LinkedIn.  LinkedIn emails are different–if a recruiter has paid for a premium account, it’s easy enough to respond with a quick ‘not interested’ if the position isn’t for me.  But if a recruiter is trying to connect on LinkedIn without having met me at all, they are not serious.  If a recruiter is serious about getting in touch with me, I can be found in about a thousand different ways online.  I think recruiters who send LinkedIn requests are lazy and not worth responding to.
  • As above, I respond to recruiter emails, even if is just ‘no thanks’.  Again, basic human dignity.
  • Finally, I’m trying to cultivate a few good recruiters.  Whether these are referrals from a friend, or recruiters that I’ve met at an event, there are a few that stand out.  One recruiter (Dave from Technical Integrity) actually introduced me to a few of his friends so I could ask some exploratory questions about a different sector.  You better believe that I’ll recommend him and think of him when I have friends who are looking for work.  That company focuses on startups and full time employees, so I haven’t had the experience of working for them, but when my needs change, I will certainly see if they are looking.  The idea is to have the relationship last longer than a single contract.

Reviewing my list, I boil it down to:

  • recruiters are human beings, treat them like it
  • they don’t work for you
  • set your expectations and boundaries
  • software development is hot now, but won’t be forever

Any tips you have on engaging with recruiters?

Mis en place

Professional chefs have a concept termed ‘mis en place’.  While I know very few professional chefs, the idea appeals to me.  It basically means ‘keep your workplace in order, before you start doing anything important’.  (Of course, you can read about ‘mis en place’ more on Wikipedia.)

I find that this is very true of software development as well.  If you don’t have your workplace in order, you will suffer.  Now, the definition of ‘workplace’ varies–someone building a webapp has a different work environment than someone building software that will run on ASIC chips.  I’ll speak to my workplace, but I challenge you to review your workplace and see what, if anything, is out of sorts.

I’m going to break this up into three different segments.  First is the physical, then organizational, then the coding environment.

First off, the physical workspace.  This includes your desk, your monitor and computer setup, your work environment (headphones, bull pen), and your view.  It also includes your body–how you are sitting, how often you move around.  My current physical workspace is a desk with perhaps a bit too much clutter on it, a desktop machine (I know! they still sell them, apparently.  And you can get a lot of machine for not much money), and a view outside.  My work environment, since I’m working from home, is alternately morgue like and circus like, depending on who comes down to the basement.

Then there’s the organizational component.  Since I’m now contracting, this is something I have less control over, but it is important to organize your working environment in terms of the organization as much as you can.  Are you working on the right things?  Are the right people on the team?  Are they spending the right amount of time on the project?  In business speak, is everything aligned?  If the answer is no, how can you fix this?

Finally, there’s the actual hands on coding environment.  This can include everything from having a continuous deployment environment to your shell aliases to your software stack to your IDE (if you lean that way).  This is what I originally thought this post was going to be about, because this aspect of the workplace is where the rubber meets the road, so to speak.

Problems with any of these can cause a project to fail.  So make sure your work environment as clean, ordered, and ready to go as possible before you start a new project, or if you are in the middle of one, take some time to clean things up as you go.  You’ll be happier and more productive.

The Tragedy of Mobile Development

unicorn photo

Photo by Origamiancy

The web, for all its warts, pushed a major advance in software construction.  The separation of concerns.  This is the idea that, just as you wouldn’t have the same person paint a house as lay the foundation, you should have different people design the user interface and the internal guts of software applications.  The skillsets are simply different.

Since modern web applications exist in the browser and are primarily CSS and HTML, with some amount of javascript, it is fairly easy to split up development between designers and developers.  Designers could focus on beautiful, functional user interfaces and developers could focus on making the application work.  HTML and CSS were the common interface, and they weren’t complicated (neither is a full featured programming language).

This separation is more difficult to maintain since the rise of the single page application, which pushes a lot of logic and functionality into the browser with javascript, but is still possible.

But, the mobile device, with its native apps, is a huge step backwards.  Sure, there’s still plenty of scope for people who just want to make things work–APIs and server side logic are a large part of most mobile services.

But for the client side, it’s not enough for designers and UX folks to be fluent in the traditional tools.  They can still do mockups, but the UI for mobile devices is constructed in code (objective C or android java, primarily).  That means that the finer points of UI must be added by developers.  No longer can they be delivered by HTML and CSS.

And hence, the rise of the proverbial unicorn, who can do mobile development, design and UX.  And makes very good money, since there are very few of these.

Is this sour grapes?  Nope.  I know that one of my limitations is my aesthetic sense.  (If I harbored any illusions about this, my wife would be quick to disabuse me.)  And I think that the unicorns should ride this market for all it is worth.  It just frustrates me that we came so far with web applications and separate of concerns and now are taking a huge step back.

Denver Bootstrappers Lunch

boots photo

Photo by liftarn

My friend and former colleague Corey Snipes has been working to get a Bootstrapper’s meetup off the ground in Denver. This is a small group (limited to 12, I believe) of people who are building products (typically software) and self funding. I believe most of the members are in the solopreneur mode (I know Corey is).

I imagine this kind of support group would be fantastic–certainly I had a similar group when I was a consultant in the past, and bouncing ideas off of others in similar situations made the struggle much easier.

I’ve not made this meetup yet because, a) I’m not sure I’m bootstrapping (and you know what, if you aren’t sure you’re bootstrapping, you aren’t bootstrapping!), b) I live in Boulder and Boulderites have a hard time leaving the Boulder Bubble, and c) Wednesdays in general are tough days for me to do anything outside of the house.

If you are a bootstrapper in the Denver area, take a look.

© Moore Consulting, 2003-2017 +