The power of automated testing

It took me a long time to understand the power of automated testing.  After all, it can end up being a large portion of your codebase and can be brittle.  Sometimes it feels like writing tests “gets in the way” of getting things done.  At one project I worked on, a colleague complained that it felt like you spent 5 minutes changing the production code and an hour changing the tests.  (And to be fair, sometimes that’s true, and there’s a balance to be struck between test code coverage and speed of development.  This can also indicate you need to spend time refactoring your tests, as you have multiple different test components testing the same production code.)

I like to think of tests like a gentle swaddling of your code.  It conforms as the body of your code changes, but changing that code does require some re-work of the tests.  And, if your code fails, it fails into the gentle swaddling, as opposed to the cruel outside world (bleeding all over your production users).  Alright, maybe the analogy fails :).

I write this today because I’m in the middle of a refactor of one of the scariest bits of The Food Corridor.  (Given we’re so young, it’s not that scary, but it’s quite complex–handling the creation and updating of bookings.)  There are many many paths through the code and if I didn’t have automated testing, I’d be far more worried about the changes I’m making.

So, consider this blog post to be a thank you to past me for making future me’s life easier by writing a comprehensive automated test suite.  If you don’t have one, you should.


Rails Gems: paper_trail

Rails has an amazing number of quality libraries (called ‘gems’) and while building The Food Corridor application, I was lucky enough to be able to use some of them.

If you want to easily track changes in your rails models over time, the paper_trail gem has you covered.  Simple to install, compatible with many databases, and very easy to use.  I’d point you to the docs for installation, but for usage, all you have to do is drop has_paper_trail into your model class.

After that’s deployed, every change to those models is tracked, including the entire model state.  From an operational perspective, I’m a bit worried about the size of the audit table (called versions), but thus far it hasn’t grown too fast.  It wouldn’t be hard to prune either.

I use the data this gem records for two reasons.  The primary purpose is to debug the system.  When there’s an issue because the system doesn’t act as expected, or a question on when a model changed, I can go back and see how the model has changed over time.  I can do this either by directly inspecting the versions table or via the rails console.  If I use the latter, then I can actually pull back the model object as it existed in the past.  This has been tremendously helpful in tracking down bugs.

The secondary use of this data is to allow the system to see changes over time, and execute code based on those changes.  This doesn’t happen often, and I’d prefer for such changes to be captured more explicitly, but in at least one case this gem has allowed me to fix an issue in a time efficient manner.

If you’re building a complicated system that changes over time, as most real world applications do, having some kind of audit trail can be extremely helpful.  paper_trail makes it trivial to get an audit trail set up in your rails app.


Useful Tool: Delighted for NPS reports

So, I was looking to add a simple widget to collect net promoter scores (often called NPS).  I was astonished at the dearth of options for standalone NPS tracking.  I assume that most customer relationship software has an NPS tracker embedded in it, but we didn’t want to use anything other than a simple standalone widget.

Two options that popped up: Delighted and Murm.  I did a quick spike to evaluate each of these tools.  At the time I reviewed it, I couldn’t get Murm to work, and they didn’t respond to my customer support request.  Delighted, on the other hand, did so.  They let me have access to web display, which was a beta feature at the time and worked with us on price.  It was trivial to install, though I’m still a bit unclear how the form determines when to display.  Highly recommended.

The nice thing about having a NPS tracker on your website is you can get direct feedback from your users.  This has led to numerous useful conversations and feature requests, as someone who is using our software for the first time brings new clarity to confusing features or user interface.  Plus, it is a great number to track.


Talking to your customers

One of my favorite parts of The Food Corridor is talking to customers.  As the main technical force there, it’s a great opportunity for me to interact with folks whose lives my work is (hopefully) making better (and sometimes making worse).

The two main ways I do this:

  1. I do customer service.  We have zendesk and a common email inbox and I take time away from developing to answer emails.  This gives me a feel for the rough edges of our product and helps me build empathy for our users (“why couldn’t they see that you just click here and then here and then… oh, that’s why”).  It also has led to a number of bug reports that make the product better.  I also answer phone calls from our google voice number.
  2. I schedule a monthly meeting with some of our bigger customers.  These tend to be 15-60 minutes long.  This meeting lets me hear directly from them what they like about the platform, and more importantly, what is missing or broken.  I rotate among a number of different customers because I don’t want to be pulled too far in one direction, and they all have slightly different needs, but hearing from them regularly helps me triangulate.  It’s important to capture what they say in some kind of tracking system, even if you don’t execute against them for a while.  This in person call also lets me let the customer know of certain other features that are new and/or may be of interest.  Frankly, this meeting can be exhausting because of the wish list aspect of it (“oh man, what would it take to implement that?”) but I try to avoid that and just be an open listener.  I think that the customers also enjoy direct access to a developer.  This certainly doesn’t scale as well as option #1.

If you are going to pursue this:

  • make it a priority and realize it is going to affect your ability to deliver code
  • don’t get defensive when your product is criticized
  • take notes
  • seek out customers with a variety of perspectives
  • don’t commit to anything new in this call, but do let them know high level roadmap if they ask
  • ask them about items outside of your product if you have time.  This can clue you in to other problems they may have.

Customer service a different activity from developing software.  It’s very choppy, and you can encounter folks that are … having a rough day and perhaps taking it out on you.  But it also is one of the best ways to make sure that you, the person building a solution, stays in touch with the people using the solution.


Things I wish I knew about Stripe

Caterpillar

Striped, but not charging your credit card

So, at The Food Corridor, we’ve been using Stripe happily since we launched in June of 2016.  As a developer, I’d used Stripe before in a couple of different ways, but this has definitely been my most sustained use of the payment service.  (If you don’t know what Stripe is, it is an API that makes charging customers as easy as an API call.  More here.)

I wanted to outline some of the things I’ve learned from months of using Stripe.

  • Stripe supports pulling money directly from bank accounts, via ACH, but it really isn’t the same ACH as your bank lets you do.  This is because Stripe isn’t a bank.  The biggest thing to be aware of here is that Stripe ACH takes 7 days to arrive in your bank account.  Another issue is that you have to do verification.  They have two ways of doing verification–micro deposits and Plaid.  Plaid is instant, but only supports major banks, which was a non starter for us (updated 9/8: Plaid supports around 1000 banks now).  The code for micro deposits is straightforward, but be prepared for some customer support issues.  Stripe deposits two amounts and withdraws just one amount, which was confusing for some of our users.  It also takes a couple of days, so if your users are hot to spend money, Stripe ACH may not be a fit.  The win?  Definitely cheaper.  (And I didn’t find any other service that would support both credit card and ACH transactions that was developer friendly.)
  • Don’t forget to set up your webhooks out of the gate.  Stripe mentions this, but I glossed over it in the early days, and missed some events that were important.  (The most relevant is that ACH is asynchronous, so when an ACH transfer fails, it is reported via webhook.  If bank account verification doesn’t work, you’ll get a different kind of webhook.  Review the docs and set up webhooks for all the ACH events.)  If you don’t have time for a full featured webhook processing implementation, Zapier can just send the webhook data to your email. This can be a great stopgap solution.  Or you can use stripe_event.
  • Per support, if a webhook post fails (because your app is down, for example), they are retried once an hour for 72 hours.
  • Speaking of stopgap solutions, the Stripe Dashboard is fantastic for manual processes.  Just because you can automate everything via an API, doesn’t mean you should.  There can be some complicated edge cases with payment processing, especially around refunds, but they can easily be handled with a google doc of instructions and the Stripe Dashboard.  I have found only one use case that the API can handle that the dashboard cannot (a partial refund of an ACH transaction).
  • I have found Stripe support to be excellent, quick and knowledgeable.
  • Occasionally customer charges will be declined because of bank fraud triggers.  Expect to occasionally ask your customers to call their bank.  (I think this has happend about once every third month).
  • Disputes are a total pain, because the process is opaque and slow (expect a resolution in about two months and know you are not in possession of the payment during that time).
  • Make sure to capture the payment id anytime you charge a card or run ACH.  It will make future automation a lot easier.
  • Monthly plans are complicated, so if you can lean on Stripe for management, even if you are doing manual plan management (applying coupons, adding, or removing users from plans via the dashboard), do that.
  • The first payment you charge takes 7 days to move from stripe to your​ bank account.  This is for fraud protection.  Payments thereafter typically take 2 days (but it depends on your country and industry).

And here are some special tips if you are using Stripe Connect (their marketplace product).

  • Read the docs!
  • Remember that first payment timeline?  It applies to every one of the connected accounts.  Think about charging your own credit card as soon as you connect an account to help with customer cash flow.
  • Consider whether you want to use managed vs standalone accounts.  Managed accounts are a lot more work but allow you to have a seamless UX that you control.  Standalone accounts, which we use, are far quicker to setup.  I think this depends on the number of sellers you have in your marketplace.
  • You also want to think about whether to place the charges on the platform account or on the connected accounts.  A major factor there is who bears the Stripe fees, the platform or the sellers.  We charged on the platform account because we wanted all our data in one place.  If you are selling plans, you can’t charge on the platform and use Stripe plans.
  • If you are charging on the platform account, and are using standalone accounts (where the sellers have to set up a stripe account) your sellers won’t see charge descriptions unless you manually copy the description over.  The code looks like:

# this will let the sellers know what invoice the charge was for
transfer_id = charge.transfer
transfer = Stripe::Transfer.retrieve(transfer_id, expand: ['destination_
payment'])
payment_id = transfer.destination_payment
payment = Stripe::Charge.retrieve(payment_id, {stripe_account: destinati
on_account_id})
payment.description = description
payment.save

Happy charging!


Advice for the bootstrapped developer

I participated in a panel for Boulder Startup Week, but previous to the panel, thought I’d be giving a 15 minute presentation.  I was going to talk about rules for the bootstrapped developer.  At The Food Corridor, we recently celebrated one year of having customers.  In celebration of that, here are my seven guidelines:

  1. It will take longer than you think.  For any meaningful value of “it”. expect the process to take longer than planned.  Whether that is acquiring your first customer, building your product, finding help, or anything else, plan for it to take longer than you think.  Heck, this blog post is taking longer than I thought it would.
  2. Know your runway.  It’s important to know your financial runway (both for the company and for yourself).  This is fairly easy to calculate–just find out your burn rate and your money in the bank.  If you are looking at your personal financial runway, don’t forget to allow some buffer for you to find a different job–typically you won’t be able to step from a cratered startup on Friday into a full time job on Monday.  However, more important than financial runway is emotional runway.  Talk about the stress with your spouse (if applicable), think about the other emotional difficulties you may encounter, and plan for some high highs and some low lows.
  3. Extend your runway.  Again, for the financial runway, lower your burn rate as much as you can.  This will mean cutting back on spending and savings.  Make sure the family is on board.  Then you will look to find other sources of money to feed and clothe yourself.  This may include digging into savings, borrowing against assets (HELOCs work for this), borrowing from relatives, pitch competitions (TFC won two), selling assets, taking contract work, having a spouse who earns enough for the household, and/or moonlighting.
  4. Talk to your customer. This was one of the great assets that one of my co-founders brought to the table.  She had deep domain expertise and had many connections to potential customers.  We had numerous customers give us feedback, including via interviews, a month long beta test, regular product advisory councils and surveys.  You will note that this point implies you can find your customer and communicate with them in a cost effective manner.  Find where your customer congregates online.  If you don’t find a place, make one and invite your potential customers to join.
  5. Everything is borked all the time. In a startup, you just don’t have time to do everything correctly.  It can be embarrassing, but if you are building something that solves customer pain, and you’ve found the right early adopters, the customers will stick around.  Just keep improving things.  And accept a certain amount of brokenness.
  6. Know your market. This is related to talking to your customer, as that is one fantastic way to learn about your market.  There are other ways too–market research, online reading, etc.  However, as you build your product, you will surprised by what your market wants.  For instance, we thought our market would want a better UX, but were surprised when someone said our v1 UX was “great”.
  7. Make your own rules. Know that my advice and rules above are based on my experience, with my co-founders, product and skillset, in this time.  You need to do the reading and figure out what your rules are.

Using Amazon Mechanical Turk

chess-1215079_640So, after over a decade, I finally found a use case where I had the clout and the need to use mechanical turk. I wanted to write about my experiences.

What I used it for: We were looking for some data on businesses.  We had business name, city and state, and wanted full contact information.  We paid a dime for each listing, and asked for email address and physical address.  We asked about each listing twice so that we’d have some kind of double check.

How effective was it? This varied.  If you were using the master workers, it was very effective, but slower.  If you open it up to all workers, you have to review their work more closely.  The few times I rejected someone’s task, they wrote back and asked why and tried to make it right, which was a testament to the power of the system (it records rejections).  Make sure you break the work into a couple of smaller groups so you can iterate on your instruction set (when workers asked questions on the first set, the answers went into the instructions for the second set).  We still had to review all the listings and double check any that didn’t match between both task answers, but that was a lot quicker than googling for each business and doing the research ourselves.

How much did it cost? On the order of a couple hundred bucks to process around fifteen hundred listings.

What kind of time savings did we see? Assume we had 1500 business names, and it took us 90 seconds to google the business name and find the information.  That is 1500 listings * 1.5 minutes == 37.5 hours, and this is on the low end.  Instead, it took about 2-3 hours of setup, and then 36 hours of calendar time (when I was able to do other things like sleep and work on other problems), and we were done.  Then I would say it was about 7-10 hours of review. So you are trading a couple hundred bucks for at least 20 hours of saved time.

Would I do it again? I think mturk is perfect if your problem has the following three attributes: more money than time, a task that is extremely simple, and time to review the finished product.

Other tips? You have to build it some kind of sampling for correctness. I have no idea what the quality is if you pay more than a dime per task.  Make sure you think about edge cases.  Provide tips to your workers (“check whois records as well as google”).


Bare minimum of ops tasks for heroku

Awesome, you are a CTO or founding engineer of a newborn startup.  You have an web app up on Heroku and someone is paying you money for it!  Nice job.

Now, you need to think about supporting it.  Heroku makes things way easier (no racking and stacking, no purchasing hardware, no configuring apache) but you still to set up some operations.

Here is the bare minimum you need to do to make sure you can sleep at night.  (Based on a couple of years of heroku projects, and being really really cheap.)

  • Have a staging environment
    • You don’t want to push code direct to prod, do you?
    • This can be a free dyno, depending on the complexity of your app.
    • Pipelines are nice, as is preboot.
    • Cost: free
  • Have a one line deploy.
    • Or, if you like CD/CI, an automatic deploy or a one click deploy.  But make it really easy to deploy.
    • Have a deploy script that goes straight to production for emergencies.
    • Cost: free
  •  Backups
    • User data.  If you aren’t using a shared object store like S3, make sure you are doing a backup.
    • Database.  Both heroku postgresql and amazon RDS have point and click solutions.  All you have to do is set them up.  (Test them, at least once.)
    • Cost: freeish, depending on the solution.  But, user data is worth spending money on.
  • Alerting
    • Heroku has options if you are running professional dynos.
    • Uptimerobot is a great free third party service that will check ports every 5 minutes and has a variety of alert options.  If you want SMS, you have to pay for it, but it’s not outrageous.
    • Cost: free
  • Logging
    • Use a logging framework (like slf4j or the rails logger, and mark error conditions with a string that will be easy to search for.
    • Yes, you can use heroku logs but having a log management solution like papertrail will make you much happier.  Plus, it’s free for 2 days of logfiles.
    • Set up alerts with papertrail as well.  These can be more granular.
    • Cost: free
  • Create a list of third party dependencies.
    • Sign up for status alerts from these.  If you have pro slack, you can have them push an email to a channel.  If you don’t, create an alias that receives them.  You want to be the person that tells your clients about outages, not the other way around.
    • Cost: free
  • Communication
    • Internal
      • a devops_alert slack channel is my preferred solutions.  All deploys and other alerts go there.
    • External
      • create a mailing list for your clients so you can inform them of issues easily.  Google groups is fine, but use whatever other folks are using.  Don’t use an alias in your email–you’ll forget to add new clients.
      • do not use this mailing list for marketing purposes, unless you want to offload the burden of keeping the list up to date to the marketing department.
      • do make sure when you gain or lose clients you keep this up to date
    • Run through a disaster in your mind and make notes on how you would communicate the issue, both internally and externally.  How often do you update your team?  How often do you update your clients?  What about an internal issue (some of your code screwed up) vs an external issue.  This doesn’t need to be exhaustive, but thinking about it ahead of time and making some notes will help you in the crisis.
    • Cost: free

All of this is probably a four hour project, max.

But once this is done, you’ll rest easier at night, knowing you have what you need to troubleshoot and recover from production issues.


Extending an existing Rails application that wasn’t meant to be extended

I am modifying an existing open source rails 4.2 app and wanted to keep my changes (some of which are to models, some to controllers, some to views) as separate as I can, so that when a new release of the app comes out, I won’t be in (too much) merge hell.

This app was not designed to be extended, which makes things more interesting.

For the views, I’m just doing partials with a prefix (_xxx_user_cta.haml).

For the models and controllers, I started out hacking the code directly, but after some digging around I discovered how to monkey patch (I believe that is what it is called) the classes.

In the config/application.rb file, I added the following code:

config.to_prepare do
Dir.glob(Rails.root + "app/decorators/**/*_decorator*.rb").each do |c|
require_dependency(c)
end
end

And then, if I want to make a change to app/models/person.rb, I add the file app/decorators/models/person_decorator.rb. In that file is something like this:

Person.class_eval do
# ... changes
end

This lets me add additional relations, helper methods, and other classes to extend existing functionality. I try to prefix things with a unique identifier (xxx_set_timezone rather than set_timezone) to lessen the chances of a collision, because if a method is added to the Person class with the same name as a method in the decorator, the decorator will win.

Write tests around this new functionality so that if anything changes, I’m aware and can more easily troubleshoot.

The downsides of this approach is that it is harder to track logic, because instead of everything in one file, it is now in two. (I don’t know if there are memory or performance implications.) However, that is a tradeoff I’m willing to make to make it easier to keep up with the upstream development and to pull said development in as often as possible.

I’m still fairly new to rails and didn’t know if this is the only or best way, but thought I’d share.


Joining The Food Corridor

After I left 8z, I contracted for about a year and a half. It was great fun, moving between projects, meeting a lot of new developers, and learned a lot of new things. I worked on evaluating software products and processes, supporting machine learning systems, large workflow engines, and, most recently, backend systems to stop distracted driving (they’re hiring, btw).

But I saw an email from Angellist early this year about a company looking to build a marketplace for kitchen space. (Aside: if you are interested in the labor market for startup professions, Angellist emails are great–they not only give you the company name and job description, but also typically include equity and salary–very useful information.) I replied, the conversation started, and I did some research on the company. They were pre-revenue, but the founder had been grinding it out for months and had an extensive background in the industry. Was clear they weren’t a fly by night, “we just had an idea for an app and need someone to build it” operation.

After discussions, interviews and reference checks, it became clear that this was a fantastic opportunity to join an early stage startup as a technical co-founder. So, I’m thrilled to announce that I have joined The Food Corridor as CTO/Co-Founder.

Why does this opportunity excite me so?

  • By increasing visibility and availability of shared kitchen space, it can grow the local food system, especially value add producers, across the country
  • There’s a real need for some innovative software and process solutions that we can solve
  • The founding team has the diverse set of skills needed to run a great company
  • I am looking forward to learning about the business side of a software company
  • It’s right at the intersection of two of my passions–food and technology

If you are interested in following along with the TFC journey, there’s a monthly newsletter that will be focused on shared kitchen topics.

As far as the blog, I expect to be heads down and building product, but will occasionally pop up and post.

Here’s to new adventures!



© Moore Consulting, 2003-2017 +