Heroku 503 errors

One day recently I woke up to a text from our monitoring service (part of minimal heroku operations tasks) saying our application was down.

Darn it.

Here’s the recreation of my steps in troubleshooting this issue:

  • Login, see that the app is indeed down.
  • Look at newrelic.
  • Look at the logs (papertrail!).
  • Look at the deployment history.
  • Note when the issue started–curses, not when I did a deploy.
  • Open a ticket with heroku (after doing some research). (Love their support.)
  • Double check that the database is good and hasn’t hiccuped.
  • Look at the logs more.
  • Add more dynos, see if that helps.
  • Google the error message.
    •  <app-name> heroku/router: at=info method=POST path=”<url>” host=app.thefoodcorridor.com request_id=8a17648f-2d84-46ea-abf2-5903be894a2c fwd=”″ dyno=web.3 connect=1ms service=4ms status=503 bytes=477 protocol=https“Notice that the issue is being stated right in the log file (passenger request queue filling up).  Here are sample error messages?
    • <app-name> app/web.3: [ 2017-07-12 14:47:44.3688 65/7f652dffd700 age/Cor/Con/CheckoutSession.cpp:261 ]: [Client 3-281] Returning HTTP 503 due to: Request queue full (configured max. size: 100)
  • Find some posts about the error message.  Here and here.
  • Start researching how to increase request queue size.
  • Talk a walk to clear my head.
  • Think about what external services we call, as that seems to be what might cause the request queue to back up.
  • Read another post that says restarting passenger helped.
  • Restart all dynos.
  • Problem disappears.
  • Look at logs more closely.
  • Last dyno to be restarted was the only problematic dyno.
  • Add comment to ticket about this being the cause.
  • Heroku confirms that the issue may have been the dyno: “sometimes individual dynos will hang and cause errors with 503 responses”
  • Write note to customers about the issue explaining how access to app was affected.
  • Lower number of dynos.
  • Breath a sigh of relief.

Interview with a boot camp grad

Lots of folks are moving into development and technical fields these days.  I remember that happening during the dot com book, but back then folks just read one of those “Learn Java in 24 Hours” books.

Nowadays there are a profusion of boot camps that help people gain the skills they need to be a developer. I have interacted with a few of these grads and was interested in learning more about their experience. Noel Worden, one of the organizers of Boulder.rb and a blogger, agreed to an interview.

– what was your background before you were a developer?

I got my degree in fine art photography, spent 5 years working in NYC as a Digital Technician in the photo industry, then moved to Colorado and was a cabinetmaker for 3 years.

– how did you land your first job?

I found the posting on the Denver Full Stack meetup whiteboard (https://www.meetup.com/fullstack/)

– what surprised you about the software industry?

How willing everyone is to help. It’s very different in the photo world, [which is] much more competitive, it’s hard to find guidance and mentorship.

– how did you pick your boot camp?

Bloc had one of the longest curriculums I could find in an online program. I figured when push came to shove, having more experience under my belt couldn’t hurt when competing with other junior [developers] looking for that first job. Bloc also has a part-time track, which allowed me to still work [a] full-time job while going to school.

– what is good about the job? What is challenging?

I appreciate the balance of being challenged but knowing I have the full support of any other engineer on the team if I need to reach out for assistance. Canvas United [ed: his current employer] has a lot of projects that I’ve been maintaining lately, and all are running different versions of Rails, which makes for interesting challenges. 

– what do you see current boot camp students doing that you’d advise against?

Not getting out and networking while going to school. This industry is all about networking and if you’re hoping to capitalize on the advantages of a good network you have to be building it while still in school.

– why did you want to transition into technology and development?

I wanted/needed a career path where the leaning curve wouldn’t plateau. I’m not a ‘cruise control’ kind of person, as soon as the challenge isn’t there for me anymore I lose interest.

– how can employers help boot camp grads in their first job?

Be ok with answering questions, and be transparent with the employee about the proper channels to ask those questions. Also, be upfront about the fact that it’s ok to fail (assuming that’s the case) a few time before you get to the best solution [ed: if it isn’t ok to fail, find a new job!]. Also also, a healthy balance of low hanging fruit and multi-day problems. Nothing kills my morale faster than ticket after ticket of problems that grind me into the ground.

– should employers have any different expectations of a boot camp grad vs someone who just graduated from college or high school?

I definitely think that it takes a particular type of management style to successfully level up a boot camp grad. If you’ve hired them you must have liked something about them, play to those strengths, but also sprinkle in challenges to help that developer evolve.

[This content has been edited for grammar and clarity.]

Testing Dossier Reports in Rails

One of the things I love about developing with rails is the vast array of open source, free components that you can drop in and extend your application.  Want an invoicing system?  Way to run your javascript testsSimple admin portal?  Just drop in a gem, run bundle install and you are good to go.

One of the gems I’ve used recently was dossier, which lets you write reports in SQL or active record, and then generate them in HTML or CSV (or JSON, but I didn’t use that). One tip–if you want your CSV results to have the same formatting as your HTML results, you’ll want to follow the steps on this issue.

I wrote up a couple of SQL reports, linked them into the appropriate admin pages, and called it good. Then, the app moved on, and at one point, the schema changed. (Some of you are shaking your head, knowing what is going to happen next.) Then, the reports failed.

I had forgotten the cardinal rule–write the tests first. I confess, I wasn’t sure how to, but a bit of research revealed that it wasn’t that hard. Here’s one of my spec files.

require 'spec_helper'

describe MonthlyHoursClientReport do

  # all this does is test that the SQL is valid
  it "sql valid" do
    report = MonthlyHoursClientReport.new
    sql = report.sql
    sql = replace_placeholders(sql)
    expect{ActiveRecord::Base.connection.execute(sql)}.to_not raise_error

  def replace_placeholders(sql)
    sql = sql.gsub(":kitchen_id",1.to_s)

This just gets the SQL from the dossier report and tries to execute the SQL in the test database. Super simple, but enough to catch the error I encountered. If/when I get more time, I could definitely add some more tests with some data in the test db to make sure the SQL is giving correct results, but I tend to be pretty confident in my SQL queries, especially when they don’t have any group by or having clauses.

Anyway, happy testing.

Updating Stripe bank accounts: “A bank account with that routing number and account number already exists for this customer.”

So, if you want to handle ACH transactions with Stripe, you can. Some limits to include the length of time for the transaction (5 business days on top of stripes 2 business day transfer duration) and support only for US accounts, but the API is nice and the price is pretty nice too (0.8% up to $5).

But if you are trying to do recurring billing with Stripe and ACH and you want to let your customer change their default charge source between credit card and bank accounts as a payment source (or even two different bank accounts), you’re going to run into a roadblock. While Stripe will happily accept new credit information with the exact same card number, expiration date and CVC code, and just create a new source for each entry, it is not so forgiving with bank accounts. Instead, you’ll get this error message: "A bank account with that routing number and account number already exists for this customer." if you try to change the default source to an existing bank account record in Stripe.

I found some code with this error message, but it actually isn’t complete. It’s not best to examine the error message and switch on that, but I didn’t see a specific exception class for this type of exception.

For a complete solution, you need to check the stripe tokens routing number and last 4 digits of the account number. If a user has two different bank accounts that match both in the last 4 of the account number and the routing number, well then, I think you are out of luck.

Here’s the complete ruby code, making sure to match the current request’s routing id number just in case your user wants to switch between multiple bank accounts for their default charge.

    def update_customer_from_token(customer,stripe_token)
      # takes the stripe customer object and the new token 
      # from the stripe indicating the changed payment method

      success = false
      Stripe.api_key = ENV["stripe_secret_key"]
        new_pmt_obj = customer.sources.create({:source => stripe_token})

        customer.default_source = new_pmt_obj.id
        success = true
      rescue Stripe::InvalidRequestError => e
        # special case where the bank account already exists, let's use that.
        if e.message == 'A bank account with that routing number and account number already exists for this customer.'
          tokobj = Stripe::Token.retrieve(stripe_token)
          customer.sources.each do | src |
              if src.object == 'bank_account' && src.routing_number == tokobj.bank_account.routing_number && src.last4 == tokobj.bank_account.last
                customer.default_source = src.id
                success = true
            rescue => e
              Rails.logger.error(STRIPE_ERROR_PREFIX+" 4 unable to update customer for "+customer.inspect+", "+e.inspect)
          Rails.logger.error(STRIPE_ERROR_PREFIX+" 3 unable to update customer for "+customer.inspect+", "+e.inspect)
      rescue Stripe::CardError => e
        Rails.logger.error(STRIPE_ERROR_PREFIX+" 1 unable to update customer for "+customer.inspect+", "+e.inspect)
      rescue => e
        Rails.logger.error(STRIPE_ERROR_PREFIX+" 2 unable to update customer for "+customer.inspect+", "+e.inspect)

Or, you could just let the user choose from a drop down list of their existing sources which one they want to be the default. That might be a cleaner solution.

Presenting on Stripe tomorrow

Excited to say that I’ll be presenting on my company’s use of Stripe at Boulder.rb tomorrow.  Working title of my talk: “Do you like Money?”.

What I’ll cover:  an overview of the service, real code, testing, operational practices, and gotchas.  This will be based on my experience at The Food Corridor, where we’ve processed over $250k of transactions through Stripe.

Hope to see you there!

Extending an existing Rails application that wasn’t meant to be extended

I am modifying an existing open source rails 4.2 app and wanted to keep my changes (some of which are to models, some to controllers, some to views) as separate as I can, so that when a new release of the app comes out, I won’t be in (too much) merge hell.

This app was not designed to be extended, which makes things more interesting.

For the views, I’m just doing partials with a prefix (_xxx_user_cta.haml).

For the models and controllers, I started out hacking the code directly, but after some digging around I discovered how to monkey patch (I believe that is what it is called) the classes.

In the config/application.rb file, I added the following code:

config.to_prepare do
Dir.glob(Rails.root + "app/decorators/**/*_decorator*.rb").each do |c|

And then, if I want to make a change to app/models/person.rb, I add the file app/decorators/models/person_decorator.rb. In that file is something like this:

Person.class_eval do
# ... changes

This lets me add additional relations, helper methods, and other classes to extend existing functionality. I try to prefix things with a unique identifier (xxx_set_timezone rather than set_timezone) to lessen the chances of a collision, because if a method is added to the Person class with the same name as a method in the decorator, the decorator will win.

Write tests around this new functionality so that if anything changes, I’m aware and can more easily troubleshoot.

The downsides of this approach is that it is harder to track logic, because instead of everything in one file, it is now in two. (I don’t know if there are memory or performance implications.) However, that is a tradeoff I’m willing to make to make it easier to keep up with the upstream development and to pull said development in as often as possible.

I’m still fairly new to rails and didn’t know if this is the only or best way, but thought I’d share.

Early Product Lessons

fence-238475_640I wanted to jot down some lessons I’ve learned being an early stage technical founder of an unfunded startup, from no product or revenue -> product and revenue. (Of The Food Corridor, if you’re interested in the startup.) I had the luxury of a co-founder who had spent years immersed in the problem space and months researching the niche. If you can find that, it really really helps in product development.

That said, here are some other lessons. For an idea of our timeline, we did a build or buy or both evaluation in March, started building in April, did beta testing in May and launched June 1.


Determine features through demand/pull, rather than push

Once you have a product that you can show users, show it to them!

It will be embarrassing.  Record all their feedback and note patterns (we did a month of beta testing, as noted above). Then, let the user requests pull features from you, rather than push features to them. This serves a couple of purposes:

  • people will know that you are hearing them, and will be more forgiving of inevitable issues
  • you will build features that people want to use
  • you’ll develop a sense of users needs
  • you’ll learn to politely say no to requests that are off base/only useful to one user

Everything is broken

Everything is borked, all the time. At an early stage startup you just don’t have time to do everything right (nor should you, because the wrong thing perfectly engineered is a waste). So there will be features that are half done, or edge cases unhandled, or undocumented build systems. Do the best you can, and realize that it gets better, but make your priority getting something out that users can give feedback from. “Usage is like oxygen for ideas.” – Matt Mullenweg

You have to walk a fine line between building something quickly and building something that you can build on later.  Get used to ambiguity and brokenness and apologizing to the customer.  (But not too used to the apologies!)

UX/UI polish is relative

Our app is a number of open source gems smashed together with some scaffolded ruby code.  The underlying framework had a decent look and feel, but there are definitely some UI and UX holes.  I thought I’d have to spend some time working on those, but our customers thought the product was beautiful and useful.  My standards were different than their standards.

That doesn’t mean that the app can look horrible, but a plain old bootstrap theme or one of the other common CSS themes is ok. You need to know your audience–many people are stuck using a mix of software and are used to navigating clunky user interfaces. If your interface is just decent, but still solves the problem, you’ll be OK.  Of course, you’ll want to solve gross UX issues eventually, but a startup is all about balance.  A friend of mine gave me the advice: “don’t allow your users to make any mistakes”.

Favor manual process for complex edge cases

There have been a couple of situations during the build where a lot of work was needed to handle an edge case. For example, prorating monthly plans. Once you start thinking about prorating in depth, it turns out to be a really interesting problem with a lot of edge cases. But guess what?  For your startup, edge cases can be a wild goose chase.

When an edge case rears its head, you should consider the following options (in preferential order).

  • can you outsource the complexity (Stripe handles proration, for example, and I guarantee you they handle edge cases you don’t).
  • can you make it a manual process? If it doesn’t happen that often and/or a real time response is unneeded, you can often get by with a manual solution. This may be partly automated, for example, an SQL query that generates an email to a human who can handle the exceptional situation.
  • if neither of the above apply, can you defer it? Maybe for a few months, maybe for just a few weeks. But sometimes requirements change and you learn things from users that may make this edge case less important.
  • if all of the above don’t apply, you may need to bite the bullet and write code.

Back end and front end development doesn’t have to be synchronized

Most users equate the front end with the complete product. Most developers know that, just like an iceberg, there’s a lot of back end processing hidden in any project. But guess what? When you are getting feedback from users, some of the backend processes need to work, but many don’t. For example, we had a billing system that handled monthly invoices. We didn’t need to build the billing system while we were getting feedback from users on what type of charges they needed to handle. We did, however, need to know that we could build it. So make sure you can build the backend system to support your front end system, perhaps by building one path through, but defer the full build-out until you have to.

What about you?  Any tips for early stage product engineering?

Getting started with sharetribe development–vagrant style

I have recently spent a fair bit of time working with Sharetribe, an open source, MIT licensed marketplace platform that also powers a hosted solution.

First, let me say that the software is the least significant piece of a marketplace (like AirBnB).  The least significant! (Check out the Sharetribe Academy for some great content about the other steps.)  

But it is still a necessary component.  If you can get by with the hosted solution to prove out your idea, I suggest you do so–$100/month is a lot cheaper than hours of software development. There may come a time when you want to customize the sharetribe interface beyond what javascript injection can do.  If this is the case, you need a developer.  And that developer needs an environment.  That’s what this post is really about.

The sharetribe github readme explains the installation process pretty well, but I find it tedious, so I created a quick start vagrant VM. This VM has a sharetribe installation ready to go.  I use vagrant 1.6.3 and Virtualbox 5–google around for instructions on how to get those up and running. The guest VM is Ubuntu 14.04. This VM uses rvm to manage ruby versions, but I couldn’t be bothered with nvm. It will install sharetribe 5.8.0 and all needed components.

Assuming you have vagrant and virtual box installed, download the Vagrant file and put it in the directory where you want to work. Edit it and change any options you’d like. The options I changed were the port forwarding (I set it to 3003), networking options, and the amount of memory used (I allocate 4GB).

Then run vagrant box add http://www.mooreds.com/sharetribe/sharetribe-base-mooreds.box --name sharetribe-base-mooreds to download the file. It’s downloading a large file from my (small) server, so expect it to take a while (hours).

Then run vagrant init.

When you can login (password is vagrant if you use vagrant putty or you can use vagrant ssh) you can go to the sharetribe directory and do the following:

  • fork the sharetribe repo
  • update your git remote so that your origin is your forked repo (and not mine, because you won’t get write access to mine)
  • create a branch in your repo off of the 5.8.0 tag. There’s one startup script I tweaked a bit, but you can just ignore those changes.
  • update your mysql password: SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass');
  • start mailcatcher listening to all interfaces: mailcatcher --ip=
  • start the rails/react server: foreman start -f Procfile.static
  • visit lvh.me:3003 and start your server
  • set up your local super user and first marketplace
  • edit the url you are redirected to to have the correct external port on it (from the vagrant settings): from http://testdev.lvh.me:3000/?auth=baEOj7kFrsw to http://testdev.lvh.me:3003/?auth=baEOj7kFrsw for example

This is running sharetribe 5.8.0, and I’m sure there will be follow on releases. Here’s how to sync the releases coming from the sharetribe team with your current repo. I’ve taken the liberty of creating an upstream branch for you.

This doesn’t cover deploying the code anywhere–I’d recommend this gist. Make sure you read the comments! Or I can install a vanilla version of sharetribe to heroku for a flat fee–contact me for details.

Perils of ORM caching

So, I was working a rails4 project today and added an after_create method to a model (call it model A) that checked on a related object (call it model B) to see its state, and if it met a certain criteria, did something to the model A object being created. The specifics don’t really matter, but I was using zeus to run my rpsec tests.

This caused three tests to fail in entirely unrelated sections of the application.

What on earth was going on?

Well, first I used git bisect to determine the exact commit that caused the issue.  (As far as I’m concerned, the existence of git bisect confirms my belief to ‘commit early, commit often’).

Then I dug in.  It appears that each of the tests was tweaking the model B object, and testing some aspect of the change, usually through the model A object.  Before I added the after_create method, the model B object wasn’t loaded into the ActiveRecord in memory network graph tied to the model A object when the test saved the model A object initially, but was loaded from the database when the method under test executed.

After the after_create method was added, the model B object was loaded into the in memory network graph tied to the model A object.  Then the test tweaked the model B object in the database, but didn’t reload the model A object, which had a dirty/old version of the model B object.

A simple reload of model A (and its network graph) fixed it (or a repositioning of when I modified the model B object), but it was quite a subtle testing bug to track down.

© Moore Consulting, 2003-2017 +