Rails Gems: Ahoy

Sometimes you want more analytical detail than Google Analytics or Heap or other analytics offerings allow.  If you have an internal datastore that you want to match visits up with, you can either pull the tracking data from the web analytics tool, push your data up to the tool, or find some other way to get the web analytics data into your internal datastore.

If you choose the third option, ahoy is the rails gem for you.  It’s a simple install and will track visits and visitors (both signed in and anonymous), user agents, time of visit, and more.  You can then use it to correlate with internal goals.  For instance, if you have a funnel: ‘adwords -> visit -> signup -> create profile -> join group -> participate in group’, you may want to track each step of the funnel.  You may want to know how many groups each adwords click joining user joins.  There may be aspects of your application that are not accessible via the web that you want to correlate with external indicators (‘how often does billing fail with people who use safari’ is a (fartfetched) example). Answering these questions may be easier to do with SQL than it would be with leveraging an external tool.

However, cohort analysis and other sophisticated statistical analysis may be harder with this data, and if you are looking at doing that you may want to make the investment in pushing data up to one of the other tools, or tagging your application such that all relevant goals are measured by the web tool.

Regardless, ahoy is simple to set up and if you’re looking to pull in web tracking data into your datastore for additional insights, I highly recommend it.


The power of automated testing

It took me a long time to understand the power of automated testing.  After all, it can end up being a large portion of your codebase and can be brittle.  Sometimes it feels like writing tests “gets in the way” of getting things done.  At one project I worked on, a colleague complained that it felt like you spent 5 minutes changing the production code and an hour changing the tests.  (And to be fair, sometimes that’s true, and there’s a balance to be struck between test code coverage and speed of development.  This can also indicate you need to spend time refactoring your tests, as you have multiple different test components testing the same production code.)

I like to think of tests like a gentle swaddling of your code.  It conforms as the body of your code changes, but changing that code does require some re-work of the tests.  And, if your code fails, it fails into the gentle swaddling, as opposed to the cruel outside world (bleeding all over your production users).  Alright, maybe the analogy fails :).

I write this today because I’m in the middle of a refactor of one of the scariest bits of The Food Corridor.  (Given we’re so young, it’s not that scary, but it’s quite complex–handling the creation and updating of bookings.)  There are many many paths through the code and if I didn’t have automated testing, I’d be far more worried about the changes I’m making.

So, consider this blog post to be a thank you to past me for making future me’s life easier by writing a comprehensive automated test suite.  If you don’t have one, you should.


Rails Gems: paper_trail

Rails has an amazing number of quality libraries (called ‘gems’) and while building The Food Corridor application, I was lucky enough to be able to use some of them.

If you want to easily track changes in your rails models over time, the paper_trail gem has you covered.  Simple to install, compatible with many databases, and very easy to use.  I’d point you to the docs for installation, but for usage, all you have to do is drop has_paper_trail into your model class.

After that’s deployed, every change to those models is tracked, including the entire model state.  From an operational perspective, I’m a bit worried about the size of the audit table (called versions), but thus far it hasn’t grown too fast.  It wouldn’t be hard to prune either.

I use the data this gem records for two reasons.  The primary purpose is to debug the system.  When there’s an issue because the system doesn’t act as expected, or a question on when a model changed, I can go back and see how the model has changed over time.  I can do this either by directly inspecting the versions table or via the rails console.  If I use the latter, then I can actually pull back the model object as it existed in the past.  This has been tremendously helpful in tracking down bugs.

The secondary use of this data is to allow the system to see changes over time, and execute code based on those changes.  This doesn’t happen often, and I’d prefer for such changes to be captured more explicitly, but in at least one case this gem has allowed me to fix an issue in a time efficient manner.

If you’re building a complicated system that changes over time, as most real world applications do, having some kind of audit trail can be extremely helpful.  paper_trail makes it trivial to get an audit trail set up in your rails app.


Heroku 503 errors

One day recently I woke up to a text from our monitoring service (part of minimal heroku operations tasks) saying our application was down.

Darn it.

Here’s the recreation of my steps in troubleshooting this issue:

  • Login, see that the app is indeed down.
  • Look at newrelic.
  • Look at the logs (papertrail!).
  • Look at the deployment history.
  • Note when the issue started–curses, not when I did a deploy.
  • Open a ticket with heroku (after doing some research). (Love their support.)
  • Double check that the database is good and hasn’t hiccuped.
  • Look at the logs more.
  • Add more dynos, see if that helps.
  • Google the error message.
    •  <app-name> heroku/router: at=info method=POST path=”<url>” host=app.thefoodcorridor.com request_id=8a17648f-2d84-46ea-abf2-5903be894a2c fwd=”216.191.191.58″ dyno=web.3 connect=1ms service=4ms status=503 bytes=477 protocol=https“Notice that the issue is being stated right in the log file (passenger request queue filling up).  Here are sample error messages?
    • <app-name> app/web.3: [ 2017-07-12 14:47:44.3688 65/7f652dffd700 age/Cor/Con/CheckoutSession.cpp:261 ]: [Client 3-281] Returning HTTP 503 due to: Request queue full (configured max. size: 100)
  • Find some posts about the error message.  Here and here.
  • Start researching how to increase request queue size.
  • Talk a walk to clear my head.
  • Think about what external services we call, as that seems to be what might cause the request queue to back up.
  • Read another post that says restarting passenger helped.
  • Restart all dynos.
  • Problem disappears.
  • Look at logs more closely.
  • Last dyno to be restarted was the only problematic dyno.
  • Add comment to ticket about this being the cause.
  • Heroku confirms that the issue may have been the dyno: “sometimes individual dynos will hang and cause errors with 503 responses”
  • Write note to customers about the issue explaining how access to app was affected.
  • Lower number of dynos.
  • Breath a sigh of relief.

Interview with a boot camp grad

Lots of folks are moving into development and technical fields these days.  I remember that happening during the dot com book, but back then folks just read one of those “Learn Java in 24 Hours” books.

Nowadays there are a profusion of boot camps that help people gain the skills they need to be a developer. I have interacted with a few of these grads and was interested in learning more about their experience. Noel Worden, one of the organizers of Boulder.rb and a blogger, agreed to an interview.

– what was your background before you were a developer?

I got my degree in fine art photography, spent 5 years working in NYC as a Digital Technician in the photo industry, then moved to Colorado and was a cabinetmaker for 3 years.

– how did you land your first job?

I found the posting on the Denver Full Stack meetup whiteboard (https://www.meetup.com/fullstack/)

– what surprised you about the software industry?

How willing everyone is to help. It’s very different in the photo world, [which is] much more competitive, it’s hard to find guidance and mentorship.

– how did you pick your boot camp?

Bloc had one of the longest curriculums I could find in an online program. I figured when push came to shove, having more experience under my belt couldn’t hurt when competing with other junior [developers] looking for that first job. Bloc also has a part-time track, which allowed me to still work [a] full-time job while going to school.

– what is good about the job? What is challenging?

I appreciate the balance of being challenged but knowing I have the full support of any other engineer on the team if I need to reach out for assistance. Canvas United [ed: his current employer] has a lot of projects that I’ve been maintaining lately, and all are running different versions of Rails, which makes for interesting challenges. 

– what do you see current boot camp students doing that you’d advise against?

Not getting out and networking while going to school. This industry is all about networking and if you’re hoping to capitalize on the advantages of a good network you have to be building it while still in school.

– why did you want to transition into technology and development?

I wanted/needed a career path where the leaning curve wouldn’t plateau. I’m not a ‘cruise control’ kind of person, as soon as the challenge isn’t there for me anymore I lose interest.

– how can employers help boot camp grads in their first job?

Be ok with answering questions, and be transparent with the employee about the proper channels to ask those questions. Also, be upfront about the fact that it’s ok to fail (assuming that’s the case) a few time before you get to the best solution [ed: if it isn’t ok to fail, find a new job!]. Also also, a healthy balance of low hanging fruit and multi-day problems. Nothing kills my morale faster than ticket after ticket of problems that grind me into the ground.

– should employers have any different expectations of a boot camp grad vs someone who just graduated from college or high school?

I definitely think that it takes a particular type of management style to successfully level up a boot camp grad. If you’ve hired them you must have liked something about them, play to those strengths, but also sprinkle in challenges to help that developer evolve.

[This content has been edited for grammar and clarity.]


Testing Dossier Reports in Rails

One of the things I love about developing with rails is the vast array of open source, free components that you can drop in and extend your application.  Want an invoicing system?  Way to run your javascript testsSimple admin portal?  Just drop in a gem, run bundle install and you are good to go.

One of the gems I’ve used recently was dossier, which lets you write reports in SQL or active record, and then generate them in HTML or CSV (or JSON, but I didn’t use that). One tip–if you want your CSV results to have the same formatting as your HTML results, you’ll want to follow the steps on this issue.

I wrote up a couple of SQL reports, linked them into the appropriate admin pages, and called it good. Then, the app moved on, and at one point, the schema changed. (Some of you are shaking your head, knowing what is going to happen next.) Then, the reports failed.

I had forgotten the cardinal rule–write the tests first. I confess, I wasn’t sure how to, but a bit of research revealed that it wasn’t that hard. Here’s one of my spec files.

require 'spec_helper'

describe MonthlyHoursClientReport do

  # all this does is test that the SQL is valid
  it "sql valid" do
    report = MonthlyHoursClientReport.new
    sql = report.sql
    sql = replace_placeholders(sql)
    expect{ActiveRecord::Base.connection.execute(sql)}.to_not raise_error
  end

  def replace_placeholders(sql)
    sql = sql.gsub(":kitchen_id",1.to_s)
  end
end

This just gets the SQL from the dossier report and tries to execute the SQL in the test database. Super simple, but enough to catch the error I encountered. If/when I get more time, I could definitely add some more tests with some data in the test db to make sure the SQL is giving correct results, but I tend to be pretty confident in my SQL queries, especially when they don’t have any group by or having clauses.

Anyway, happy testing.



Updating Stripe bank accounts: “A bank account with that routing number and account number already exists for this customer.”

So, if you want to handle ACH transactions with Stripe, you can. Some limits to include the length of time for the transaction (5 business days on top of stripes 2 business day transfer duration) and support only for US accounts, but the API is nice and the price is pretty nice too (0.8% up to $5).

But if you are trying to do recurring billing with Stripe and ACH and you want to let your customer change their default charge source between credit card and bank accounts as a payment source (or even two different bank accounts), you’re going to run into a roadblock. While Stripe will happily accept new credit information with the exact same card number, expiration date and CVC code, and just create a new source for each entry, it is not so forgiving with bank accounts. Instead, you’ll get this error message: "A bank account with that routing number and account number already exists for this customer." if you try to change the default source to an existing bank account record in Stripe.

I found some code with this error message, but it actually isn’t complete. It’s not best to examine the error message and switch on that, but I didn’t see a specific exception class for this type of exception.

For a complete solution, you need to check the stripe tokens routing number and last 4 digits of the account number. If a user has two different bank accounts that match both in the last 4 of the account number and the routing number, well then, I think you are out of luck.

Here’s the complete ruby code, making sure to match the current request’s routing id number just in case your user wants to switch between multiple bank accounts for their default charge.


    def update_customer_from_token(customer,stripe_token)
      # takes the stripe customer object and the new token 
      # from the stripe indicating the changed payment method

      success = false
      Stripe.api_key = ENV["stripe_secret_key"]
      begin
        new_pmt_obj = customer.sources.create({:source => stripe_token})

        customer.default_source = new_pmt_obj.id
        customer.save
        success = true
      rescue Stripe::InvalidRequestError => e
        # special case where the bank account already exists, let's use that.
        if e.message == 'A bank account with that routing number and account number already exists for this customer.'
          tokobj = Stripe::Token.retrieve(stripe_token)
          customer.sources.each do | src |
            begin
              if src.object == 'bank_account' && src.routing_number == tokobj.bank_account.routing_number && src.last4 == tokobj.bank_account.last
                customer.default_source = src.id
                customer.save
                success = true
                break
              end
            rescue => e
              Rails.logger.error(STRIPE_ERROR_PREFIX+" 4 unable to update customer for "+customer.inspect+", "+e.inspect)
            end
          end
        else
          Rails.logger.error(STRIPE_ERROR_PREFIX+" 3 unable to update customer for "+customer.inspect+", "+e.inspect)
        end
      rescue Stripe::CardError => e
        Rails.logger.error(STRIPE_ERROR_PREFIX+" 1 unable to update customer for "+customer.inspect+", "+e.inspect)
      rescue => e
        Rails.logger.error(STRIPE_ERROR_PREFIX+" 2 unable to update customer for "+customer.inspect+", "+e.inspect)
      end
      success
    end

Or, you could just let the user choose from a drop down list of their existing sources which one they want to be the default. That might be a cleaner solution.


Presenting on Stripe tomorrow

Excited to say that I’ll be presenting on my company’s use of Stripe at Boulder.rb tomorrow.  Working title of my talk: “Do you like Money?”.

What I’ll cover:  an overview of the service, real code, testing, operational practices, and gotchas.  This will be based on my experience at The Food Corridor, where we’ve processed over $250k of transactions through Stripe.

Hope to see you there!


Extending an existing Rails application that wasn’t meant to be extended

I am modifying an existing open source rails 4.2 app and wanted to keep my changes (some of which are to models, some to controllers, some to views) as separate as I can, so that when a new release of the app comes out, I won’t be in (too much) merge hell.

This app was not designed to be extended, which makes things more interesting.

For the views, I’m just doing partials with a prefix (_xxx_user_cta.haml).

For the models and controllers, I started out hacking the code directly, but after some digging around I discovered how to monkey patch (I believe that is what it is called) the classes.

In the config/application.rb file, I added the following code:

config.to_prepare do
Dir.glob(Rails.root + "app/decorators/**/*_decorator*.rb").each do |c|
require_dependency(c)
end
end

And then, if I want to make a change to app/models/person.rb, I add the file app/decorators/models/person_decorator.rb. In that file is something like this:

Person.class_eval do
# ... changes
end

This lets me add additional relations, helper methods, and other classes to extend existing functionality. I try to prefix things with a unique identifier (xxx_set_timezone rather than set_timezone) to lessen the chances of a collision, because if a method is added to the Person class with the same name as a method in the decorator, the decorator will win.

Write tests around this new functionality so that if anything changes, I’m aware and can more easily troubleshoot.

The downsides of this approach is that it is harder to track logic, because instead of everything in one file, it is now in two. (I don’t know if there are memory or performance implications.) However, that is a tradeoff I’m willing to make to make it easier to keep up with the upstream development and to pull said development in as often as possible.

I’m still fairly new to rails and didn’t know if this is the only or best way, but thought I’d share.



© Moore Consulting, 2003-2017 +