Tonight I’m doing a lightning talk titled “3 Gems That Saved My Bacon” at the Boulder Ruby Meetup. 5ish minutes or so. Hope to see you there. Plus, free pizza and beer, courtesy of our sponsors.
Really, you should come. It’s fun.
The Rails Low Level Cache
I think the Rails low level cache is the bees knees. It lets you store complicated values easily, manage TTLs, and improve the performance of your application.
I’d be remiss in stating that you shouldn’t cache until you need to. Caches will introduce additional complexity into your application, make troubleshooting harder, and in general can be confusing. How do you know if you need to introduce a cache? Profile. Using rack-mini-profiler is a great quick and dirty way to profile your code.
The rails low level cache is fairly simple to configure (especially if you’re using a PaaS like Heroku–you can drop in a managed memcached service easily).
From there, you need to build a cache key. This is a string, up to 250 characters in length, that uniquely identifies whatever you’re trying to cache. Basically anything that would cause a change in the contents of the cached object should be included in this key. However, since you are going to calculate this key every time the value is requested, you don’t want the cache key to be expensive to calculate, otherwise the value of the cache will be degraded. Good things for the key: timestamps, max updated_at values for a collection, user ids or roles.
Then, putting a value in the cache is as easy as:
def pull_from_cache
cache_key = 'mykeyval'
myval = Rails.cache.fetch(cache_key, expires_in: 1.hours) do
@service.build_expensive_object()
end
end
In the case above, the cached value will automatically expire in 1 hour. If you don’t set the expiration time, then the cached value will eventually be removed via LRU. How long that is depends on the size of your cache and what else you are putting in there.
If you want to test that something is being put in the cache, you can run a unit test and see how many times build_expensive_object() is called.
#... object.service = @service expect(@service).to receive(:build_expensive_object).once object.pull_from_cache object.pull_from_cache #...
In a production troubleshooting situation, you may need to evict something from the cache. You can do so from the rails console by finding the cache key and running Rails.cache.delete('key').
Using the low level cache is a great way to push read heavy hot spots of your rails application (database queries or other complicated calculations) into memory and make your application faster.
Useful Rails Gems: Pretender
I’m constantly amazed at how productive you can be with rails. It simply lets you work on typical webapp problems at a much higher level. At 8z, we had a web application and a customer support team. Occasionally the customer support person had to ‘impersonate’ a normal user to troubleshoot an issue. We built a piece of software that let them assume that role. (We called it ‘sudo‘, obviously.) It’s been a few years, but as I recall it was complicated and error prone, lived on a different domain and wasn’t fully functional.
I needed to add similar functionality to a rails web app, and was able to find a couple of gems that looked useful. I selected pretender, mostly on the basis of documentation and google search results placement. I followed the instructions, tweaked a few settings and was off to the races in about an hour. (Note this isn’t a fair apples to apples comparison of the underlying technologies, due to the differences in available open source libraries between the mid 2000s and the late 2010s.)
Now, all this gem does is what it says it does. It lets a certain user or set of users in your application pretend to be another user. It doesn’t handle auditing or anything else you might want with an elevated privilege system.
But it does do what it does well.
Railsconf Call For Proposals
Railsconf, a conference focused on, well, Ruby on Rails, is happening in Pittsburgh in April this year. I attended last year and it was a fantastic experience. I enjoyed the people I met, the problems I saw discussed, and the size and content of the presentations and workshops. It definitely had a vendor experience (thank you, Heroku, for the t-shirts), but wasn’t too explicit.
Railsconf organizers are now accepting proposals for workshops, panels and speaking. For some reason the CFP isn’t on the website, but I noticed it was announced via twitter. I just submitted, so I can’t speak to the entire process, but the initial submission was pretty painless, just few sections on what you’re interested in presenting and why you might be a good fit.
They actually have a blind submission format, so I had to edit my initial submission to remove any reference to my identity. Seems like a good idea.
So, go forth and submit!
Useful Rails Gems: dossier
Writing reports is something that is crucial to any business application. There are a number of nice reporting solutions. I haven’t done a survey recently, but I know that Pentaho has an offering and Crystal Reports is another one. Note that the gorilla in the room is MS Excel, because most anyone doing a report is going to want to pull it down and do additional data manipulation. You should never try to compete with Excel.
For The Food Corridor, we didn’t need a full featured reporting solution, but we did need to pull data from the SQL database and expose it in CSV and HTML tables. The dossier gem takes care of this.
What I love about dossier is that you can write the reports in either Active Record syntax or in SQL. I know SQL pretty well, so it’s easier for me to use that avenue, but if you’re just getting started with Rails, I could see how the Active Record syntax would be helpful. After writing the query, dossier takes care of much of the grunt work of CSV generation or HTML table generation (there are other output formats, but I don’t use them, so can’t speak to them).
You can preform arbitrary formatting on the SQL response, including hiding columns and running ruby code on the value pulled back from the database. The report class is just a ruby object, so you can also reuse chunks of logic or SQL across reports. Column names are turned into report headers. (I wrote previously about testing such reports.)
Authentication is a bit weird out of the box but works. There’s also documentation for other authentication/authorization options. Dossier can also take database configuration from the initializer file so you can easily offload your reporting traffic to a read only replica. (We’re not there yet in terms of traffic, but I can see the day coming.)
The dossier gem is so easy to user that I can get a full featured reporting solution created in only an hour beyond writing the SQL.
Heroku pipelines and asset_paths and font downloads being cancelled
My font downloads were being cancelled when I was using a CDN in front of a rails app hosted on Heroku. I was testing out using heroku pipelines, which lets you promote a slug built in one environment directly to another. Rails was also serving the CSS/JS/font files, but was behind a CDN and config.action_controller.asset_host was set to the cloudfront URL, so most clients were pulling from the CDN.
The issue I ran into was that in order to get my fonts to show up correctly when I was compiling my slugs on production deploy (before I moved to heroku pipelines), I needed to use erb and the asset_path tag. So my font css file looked like this:
@font-face{
font-family:'FontAwesome';
src:url(<%= asset_path('fontawesome-webfont.eot?v=3.0.1') %>);
src:url(<%= asset_path('fontawesome-webfont.eot?#iefix&v=3.0.1') %>) format('embedded-opentype'),
url(<%= asset_path('fontawesome-webfont.woff?v=3.0.1') %>) format('woff'),
url(<%= asset_path('fontawesome-webfont.ttf?v=3.0.1') %>) format('truetype');
font-weight:normal;
font-style:normal }
However, this caused the asset_path in this CSS file to be set to the full CDN hostname plus the asset path, something like https://d123.cloudfront.net/assets/fontawesome-webfont.eot (when the slug was compiled on staging). When the slug was compiled on production, the value was set to https://d256.cloudfront.net/assets/fontawesome-webfont.eot. When I was using heroku pipelines, the slug was unchanged between staging and production, just as advertised. However, this meant that the CSS was being served off of the production CDN host (d256) and the fonts in that CSS file were referencing the staging CDN host (d123). Browsers didn’t like that. This question indicates that this is due to cross resource permissions not being set correctly.
What were my options?
I could return to deploying slugs by compiling them on staging and production. However, I like the concept of pipelines and the immutable artifact.
I could try to find some way to post process the slug and change that link in the font CSS. But that didn’t seem possible, based on documentation, and kinda defeats the purpose. I did discover the heroku release phase.
I tried some of the other solutions outlined on this question to tweak how the asset pipeline compiled the font references, but nothing else worked.
I could have tried to set the permissions correctly for the font download. One worry there would be having the production environment depend on the staging CDN. Never a good idea to mix environments.
I also could have tried relative paths to the font files in the CSS file. I didn’t do this because I thought of my final solution first.
The solution I finally went what was using a CDN that hosted the font files of the correct version and used those. I also chose this solution because we won’t be using these fonts (which we primarily use for icons) for long–in the near future we’ll be removing the icons from the application.
If the images in your Rails image_tag calls don’t have a checksum…
This last week, I spent a lot of time learning about how Rails serves static files, how it interacts with a CDN like CloudFront, and how misconfigurations can really screw up your application. I wanted to document this here so that if I run into these situations again, I can troubleshoot them more easily.
Problem #1: We did an large release, with a lot of moving pieces
We (The Food Corridor) recently engaged with a consulting company to do a refresh of the look and feel of our application. They don’t just do UX and design, but also implementation using overseas developers and QA. I was excited to let said developers focus on look and feel (not my strength) but made a mistake in not setting up an entire environment for them. Instead, I let them use our staging environment. Things took longer than predicted (as they always do, and some of it was due to my availability). I was doing some tidying up, changing the deploy process, etc, and ended up merging a lot of changes into our codebase. This meant that the first release had a lot more risk that previous releases–there was some of the new look and feel as well as all my changes.
That made troubleshooting any issues that came up with the release difficult, because it wasn’t clear what caused it–was it deployment changes, some of my code, some of the look and feel code?
Solution #1: Set up a ‘review’ environment for any long running branches. I haven’t gone the full ‘review app’ path yet, but based on the docs, it doesn’t seem too hard to set up. For now we just have one review environment that can be shared.
Problem #2: Certain images appeared on staging but not on production.
This was one of the issues that caused some heartache during the first release. There were some images (svgs, to be exact) that were present on staging and not on production (from the browser, you’d get a 404). But staging and production had the exact same codebase, including images. We were doing a fresh deploy to production. What was the issues?
I rolled back a lot of the changes I’d made to make sure it wasn’t a deployment issue (turning off pipelines, unsetting environment variables, etc). No love. We were still seeing 404s. Looking at the HTML, on production the image had a different name, without the hash at the end. From slack:
Interesting. on prod the envelope image is here: <limg alt=”Message” src=”https://d1soonciftqo56.cloudfront.net/images/tfc/message.svg”>
and on staging it is here: <img alt=”Message” src=”https://d1wspyydkkjqvw.cloudfront.net/assets/tfc/message-c9189c257a23964ea6b97b89416b25a4.svg”>
… the svg isn’t being compiled on production for some reason
That led me to learn about the heroku asset pipeline and in particular the way rails4 apps are treated. I then dove into the compiled slug, and then saw that the message-c9189....svg image was present in the staging slug under public/assets but that there was no message.svg file in production. There was, however, a message-text-89ab....svg file.
Solution #2: The staging environment had some old copies of the image files. The files had been renamed, but the image_tag calls still referenced those old files. We had to update them. I also updated the build process to run a heroku repo purge-cache every time staging was built so that we wouldn’t have any of those old files lying around, using the heroku repo plugin/. (It’s fine to make a mistake, but try not to make the same mistake twice.)
Seek The Golden Comment: “zeus exit status 1” fix
This bit me yesterday, so I wanted to get it written down. zeus is a preloader that makes running tests faster. It can be a bit finicky about the gems available, even when using rvm.
Yesterday, I tried to upgrade a rails 4.2 app to rails 5. (I failed.) When I checked out my source branch to work on a different issue, ran a bundle install, and then a zeus start, I saw this:
zeus rake
zeus runner (alias: r)
zeus console (alias: c)
zeus server (alias: s)
zeus generate (alias: g)
zeus destroy (alias: d)
zeus dbconsole
zeus test (alias: rspec, testrb)
exit status 1
And then zeus exited. I did some google searches and turned up these two issues: 118 and 237. Lots of folks having similar issues.
After reading carefully through these issue, and trying some of the suggested fixes as I did so, I arrived at the golden comment. I reproduce it here in its beautiful entirety:
There’s a lot of “try this” in here, but no actual debugging steps. Here’s how you can find the actual issue:
zeus –log ZEUS.LOG start then cat ZEUS.LOG
Excellent! Using this logfile I quickly determined that the root issue was a collision in my json gem versions and was able to get zeus running again.
But that’s not really the point. The point is that this user (thank you Steven!) didn’t just provide an answer, he provided the means for me to diagnose and find my own answer.
I wish github had some way of calling out highly recommended comments, as if I’d seen his comment first, it would have saved me some time.
Just goes to show, you should always read the comments fully and look for the golden one when you are troubleshooting.
Rails Gems: Ahoy
Sometimes you want more analytical detail than Google Analytics or Heap or other analytics offerings allow. If you have an internal datastore that you want to match visits up with, you can either pull the tracking data from the web analytics tool, push your data up to the tool, or find some other way to get the web analytics data into your internal datastore.
If you choose the third option, ahoy is the rails gem for you. It’s a simple install and will track visits and visitors (both signed in and anonymous), user agents, time of visit, and more. You can then use it to correlate with internal goals. For instance, if you have a funnel: ‘adwords -> visit -> signup -> create profile -> join group -> participate in group’, you may want to track each step of the funnel. You may want to know how many groups each adwords click joining user joins. There may be aspects of your application that are not accessible via the web that you want to correlate with external indicators (‘how often does billing fail with people who use safari’ is a (fartfetched) example). Answering these questions may be easier to do with SQL than it would be with leveraging an external tool.
However, cohort analysis and other sophisticated statistical analysis may be harder with this data, and if you are looking at doing that you may want to make the investment in pushing data up to one of the other tools, or tagging your application such that all relevant goals are measured by the web tool.
Regardless, ahoy is simple to set up and if you’re looking to pull in web tracking data into your datastore for additional insights, I highly recommend it.
The power of automated testing
It took me a long time to understand the power of automated testing. After all, it can end up being a large portion of your codebase and can be brittle. Sometimes it feels like writing tests “gets in the way” of getting things done. At one project I worked on, a colleague complained that it felt like you spent 5 minutes changing the production code and an hour changing the tests. (And to be fair, sometimes that’s true, and there’s a balance to be struck between test code coverage and speed of development. This can also indicate you need to spend time refactoring your tests, as you have multiple different test components testing the same production code.)
I like to think of tests like a gentle swaddling of your code. It conforms as the body of your code changes, but changing that code does require some re-work of the tests. And, if your code fails, it fails into the gentle swaddling, as opposed to the cruel outside world (bleeding all over your production users). Alright, maybe the analogy fails :).
I write this today because I’m in the middle of a refactor of one of the scariest bits of The Food Corridor. (Given we’re so young, it’s not that scary, but it’s quite complex–handling the creation and updating of bookings.) There are many many paths through the code and if I didn’t have automated testing, I’d be far more worried about the changes I’m making.
So, consider this blog post to be a thank you to past me for making future me’s life easier by writing a comprehensive automated test suite. If you don’t have one, you should.
Follow me on Twitter
Follow me on Blue Sky