This last week, I spent a lot of time learning about how Rails serves static files, how it interacts with a CDN like CloudFront, and how misconfigurations can really screw up your application. I wanted to document this here so that if I run into these situations again, I can troubleshoot them more easily.
Problem #1: We did an large release, with a lot of moving pieces
We (The Food Corridor) recently engaged with a consulting company to do a refresh of the look and feel of our application. They don’t just do UX and design, but also implementation using overseas developers and QA. I was excited to let said developers focus on look and feel (not my strength) but made a mistake in not setting up an entire environment for them. Instead, I let them use our staging environment. Things took longer than predicted (as they always do, and some of it was due to my availability). I was doing some tidying up, changing the deploy process, etc, and ended up merging a lot of changes into our codebase. This meant that the first release had a lot more risk that previous releases–there was some of the new look and feel as well as all my changes.
That made troubleshooting any issues that came up with the release difficult, because it wasn’t clear what caused it–was it deployment changes, some of my code, some of the look and feel code?
Solution #1: Set up a ‘review’ environment for any long running branches. I haven’t gone the full ‘review app’ path yet, but based on the docs, it doesn’t seem too hard to set up. For now we just have one review environment that can be shared.
Problem #2: Certain images appeared on staging but not on production.
This was one of the issues that caused some heartache during the first release. There were some images (svgs, to be exact) that were present on staging and not on production (from the browser, you’d get a 404). But staging and production had the exact same codebase, including images. We were doing a fresh deploy to production. What was the issues?
I rolled back a lot of the changes I’d made to make sure it wasn’t a deployment issue (turning off pipelines, unsetting environment variables, etc). No love. We were still seeing 404s. Looking at the HTML, on production the image had a different name, without the hash at the end. From slack:
Interesting. on prod the envelope image is here: <limg alt=”Message” src=”https://d1soonciftqo56.cloudfront.net/images/tfc/message.svg”>
and on staging it is here: <img alt=”Message” src=”https://d1wspyydkkjqvw.cloudfront.net/assets/tfc/message-c9189c257a23964ea6b97b89416b25a4.svg”>
… the svg isn’t being compiled on production for some reason
That led me to learn about the heroku asset pipeline and in particular the way rails4 apps are treated. I then dove into the compiled slug, and then saw that the
message-c9189....svg image was present in the staging slug under
public/assets but that there was no
message.svg file in production. There was, however, a
Solution #2: The staging environment had some old copies of the image files. The files had been renamed, but the
image_tag calls still referenced those old files. We had to update them. I also updated the build process to run a
heroku repo purge-cache every time staging was built so that we wouldn’t have any of those old files lying around, using the heroku repo plugin/. (It’s fine to make a mistake, but try not to make the same mistake twice.)