Awesome, you are a CTO or founding engineer of a newborn startup. You have an web app up on Heroku and someone is paying you money for it! Nice job.
Now, you need to think about supporting it. Heroku makes things way easier (no racking and stacking, no purchasing hardware, no configuring apache) but you still to set up some operations.
Here is the bare minimum you need to do to make sure you can sleep at night. (Based on a couple of years of heroku projects, and being really really cheap.)
- Have a staging environment
- Have a one line deploy.
- Or, if you like CD/CI, an automatic deploy or a one click deploy. But make it really easy to deploy.
- Have a deploy script that goes straight to production for emergencies.
- Cost: free
- Backups
- User data. If you aren’t using a shared object store like S3, make sure you are doing a backup.
- Database. Both heroku postgresql and amazon RDS have point and click solutions. All you have to do is set them up. (Test them, at least once.)
- Cost: freeish, depending on the solution. But, user data is worth spending money on.
- Alerting
- Heroku has options if you are running professional dynos.
- Uptimerobot is a great free third party service that will check ports every 5 minutes and has a variety of alert options. If you want SMS, you have to pay for it, but it’s not outrageous.
- Cost: free
- Logging
- Use a logging framework (like slf4j or the rails logger, and mark error conditions with a string that will be easy to search for.
- Yes, you can use
heroku logs
but having a log management solution like papertrail will make you much happier. Plus, it’s free for 2 days of logfiles. - Set up alerts with papertrail as well. These can be more granular.
- Cost: free
- Create a list of third party dependencies.
- Sign up for status alerts from these. If you have pro slack, you can have them push an email to a channel. If you don’t, create an alias that receives them. You want to be the person that tells your clients about outages, not the other way around.
- Cost: free
- Communication
- Internal
- a devops_alert slack channel is my preferred solutions. All deploys and other alerts go there.
- External
- create a mailing list for your clients so you can inform them of issues easily. Google groups is fine, but use whatever other folks are using. Don’t use an alias in your email–you’ll forget to add new clients.
- do not use this mailing list for marketing purposes, unless you want to offload the burden of keeping the list up to date to the marketing department.
- do make sure when you gain or lose clients you keep this up to date
- Run through a disaster in your mind and make notes on how you would communicate the issue, both internally and externally. How often do you update your team? How often do you update your clients? What about an internal issue (some of your code screwed up) vs an external issue. This doesn’t need to be exhaustive, but thinking about it ahead of time and making some notes will help you in the crisis.
- Cost: free
- Internal
All of this is probably a four hour project, max.
But once this is done, you’ll rest easier at night, knowing you have what you need to troubleshoot and recover from production issues.
Pingback: Heroku 503 errors | Dan Moore!