Skip to content

Debugging options for AWS Lambda functions

AWS Lambda lets you write a ‘function as a service’, and run code from 100ms to 5 minutes in execution time without maintaining any servers.  This code has few limitations, but one of the issues I’ve always encountered is debugging lambda functions.

I mentioned it at a past meetup and encountered the following solutions:

I think the right debugging option depends on the complexity of your code and the urgency of the situation, but could see using all of these at different times.

Bonus: here’s a post on how to continuously deploy your lambda functions.

AWS | moore | | Leave a comment

Supporting your local newspaper

We are subscribers to the local paper (the Daily Camera).  It’s not perfect (and there are of course online subscription options), but the experience of reading newspaper is different than the experience of reading online.  They both have strengths.

Online:

  • read from anywhere: anywhere that has an internet connection, of course
  • searchability: you can search for a particular word within an article or a specific article within an edition
  • shareability: super simple to pass on an article that you think a friend would enjoy via email, text, etc

Offline/paper:

  • discoverability: often you’ll discover stories that you didn’t know you needed to read
  • focus: when you’re reading the paper, you aren’t distracted by any notifications or other applications
  • browseability: you can easily switch between sections and discover related articles
  • shareability: when you read an article you like, you can point it out to a family member and they can immediately read it if they like

There’s also a comforting ritual about it–wandering out to the driveway every morning and enjoying the early morning across the seasons, younger family members seeking out the funnies, folding the newspaper and reading it on the couch.

Supporting local news organizations is much like supporting local shops or local farms. In all of these cases, you’re trading efficiency for resiliency (the centralized solution is in general more efficient, but concentration means more chokepoints).  If you don’t support local options, eventually they won’t be available. If that’s OK with you, that’s fine, but it’s not OK with me.

Technology and Society | moore | | Leave a comment

AWS Advent Calendar

There’s an AWS advent calendar, where new articles will be posted about various aspects of AWS starting Dec 25. If you’re interested in writing or reviewing the articles, feel free to sign up.  There are also some great posts from 2016, covering topics such as how to analyze VPC flow logs, cost control, lambda and building AMIs with packer.

There are also articles from years past.  I haven’t examined them closely, but I’d be wary of them, simply because the AWS landscape is changing so rapidly, and an article from three or four years ago may or may not be applicable.

AWS | moore | | 1 Comment

Rails Gems: Ahoy

Sometimes you want more analytical detail than Google Analytics or Heap or other analytics offerings allow.  If you have an internal datastore that you want to match visits up with, you can either pull the tracking data from the web analytics tool, push your data up to the tool, or find some other way to get the web analytics data into your internal datastore.

If you choose the third option, ahoy is the rails gem for you.  It’s a simple install and will track visits and visitors (both signed in and anonymous), user agents, time of visit, and more.  You can then use it to correlate with internal goals.  For instance, if you have a funnel: ‘adwords -> visit -> signup -> create profile -> join group -> participate in group’, you may want to track each step of the funnel.  You may want to know how many groups each adwords click joining user joins.  There may be aspects of your application that are not accessible via the web that you want to correlate with external indicators (‘how often does billing fail with people who use safari’ is a (fartfetched) example). Answering these questions may be easier to do with SQL than it would be with leveraging an external tool.

However, cohort analysis and other sophisticated statistical analysis may be harder with this data, and if you are looking at doing that you may want to make the investment in pushing data up to one of the other tools, or tagging your application such that all relevant goals are measured by the web tool.

Regardless, ahoy is simple to set up and if you’re looking to pull in web tracking data into your datastore for additional insights, I highly recommend it.

Rails Useful Tools | moore |

Hiring: Apples and Oranges

I ran across this great article about hiring in my twitter feed.  (Incidentally, the author is seeking employment at the moment.)  Here’s the author’s take on what’s wrong with hiring in tech today:

Job postings focus on the current tooling and products rather than factor in future plans. Interviews focus on the immediate technical knowledge and abilities of candidates, rather than on the candidates’ capabilities for learning and adapting. Companies hire the wrong people for the wrong reasons, then are surprised to find they can’t scale or adapt to meet strategic goals.

It really resonated.  It crystallized what I’ve seen reading and writing job descriptions over the past 20 years, which is typically an over focus on hard skills and an underfocus on softer skills.  From a higher level, it’s a focus on immediate needs vs long term compatibility (perhaps the emphasis on ‘fit’ is a counter to that).  Job postings focus on current needs to the exclusion of adaptability because it’s a lot harder to measure adaptability.  After all, everyone is capable of change in some aspects of their lives, so how do you know if an employee can handle change in the particular manner you need?

The entire article is worth a read.  Next time you write a job requisition, think about whether you need someone to hit the ground running with a particular technology, or whether you can afford to hire someone who’ll have to either be trained up or transfer skills from a similar but not identical concepts (from rails to django or vice versa, for example).  If the former, do they need to be an employee, and if so, what will happen to them when the exact need is done?  If the latter, you’ve vastly expanded your talent pool.

Jobs Recruiting | moore | | Leave a comment

The wonders of outsourcing devops

I have maintained a Jenkins server (actually, it was Hudson, the precursor). I’ve run my own database server.  I’ve installed a bug tracking system, and even extended it. I’ve set up web servers (apache and nginx).

And I’ll tell you what, if I never have to do any of these again, I’ll be happy as a clam. There are so many tools out there that let you outsource your infrastructure.  Often they start out free and end up charging when you reach a certain point.

By outsourcing the infrastructure to a service privder, you you let specialists focus on maintaining that infrastructure. They achieve scale that you’d be hard pressed to. They hire experts that you won’t be able to hire. They respond to vulnerabilities like it is their job (which it is).

Using one of these services also lets you punch above your weight. If you want, with AWS or GCP you can run your application in multiple data centers around the globe. With heroku, you can scale out during busy times, and you can scale in during slow times. With circleci or github or many of the other devtool offerings, you can have your ci/cd/source repository environment continually improved, without any effort on your part (besides paying the credit card bill).  Specialization wins.

What is the downside? You lose control–the ability to fine tune your infrastructure in ways that the service provider may not have thought of.  You have to conform to their view of the world.  You also may, depending on the service provider, have performance impacted.

At a certain scale, you may need that control and that performance.  But, you may or may not reach that scale.

It can be frustrating to have to workaround issues that, if you just had the appropriate level of access, you would be able to fix quickly.  It’s also frustrating having to come up to speed on the docs and environment that the service provider makes available.

That said, I try to remember all the other tasks that these services are taking off my plate, and the focus allowed on the unique business differentiators.

AWS Programming Useful Tools | moore | | Leave a comment

The power of automated testing

It took me a long time to understand the power of automated testing.  After all, it can end up being a large portion of your codebase and can be brittle.  Sometimes it feels like writing tests “gets in the way” of getting things done.  At one project I worked on, a colleague complained that it felt like you spent 5 minutes changing the production code and an hour changing the tests.  (And to be fair, sometimes that’s true, and there’s a balance to be struck between test code coverage and speed of development.  This can also indicate you need to spend time refactoring your tests, as you have multiple different test components testing the same production code.)

I like to think of tests like a gentle swaddling of your code.  It conforms as the body of your code changes, but changing that code does require some re-work of the tests.  And, if your code fails, it fails into the gentle swaddling, as opposed to the cruel outside world (bleeding all over your production users).  Alright, maybe the analogy fails :).

I write this today because I’m in the middle of a refactor of one of the scariest bits of The Food Corridor.  (Given we’re so young, it’s not that scary, but it’s quite complex–handling the creation and updating of bookings.)  There are many many paths through the code and if I didn’t have automated testing, I’d be far more worried about the changes I’m making.

So, consider this blog post to be a thank you to past me for making future me’s life easier by writing a comprehensive automated test suite.  If you don’t have one, you should.

Rails Testing The Food Corridor | moore | | Leave a comment

Customer support best practice: Shared inbox

So, both at The Food Corridor and at 8z, I saw the power of a shared inbox for support.  Sure, there are tools like Zendesk out there, but when you are just getting started, having a common inbox (typically in gmail) is a great idea.  It’s free, everyone knows how to handle it, it has a great mobile client, and it is very flexible.  It’s worth noting that both of these companies had relatively small support teams (less than 10)–once you get to a larger team, more specialized tools will be helpful.

What should you call it?  You could call it ‘support@example.com’ or ‘help@example.com’, but I’ve seen a lot of folks go with ‘hello@example.com’, which can be used for all kinds of communication.

You can also use this as the master account for other subscriptions you have, whether of business specific newsletters or SaaS tools.  If someone is signing up for something that other employees may need to use, use this as the email address.  (You may want to set up a second inbox or alias for technical tools.)  You can also augment the inbox with plugins.  Rapportive was one that I’ve tried in the past, and I love Streak for easy CRM and email scheduling.

When you are responding to customer requests from this inbox, should you sign with your name or not?  It used to drive me crazy when folks wouldn’t sign emails at 8z, because I liked to know who I was corresponding with.  Now that I’m doing customer support, I see the benefit of being anonymous.  When you have different folks signing emails, it can confuse the customer (“who am I dealing with again”).  So I’d recommend signing or not signing depending on the context.  If this is a person who you haven’t dealt with before and they have no context for who you are, leave it unsigned (or, as we do at The Food Corridor, sign with something generic–“The Food Corridor Gnomes”).  If they are an existing client, then sign it.

Finally, keep this inbox clean, otherwise the value will decrease dramatically.  Make sure you archive anything that is taken care of.  Take shifts of who ‘owns’ checking the inbox.  I’ve definitely seen folks re-forward an email so it goes to the top of the inbox, or using slack to coordinate responses.  Forward questions or information for specific individuals or teams off to their email accounts, then archive.

Business Useful Tools | moore | | Leave a comment

Rails Gems: paper_trail

Rails has an amazing number of quality libraries (called ‘gems’) and while building The Food Corridor application, I was lucky enough to be able to use some of them.

If you want to easily track changes in your rails models over time, the paper_trail gem has you covered.  Simple to install, compatible with many databases, and very easy to use.  I’d point you to the docs for installation, but for usage, all you have to do is drop has_paper_trail into your model class.

After that’s deployed, every change to those models is tracked, including the entire model state.  From an operational perspective, I’m a bit worried about the size of the audit table (called versions), but thus far it hasn’t grown too fast.  It wouldn’t be hard to prune either.

I use the data this gem records for two reasons.  The primary purpose is to debug the system.  When there’s an issue because the system doesn’t act as expected, or a question on when a model changed, I can go back and see how the model has changed over time.  I can do this either by directly inspecting the versions table or via the rails console.  If I use the latter, then I can actually pull back the model object as it existed in the past.  This has been tremendously helpful in tracking down bugs.

The secondary use of this data is to allow the system to see changes over time, and execute code based on those changes.  This doesn’t happen often, and I’d prefer for such changes to be captured more explicitly, but in at least one case this gem has allowed me to fix an issue in a time efficient manner.

If you’re building a complicated system that changes over time, as most real world applications do, having some kind of audit trail can be extremely helpful.  paper_trail makes it trivial to get an audit trail set up in your rails app.

Rails The Food Corridor Useful Tools | moore | | Leave a comment

Re:Invent Videos

AWS Re:Invent is supposed to be a great conference.  I have thus far been unable to attend, but the videos of the presentations are posted online with about a day’s lag.  So, like most conferences, you really should be networking and meeting people face to face rather than attending the presentations.

Here’s the AWS Youtube channel where you can watch all the videos, or just sample them.

I’ve found the talks to be of varying quality.  Some just rehash the docs, but others, especially the deep dives, discuss interesting aspects of the AWS infrastructure that I haven’t found to be documented anywhere (here’s a great talk about Elastic Block Storage from 2016).  The talks by real customers also give a great viewpoint into how AWS’s offerings are actually implemented to provide business value (here’s a great talk from 2016 about using Amazon Machine Learning to predict real estate transactions).

It’s a sprawling conference, well suited to AWS’s sprawling offering, and I bet no matter what your interest, you will be able to find a video worth watching.

Amazon Machine Learning AWS Video | moore | | Leave a comment