Skip to content

The most underappreciated AWS service

Three letters for you.

V. P. C.

Amazon Web Services’ Virtual Private Cloud is often the first real hurdle for developers and others looking to understand cloud systems. I know it was for me; I’d not had much networking experience when I first encountered it. To really grok VPC, you need to have at least a mild understanding of network architecture, including subnets and gateways. In my experience, such knowledge isn’t par for the course with software developers.

Most software developers can quickly understand EC2 (oh, a virtual machine) and S3 (ah, a gigantic disk drive). However, VPC’s networking abstractions are tougher. However, VPC is magic.

Let’s talk a bit more about this. AWS built a performant software defined networking layer. But they didn’t just port all the concepts from the physical world into the cloud. At least, when I read the Unix and Linux Sysadmin’s Guide, which dives deep into such things, that’s how it looks to me.

Instead, AWS followed the 80/20 rule, giving developers and architects flexibility to build real network architectures. These architectures can support real world applications, with real world security needs and compliance concerns.

Yes, you can also achieve such separation using different AWS constructs (and should!). IAM and Organizations spring to mind. But for many, making sure that they could easily port their current security posture to AWS made a cloud transition easier.

VPC simplifies the networking layer enough that even developers with little networking experience can understand it (like me!). AWS has even provided network logging so that, should you need to delve deep into your networking layer for troubleshooting or auditing, you can.

VPC is also fundamental to other services. EC2 instances are still the majority of AWS budgets, according to Corey Quinn’s 2020 Re:Invent rebuttal.

VPCs are where those “machines” exist. The fancy managed services AWS wants you to use so they can lock you in? Often they are accessed by dropping ENIs into your VPC.

If not, these services are running in VPCs managed by AWS, therefore inaccessible to you except through tightly managed interfaces. See, the security works!

To top it all off, VPC is free and ubiquitous.

AWS’s VPC is the water to cloud engineers’ fish; we don’t even see it, let alone appreciate it.

Book Review: The Cuckoo’s Egg

I recently finished “The Cuckoo’s Egg”, by Clifford Stoll. It was a fascinating non-fiction book exploring the foundations of computer security in a personable format.

The setting is the mid 1980s. The author discovers something weird on his academic computer system. There’s an unexplained charge of 75 cents. He digs deeper, discovers that someone who’s left the university is logging in.

After further investigation, he discovers that the user who is logging in is an intruder. After discussing the situation with his boss, he gets three weeks to find out who they are. He figures that’s plenty of time.

The investigation ends up taking a year.

It also extends far beyond his academic systems, both in scope and effort. Stoll talks to numerous government agencies and private organizations, letting them know they’ve been attacked and getting their assistance tracing the hacker. He sleeps under his desk. He rigs up a pager so that he can know which accounts the hacker is using. Stoll sets up printers so that every word the hacker types is recorded, unbeknownst to him.

It’s quite the tale. As someone who has worked with software for years, I really appreciated the historical nature of it. When I became aware of the internet, in my youth, some of the groups and communities he mentions were still around; I remember reading and posting to usenet. But many of the systems were before my time. I’ve never touched a computer running VMS, for instance.

But, for all the history, the people problems were the same: users not changing passwords, system managers not locking their software down, bureaucrats happy to take information but not willing to share. Let’s just say, mistakes were made.

I also enjoyed the author’s interspersal of lived experiences. We don’t simply follow one computer nerd tracking another. We also learn about milkshakes, parties in San Francisco, curry nights and his first experience with the microwave. While some phrases and analogies are repeated (“should we thank someone who goes to a little town and robs people to illustrate they should lock their doors” pops up at least twice), in general the book is pretty readable. Stoll’s personal stories and musings help that readability immensely.

All in all, a great book if you are interested in the history of computing or modern security practices. If you’re interested in learning more, you can check out a paper he wrote based on the same experience for ACM.

Book Review: Algorithms to live by

I finished Algorithms to Live By, by Brian Christian and Tom Griffiths. I enjoyed it immensely.

The premise of the book is that computer programs make decisions with algorithms all the time. There’s math behind how they do so, including tradeoffs and time considerations. Human beings face some of the same decisions and there’s no reason we can’t use this knowledge to live better lives.

Yes, this is kind of a self help book—“you can live a better life by thinking like a computer”. But it’s math, folks.

I actually recommended it to my SO because I feel like she’d understand me better after reading it. I’m always talking about how much of the stress in our life is due to resource contention.

The book covers a wide swathe of decision making. Here are some examples of the broad categories and some specific references:

  • when to decide to stop looking for a house or a partner
  • when to explore new knowledge vs exploit current knowledge in the context of clinical trials
  • how randomness can lead to better outcomes
  • how caches can help you determine what of your wardrobe to keep
  • how overfitting warps sports like fencing

As illustrated above, this is not a book about theory, but is actually hands on. I don’t recall seeing a single equation, though there are graphs. And the authors do mention plenty of researcher names, provide footnotes and have a twenty page bibliography. So if you want to learn more about the formal proofs, the info is there.

It’s hard to choose just one takeaway from this book, but if I had to pick one, it’d be the fact that game theory shows that you need external forces to avoid the tragedy of the commons, and that emotions may play a role in providing that.

Here’s an excerpt if you want a deeper look.

If you spend any time thinking about how you can make decisions better, Algorithms to Live By is worth reading.

A quick look at xkit

I was prototyping a small app in xkit and wanted to document this useful tool. When I first saw this launch on HackerNews, I couldn’t quite understand what the purpose was. But now that I’ve spent a bit of time playing with it, I understand it a bit more.

Suppose you are writing a recipe management SaaS and realize that you want to integrate with some other services. Perhaps you want to be able to export the steps of a recipe to a Trello board, or to a Google doc, or to a PDF.

These are all services available on the internet with an API which will allow end users to give your application access to their accounts. This lets you publish to each user’s Google docs account or Trello board.

(I’m not as familiar with services offering PDF generation functionality, but a quick search turns up some options, including some that you can self host.)

There’s a fair bit of hoop jumping in terms of setting up API keys and OAuth consent screens, however.

And this is the problem that xkit solves. If they’ve already written the connection (here’s a list), it is quite simple to add the ability for a user to connect to the service. With no previous experience, I was able to connect to Trello in about an hour. The user experience of connecting the external SaaS application is really smooth and far better than something I could whip up quickly.

If they haven’t written a connector, I don’t believe you can write one yourself. For example, for that PDF service, you’d need to contact the xkit folks and ask them to add one.

This is different than, say, Zapier, because it’s operating at a different level. Zapier is excellent (and has been for years) at letting users connect their apps. But xkit lets you let your users connect apps, basically letting you build a mini Zapier (in terms of connectivity, not functionality).

You can also host your own app catalog if you want to. I didn’t get into this too much, though, so it’s unclear what the benefits of that are.

They provide a user data store out of the box, but also integrate with a number of other providers (including FusionAuth). This means you can leverage your existing auth solution and still get the easy integration with other third party APIs.

Their pricing seems reasonable, given what they take off your plate.

Nothing’s perfect, however. I found a few documentation bugs, which I let them know about (they host their docs on readme.com and I found the suggestion process delightful). When I tried to sign up, the service was down, but a quick Tweet exchange resolved the issue within 30 minutes.

It is bizarre to me as an authentication focused company that they don’t have a “forgot password” link on their login pages. The documentation is javascript heavy, with nary a mention of other languages, but that’s understandable as they’re just starting out. It’s also strangely video heavy, which I found a bit distracting; that, however, could just be my learning style.

All in all, if you are looking to integrate third party APIs which require OAuth interactions on the part of your users, you’d be well served to take a look at xkit.

Book Review: A Memory Called Empire

I was at a bookstore the other day and was rummaging around for an escape book. I had picked up one book based on staff recommendation, but came across another that had won the Nebula. A Memory Called Empire appeared to be an award winning space opera novel.

I thought, why not, and picked it up. I was not disappointed.

There are two main cultures of different power and viewpoints. The action follows an new, unprepared ambassador from one culture to the other. The cultures are coherent and yet alien. Alien to each other and to me. Competitive poetry, internal reflection, and constant political intrigue define one culture. We get drips and drabs of the other, less powerful civilization, but you learn enough to be appreciative of their scrappyness and reverence for the collective.

The cultures are never described to the detriment of the action. The ambassador is dropped into a political mess and acts and reacts to help save her nation and herself. Whether she is trying to meet powerful people to negotiate for her people, reading correspondence, or escape danger, there’s no downtime. The entire book happens in span of about two weeks.

I also enjoyed the character development. Many of the characters pop in and out of the storyline, but you follow a few main characters for a while. You get to understand and appreciate the way they interact with each other. It doesn’t feel forced at all. At the same time, the “otherness” of the ambassador provides a constant tension which is understandable to anyone who has been dropped into an uncomfortable situation.

The plot revolves around a mystery. But even when it is unveiled, there’s still plenty of excitement, as a confluence of outside, political events ensnare the protagonists.

Some books are so good they keep you reading late into the night. This was one of them.

What I wish I had known when I was starting out as a developer

As I get older, I wish I could reach back and give myself advice. Some of it is life advice (take that leap, kiss that girl, take that trip) but a lot of it is career advice.

There’s so much about being a software developer that you learn along the way. This includes technical knowledge like how to tune a database query and how to use a text editor. It also includes non technical knowledge, what some people term “soft skills”. I tend to think they are actually pretty hard, to be honest. Things like:

  • How to help your manager help you
  • Why writing code isn’t always the best solution
  • Mistakes and how to handle them
  • How to view other parts of the business

These skills took me a long time to learn (and I am still learning them to this day) but have had a material impact on my happiness as a developer.

I am so passionate about this topic that I’ve written over 150 blog posts. Where are these? They’re over at another site I’ve been updating for a few years.

And then I went and wrote a book. I learned a bunch of lessons during that process, including the fact that it’s an intense effort. I wrote it with three audiences in mind:

  • The new developer, with less than five years of experience. This book will help them level up.
  • The person considering software development. This book will give them a glimpse of what being a software developer is like.
  • The mentor. This book will serve as a discussion guide for many interesting aspects of development with a mentee.

The book arrives in August. I hope you’ll check it out.

Full details, including ordering information, over at the Letters To A New Developer website.

Thoughts on static site generators and WordPress

Blog in scrabble lettersIn my last few jobs, I’ve done a lot more writing. I’ve learned to work with static site generators (SSGs), such as jekyll and 11ty. I even moved a database driven side project to Netlify. Here’s an interesting survey about SSGs from Redmonk, if you want to learn a bit more. I am also a longtime WordPress user, and think it has some tremendous strengths. I wanted to capture my thoughts on these two options for building your website.

Here’s why I’d pick WordPress for a content heavy site:

  • Avoid any kind of compilation pipeline or step
  • The authors don’t have technical chops and want WYSIWYG
  • You want more than a blog, with additional functionality pulled from the wide world of plugins. Also, themes!
  • You want to allow people to just write, without software getting in the way

Here’s why I’d pick a static site generator for the same:

  • Need performance and scalability at a low low price
  • Wanted ‘set it and forget it’ security (you can’t hack a static HTML page)
  • Authors are technical enough to write markdown and to leverage the data driven possibilities of an SSG
  • No need for interactive functionality, beyond what JavaScript/JAMStack can provide

Like any other choice in software engineering, these two solutions offer tradeoffs.

I think that SSGs are far better for simpler websites, and in some ways they are a more sophisticated version of very early websites. I remember writing a perl program to take a usenet file and turn it into a set of web pages in the last 1990s. (It was jokes, if you must know the content.) When you build and deploy a static site, you know exactly what you’re getting, but you can also extract common functionality to shared files. You will have to compile it, however, and typically uses a version control system, which can be a lot for some non technical folks.

WordPress is fantastic for just getting going. I can get a blog started in 15 minutes that can be used by anyone who knows the basics of Microsoft Word. The flip side of the speed to first page is operational complexity, slower performance (all things being equal), as well as a more complicated application to secure.

I will say that as a developer, SSGs are growing on me.

PS I know some people have combined them both, and using WordPress as the backend and publishing a set of static pages is really appealing. I’ve done some preliminary work on this, but haven’t found a great solution out there.

Use managed services. Please.

“Use managed services.”

If there was one piece of advice I wish I could shout from the mountains to all cloud engineers, this would be it.

Operations, especially operations at scale, are a hard problem. Edge cases become commonplace. Failure is rampant. Automation and standardization are crucial. People with experience running software and hardware at this scale tend to be rare and expensive. The mistakes they’ve made and situations they’ve learned from aren’t easy to pick up.

When you use a managed service from one of the major cloud vendors, you’re getting access to all the wisdom of their teams and the power of their automation and systems, for the low price of their software.

A managed service is a service like AWS relational database service, Google Cloud SQL or Azure SQL Database. With all three of these services, you’re getting best of breed configuration and management for a relational database system. There’s configuration needed on your part, but hard or tedious tasks like setting up replication or backups can be done quickly and easily (take this from someone who fed and cared for a mysql replication system for years). Depending on your cloud vendor and needs, you can get managed services for key components of modern software systems like:

  • File storage
  • Object caches
  • Message queues
  • Stream processing software
  • ETL tools
  • And more

(Note that these are all components of your application, and will still require developer time to thread together.)

You should use managed services for three reasons.

  • It’s going to be operated well. The expertise that the cloud providers can provide and the automation they can afford to implement will likely surpass what you can do, especially across multiple services.
  • It’s going to be cheaper. Especially when you consider employee costs. The most expensive AWS RDS instance is approximately $100k/year (full price). It’s not an apples to apples comparison, but in many countries you can’t get a database architect for that salary.
  • It’s going to be faster for development. Developers can focus on connecting these pieces of infrastructure rather than learning how to set them up and run them.

A managed service doesn’t work for everyone. If you need to be able to tweak every setting, a managed service won’t let you. You may have stringent performance or security requirements that a managed service can’t meet. You may also start out with a managed service and grow out of it. (Congrats!)

Another important consideration is lock-in. Some managed services are compatible with alternatives (kubernetes services are a good example). If that is the case, you can move clouds. Others are proprietary and will require substantial reworking of your application if you need to migrate.

If you are working in the cloud and you need a building block for your application like a relational database or a message queue, start with a managed service (and self host if it doesn’t meet your needs). Leverage the operational excellence of the cloud vendors, and you’ll be able to build more, faster.

When in doubt, test it out

When I taught AWS certification courses, I’d often get questions about how a service behaved under load or other unusual circumstances. Frequently I could answer from personal experience or by asking other instructors; occasionally class members provided their insights. Sometimes I could dig up relevant vendor documentation.

However, my default answer was:

“Test it for yourself. There’s no substitute for testing.”

This is one of the great advantages of the cloud. When you have a question about the performance or behavior of a service or system, spin it up and test it. This will cost you money and some time configuring the system, but certainly will be cheaper than ordering hardware, racking it and then also configuring the system. When you’re done with your testing, you can tear down the infrastructure and never worry about it again. Sure beats shipping a server back to the manufacturer.

Of course, no testing scenario can replicate production perfectly. But you can get pretty close (especially if you can reuse production traffic).

When you do test, start by documenting what you want to achieve. What is the question you are trying to answer? Make sure to seek feedback from other team members and/or search online, as it’s possible someone has already answered your question. If you do find answers, understand under what circumstances the tests were performed, as the cloud and the offered services change over time.

Some examples of cloud infrastructure questions you might want to answer:

  • How do EBS volumes of different sizes and types perform under load?
  • When a Kubernetes cluster running on GKE is under load, what happens when you add an additional node? An additional pod?
  • What happens when you turn off a NAT gateway while a file is being uploaded to S3 from an EC2 instance in a private subnet (without an S3 VPC endpoint)?
  • What is the cold start time for an empty Azure function? What about a function loading your dlls?

Think about what steps you are going to take to try to answer the question.

With your question and methodology spelled out, spin up your testing environment. Having your infrastructure represented as code will make this quick, especially if you have a complicated environment. If you are creating the test environment manually, record settings and other configuration in a text file to be able to re-create the environment later.

Run your tests. If you are load testing, find an open source or commercial load testing tool. What you need depends on your goals: you need a different tool to test 100k+ simultaneous users on a website than you do when trying to understand how an internal API handles 100 requests/second.

Review the data to see if your questions are answered. More questions or areas of interest may appear. Adjust your tests to answer them.

Once you have your answers to the desired level of certainty, tear down your testing infrastructure.

Document what you tested, how you tested and your results. Circulate this internally to help your team. If possible, publish it on your company blog to both help others in the same boat and to boost your company’s standing in the community.

All the vendor documentation in the world is no substitute for rolling up your sleeves and testing.

Joining FusionAuth

I wanted to let y’all know that I’ve joined FusionAuth as a developer advocate. I’ll be working to help our customers succeed and promote the virtues of standards based user management systems. I get to write a lot of content and example applications against a full featured API.

I’ve built enough systems to know two things:

  • Users and their behavior are almost always a key part of any software application.
  • User management is difficult to get right, especially if you want to use secure best practices and standards such as OAuth.

FusionAuth wants to elevate everyone’s user identity management system. The community edition is free and will always be. (It’s important to note that it is free as in beer, not free as in speech, but almost all of the development happens in the open.) If you want to run FusionAuth on your own forever, that’s great! You get a secure user store that supports OAuth, SAML and two factor authentication, free forever. We’ll happily provide you “best effort” support in our forums and we’ve seen the community help each other out too (most notably in the creation of helm charts to run FusionAuth on Kubernetes).

If, on the other hand, you find value in FusionAuth and want guaranteed support, custom development, or hosting, we’re happy to sell that to you. The price is often a fraction of the other solutions out there. Another differentiator for FusionAuth is that you can host it wherever you want: in your data center, in your cloud, or on our cloud servers. Not every client needs that level of control, but many do.

I really love the business model of providing a ton of value to your end users and monetizing only a small percentage of them with unique needs. (I’ve been involved in this type of business before.) The business thrives and there’s a ton of consumer surplus generated.

I’m really excited about this opportunity. It’s a nimble company with a passionate team based in Denver. If you need a user identity management system built from the ground up for developer happiness, please check us out.