Skip to content

Monthly Archives: March 2018

Let AWS RDS handle database scutwork

Amazon DatabasesRDS is a service I’ve mentioned in the past, but it’s fantastic. You can outsource large chunks of database administration to AWS. Tasks you can forget about include backups, failover, read only replicas, and OS and DB upgrades.

This is a great fit for spinning up databases for small scale to large scale systems and prototyping.

Things to keep in mind if you start using RDS:

  • The database is launched into a VPC and will have a security group around it. You’ll need to allow IP addresses or security groups access to the port the database is living on or your connections will time out.
  • The database RDS creates is a normal database that you can manage like you can any other database you have set up and installed, but there are certain limitations (for example, no MySQL UDFs). Read the documentation and understand the limitations, but be aware they are constantly changing. I suggest subscribing to the AWS Database blog RDS category for updates.
  • RDS uses EBS under the covers and has the performance constraints of that technology. For the largest scale production systems you’ll want to test before jumping in whole hog.
  • If you are using MySQL or PostgreSQL and are running into concurrency problems, Aurora may be worth evaluating.
  • If you want to have backups past thirty five days for peace of mind of compliance concerns, you’ll need manual snapshots.
  • RDS only supports certain RDBMS and limits databases to certain sizes. If you want to run anything else on AWS, you will need to self manage your DB on EC2 or look at other data management solutions. Here are some other gotchas.
  • When using RDS you aren’t freed from all database administration tasks. There are still users to manage, indices to add, and queries to tune. Most of your RDBMS skillset is applicable to RDS, however. You’ll also need to determine when to schedule DB and OS upgrades, backups and how to size your instances. You still need to set up the optimal architecture of an RDS system including standbys and read only replicas and do other configuration both at the network and database level.
  • You can manage RDS system attributes via cloudformation, terraform and the CLI in the same way you can manage other AWS infrastructure. That said, the RDS system is stateful so you can’t treat it entirely as “cattle”.

You can learn more about RDS in the extensive documentation.

Swagger looks pretty good from here

SwaggerA few years ago I was working on an API that my client was going to make available to some of their clients. I used Swagger, which I’d heard about from a Gluecon presentation.

I was unimpressed. I recall having difficulty getting the online tool to work and the documentation generated was poor. This could very well have been user error or my misunderstanding of the sweet spot of the tool, but for whatever reason it wasn’t a fit for the problem.

Fast forward a few years and I was talking to a company about a position. They were planning to use Swagger to generate their API SDKs. They had an API which was crucial to their business and were currently supporting a large number of SDKs on a bespoke basis. I reviewed the documentation of Swagger and downloaded the 2.3 version. I was very quickly able to generate a number of client and server stubs using the codegen project. They have a long list of supported languages, but I quickly generated ruby, csharp and perl client bindings and a simple rails5 and spring server. I didn’t push these through to production and I’m sure that if I had I’d have learned about the rough edges (it’s always nice to check out Stack Overflow for rough edges for any technology–here’s questions for swagger). Regardless, the ability to take a simple JSON configuration file and create API front ends and backends in minutes was quite impressive.

Note that the codegen tools are all in java, so if leveraging that technology gives you the willies, you might want to look around for another solution. Note also that Swagger has now been moved to the OpenAPI project (as of version 2.0 of Swagger). If you want to know more about that, here’s the blog post announcing that move. Here’s a look at features of the next version.

If you are developing an API first company (and there are good reasons to be that), I’d recommend taking a long hard look at Swagger. The speed of adding SDK support as well as the community around this tooling look to be huge advantages.

Blockchain now in the trough of disillusionment?

ChainsLooks like the blockchain may be headed into the trough of disillusionment. See also Kevin Owocki’s thoughts on ICOs.

This happens to every technology. There is wild optimism over the usefulness (remember Iridium?), which is overblown. It even happened in the 1800s with trains. That said, I’m sure there are uses for blockchain technology beyond stores of value, and am looking forward to seeing those emerge. For a fascinating read on the rollout of technology, see The Deployment Age.

Remote work opens up the labor pool

PoolThis tweet from Justin Searls (from TestDouble) discusses how companies who hire remote developers don’t have any issue finding applicants (to the point where they don’t need to pay recruiting fees). This doesn’t surprise me at all. There are a number of reasons why remote work will lead to more job applicants.

  • There’s a larger labor pool. This is the biggest advantage, and accrues to both sides of the labor market. (There’s a larger labor pool, but there’s a larger set of hiring companies as well.) Even if you are in SF, the labor pool for a particular technical skill is going to be larger if you include remote work (because your pool will include everyone in SF as well as everywhere else). If you can handle the legal tangle, your labor pool can extend past national boundaries as well.
  • The remote developer has more time. Typically the “commute” for a remote worker is on the order of minutes or seconds, as they walk to a different room in their house. Compare this with tens of minutes or hours of driving, public transit or biking. A commute to an office is a hidden job expense which can total hours of unpaid labor every week (or you could look at it as the price you pay for living where you want to live, as opposed to where your employer is located).
  • Remote work promotes different, less intrusive communication practices. Meetings take place on slack. PRs are reviewed online, rather than code review happening by looking over someone’s shoulder. Aysnchronous communication is favored (although that can certainly happen within colocated companies as well). These processes are far better for knowledge sharing than anything else, especially as a team scales.
  • Similarly, flow is easier to achieve when you are a remote worker, as long as you turn off your notifications (and set up your office appropriately). Depending on your workload this can be valuable and/or appealing.
  • Remote developers in general have more control over their environments. Most people enjoy this control (which, by the way, is free or cheap to the company).
  • People don’t have to uproot their lives to work for an interesting company.
  • There’s a larger labor pool. This is so important it’s worth listing twice.

There are of course many benefits to being onsite.

  • Easy, free, higher bandwidth communication that doesn’t require downloading yet another video conferencing plugin. Humans have been communicating face to face for hundreds of thousands of years and are finely tuned communication machines. If communication is a wall of text, context can be lost and misunderstanding can ensue.
  • Interaction outside of work is easier. This can be more important for different folks at different times in their career.
  • Water cooler/serendipitous department cross pollination can occur when people are in the same physical space.
  • Some types of technical discussion are just easier in person (I still haven’t find a truly great online whiteboarding application).
  • Some folks have a harder time staying focused or being motivated at home.

There are of course solutions for remote workers for all of the above onsite advantages. Every company and worker needs to determine what works best for them.

But just like e-commerce changed retail and social networks changed keeping in touch with old friends and work colleagues, I expect remote work to remake the modern team organization structure.

PS For more on this, consider this subsequent tweet from Justin.

Ephemeralization is the future

In the same vein as “write less software“, this is a great post about ephemeralization. To ephemeralize software is to make it disappear either by pushing it to another provider or just removing the feature. From the post:

Proposals to ephemeralize a component or feature will sometimes be met with emotionally-charged responses from your team. It’s totally reasonable to feel attached to a component everyone has worked hard on and has been important historically. But realize that the component has value because it got you to where you are today, not necessarily because of its ongoing existence in the future.

Path dependence of software is very real. What you’ve done before determines what you can easily do in the future. But you can take large right turns if you are prepared to make the investment. As the above quote notes, it’s not just an investment in time, knowledge and money, it can also be an emotional investment.

“Run less software”

RunningThis post from the folks at Intercom makes some really good points about the benefits you can get from leveraging other software solutions. It’s an interesting article, but the source works at a company that offers a SaaS solution to help you help your customers (I’m a fan). You can read some interesting discussion at HN. An interesting quote from the post:

Choosing standard technologies is very similar to this, only in software. You need to constrain yourself to solving problems mostly, but not exclusively, with a small, specific set of standard tools. By doing this over time we become experts in them. Then, we are able to build better and faster solutions from them.

From my experience with small teams, having a standard software stack is a big win. It lets you reuse business logic and skills across your business. (The spread of microservices may mean this matters less in the future, though with a small team and product microservices may be overkill.)

However, I wish Rich had titled it “write less software” because that is really what he is advocating for. When I use AWS or Intercom or Gmail, I’m not really running less software. In fact, I may be running more. Since I don’t have insight into those applications, I really don’t know. But I’m not concerned about writing, reading and maintaining as much software, and that is the true win.

AWS Quick Starts

StopwatchIf you are looking to stand up an application quickly, I often recommend the AWS marketplace. This service has thousands of vendor maintained solutions and is a great way to get going quickly. Note that some of the solutions have extra per hour charges, and if that is the case per second billing won’t apply. These solutions are focused on individual AWS EC2 instance images (so you can quickly stand up a phpbb instance or a redmine server, for example).

However, another good option is AWS Quick Starts. These are recipes for deployments, possibly of multiple virtual machines, and are aimed at handling larger business problems. There are over 80 listed on the Quick Start page right now, ranging from creating a data lake to a HIPAA reference architecture to running devops tools like consul and bitbucket. These solutions may or may not carry additional charges, so make sure review licensing and billing information as well as functionality.

If you are thinking about setting up a complex system in AWS, it’s worth some time to see if someone has put a reference Quick Start together. It may not fit your needs perfectly, but can be a good place to begin.

Questions to ask interviewers

Person in suitI’ve been doing a fair amount of interviewing lately (looking for work, not hiring) and there’s always that moment when the interviewer asks “so, do you have any questions for me”. Here are some of my favorite questions:

  • How long have you been working at company YYY? This helps me understand their perspective on the company. If they are new, they’ll have fresh eyes. If they have been at the company for years, it’s interesting to understand why they’ve stayed. This can also lead to a discussion of turnover.
  • What is a typical day like for this position? This helps me understand what the company emphasizes. This can also lead to a discussion of the development cycle (weekly releases, etc).
  • Who are your typical customers? When talking to a company, I always want to know who they sell to. This shows I have an interest in the company beyond just the tech work I’ll be doing. I want to hear words like “wide variety of clients”, “we find most of our customers by referral” or “we’re focused on niche YYY”. Of course I can always do research and see how the company markets itself and see if what I am hearing from the interviewer is consistent with how the company presents itself.
  • What do you do when folks are on the bench (if it is a consulting company)? “The bench” is where consultants who aren’t making money for the company are. What kind of projects are they working on? How often are folks on the bench?
  • Finally, what is the worst part about working at company YYY? This is the inverse of the “what is your biggest weakness” question that is sometimes asked. But every company has its warts. I’m probably going to find out anyway, but it’s good to ask and get the interviewer’s perspective.

What are your favorite questions to ask interviewers?

Management Debt

This post by Ben Horowitz is well worth the read as he discusses a few types of management debt, where a short term decision has long term consequences that you’ll have to expend energy to repay.

This quote stuck out:

Companies execute well when everybody is on the same page and everybody is constantly improving. In a vacuum of feedback, there is almost no chance that your company will perform optimally across either dimension. Directions with no corrections will seem fuzzy and obtuse. People rarely improve weakness that they are unaware of. The ultimate price you will pay for not giving feedback: systematically crappy company performance.

If you’ve read “The Hard Thing About Hard Things”, you’ll remember he talks about giving feedback all the time so that when you have to give feedback it’s not exceptional (he also talks about how this is not a normal human capacity). When you don’t set up systems for this feedback eventually you end up with a rudderless ship. There’s a reason that systematized feedback methods (the dreaded yearly review, among others) exist in almost every organization.

Book Review: Working With Coders

Woman with 1s and 0sSoftware is so integral to business processes and relatively inexpensive compared to labor that I believe every company is going to be a custom software company, in the same way that every company is an accounting company or every company uses paper. I happened on an interesting blog post and saw the author had written a book, “Working With Coders”. How non technical folks interact with coders is a topic of perennial interest to me, so I picked it up after reading the first few pages on Amazon. The book is written for clients, CEOs or project managers who are going to be working with developers to deliver applications that will provide business value.

Frankly, I couldn’t put it down.

The author, Patrick, is an engaging, opinionated writer. He breaks down complicated concepts into easily digestible pieces. Where there’s more to the story, there’s a footnote with a snarky comment or a link to more information. Patrick also provides nuts and bolts examples to show why something that seems simple to change is not (scaling text in a browser, for example). He also covers how big decisions like language, frameworks and library choices at the beginning of a project constrain freedom and choices further down.

Patrick covers what developers do, how they think, and why projects often fail. I thought his explanation of the benefits of agile development was darn good, and his explanation that even agile projects fail more often then they succeed was pretty depressing. He also discusses how the house construction metaphor for building software is just a big fat untruth.

I also enjoyed the section about testing in general, the various types of testing, and where they make sense. There’s also a section on finding coders, including a good explanation of why not to hire them as employees (you might be better off just hiring a development shop, depending on your needs). The chapter on how to deal with common issues (“the team hates each other”, “we’re behind schedule”) was worthwhile. His solutions won’t work for everyone. Maybe you’ll want to deal with these issues differently, but considering them before they happen will only help you prepare.

Of course, I also enjoyed the chapter on how to keep coders happy (continuous learning, quiet, a fast computer). In general the author is careful to avoid stereotypes, but does do a good job of covering common themes. I haven’t met too many developers who love working in bullpen environments.

I am definitely not the target audience. Neither is someone who is an experienced manager of developers. However, I am a subject of the book, so it resonated with me and I definitely found myself nodding along. There aren’t too many books I have wanted to distribute copies of (the two others are “The Hard Thing About Hard Things” and “Climate Wars”), but this is one.

If you work in a consulting practice with inexperienced clients or if you work in a product company with an owner or higher up that isn’t technical, reading this book will give you insights into their questions and thought processes. And if you can find a way to give them this book without being condescending (“hey, I found this book fascinating for helping facilitate conversation, maybe you will too”), both they and you will benefit.