Skip to content

Monthly Archives: June 2018

Startup COOs: What do they do?

I really enjoyed this post about what startupo COOs do. It was interesting because it wasn’t just opinion, there was also data. I particularly enjoyed the evolution of the author’s role over time, and the percentage of COOs that owned various responsibilities. To be fair, it was a small set of respondents in one geographic area, but still interesting.

However, the money quote was:

I actually have [simple way] of explaining what I do, and I would sum it up this way: taking things off my CEO’s plate, and figuring out how to thoughtfully scale the company.

Of course every company’s different, but I’ve seen the pattern of a visionary and an operator in a couple of companies, and it’s powerful.

Business Startup | moore |

Rails Views Cached In Production Environment

Railroad tracksI was troubleshooting a data issue in a production environment. It wasn’t heroku, rather a rails environment hosted on AWS. It was Rails 4.2, ruby 2.2.3.

First off, it’s worth noting that there were two or three bugs that were commingled and causing issues for our client. A number of folks had spent a long time trying to troubleshoot the issue. At this point, I was tasked with taking a look and had access to all the environments. The problem only seemed to appear on production, and appeared to be a data issue. I was editing views directly on production to track down where the data issue appeared, as well as running queries on the production database and using the rails console to see what rails thought was happening. In other words, it was a hot mess. However, this debugging story isn’t the point of this post. Rather, I ran into the most peculiar situation and wanted to document it so that if I ever ran into it in the future, I would remember it.

Basically, I had a view that looked something like this:

text
<% cache('[key]') %>
other text
<% end %>

I changed text to be new text which included some useful debugging information. Debugged the problem and went on my merry way. The next day, early, I realized that I hadn’t changed it back, so logged back into prod and changed it back to text. Reloaded the page and didn’t see the change. What? Tried to clear the cache using the rails console and Rails.cache.delete(). No change.

After lots of googling, I realized that the view text, outside of cache tags, is cached in some other fashion. I finally figured out how to reset the cache by following these steps:

  • edit config/environments/production.rb
  • set config.cache_classes=false
  • restart passenger by touching tmp/restart.txt (see here for more on that)
  • reload the page, and now I could see text instead of new text
  • set config.cache_classes=true
  • restart passenger by touching tmp/restart.txt

This only happens when you both have a mutable production environment and are changing the view files in that environment. This won’t occur if you were using a platform like Heroku, or if you never troubleshot on production.

AWS Rails | moore | | Leave a comment

(Links To) Advice For Someone Selling a SaaS Business

Sold signI ran into someone at a meetup recently who’d built a SaaS that had a pretty decent MRR. Enough to support one person. Which is a huge achievement!

He was wondering what options he had to either grow or exit the business. This is something I’ve been reading about for a number of years, so I had some advice. I thought I’d write it down so that others could benefit (or chime in). These are resources I’ve found insightful.

This is a great first hand account by patio11 of selling a software business (it wasn’t SaaS, rather a one time digital product sale, but I think there are a number of common themes). He mentions the broker he uses, the due diligence process, and what you can do now to set yourself up for success (have separate accounts, for one).

I can’t even talk about SaaS without telling folks to raise their prices. It’s a reflex now. Amy Hoy has two great posts on this: grandfathering and new features (with a lot of communication mixed in). I experienced this myself at a previous startup, where we almost doubled our monthly subscription price in 18 months.

Finally, here’s an interesting post from a venture capitalist about how private equity is a new exit option for SaaS companies. In that vein, I chatted briefly with one such PE firm, SureSwift Capital, about part time work a year or so ago. I don’t know how they are to work with (the position wasn’t a fit) but at the time they were focused on acquiring SaaS companies with good MRR and helping them grow.

Business Startup | moore | | Leave a comment

What’s New With Ruby?

Red Rubyish DiamondFor the past couple of months I’ve been doing a short segment at the beginning of the Boulder Ruby Meetup called “What’s happened in Rubyland?”

I basically look at 3-4 blogs and google searches and see what is happening. Of course, far more than what I can collate is happening (I don’t look at any major gem releases, for example) but this gives quick insight into major happenings.

Here are the past three editions. Enjoy the starkness of my presentation.

PS We’re always on the lookout for speakers. Let me or tweet the organizers if you’re interested.

 

Dynamic Languages Rails | moore | | Leave a comment

Load Testing Weirdness With AWS Aurora

Confused personSo I was doing a load test and saw behavior that reminded me that sometimes you just need to test.

Ran a test with 1500 requests/second with multiple servers (20ish) and smaller number of bigger servers (2-3). Saw some weird behavior with a number of 500 errors (bad gateway). Didn’t see these errors under a lower load.

Looked at the database (an aurora cluster with a single read and a single write instance) and saw that it was maxed out (cpu pegged, connections at max, couldn’t even connect at times.

Thought I need to upgrade the database. I upgraded the write instance. It was late and I failed to notice that that upgrade flipped the read and the write instances. So now the read instance was at the bigger server size and the write instance was at the smaller (original) server size. Then I re-ran the load test and everything went swimmingly (response time under 500 ms, where before it had spiked to 100 secs or more).

Great, problem solved. The larger instance size solved it.

But wait, it didn’t. The app was connecting to the primary endpoint, which is the master write node. I didn’t believe it, so I double checked and matched test times against connection spikes to the db.

So somehow, the flipping of the database to have a different primary Aurora instance (but no change in db size) caused a radical change in system behavior under heavyish loadfor a distributed php application.

Mysteries.

Amazon RDS AWS Databases Testing | moore | | Leave a comment

Using AWS for load testing experimentation

Someone with heavy weightThe cloud is amazing for load testing your system. If you design your system to be behind a load balancer (which, in many applications, means pushing state to a database and having stateless compute nodes), you can easily switch out those nodes in different scenarios.

I just load tested a system I’m working on and changing out the compute nodes was fairly easy. Once I’d built a number of servers (something I scripted partially but didn’t fully automate because the return wasn’t there) and troubleshot some horizontal scaling issues that popped up in the application, I was able to:

  • take a server out of service behind the load balancer
  • stop it
  • change the instance type
  • start it
  • re-run any needed config changes on the server
  • update DNS if needed (depending on if you have a pinned IP address or not)
  • add it back to the load balancer

Swap out a few instances and you have a new setup for your load test. When you are done, follow the process in reverse to save yourself some money.

Incidentally, increasing the number or size of compute nodes didn’t have the desired effect of being able to handle more load.

What turned out to be the root issue? The database was pegged, both in terms of CPU and connections. Just goes to show that when you’re load testing, you really need to be looking at different aspects of the system, thinking about where your weak bottlenecks are, and use the scientific method of hypothesis, experiment, result.

AWS Cloud Computing | moore | | 1 Comment