Skip to content

Online community power outside of devrel

I’ve been working in devrel for while and one of the key components is community–you want to support the developer community around your product or the larger ecosystem. I witnessed this same focus work outside of the developer community and still feel an online curated community is an under appreciated asset. Like all assets, you have to work to attain it, but then after you have it, it pays dividends.

The setting: a little startup in a decidedly nontechnical niche. I’m not going to be too specific because of agreements I made with the company after I left. They did the following:

  • Built up deep expertise in the field before they even started the community. This can’t be rushed, but if you spend a year or two focusing, that’s a a good start. (They’d spent 5 or 10, which is of course better.) This is a critical step that can’t be short cut. If you don’t have this, you’ll have a more difficult time with everything, because you won’t have foundational knowledge about the community for which you are trying to build an online presence.
  • Started an invite only Facebook group. The actual group software matters less than going to where people are. In this case, the target market already hung out on Facebook.
  • Invited people to the group. Not just clients, but anyone who was affiliated with the community.
  • Had a clear code of conduct, especially in relation to commercial activities (aka spam). Discussion of business related issues was encouraged, but unrelated commercial links were removed.
  • Seeded discussions with questions heard elsewhere. Answers were highlighted in blog posts and newsletters as well as shared with the group.
  • Spent time encouraging group members to share their experiences.
  • Promoted the online community to other offline communities with similar interests.

In time (years!, not months), this Facebook group became extremely valuable to the company, in a number of ways:

  • The questions people were asking in the forum could be turned into blog posts, which were helpful for SEO. That is, the company learned from the forum what kinds of questions people the company wanted to target were asking. (Ask if you can do this before you publish what community members may consider to be private! Just ask, sometimes people want their wisdom shared.)
  • The company could see if members were looking for the solution the company was selling. They could then reach out outside of the group and do a gentle ask. Member demand for services could be monitored over time, as group members shifted employers and/or situations.
  • When talking to prospective customers who were not in the group, a membership was an easy, free offer of value. The answers in the group provided insight and many prospects were happy to join, even if they didn’t want to purchase anything from the company.
  • People found the online community and then learned about the company supporting the community. A definite halo effect.
  • If the company has a new offering, they can do market research easily. They know where to find their target customer.

This onlinecommunity definitely isn’t free, even if you use a solution like Facebook groups with no hosting costs. (Facebook groups is not the best option, in my opinion, in terms of long term flexibility, because Facebook owns the user and you don’t have access to information like their email. But at the same time you have to go where the prospective members of your online community are, and anything you can do to reduce friction is crucial.) Moderation and content seeding are also crucial to getting the online community going.

But now that the community is up and running, it is a sustainable moat.

If you run an non-technical business, do you have an online community that you participate in? Do you know of one? If you don’t and can’t find one with a little bit of googling, think about setting one up. It can be a long term sustainable advantage for your business. Like any other assets, you have to invest, but it’ll pay dividends.

Community Devrel | moore |

You should use forums rather than Slack/Discord to support developer community

Hot take after a year or so of trying to build a developer community. If you can pick only one, use forum software rather than synchronous chat software for community building around a developer platform.

While there are tradeoffs in terms of convenience and closeness, for most developer communities a public, managed forum is better than a private, unsearchable Slack.

There are a few key differences between forum software (which includes packages like nodebb, forem, discourse, and others) and chat software (like Slack, Facebook groups, or Discord).

First, though, it’s important to know what you are trying to accomplish. If you are trying to get immediate feedback from a small set of users, then synchronous solutions are better. You can be super responsive, your users will feel loved, and you’ll get feedback quickly. However, synchronous solutions go beyond chat and include phone and video calls. The general goal of validating user feedback at an early stage is beyond the scope of this post.

But as you scale with a chat solution, major problems for the longevity and value of the community emerge.

Problem #1: the memory hole

This occurs when there’s a great answer to a common question, but it isn’t available or is hard to find. This matters more for community Slacks than other synchronous solutions, since Slack limits free plans to 10,000 messages.

It still exists for solutions such as discord because older messages scroll up. Search isn’t great. As a participant it feels easier to just re-ask the question. If the community is vibrant and willing to help newcomers, such questions get answered. If not, they languish or are ignored, frustrating the new participants. Not good.

You can work around the memory hole as a community member by extracting and reifying interesting chat posts. I have done this by generalizing and publishing the messages as blog posts, to a newsletter, or even to a google sheet. But this is additional work that may not be done regularly, or at all.

Side note: for some communities, discussing current events or just chatting with friends, this is actually a feature, not a bug. Who needs to remember who said what six months ago when conversing with friends.

But for developer communities, friendly chat is important, but so is sharing knowledge; the memory hole actively thwarts the latter.

Forums, on the other hand, are optimized for reading. nodebb even suggests related posts as you begin a topic, actively directing people to older posts that may solve their issue without them ever posting.

And if published on the internet, forums are searched via Google.

Problem #2: Google can’t see inside chats

Google is the primary user interface for knowledge gathering among developers.

I hope this statement isn’t controversial. It is based on personal experience and observation, but there’s also some research.

I have seen many many developers use Google as soon as they are confronted by a problem. Youtube, books, going to a specific site and using that search: these all are far less used alternatives when a developer has a question.

Google has made it so easy to find so much good information that most developers have been trained when they face a problem to open a new tab, type in the search term and trust the first page of results.

Chat systems don’t work well with this common workflow, because all the content is hidden.

This means when you use a chat for a developer community, you don’t get compounding benefits when someone, either a team member or a community member, answers a question well or has an insightful comment that would be worth reading. Very few folks ever benefit in the future.

With chat, people who aren’t present at message posting or soon thereafter never learn from that knowledge.

Problem #3: synchronous communication is synchronous

When you are in a chat system, the information is ephemeral. This means that valuable comments can be lost if there is a flurry of other messages.

People can feel ignored even though the reality is that they just posted at an inopportune moment. This feeling can be intimidating; I’ve definitely felt miffed as a question or comment I posted was ignored or unseen and other people’s questions were answered. “Was it me? My topic? Are other people more welcome here than I am?”

People who like to answer questions may feel the need to do so quickly. This may interfere with time for deep work. The Pavlovian response is real; I’ve felt it myself. It feels “better” to write a response to help someone than it does to write a document that will help many, because the former is so concrete and immediate.

When you pick a chat solution, you are optimizing for this kind of response.

Problem #4: less capable moderation tools

Forums have been around a long time. It’s a well understood problem space. There’s a rich set of functionality in most of them for handling the more frustrating aspects of online community management (see also “A Group is its own worst enemy”).

Chat applications have uneven support for this aspect of community management; I have heard Discord is pretty good, but Slack is not.

Remember that when you are running a community, you will inevitably attract trolls and spammers. Make sure you have the tools to protect the community from abuse.

In addition, make sure you have the time/energy. The community may be able to police itself when it gets to a certain size, but initially and for a long while, with a chat solution you may need to be ready to jump in and moderate.

Forums still require attention, don’t get me wrong, but the tooling and the separation of topics means they aren’t quite as vulnerable. They are a higher value target, because of Google, however.

Problem #5: you’re missing out on long tail content

This issue is related to problem #2, but slightly different. When you are building developer tools, there is a wide surface area of support needed. Questions from developers help define that space. When they go to a chat, someone needs to capture the questions and make them public to help future developers. You can capture this knowledge in formal docs.

When using a forum, the answers are made available to the long tail of searchers without any effort at all. A company I worked for got about 5-6% of their traffic from their forum pages.

That traffic was essentially free because the time to answer the questions was required with either solution. (This assumes the question would have been asked in either a chat or a forum.)

Problem #6: questions can be flippant

When I am talking in person to someone and they ask a question, I don’t expect them to have done a ton of research or thinking about it. It’s a conversation, after all.

The same attitude occurs during real time chat.

For technical questions, this can be frustrating because you want to help immediately (see problem #3) and yet you don’t have all the information you need. In async discussions, because they are async, more context is typically provided by the questioner.

This makes it easier for people who want to help to do so.

Why do people use Slack/Discord/etc?

Wow, so many problems with chat and all these reasons are why forum software is better.

So why do so many folks building developer communities choose solutions like Slack or Discord?

There are two motivations that I can see.

One from the company perspective and one from that of the developer.

From the company side: it helps build community between members.

I don’t know about you, but I am much more likely to pitch in and help when I have had a conversation with someone than if some rando drops by with a question and leaves.

A Slack can begin to feel like a real community, where you know people. It doesn’t feel as transactional when I see a question in a Slack when I’ve seen the questioner post other messages or share a bit about themselves. This type of interaction can happen in a forum, but seems more common in a Slack. This type of interaction makes the community more sticky and people more likely to help. A minor benefit is that chat can be hosted elsewhere for free so the startup cost and friction is low.

From the developer’s side: when I run into an issue or problem, I want an answer as soon as frickin’ possible. It’s blocking me, otherwise I wouldn’t have asked it.

Sure, I can context switch, but that has its own costs. So there’s tremendous value from a knowledge seeker’s side to pick a synchronous method of asking questions.

If you had a burning question to ask, which would you prefer? Hopping on the phone or sending a physical email. That’s the allure of the chat platforms.

I will say that some forum software has built chat in, but that isn’t going to get you an answer immediately.

What’s right for you?

Well, what do you want to emphasize? Long term aggregation of knowledge and a culture of completeness, or community and a culture of immediacy.

As alluded to initially, you can of course use both tools at different times in your community’s evolution. I think the longer you build, the more you’ll move to a forum or other public knowledge sharing solution.

Here’s a tweet survey that I ran a month ago asking how developers wanted to get tech help. (Something else turned out to mostly be “well written documentation”, from the thread responses.)

 

Devrel Tips | moore |

Advice for finding a technical co-founder

So, you’re a non-technical founder. You have a great business idea. You have some market validation.

You’re ready to find a technical co-founder who can help build out your startup. You’ll need some coding done, and you imagine there are a whole host of other technical things that need doing. You want a partner who can help you.

You start looking and realize a few things:

  1. Anyone you pick is going to have a huge role in the success or failure of your business.
  2. You don’t have many developers in your network.
  3. It is hard to find technical people willing to work for equity in your unproven startup.
  4. Any time you post looking for a technologist co-founder, agencies (many offshore) respond, happy to help for $$.

I have had the “how can I find a technical co-founder” conversation a few times in the last week so wanted to share my perspective. This is based on my observations and reading, as well as having been a founder (two times, one successful) and early stage employee or contractor multiple times.

Where are you trying to get

Knowing this is important, as it affects the rest of your search.

Are you trying to get to an MVP you can raise money on? A partner you can work with for a long time (if so, consider the 5 Cs)? An application you can sell to someone? A refinement of your current vision which may have technical uncertainties? Advice if what you are trying to build is possible? A technical network so you can hire quickly?

All of these are valid answers, but different ones shift where you look.

It could be because of who I am rather than indicative of the larger need, but most of the folks I’ve talked to really don’t need a CTO. They need a founding engineer. I wrote about the difference between the two; the latter may grow into the former.

What do you have

You also need to get clear on what you have. The people I talk with often have the following situation:

  • no money (raised or in savings)
  • no tech skills/network
  • an interesting idea
  • a no-code or partially coded solution proving out part of their vision
  • some level of market validation

If you aren’t in a similar spot, you can read the rest of this post with some skepticism. Immediately below I give advice on what to do if this doesn’t apply to you.

If you have money, it’s easy: hire someone.

If you have a network, reach out to that network. Most people are happy to help. It won’t guarantee you a partner, but it can lead you down the path.

If you don’t have an interesting idea, well, you’re in trouble. That’s one of the key things you can bring to the table to interest a tech co-founder.

If you haven’t proven out some aspect of your idea with the tools you have at hand, that’s worrisome. Why not?

What can you offer

Being realistic about what you have means you can be realistic about what you can offer.

Here’s a fact: right now, technically competent people can be hired by companies willing to pay them money pretty quickly.

The idea that you are so passionate about that you’ve quit your job or are working on it full time off hours (if not, that’s a serious red flag to any developer)? That idea is exciting and unique for you, but it is one of many ideas the developer has heard about from passionate would be founders. In addition to not being bought in to your vision, technical talent has the following other considerations:

  • there is significant opportunity cost to joining a startup for equity only. It depends on market and experience, but call it a net swing of -$150k/year ($100k in foregone salary and $50k of debt/savings drawdown for living expenses). Yes, you might raise money, but will you pay market salaries to a co-founder then?
  • the risk is front loaded. I know tech folks who joined a startup, worked their butts off to build something, only to discover that their partners were unable to market or sell what they’d built. Whoops! That was a waste.

So, how can you mitigate those risks?

Don’t stint with the equity. Double digits for sure. Make sure to leave some aside for dilution and/or an employee pool.

In fact, thinking intelligently about equity is a subset of something else you should offer: understanding the business. The Nolo books are great for the fundamentals of business, but you should also understand the domain. If you’ve thought deeply about the domain and the business, you will stand out from the “idea folks” who just have, well, ideas.

Be flexible on working arrangements. In one situation where I was the tech co-founder, I worked full time on the company for the first four months, then half time after that because I contracted to pay the bills. Bring finances up in your initial discussions. Be honest and upfront about living expenses and how everyone has different needs. This will filter out some folks, but that’s ok. Better to do that early.

Push forward on your own. You can get a long way with no-code tools and manual processes. This is a grind, but a tech founder will be impressed by this effort if you get results. If you don’t get results, that might be the market telling you something. You can also start an accountability email list for $0 to share your progress.

You may even decide you want to code. Depending on the complexity of your idea, it may be possible for you to build something using free resources. Or buy a book–you are going to invest years of your life, so buying an intro book on coding is cheap. I’d suggest the Rails Tutorial if you are building a webapp.

I, and other devs, judge non technical founders by their ability to get sh*t done. Show that you can.

Know who you are looking for

Based on where you are trying to get and what you have to offer, you should be able to narrow down who you are looking for. Think about the following factors:

  • risk tolerance, including emotional and financial
  • experience
  • domain expertise
  • personal chemistry (you are going to be spending a lot of time with this person, they’ll know your triumphs and low points like no one else)

Write this down. It’s the start of your job description. You should flesh it out with your vision, experience, and what you’ve accomplished so far.

You want a written job desc because you’ll want to share it with anyone who is interested in helping. That’s the first thing I ask for. Put the job desc at at a URL. It doesn’t need to be fancy, a public google doc is fine.

This lets people easily help by sharing the job with anyone or any group they might think a potential co-founder frequents.

Where to look

Share that job desc with your network. You never know who may know someone.

Tech folks congregate in groups. This is often in a slack. Google for “<your area> tech slack” and you should be able to find something. Join, but watch for a while before posting your job desc. There may be certain channels where it is appropriate. And you’ll get more response if you aren’t a “drive by” poster.

There are also meetups. Google for “<your area> tech meetup”. Again, these are communities, not bulletin boards. Be prepared to join and interact for a while before making any pitches. Could be months.

Angellist is a good site on which to create a company listing and post your jobs. They have an email that goes out to interested folks. That’s how I first made contact with one of my co-founders.

Another option is targeted outreach. Tech folks, as mentioned above, congregate in online communities. They often comment. You can search for comments related to your idea/business and see who is talking about it. They may have contact info in their profile.

For instance, I had someone reach out to me because I’d commented several times about agriculture, and that was the domain of their business. This is a bigger time investment, but can lead to interesting connections. Reddit and Hacker News are good places to start. Doing this helps you build your tech network, because even if the commenter isn’t a fit for a co-founder position, they may pass along the job desc to someone else who is. They may also be interested in remaining in touch; if the company grows or their circumstances change, there may be a possible fit in the future.

When you find the person

Great, you think you have found a co-founder.

Have their skills validated by other technical people. You don’t have the knowledge or perspective to evaluate the co-founder. You are excited about possibly working with someone who can help make your dreams a reality. Sometimes an incubator can provide these advisors, other times your network. (As an aside, you should be trying to build your tech network. When you talk to someone who is helpful, see if you can add them to your startup accountability list.)

When people share equity in a business, it’s like getting married without the sex. Date first. See if there are a couple of projects you can work on before jumping in. Offer to pay them something for these, even if it isn’t full market rate.

Set up vesting for founder equity. Everyone should be on the same page if you are starting a true partnership. This will save you heartache if someone has a change of perspective or goals two years into the startup.

In general, plan for the worst, hope for the best. Discuss exits, future roles (do they want to build a team or continue to be an IC), co-founder departures, and long term vision. Do this even though the business model and organization is imaginary at this point. Having these conversations while everyone is excited about the company will make the inevitable tough times just a little less tough.

Make sure you discuss who will be in control. If you are partnering with a childhood friend, maybe 50-50 ownership will work. I’ve definitely seen it blow up because there’s a tension between the “owner” role and the “CEO” role, so even then I’d be hesitant. But if this is someone you just met, you should stay in control with more than 50% of the shares. Make that clear from the get-go so there is no confusion. That said, you aren’t partnering with this person to micro-manage them. Be clear about the areas they’ll have autonomy in.

I know I had issues with my lack of control when I co-founded companies, but that’s all the more reason to have that discussion earlier rather than later.

When it comes down to it, the technologist is crucial for many companies, but the non technical founder is even more critical. It’s a bit like the product and company is a racecar. The technical founder builds the car, but the CEO drives it. Both are important, but you can often find other car builders once the car is running (to mangle my metaphor a bit).

Conclusion

It’s going to be a long haul to find the right person for your startup.

It might be easier to try to raise some money to pay those ravenous agencies (I’ve worked there, so I can call them that) or hire a freelancer.

But the right co-founder will be someone you can trust, someone who will experience the depths and heights of startup life.

Best of luck finding them!

Business Startup | moore |

Open source software business model concerns in the age of the public cloud

A few years ago at Gluecon I was sitting at the lunch table with some new acquaintances. The talk turned to business models and someone mentioned the chilling effect of the public cloud providers on open source businesses. Companies whose main product was open source software (OSS) originally had a straightforward business model: sell support and consulting. (I’m leaving aside sponsorships, donations, dual licensing and other smaller streams of income. I’m focusing on what I’ve seen work to build larger, stable companies. If you are solopreneur you have a different set of options.)

This shifted over time to withholding certain features (the “open core” model) and licensing them to customers who wanted them. Open core was better for companies because it is more scalable and profitable to sell software than it is to sell labor (which is what support and consulting essentially are). Customers won because a certain set of them needed features that were likely not core to the software, such as single sign on or auditing, but were critical for their use cases. These customers tended to be businesses with money to spend. Open core, however, has some problems.

The fundamental one is “what should be OSS and what should be licensed” becomes a pressing question. Building a product is tough enough in general, but now there’s an additional marketing, product and engineering complexity.

The SaaS Business Model Cometh

With the rise of SaaS a third monetization option appeared: offer their software as a service and run it on behalf of their customers. NodeBB, ElasticSearch  and others all do this. The customer wins because they have a lower TCO and benefit from continual updates and the company wins because SaaS combines the margins of open core with recurring revenue. This has even caused some companies to move away from open core. The community remains satisfied because if they want, they can self-host.

Enter the cloud companies. Cloud companies could and did take open source projects, packaged them up and offered them as a service for a low monthly fee.

As a customer, and I’ve been such a customer many times, using an offering from these providers has definite benefits:

  • The cloud operators know how to run software at scale.
  • They are already part of the accounting infrastructure. There’s no additional procurement process.
  • They don’t typically deprecate software (killedbygoogle notwithstanding), so as a customer you are less worried ongoing support and availability.

Adoption by a cloud provider has benefits for the open source project, too.

  • It’s proof the software has reached a large level of customer demand, otherwise the big clouds wouldn’t bother.
  • Improvements to the software or documentation may be donated back.
  • It increases the number of folks who use the software or know about it.

However, those benefits may not balance out the loss or threat of loss of revenue from SaaS hosting. This was the chilling effect mentioned around that lunch table. When I read or ask folks about this issue, I’ve heard three separate proposed solutions:

  • Be better
  • You should be so lucky
  • Close your source or re-license

Be Better

First, as the prime mover behind the product, a company should be able to be better than the cloud operators. Not necessarily better at running all software, but better at running theirs. After all, they built it. This does assume that the best people to operate a service are the ones who built it. This may or may not be true. I think it is definitely true at a small scale–they are going to know which knobs to tweak, and if they run into an issue they can immediately fix it in the code base. As software gets more and more popular, the ability to run it well diffuses, however.

The company behind any project also should certainly have a better sense of customer needs and the ability to map those to its internal roadmap. Of course, being an open source project means that the advantages are fleeting. If anything is turned into code and released, it made available to competitors.

An even bigger issue is that this relies on customers being willing to stick with the company on the bleeding edge. For rapidly growing or changing software that may make sense. Where major functionality is being built regularly, upgrades are an easy sell. But OSS isn’t adopted by the cloud provider until it has a large audience, which means they are likely very functional for a lot of use cases. This means that the company won’t get as many folks on the upgrade train, which makes the “be better” argument more problematic. If the cloud provider can offer a “good enough” experience, then their other advantages start to dominate.

You Should Be So Lucky

This is the argument that the chances of an application becoming popular enough to be adopted by a public cloud are so low that it’s not worth spending time thinking about. I asked Steve O’Grady about the OSS business model predicament at a webinar once and this was his answer.

There’s a lot of merit to this. The biggest obstacle to an open source company succeeding is not anyone taking the software and running it, it’s the software not being useful or known. Obscurity has killed many a company, far more than AWS adopting their software.

But when a company reaches a certain size, this advice is less of a solution and more naming a strategic threat. Yes, they are lucky to have their solution be popular. But what can they actually do about the threat to what may be a sizable portion of their revenue? This argument doesn’t help with that.

Close Your Source

Depending on how the company developed the software, they could relicense or close source. This may require some prep work. First, make sure the software is all owned by the company; have all contributors assign over their copyrights. This will make it more difficult for you to reap some of the benefits of open source: easy contributions from others.

The company can also choose to relicense only certain portions of the codebase, taking a step towards open core. In any case, prepare to consult an IP lawyer.

The company must also be ready for the blowback from the community. Which hurts more, losing something that was once free or paying for something from the get-go? For me, it’s the former. Many of the marketing, engineering and product benefits gained from being open source will be lost; this is the price to pay for obviating the cloud vendor SaaS threat.

The firm could also dual license the software or use something like the Business Source License, which strikes a different balance, but still has some attributes of an open source license.

Alternatives

None of these sound very fun. I think the answer is to back up and consider whether to make a project open source initially. Consider this very carefully initially to avoid pain.

The benefits of open source for a business’ core offering are:

  • Lower friction of adoption
  • Contributions from outside an organization (these must be fostered, they may be free in terms of money but not time)
  • Increased rigorousness for the eng team (coding in public can make code better)
  • Easier to recruit because devs like working on open source
  • More eyes on the code means shallower bugs
  • Marketing halo

Are these worth the risk? I can’t answer that authoratively. If they are, another option might be to prepare for SaaS revenue cannibalization; this may mean going open core, focusing on consulting or support, or possibly never even building a SaaS solution.

There’s more than one way to be open. A company can do open development, inviting customers in to the product process. Would being more open with the development process (using public GitHub issues, for example, instead of an internal issue tracker) allow a company to get some level of contribution from outside the org? This would also imposes that “spotlight level rigor” on the eng team. Could the company make the software “free as in beer”, rather than “free as in speech” to lower adoption costs?

A firm can also release software as open source without taking any feedback from the community (“throw it over the wall”), which may be a good choice if the software isn’t core. Big companies do this every day. Facebook gladly open sources their computer plans and React, but certainly wouldn’t open source the core Facebook application.

In conclusion, if you are planning to build a company on an open source project, think about the main revenue flows for that company and what you’ll do in the unlikely, but hopeful, event the project succeeds.

Business Cloud Computing | moore |

Slack failing to open with NS_ERROR_DOM_ABORT_ERR error

I use Firefox as my primary browser (version 84 as of today). As part of this, I keep logged into a number of slack channels. These are some of my favorite communities and I go there to chat with folks, learn about interesting topics and hunt for neat links to resources I didn’t know about. Occasionally I’ll even ask a question or two.

Recently, I saw some weird behavior. The slack channels weren’t updating. The notice at the bottom of the browser bar was something like “Slack is trying to connect”.

Hmm, I thought, that’s weird. I tried reloading the page. I couldn’t do it. That is, even hitting control-f5 didn’t change anything. However, I could open other website just fine.

I tried quitting all my browser tabs and restarting. Made sure Firefox was up to date. Checked the Slack status page to see if slack was down. Opened a new tab and typed the slack channel url into the address bar.

Nothing.

So, I popped open the dev tools and looked at the network tab. I saw this error when loading app.slack.com:

NS_ERROR_DOM_ABORT_ERR

That turned up this bug. From a scan, several other folks where having this issue, with slack and other sites. This comment shared the solution:

Clearing storage for app.slack.com fixed the issue, and the Slack workspace loads correctly now.

All I had to do was clear the storage for app.slack.com and Slack started working again, magically. Even though I was warned that it might force me to log in again, I didn’t have to do so.

Tips Web Applications | moore |

The most underappreciated AWS service

Three letters for you.

V. P. C.

Amazon Web Services’ Virtual Private Cloud is often the first real hurdle for developers and others looking to understand cloud systems. I know it was for me; I’d not had much networking experience when I first encountered it. To really grok VPC, you need to have at least a mild understanding of network architecture, including subnets and gateways. In my experience, such knowledge isn’t par for the course with software developers.

Most software developers can quickly understand EC2 (oh, a virtual machine) and S3 (ah, a gigantic disk drive). However, VPC’s networking abstractions are tougher. However, VPC is magic.

Let’s talk a bit more about this. AWS built a performant software defined networking layer. But they didn’t just port all the concepts from the physical world into the cloud. At least, when I read the Unix and Linux Sysadmin’s Guide, which dives deep into such things, that’s how it looks to me.

Instead, AWS followed the 80/20 rule, giving developers and architects flexibility to build real network architectures. These architectures can support real world applications, with real world security needs and compliance concerns.

Yes, you can also achieve such separation using different AWS constructs (and should!). IAM and Organizations spring to mind. But for many, making sure that they could easily port their current security posture to AWS made a cloud transition easier.

VPC simplifies the networking layer enough that even developers with little networking experience can understand it (like me!). AWS has even provided network logging so that, should you need to delve deep into your networking layer for troubleshooting or auditing, you can.

VPC is also fundamental to other services. EC2 instances are still the majority of AWS budgets, according to Corey Quinn’s 2020 Re:Invent rebuttal.

VPCs are where those “machines” exist. The fancy managed services AWS wants you to use so they can lock you in? Often they are accessed by dropping ENIs into your VPC.

If not, these services are running in VPCs managed by AWS, therefore inaccessible to you except through tightly managed interfaces. See, the security works!

To top it all off, VPC is free and ubiquitous.

AWS’s VPC is the water to cloud engineers’ fish; we don’t even see it, let alone appreciate it.

AWS | moore |

Book Review: The Cuckoo’s Egg

I recently finished “The Cuckoo’s Egg”, by Clifford Stoll. It was a fascinating non-fiction book exploring the foundations of computer security in a personable format.

The setting is the mid 1980s. The author discovers something weird on his academic computer system. There’s an unexplained charge of 75 cents. He digs deeper, discovers that someone who’s left the university is logging in.

After further investigation, he discovers that the user who is logging in is an intruder. After discussing the situation with his boss, he gets three weeks to find out who they are. He figures that’s plenty of time.

The investigation ends up taking a year.

It also extends far beyond his academic systems, both in scope and effort. Stoll talks to numerous government agencies and private organizations, letting them know they’ve been attacked and getting their assistance tracing the hacker. He sleeps under his desk. He rigs up a pager so that he can know which accounts the hacker is using. Stoll sets up printers so that every word the hacker types is recorded, unbeknownst to him.

It’s quite the tale. As someone who has worked with software for years, I really appreciated the historical nature of it. When I became aware of the internet, in my youth, some of the groups and communities he mentions were still around; I remember reading and posting to usenet. But many of the systems were before my time. I’ve never touched a computer running VMS, for instance.

But, for all the history, the people problems were the same: users not changing passwords, system managers not locking their software down, bureaucrats happy to take information but not willing to share. Let’s just say, mistakes were made.

I also enjoyed the author’s interspersal of lived experiences. We don’t simply follow one computer nerd tracking another. We also learn about milkshakes, parties in San Francisco, curry nights and his first experience with the microwave. While some phrases and analogies are repeated (“should we thank someone who goes to a little town and robs people to illustrate they should lock their doors” pops up at least twice), in general the book is pretty readable. Stoll’s personal stories and musings help that readability immensely.

All in all, a great book if you are interested in the history of computing or modern security practices. If you’re interested in learning more, you can check out a paper he wrote based on the same experience for ACM.

Books Mistakes Programming | moore |

Book Review: Algorithms to live by

I finished Algorithms to Live By, by Brian Christian and Tom Griffiths. I enjoyed it immensely.

The premise of the book is that computer programs make decisions with algorithms all the time. There’s math behind how they do so, including tradeoffs and time considerations. Human beings face some of the same decisions and there’s no reason we can’t use this knowledge to live better lives.

Yes, this is kind of a self help book—“you can live a better life by thinking like a computer”. But it’s math, folks.

I actually recommended it to my SO because I feel like she’d understand me better after reading it. I’m always talking about how much of the stress in our life is due to resource contention.

The book covers a wide swathe of decision making. Here are some examples of the broad categories and some specific references:

  • when to decide to stop looking for a house or a partner
  • when to explore new knowledge vs exploit current knowledge in the context of clinical trials
  • how randomness can lead to better outcomes
  • how caches can help you determine what of your wardrobe to keep
  • how overfitting warps sports like fencing

As illustrated above, this is not a book about theory, but is actually hands on. I don’t recall seeing a single equation, though there are graphs. And the authors do mention plenty of researcher names, provide footnotes and have a twenty page bibliography. So if you want to learn more about the formal proofs, the info is there.

It’s hard to choose just one takeaway from this book, but if I had to pick one, it’d be the fact that game theory shows that you need external forces to avoid the tragedy of the commons, and that emotions may play a role in providing that.

Here’s an excerpt if you want a deeper look.

If you spend any time thinking about how you can make decisions better, Algorithms to Live By is worth reading.

Books Programming | moore |

A quick look at xkit

I was prototyping a small app in xkit and wanted to document this useful tool. When I first saw this launch on HackerNews, I couldn’t quite understand what the purpose was. But now that I’ve spent a bit of time playing with it, I understand it a bit more.

Suppose you are writing a recipe management SaaS and realize that you want to integrate with some other services. Perhaps you want to be able to export the steps of a recipe to a Trello board, or to a Google doc, or to a PDF.

These are all services available on the internet with an API which will allow end users to give your application access to their accounts. This lets you publish to each user’s Google docs account or Trello board.

(I’m not as familiar with services offering PDF generation functionality, but a quick search turns up some options, including some that you can self host.)

There’s a fair bit of hoop jumping in terms of setting up API keys and OAuth consent screens, however.

And this is the problem that xkit solves. If they’ve already written the connection (here’s a list), it is quite simple to add the ability for a user to connect to the service. With no previous experience, I was able to connect to Trello in about an hour. The user experience of connecting the external SaaS application is really smooth and far better than something I could whip up quickly.

If they haven’t written a connector, I don’t believe you can write one yourself. For example, for that PDF service, you’d need to contact the xkit folks and ask them to add one.

This is different than, say, Zapier, because it’s operating at a different level. Zapier is excellent (and has been for years) at letting users connect their apps. But xkit lets you let your users connect apps, basically letting you build a mini Zapier (in terms of connectivity, not functionality).

You can also host your own app catalog if you want to. I didn’t get into this too much, though, so it’s unclear what the benefits of that are.

They provide a user data store out of the box, but also integrate with a number of other providers (including FusionAuth). This means you can leverage your existing auth solution and still get the easy integration with other third party APIs.

Their pricing seems reasonable, given what they take off your plate.

Nothing’s perfect, however. I found a few documentation bugs, which I let them know about (they host their docs on readme.com and I found the suggestion process delightful). When I tried to sign up, the service was down, but a quick Tweet exchange resolved the issue within 30 minutes.

It is bizarre to me as an authentication focused company that they don’t have a “forgot password” link on their login pages. The documentation is javascript heavy, with nary a mention of other languages, but that’s understandable as they’re just starting out. It’s also strangely video heavy, which I found a bit distracting; that, however, could just be my learning style.

All in all, if you are looking to integrate third party APIs which require OAuth interactions on the part of your users, you’d be well served to take a look at xkit.

APIs Useful Tools Web Applications | moore |

Book Review: A Memory Called Empire

I was at a bookstore the other day and was rummaging around for an escape book. I had picked up one book based on staff recommendation, but came across another that had won the Nebula. A Memory Called Empire appeared to be an award winning space opera novel.

I thought, why not, and picked it up. I was not disappointed.

There are two main cultures of different power and viewpoints. The action follows an new, unprepared ambassador from one culture to the other. The cultures are coherent and yet alien. Alien to each other and to me. Competitive poetry, internal reflection, and constant political intrigue define one culture. We get drips and drabs of the other, less powerful civilization, but you learn enough to be appreciative of their scrappyness and reverence for the collective.

The cultures are never described to the detriment of the action. The ambassador is dropped into a political mess and acts and reacts to help save her nation and herself. Whether she is trying to meet powerful people to negotiate for her people, reading correspondence, or escape danger, there’s no downtime. The entire book happens in span of about two weeks.

I also enjoyed the character development. Many of the characters pop in and out of the storyline, but you follow a few main characters for a while. You get to understand and appreciate the way they interact with each other. It doesn’t feel forced at all. At the same time, the “otherness” of the ambassador provides a constant tension which is understandable to anyone who has been dropped into an uncomfortable situation.

The plot revolves around a mystery. But even when it is unveiled, there’s still plenty of excitement, as a confluence of outside, political events ensnare the protagonists.

Some books are so good they keep you reading late into the night. This was one of them.

Books | moore |