Skip to content

Should your customers be your devrels?

Saw this spicy take:

If Twitter goes away, this is from Stefan Avram, who said “You shouldn’t have devrels. Your customers should be your devrels”. (Stefan is a cofounder of Wundergraph and head of growth. Good on him! It’s hard to found a company.)

I wanted to talk about this statement because, well, I’m a devrel (a developer relations professional) and I don’t agree with it.

First, let’s rule out a few obvious points.

  • If you aren’t targeting developers as users, you shouldn’t have devrel as a function.
  • Even if you have a devtool, if your main sales channel is top-down, devrel is not super useful.
  • If your product doesn’t have some kind of free version, devrel is going to have a tough time. Not everything has to be free, but it’s really tough to get a developer to offer both time and money to learn about your product.

I hope this goes without say, but it is fantastic if your customers are advocates for your product.

However, I think this approach is a bit naive. It’s a bit like saying “You shouldn’t have sales people. Your customers should be selling your product.”

The truth is that there will be some subset of your users who are enthusiastic and advocate for your product if it is good enough to buy. I have interviewed some for FusionAuth. You should do everything you can to encourage this behavior, including:

  • talking to them
  • learning about their companies and successes
  • giving them swag
  • sharing their stories far and wide

But these folks’ job isn’t to promote or support other users’ use of your product. They’re busy building their product, company or project. As they should.

A high functioning devrel team can offer:

  • quality content
  • support for a community
  • up to date, relevant code

This team should be working to improve all of these and make the experience of all of your potential users better and better.

And they’ll care about the company’s overall mission. And they’ll have inside access to the codebase, the engineering team, and the product roadmap. And they can offer reified, constructive feedback from users to your teams.

That’s a large number of important business functions. If they are performed by customers, who are, again, busy working on their own stuff, they’ll be done at best haphazardly and at worst not at all.

Think about it. How many blog posts have you written about a product? How many of them were deep dives?

I can say that I’ve done it for a few products (stripe, zapier, AWS, lob) and none of those are deep enough to qualify as anything other than the briefest introduction to a product. A developer who encounters posts like these will either be intrigued and want to dig in further or bounce, but they certainly won’t have enough info to make a decision or use the product.

By making documentation, community and code examples the developer relations team’s problem, you’ll get more focus and better results.

And more users and happier customers.

How to solve the “this application is slow” type of problem

How do you feel when your boss says “the application is slow, please speed it up”?

Personally, my heart sinks and then I get excited. Though they can be frustrating, this kind of thorny performance issue is fun if you look at it the right way.

Whenever I’ve tackled problems like this, the first thing to do is define the start and the finish line, as precisely as you can.

This requires understanding the application’s behavior and architecture. Where is the data stored? How is it presented? How is it modified? Are there types of operations that happen regularly? What exactly is “slow” about the application?

Try to avoid jumping to conclusions here. I understand the temptation to make a change as soon as you think of one that might help, but it’s better to approach this systematically.

Suppose you find out the issue is the database. Operations are too slow and the CPU is not pegged. You know the type and version of the database, how the application calls it, and more.

Here, a good finish line might be “we need to be able to handle updating the main table with 50k items in 1 second”. If you don’t have a precise finish line, this type of work can be endless and frustrating. After all, it is almost always possible to “make it faster”, but you will reach the point of diminishing returns.

If possible, set up a test scenario/system that you can run through repeatedly as you make changes. If not, figure out some other way to test that changes have a positive impact.

Next brainstorm possible solutions and think of two numbers for each: level of effort and hoped improvement. Doesn’t need to be too precise, a scale of 1-5 is fine. Here’s an example for a database bottleneck:

  • upgrade the size of the database. LOE low, impact medium
  • increase the disk speed: LOW low, impact high
  • running explain plans and adding suggested indices: LOE medium, impact high
  • offloading operations to read replicas: LOE high, impact high
  • …etc, etc

Then start doing low effort, high impact changes. Run through your scenario and tests after each one. See if you get closer to the finish line. Rinse and repeat.

This type of performance issue is a case where hiring an outside consultant/contractor can make sense. You don’t have to spend a lot, since the scope of work can be limited. They can work with you to define the start and finish lines as well as possible steps if you don’t have the time or knowledge to do so. Then have an internal team take the specific actions and test each change to see if it helps.

Advice for startup founders

On an email list I’m on, someone was recently accepted to an incubator program. They asked for advice about startups. I couldn’t resist!

I wrote this (lightly edited) and wanted to post it here so it’d have a permanent spot on the internet.


I’ve been a founder or early employee/contractor of six startups over my two+ decades. Some are still kicking, but none have had an exit. Please consider that when contemplating my advice.

When building an early-stage startup:

  • talk to customers, talk to customers, talk to customers
  • revenue > funding
  • know your business domain. If possible, co-found with someone who is an expert in the area if you are not
  • choose boring technology whenever possible
  • choose technology that you know whenever possible
  • be prepared to consult or find other ways keep the company alive if you don’t have immediate revenue
  • take advantage of in-person support systems to keep your spirits up
  • exercise: it’ll help you be a better founder
  • when you have an ask for someone, make it easy for them to say yes (be specific, do your research, scale the ask to the relationship strength)
  • remember, building a company is a marathon not a sprint (but sometimes you need to sprint!)
  • VCs and founders have shared incentives (both want a successful business)
  • VCs and founders also have misaligned incentives (you get one bet, they get N bets; you need $X, they need $10X-100X)
  • understand your financial runway
  • understand your emotional runway
  • if you have a spouse or partner, make sure they are on board with the big decisions you make
  • being lucky is usually more of a factor than being good
  • you can sometimes make your own luck through hard work
  • all founders should take part in the sales process, no excuses
  • your company’s biggest competitor is customer inertia (or Excel)
  • all advice is contextual, always understand the context of the giver
  • talk to customers
  • talk to customers

Best of luck!
Dan

PS Talk to customers.


Thoughts on a freemium software product

At FusionAuth, we have a free software product that is a critical part of our business model.

A free product is pretty common in the software space because of two things:

  • Software needs to be used to determine its efficacy; a software package is not like a shovel. With a free option, money is no longer a barrier (though time is).
  • Software has zero marginal cost; once you put the effort in to build the first copy, you can create 1M copies for essentially the same cost.

However, supporting a free software product sure isn’t free. This post covers what you need to think about in terms of investing in a free product.

If FusionAuth were a public company, this is where there’d be lawyerese talking about forward looking statements and safe harbors and whatnot. Suffice it to say that this entire post is my personal opinion.

Side note: a free product is often but not always open source. Free can mean a tier of a SaaS, an open source project, or a downloadable product. Open source has additional complexities, so I’m going to focus on products that are “free as in beer” in this post. FusionAuth has a downloadable product that devs can run for free, within constraints.

We have internal discussions about tweaking the free version of the product. Options include:

  • Keeping the product as-is, but making investments in the community version. This includes bug fixes and feature improvements. Since our free product is production ready and feature rich, this is a solid choice.
  • Improving the free product. There are multiple dimensions to doing so, including:
    • Improving discoverability of the free option, such as highlighting it more on the website, advertising it, or investing in additional documentation around its features and usage.
    • Changing the license to make it usable across a greater number of use cases or with different limits.
    • Move features currently restricted to paid plans to the free product.
  • Degrading or limiting the free product. These are basically the flip side of improving it:
    • Increasing friction to find or use it.
    • Modifying the license to prohibit currently allowed use cases.
    • Clawing back features from the free plan, focusing on features useful to businesses who are likely to pay. For example, the free plan currently allows unlimited SAML connections, which many competitors throttle.

For the foreseeable future, we are following the first path, improving the free product. The free product is usable and robust, and we get substantial benefits from the community’s usage.

Let’s dig into these benefits.

The value of the free users

First, if developers are using our free software to solve their authentication needs, they aren’t using someone else’s. While you can’t pay a mortgage with developer attention, that doesn’t mean it has no value. Such attention expands our mindshare and market share. More mindshare means that people are learning about our solution. Gaining market share means they aren’t using someone else’s solution. Therefore they are neither paying a competitor money nor getting more familiar with their solution.

Second, such users spread the word about our solution. Sometimes they talk about it on social channels and sometimes on a review site. But often it is prompted by us; here is an example of one of my favorite set of blog posts, entirely drawn from community experiences. Talking to community members has opened up my eyes to the wide variety of ways our product is used. Community stories are not, however, useful as case studies for the sales process. They aren’t detailed enough. But they are still helpful to spread the word about the product, highlight our community members and catch long tail keywords.

Third, free users improve FusionAuth. They do this by:

  • Finding flaws in FusionAuth, such as bugs or regressions. Here’s an example issue, including workarounds.
  • Requesting new features. We leverage the community further by asking users to upvote such requests so we know what the community wants from the product.
  • Exercising the software by performing integrations that we would never have done. This is a variant on finding bugs. For example, a free user reported we aren’t to spec with regards to the SAML relay state; that’s never been an issue for the existing SAML integrations.

Finally, free users also offer each other support. While not all community members are active all the time (our community is more of a Google than a Facebook), a few have a presence on our forum, slack and GitHub issues.

The conversion of free users to paid users

Free users may purchase the software in the future. Once someone needs paid features, they may stop using the free plan and buy. We’ve built up trust as a solid solution in their mind and they have already integrated us. So a free user will consider paying you when they are looking to purchase.

Of course, you need  a product worth paying for above and beyond your free offering. At FusionAuth, we accomplish this in three dimensions:

  • Operating the product. Many of our customers are fine with the features of the free plan but don’t mind paying us to run it. This can include service level agreements (SLAs) as well, which are like catnip for enterprises.
  • Paid features. These are either features good when you are at a certain size (like SCIM) or enhancements of features available in the free tier, such as a more customizable registration form or MFA policy. Choosing the features to charge for is critical, but is really hard to get good data for since it is very business specific.
  • Support. Knowing you can ask questions of the engineers behind the product is valuable, especially for larger businesses.

There are two ways for free users to convert.

They can do it directly, where they use the free version to evaluate or run a “proof of concept” to ensure that our product meets their needs before they ever engage with us. We have plenty of customers who say “we’re already a few months into integrating with you” on our purchase kickoff call, and that ability to “get going” with the product without talking to a sales rep or pulling out a credit card makes the decision easier. Again, they trust the solution will solve their problem. They can also see how the company treats the free users and the community in general too.

There are also people who “kick the tires” with the free product and discover that it doesn’t meet their needs. We don’t hear about as many of those, but I have talked to a few. In this case, both parties win; getting to a quick no is not as good as a “yes” but is still pretty good. There are also people who do a self-led POC and incorrectly determine it isn’t a fit. They might miss features through lack of docs or a conceptual mismatch. We keep trying to improve our docs, education and product, but you can’t win them all!

There’s also an indirect conversion path, alluded to in the above mindshare point. Free users may use our software on a side project or to learn about authentication and OAuth in general. This lets them add FusionAuth to their toolkit. Then, later, perhaps years later, they can bring us into a project as an option to evaluate or to recommend us for purchase.

In general, free products let you build trust and de-risk purchases.

Keeping up with the Joneses, err the competition

Finally, competition matters. Lots of our competitors have a free offering. Again, this is due to the nature of software. Yay to zero marginal cost!

This is also compounded by the nature of developer tools. Devs are looking for tools to solve a problem, but once they’ve integrated one, it sticks around. One time I picked a bug tracking system in a few days (phpbt, what what!) and it was used at our company for years. This means if you have a friction free evaluation process, you stand a good chance of being embedded.

Offering a free product matters for our market positioning.

Nothing sells quite like free.

In conclusion

When you make a free product available, you are offering users something of value. Resources are scarce, and supporting the free product isn’t free.

Determining how much time and effort to expend supporting it depends on the value your company gains. You won’t always be able to calculate it in dollars and cents, but it exists nonetheless.

20 years

Wow, it’s been 20 years since I started this blog. The world and my life have changed quite a bit, but I still like to muse on technology, business and more.

Thanks for reading! I appreciate everyone who takes the time to listen to me, whether you scan it, share it, or reach out to me to let me know.

Still think blogging is one of the best things you can do if you want to:

  • engage with people
  • understand a subject
  • build a business
  • create a name for yourself
  • think deeply about a topic
  • find a job (but not quickly!)

Thanks again for 20 great years!

Docker thoughts

Docker is amazing tech for developer productivity.

You can package up the dependencies for your application into one file (a Dockerfile) or more than one file (a docker-compose file). You might use the former for an application or service and the latter if your application depends on a database, cache, or other external architectural component.

I am definitely late to the Docker party, as I remember Eric Norlin interviewing Solomon Hykes in the early 2010s at Gluecon. In 2018 I used it to stand up some consulting projects, but wondered at the value and the required investment.

I’ve recently been using it extensively at work to set up quickstarts. These tutorials let folks stand up our software quickly for evaluation purposes. Docker is a fantastic fit for this use case.

I’ve been a big fan of detailed documentation and readmes for setting up development environments, but Docker takes all that documentation (including tedious things like ‘install python 3.8.12. 3.8.11 doesn’t work’) and puts it into code that is accessible with single command.

I have a foggy memory of detailed readmes, hours of setup and dependency hell, but Docker removes all of that. Doesn’t mean your Docker images/compose files can’t get stale and won’t need love and attention, but it does mean you can control when you test and upgrade such dependencies, rather than discovering it when a new developer comes on the team and can’t get their environment set up.

Shipping software to production via containers (Docker or otherwise) has operational benefits, but I’m less familiar with those. One of the nice things about developer relations is you’re separated a bit from operations. (It’s one of the downsides as well.) I haven’t recently worked with a production system where containers were used, but I have read nice things about the approach. Here’s an article from Google which covers the technical benefits.

What about the licensing? This business model change rocked the world of happy Docker users, but when you take funding, you have to make money. Actually, if you want to survive as a company, you have to make money, but if you take funding, you have to try to make a LOT of money. That’s the game, make sure you know before you start to play. But that’s a different post.

As far as Docker licensing, as a Docker user, you have two options:

  • pay docker when you reach the threshold
  • use a docker alternative such as finch or podman

As the previous paragraph implies, Docker is a subset of containers. The benefits I mention above apply to container based systems, not just Docker systems, but Docker is the solution I’m most familiar with.

Docker, and containerization in general, feels like as big a leap as Stackoverflow, git or memory managed languages in terms of developer productivity.

Ending a side project

I think side projects are great. They let you experiment, scratch and itch, and learn in a safe and low pressure environment. They can raise your professional profile too and are great fodder for a blog. Of course, they also take precious precious free time.

These all mean that it is entirely okay to shut down a side project when it is no longer serving you.

The side project could be:

  • less fun than it used to be
  • taking more time than you want
  • an area you used to be interested in but are no longer
  • related to a job you have left

In all of these cases, you have NO obligation to keep doing something simply because you used to.

So, shut it down.

How you do so depends on what the side project is. A blog should be treated differently than a webapp which should be treated differently than a library.

There are some things you can do for any side project:

  • Set an date to end your efforts–this is something for you internally.
  • Run through the finish line–keep up the good work.
  • Announce the end and what it means for the project.
  • Thank everyone who helped you.
  • Optionally make it available for a while longer.

What a “while” is depends on the effort and money required to make the project available, as well as how useful it is in the end state. A library can live on GitHub forever, especially if you archive it so it is clear that it won’t be getting updates from you. It’ll still be available to fork and may be useful to others. A directory of local restaurants offering patios, on the other hand, will decay in usefulness pretty quickly as businesses start and close.

Ending a project is just as natural as starting it. Make sure you do it right.

Protecting a CDN source using basic auth

I have a website that is behind a content delivery network (CDN). I want to protect it from being crawled by any robots. I want all access to go through the CDN for reasons. There may be errant links to the source; I don’t care if they continue to work.

htaccess and basic auth is perfect for this situation.

I added an .htaccess file that looks like this:

AuthType Basic
AuthName "Secure Content"
AuthUserFile /path/to/.htpasswd
require valid-user

I needed to make sure the path and the file are readable by the web server user.

Then, I added a .htpasswd entry that looks like this:

user:passwdvalue

If you don’t have access to htpasswd, the typical program used to generate the password value, this site will generate one for you.

Then, I had to configure my CDN to give it the appropriate header.

Use the Authorization header, and make sure to pass the username and the password. This site will generate the appropriately base64 encoded values.

Voila. Only the CDN has access.

Now, the flaws:

  • Depending on how the CDN accesses the site, it may be possible to snoop out the username and password
  • If you ever want to get the origin site over HTTP, you’ll need the username/password

Setting up password gorilla on an ARM Mac

I love password gorilla. It’s a portable locally hosted password manager that is compatible with passwordsafe. It has all the features I need in a password manager:

  • account groups
  • password generation
  • metadata storage (so you can add a note)
  • keyboard shortcuts for copy and pasting the url, username, and password
  • local storage of secrets

It doesn’t have fancy integration with browsers, remote backup or TOTP, but I don’t need these. For browser integration, I use the clipboard. For remote backup, pwsafe files can be copied anywhere. And for TOTP, the whole point of MFA is that each factor is separate.

But recently, I started setting up an Apple M1 machine, and the password gorilla downloads failed to start. I looked at the github issues of the repository and didn’t find a ton of help, though there were some related issues.

After some poking around, here’s what I did that worked for running password gorilla on my Apple M1:

  • installed tcl-tk using brew: brew install tcl-tk
  • cloned the password gorilla repository: git clone https://github.com/zdia/gorilla.git
  • wrote a small shell script and added the directory where it lived to my path.
#!/bin/sh

cd <cloned repository directory>/sources
/opt/homebrew/opt/tcl-tk/bin/tclsh gorilla.tcl

That’s it. After I did this, I can run this script any time and password gorilla starts right up.

LLM training data, or, a broken virtuous cycle

Have you used ChatGPT?

I have and it’s  amazing.

From the whimsical (I asked it to write a sonnet about SAML and OIDC) to the helpful (I asked for an example of a script with async calls using Typescript), it’s impressive.

However, one issue I haven’t seen mentioned is training data. Right now, there are sets of training data used to teach these large language models (LLMs) what is correct about the world and what is not. This podcast is a great intro to the model families and training process.

But where does that training data come from? I mentioned that question here, but the answer is humans provide it. Human effort and knowledge are gathered on reddit, wikipedia, and other places.

Why did humans spend so much time and effort publishing that knowledge? Lots of reasons, but some include:

  • Making money (establishing yourself as an expert can lead to jobs and consulting)
  • Helping other humans (feels good)
  • Internet points (also feels good)

In each case, the human contributing is acknowledged in some way. Maybe not by the end user who doesn’t, for example, read through the Wikipedia wiki editing history. But someone knows. Wikipedia editors know and celebrate each other. Here’s a list of folks who have edited that site for a decade or more.

What about search engines? Google reifies knowledge in a manner similar to ChatGPT. But, cards notwithstanding, Google offers a reputational reward to publishers. It may be in money (Adwords) or site authority. Other applications like Ahrefs help you understand that authority and I can tell you as a devrel, high search engine ranking is valuable.

ChatGPT offers none of that, at least not out of the box. You can ask for links to sources, but the end user must choose to do so. I doubt most do, and, in my minimal experience, the links are often broken or made up.

This fact breaks the fundamental virtuous cycle of internet knowledge sharing.

Before, with search engines:

  • Publisher/author writes good stuff
  • Search engine discovers it
  • User reads/enjoys/learns from it on the publishers site
  • Publisher/author gains value, so publishes more
  • Search engine “sees” people are enjoying publisher, so promotes it
  • More users read it
  • Back to step one

After, with LLMs:

  • Publisher writes good stuff
  • LLM trains on it
  • User reads/enjoys/learns from it via ChatGPT
  • … crickets …

The feedback loop is broken.

Now, some say that the feedback loop is already broken because Google over optimized Adwords. Content farms, SEO focused garbage sites and tricks to rank are hard to stomach, but they do make money from Google’s traffic. This is especially acute with products and product reviews because the path to monetization is so clear; end users are looking to buy and being on page 1 will result in money. I agree with this critique; I’m not sure the current knowledge sharing experience is optimal, but humans have been working around Google’s limitations.

More human labor helps with this. I’ve seen this happen in two ways, especially around products.

  • Social media, where searchers are relying on curation from experts. Here end users aren’t searching so much as browsing from a subset of answers.
  • Reddit, where searchers are relying on the moderators and groups of redditors to keep spam out of the system. Who among us hasn’t searched for “<product name> review reddit” to avoid trash SEO sites? This also works with other sites like Stackoverflow (for programming expertise).

In contrast, the knowledge product disintermediation of ChatGPT is complete. I’ll never know who helped me with Typescript. Perhaps I can’t know, because it was one million little pieces of data all broken up and coalesced by the magic of matrix algebra.

This will work fine for now, because a large corpus of training data is out there and available. But will it work forever? I dunno. The cycle has been broken, and we will eventually feel the effects.

In the short term, I predict that within the next three months, there will be a creative commons type license which prohibits the usage of published content by LLMs.