Skip to content

Great GitHub actions for techdocs sites

There are two main options for techdocs sites. You can use a SaaS solution like Readme or DeveloperHub. Or you can use a static site generator like Jekyll, Hugo or Docusaurus. There are strengths to each approach, but for this post I want to look at continuous integration/continuous deployment (CI/CD) tasks you can run on staticly generated tech documentation.

Why use CI/CD for your docs site? CI/CD tools let you automate code or documentation tasks. For documentation, you can run checks to ensure consistency and correctness which make your documentation better.

GitHub Actions is the CI/CD tool I’m going to refer to, but similar functionality is available with other CI/CD solutions such as CircleCI or Jenkins. Additionally, I’ll talk about the content living on GitHub in a repo, but you could use self hosted git, gitlab, bitbucket or other version control solutions.

This post has a couple of assumptions:

  • Your content is in version control.
  • Your content is in a format that needs to be processed before it is ready for publishing. In other words, you are editing text files in a certain format (markdown, asciidoc, etc) that are then processed, rather than using a WYSIWYG editor.
  • The people writing your content are comfortable using version control and tools like markdown.

Here are tasks you should automate:

Deploying The Documentation

This is the first one and the easiest win. You should have a workflow which automatically deploys changes when a PR is merged to your primary branch. This lowers the barrier to entry for contributing to documentation, because you only need to get a GitHub PR approved. Similar to continuous deployment of a SaaS application, it makes it easy to push changes regularly. Because it is a GitHub PR, it also is easy to roll back–you just revert the PR.

You may wonder at my including this, because it seems so obvious, but I’ve been on teams where the deploy process included sshing to a server, pulling the branch, and running a script.

Automating deployment was well worth it, because it changed “I found a typo, oh man, what a pain” to “I found a typo, I can push a PR and fix it in 3 min”.

Vale

Vale is an amazing, free linter for your prose. I haven’t witnessed it used at its full potential, but it is a spell checker, word police and living style guide all in one. It’s kinda imposing to start with since there are so many “nerd knobs”, but start simple and add to it as you learn it.

Vale runs fast enough you can have it run on every PR.

Spelling

Deploy a spell checker to run against your documentation to prevent misspellings and typos.

You have two choices here:

  • spell check just the changes (vale is good at this)
  • spell check the entire site (pyspelling is good at this)

The former lets you block a PR if there’s an issue, while the latter takes a lot longer and should run on a schedule. The latter is a good fit if you have non-technical docs (perhaps from a marketing site) that you also want to check for correctness.

You should plan to add a lot of words to the ‘known words’ file because most techdocs have jargon or names that aren’t in standard dictionaries.

Look For Dead Links

Running a link checker regularly will help your users avoid 404 pages. No matter how cute they are, they annoy folks because a 404 is a dead end. Techdocs are all about enabling end users to solve their problems and a 404 page doesn’t help.

You should run the checker periodically and fix busted links as soon as possible. I haven’t found a way to run it quickly enough to link checking on a PR. (Update July 7: someone on a slack pointed me to this open source partial link checker. I haven’t tested it, but they say they use it.) I like this fast link checker which catches 404s but also will error on page anchors that don’t exist. The latter are not as annoying as the former, but still impact developer experience.

Side note: link checkers only prevent internal linkrot. You can prevent external linkrot by being assiduous about your redirects. Never let an external link be sent to the home page or, worse, a 404 page.

Check For Closed Issues

Often documentation references an externally facing issue tracker, such as GitHub issues or Jira. After all, if there is a known problem that has a workaround or enhancement planned, adding a link to an external tracker lets the dev audience know and can help them accomplish their goals or know to wait for a future release. Often issues are closed but are still referenced in the docs, which means that the link is confusing.

This job mitigates that by iterating docs, looking for issue links, then checking to see the status of the issues. If the issue is closed, this task reports the problem that the doc can be updated.

Shrinking Images

Small images lead to a faster experience for your users, but it can be hard to remember to shrink the images as you are creating documentation. Automating such image shrinking using a tool like tinypng is an easy way to improve user experience.

This is best done on a pull request, but will require committing the changes using a tool like git-auto-commit.

Generic Content Checking

You can write a shell script using grep to look for known content issues. Examples:

  • Images without alt text.
  • Documents without a title or description tag.
  • Alt text that is not a full sentence.
  • Descriptions that are not a full sentence.
  • Enforcing that every blog post is in a known category.
  • Absolute URLs that point to your docs site. Everything should be relative so it’s easy to stand up locally.
  • Markdown syntax issues

In practice, this looks like: exit `find astro/src/content/ -type f -name "*.md*" | xargs grep ']()'| wc -l |sed 's/[ ]*//g'`

That one liner looks for a markdown link with an empty URL. The above GitHub action will fail if there is a non-zero exit code.

Custom Checks

Once you start automating content quality checks, you’ll find opportunities everywhere. Some ideas:

  • If you show example applications on your techdocs website, but store the code elsewhere in a repository, you can check to see that the numbers from each source are equivalent.
  • If you have handcrafted JSON examples, making sure they parse using a tool like jq. It can be easy to miss an errant comma.
  • Making sure every API page has example code on it.

As you look at your docs, I’m sure you’ll think of others.

How To Handle Errors

Many of these tasks will throw an error when something is incorrect, such as a misspelling or syntax error. There are two ways to handle these errors.

  1. If you can run the check quickly enough, run it on every push or every opened PR and provide feedback for fixing the issue. The doc author can then handle it immediately before the PR is merged.
  2. If the check takes a long time, like spell checking your entire site, then run it on a schedule. The last person to edit the workflow will get the notification, so it’s best to catch the error and have it send an email to a shared alias to capture the issue and then fix it.

Also, configure these tasks to be run manually (the workflow_dispatch event for GitHub actions). This helps with troubleshooting or testing when a fix has been made.

Conclusion

All of these tasks can help you remove some of the toil from creating an excellent techdocs site. You don’t have to do them all at once, but adding them will reduce your effort and increase your documentation quality.

How to have great guest posts on a blog

I have another blog that I’ve been running for a few years, called Letters to a New Developer.

It is full of advice I wish I had had at the beginning of my software development career. I even turned it into a book.

One thing I’ve done the entire time I’ve been writing this is to ask for guest posts. No advice is generic; it’s all based on where and who you are. By asking for guests to share their experiences, I expose my readers to a wider variety of viewpoints. This leads to better understanding of the wide world of software.

After all, if you want to work for a FAANG, it’s better if you hear from someone who prepped, interviewed and was hired for such a company, rather than me, who will never work for such a company. Or if you are looking to understand what it’s like to work in open source, hearing from someone who has made it their career. Have imposter syndrome? Others have struggled with it too. Also works for techniques; it’s great to hear from someone who is a methodical journaler, for example.

I bet you get the point. There’s no way I could have written any of those articles.

Guest posting also helps spread the word about my blog, since a guest poster usually shares their writing with their friends and network.

For the right kind of blog, guest posts can be great. Here’s my criteria:

  • Does the blog have decent traffic? If not, folks probably won’t want to guest post. I’d say at least 25 visits a day.
  • Have you done this for a while? A years worth of regular posting shows you’re serious.
  • Does the blog present a variety of viewpoints? I wouldn’t want a guest post for this blog, because it’s all me, all the time.
  • Is the blog non-corporate? If it is a company blog, guest posts should be paid for with money, or be part of a co-marketing project.

If you want to have great guest posters, here’s my recipe for finding them and fostering them:

  • Don’t accept inbounds, unless you know the person or can vet what else they’ve written.
  • Don’t accept any content that is off-topic.
  • Set a target guest. For me, it is someone who is or was a developer with different viewpoint and perspective. I was especially interested in highlighting voices of people who were not white men.
  • Keep an eye out for interesting posts or interviews. Reach out to the authors and see if they might be game to guest post.
  • Be okay with a cross post. I’d say about 60% of my guest posts are original content, but more recently authors want to post the content on their blog first. Offer a link back.
  • Be okay with a repost, especially if the author is prominent. Review existing content and find one or two articles that fit with your blog’s theme. Ask if you can re-publish. Offer a link back.
  • Some folks will say no. They can do so for a variety of reasons (don’t want to write, don’t syndicate their content, etc), and you have to be okay with it.
  • If someone says “I can’t right now, sorry.” reply with something like “Totally get it. I’ll follow up in 6 months, unless you’d rather I didn’t.” Then, follow up. Sometimes they’ll ignore you, sometimes they’ll have more time.
  • If they are writing original content for you, offer to edit it. Find out how much editing they want. I’ve over-edited a few times and that’s a bad thing if someone is volunteering their time and knowledge.
  • If they ask about topics, advise them to focus. Lots of people want to write about “5 things you need to do” but in my experience, articles with focus on one topic are better received.
  • Write up guidelines so you can easily share. This can include audience, benefits, formatting, delivery mechanism, word count, topic ideas, and more.

Hope this helps. Happy guest blogging!

Options for scaling written content as a devrel team

If you are in developer relations, creating content is often a big part of your job. Written content is one of type of content, and a common one at that.

Written content scales well, is easily updated, can be consumed on readers’ schedules, is fairly accessible, and can be reused. It also can serve as a foundation for other kinds of content, such as talks, example apps, or videos.

At FusionAuth, I’m part of a team that creates a lot of written content. I wanted to talk about a couple of ways you can scale written content creation. Note this focuses on creating more content, but don’t forget to write the right content, which is more important than sheer volume. (That’s a whole other blog post, though I feel a bit like Fermat even mentioning it.) This post also assumes you can’t hire more in-house talent, either because of budget or because finding good devrel folks is really hard right now.

Re-use

First up, re-use your content. You can take pieces and use them in different ways. For instance, write a great piece of long form content. Then, pick a few of the most interesting paragraphs or sentences, and share those in Twitter or on other social media. You can also use these excerpts as fodder for comments on online communities or forums, if they answer a question someone else is asking.

Finally, you can also combine articles. For instance, I lightly edited a number of articles, wrote a few pieces of original topical content, and ended up with an ebook about outsourcing your auth which has been useful to share with readers and possible FusionAuth prospects. The effort was far far less than if I’d set out to write a full book on the same subject.

The next two options require increasing amounts of money, so if you only have your time to spend, focus on this option.

Find freelancers

The next option is to find freelancers or community members who are willing to write articles. At FusionAuth, we paid money for these posts, which is typical. Our rates were between $0.25 and $0.50 per word, typically including an example app that we would host in our GitHub organization and open source.

Thee downsides of freelancers are:

  • it is hard to find good ones. I did find a couple who delivered multiple good posts, but they are few and far between.
  • you have to manage them and their delivery. This can include extensive editing depending on skill level.
  • you have to give them an outline. While I tried to get folks to ‘pitch me’ with interesting ideas, that didn’t turn up much at all.
  • they are not going to know your product or space as well as you do.

If you are larger, you might be able to pay less because it’ll be more of a plum for authors being published and associated with you, but you should pay something. Developers know the value of good content.

You may also highlight articles written by someone on their own blog, but that isn’t your content and You won’t be able to re-purpose it. You could, I suppose, reach out and see if the authors would be okay with you licensing it. Haven’t done that myself; I prefer to keep it simpler and share whatever someone writes about FusionAuth.

Content agency

The final option is to hire a content agency. The active ones that I know of are draft.dev (disclosure, FusionAuth is a current client), Ritza, and Hit Subscribe. (I am sure there are others.) These agencies have different strengths and approaches, but they all take some of the management burden off of you. Often you can come to them with just an idea and they’ll build out a content brief (an outline), find someone to write it, and do an technical editing pass before delivering it to you.

These are great if you want technical content frequently. It is also fantastic if you want posts written about technologies that aren’t in your wheelhouse.

For example, Joomla is used across 1.7% of all sites on the internet and we wanted an article about SSO and Joomla. I didn’t want to come up to speed on the. So draft.dev found a Joomla expert who was willing to write an article for us about SSO and Joomla. If not for them, I doubt that post would ever have seen the light of day.

The downsides of these agencies include:

  • you have less control over the freelancer selected.
  • they will not know your product or space as well as you do.
  • you’ll still have to do some quality control and checking. You can’t outsource this entirely, because of the above point.
  • it’ll cost more money. You can expect to spend between $1000 and $3000 for a post (at the time I write this).

These are options that I know of that let you scale up your written content creation.

Moving a database driven side project to Netlify

FishermanI recently moved a side project to Netlify. Netlify, if you aren’t familiar with it, is a static file hosting service with a great workflow, free SSL certificates, and a built in CDN. I made the move because the side project was a database driven site, but didn’t really use other server side interactivity (no user generated content, etc). I haven’t invested much in the side project over the past couple of years, but it still gets tens of hits a day and is useful to a users. It’s a top hit on google for certain keywords. I didn’t want to spend time updating the underlying application, but wanted it to be more secure. The site is only updated over a month or so every year. It is then read only for the next eleven months of the year, as it provides information on a very seasonal service.

Moving to Netlify (or, frankly, any other static site provider) provides the following benefits:

  • extremely low cost (Netlify is free for the level of traffic I have).
  • faster browsing experience for the end user (due to being behind a CDN).
  • free SSL certificates, with no hassle of setting up letsencrypt myself.
  • indirection–the source site can now be anywhere. I could put it on an ec2 instance that I only boot up when I need to push a new build, or could even host it locally. This means that I don’t need to worry about the updating the source application.

I could have chosen to do this with S3/Cloudfront/AWS Certificate manager/Lambda (or using technologies from some other cloud provider). But even though I’m familiar with all the steps to do this, it was lot of work. And Netlify was free and had done all the grunt work of bringing all these technologies (or similar ones, I’m not familiar with the tech behind the site, but they do write about it a bit).

What steps did I take to move a database driven directory site to Netlify? Enough that I wanted to document this for my future self.

  • Change the DNS for the site to have a lower TTL. During the transition from your site to Netlify, users may see versions served from either the old site or the new site, and you want to minimize that.
  • Remove all forms and other server side interactivity that your site has. You can replace them with javascript driven interactivity (like Disqus for comments). Netlify has some support for functions, but I didn’t explore that.
  • Set up Netlify using a default URL (they provide it, it is something like ‘foo-bar-123.netlify.com’). I deploy from Bitbucket. (I know some people hate on Bitbucket, and I don’t like all of their UI, but they are free for private repos, which is a win for me.) This let me understand how to deploy a simple one page site.
  • Download the site. I used wget: wget -mk http://mysite.com/ . The switches set up mirroring and convert all links to be relative.
  • Move the site to a subdirectory (I used web) and configure Netlify to deploy the HTML from that subdirectory. This lets you have other scripts outside of the webroot that can help you build the site.
  • Check in the site with the downloaded content and push it up to Bitbucket.
  • Watch the site be deployed to the Netlify default URL.
  • Access the site via SSL: ‘https://foo-bar-123.netlify.com’ and look in the console for “mixed active content” errors. Fix those as applicable. (If this was for a client, rather than a side project, I’d solve all of these.) Note that if the source site is http only, you may need to hardcode https URLs.
  • See that “pretty” html pages like ‘https://foo-bar-123.netlify.com/about’ are rendered as text.
  • Ask on Stackoverflow about the problem.
  • End up solving it by generating a _headers file after downloading the site. Update Stackoverflow question with the answer.
  • Test again that the site at the default URL looks good.
  • Update the webserver config and DNS for the database driven site. I wanted it to answer to both the old addresses: mysite.com/www.mysite.com and a new address: generator.mysite.com
  • Update DNS to point mysite.com and www.mysite.com to the Netlify site. Instructions here.
  • Wait for DNS to propagate. When it does, check to make sure that SSL works.
  • Add a password for generator.mysite.com
  • Test the download process with the password (the wget command changes to wget --user=USER --password=PASS -mk http://mysite.com/
  • Write a script to do the download process.
#!/bin/sh
wget -mk  --user=USER --password=PASS http://generator.mysite.com/
mv generator.mysite.com mysitecom
cd mysitecom
find `pwd` -type f |grep  -v \\. |sed "s#`pwd`/##" > list 
for i in `cat list`; do echo "/$i" >> _headers; echo "  Content-Type: text/html" >> _headers; done
rm list
cd ..
rm -rf web
mv mysitecom web
git add web
git commit -m "pull in latest from generator.mysite.com"
# could do a git push here as well if you wanted

Now, any time that I make changes to the site, I need to run this script and push to Netlify. I could put it in cron or a scheduled lambda if I thought I was going to make changes often. For now, I’m content to run it manually. One downside is that I self-host my analytics code and that site is not SSL enabled. So I’ll need to make a change if I want to continue to get statistics.

So, in the end I have a fast, secure site that is hosted outside my infrastructure and that is easy to update. This process, while not trivial, was easy enough that it has me thinking about where else I could use it (this blog, for instance). Highly recommend.

Where I’m Writing

It’s been a bit quiet around here, but since I joined Culture Foundry I’ve been writing over on their site more. I’ve written on a number of topics, but this one about how to move files between different servers in order to scale a traditional CMS application horizontally, is one of my favs:

Compared to the other methods of syncing files, this works well with non cloud native applications. It also has the virtue of using old tested tools. You can use this system on prem or in the cloud, anywhere with SSH and rsync. This system works well with large numbers of small objects because rsync can be configured to only push new objects.

I’ll be splitting my blogging time between this site and the Culture Foundry site in the future. See you there.

 

Lessons from curating a link blog

link photo
Photo by StockMonkeys.com

I maintain a link blog about Colorado food and local food in general.  I use Tumblr, but I’m only incidentally interested in Tumblr traffic.  Tumblr hooks up to Facebook and Twitter, and pushes links there.  (I realize that I am missing interaction on Twitter and Facebook by using these networks as broadcast only, but I don’t have time to fully engage, so I thought a limited presence was better than nothing.)

Having maintained this link blog for over two years, I have learned a few things.

  • It is easy to start a project like this, but hard to finish.  There’s always more to do.  I think I’ll stop when it stops being interesting.
  • Deciding to do this is a great way to gain a broad understanding of a field while providing some value (via curating).  As you find more and more sources of links, videos, articles and audio content, you’ll gain a sense of what is happening.  Even if you don’t painstakingly read every article, you’ll still get a sense.
  • Speaking of sources, Google alerts is your friend.  I get emailed alerts on a variety of searches, and about 25% of the results are worth posting.  Facebook and twitter are additional great sources of links.
  • An RSS reader can help you if you are really diving in.
  • Giving someone notice that you’ve referenced their article via an ‘@’ mention will get you their attention.
  • Queuing up posts on Tumblr is a life saver.  This lets you stack up posts and portion them out one per day.  I typically have between 15 and 30 posts in my queue.  This makes timely posts more difficult, but frees me up to forget about the link blog for weeks at a time.
  • A link blog like this is a great use of your in between time, especially if you have a smartphone.  In five minutes I can scan and post two or three links, where five minutes is barely enough time to think of a regular blog post.  The Tumblr app is very good.
  • A linkblog is a great resource for other content generation.  I have a newsletter about local food as well, and a key section of that is interesting links.  Those are almost entirely drawn from the Tumblr.

The linkblog approach is very similar to Twitter, but differs in a few crucial ways:

These attributes make a linkblog a fine complement to Twitter.

There are some problems with this model, however.

  • Limited interaction with followers, either on Tumblr, Facebook or Twitter.
  • I’ve found that engaging on Twitter and Facebook directly is far more effective if you want content to be viewed or links to be clicked.
  • A linkblog like this is not truly building my tribe

So, if you have limited time, want to gain insight into a particular area of interest, and are OK with the drawbacks, create a linkblog.