Skip to content

Creating a CircleCI orb to authenticate a user during a build

I’m a big fan of automating all the things and I also believe in the DRY principle. I’ve been using CircleCI for years and noticed that they had added a way to abstract away some repeated configuration called orbs. I recently built one and wanted to share my experience.

If you are thinking about building an orb, first take a look at the list of existing orbs. After all, it’s better to reuse code someone else will maintain if it does what you need.

In my case, I wanted to explore building an orb. I dreamed up a use case for which no one else had written an orb. The situation is that you store user data in FusionAuth. Each time a build runs, you want to verify that the user running the build is active andvalid before continuing the build. If the user can’t authenticate, fail the build.

Set up the authentication server

I set up a FusionAuth server using their instructions on EC2. It can’t be on localhost because CircleCI needs to communicate with it during the build. The server didn’t run well on a t2.micro size so I ended up springing for a t2.large, where it worked fine. I also had some difficulty installing mysql on the Amazon Linux AMI, but this SO answer helped me out. I added a FusionAuth application and user via the admin panel. I also got an API key. I limited the API key drastically; it could only post to the login endpoint. I tested access with curl like so (ALL_CAPS strings are placeholders you’d want to replace with real values):

curl -s -o /dev/null -w "%{http_code}" -XPOST -H 'Authorization: API_KEY' \
-H 'Content-Type: application/json' \
-d '{ "loginId": "USERNAME", "password": "PASSWORD", "applicationId": "APPLICATION_ID" }' \
http://FUSION_AUTH_SERVER_HOSTNAME:9011/api/login|grep 202 > /dev/null

FusionAuth returns a 404 status code if the user isn’t authenticated successfully (they don’t exist or have an incorrect password) and 202 if the user logs in successfully (more API information here). Note that this login process may skew some of the reporting (around DAUs, for example) so you’ll want to create a separate application just for CI/CD.

The curl + grep statement above exits with a value of 0 if the user successfully authenticates and with a value of 1 if authentication fails. The former will allow the build to continue and the latter will stop it.

The check I want to run worked. Now I just needed to build and publish the orb.

 

White orb

Building the orb

An orb is reusable CircleCI configuration. You have three kinds of configuration you can re-use:

  • jobs: a normal CircleCI job, but you can also pass in parameters. A great option if you have a job you want to use across different projects; something like ‘run this formatting tool’.
  • executors: an environment in which to execute code (docker container, vm, etc).
  • commands: a set of steps that can be reused. It is lower level than a job and can be used across different jobs.

I chose to create a single command. I also created a job to run while I was developing (I added it as a project in CircleCI). You can see the full source code here.

Here are the interesting bits where I define the command to be shared (ah, yaml):

commands:
  verifyauth:
    parameters:
      username:
        type: string
        default: "user"
        description: "FusionAuth username to try to validate"
      applicationid:
        type: string
        default: "appid"
        description: "FusionAuth application id"
      hostbaseurl:
        type: string
        default: "http://ec2-52-35-2-20.us-west-2.compute.amazonaws.com:9011/"
        description: "FusionAuth host base url"
      password_env_var_name:
        type: env_var_name
        default: BUILDER_PASS
        description: "The user's FusionAuth password is stored in this environment variable"
      fusionauth_api_key_env_var_name:
        type: env_var_name
        default: FUSION_AUTH_API
        description: "The FusionAuth API key is stored in this environment variable"
    steps:
      - run: |
         curl -s -o /dev/null -w "%{http_code}"   -XPOST -H "Authorization: ${<< parameters.fusionauth_api_key_env_var_name >>}"  -H 'Content-Type: application/json' -d '{ "loginId": "<< parameters.username >>", "password": "'${<< parameters.password_env_var_name >>}'", "applicationId": "<< parameters.applicationid >>" }' << parameters.hostbaseurl >>api/login |grep 202 > /dev/null
      - run: |
         echo User authorized

The command verifyauth takes parameters. These have defaults and descriptions. Anything that you wouldn’t mind seeing checked into a source code repository can be passed as a parameter. You then call the command in your job and pass parameters as needed (we’ll see that below).

However, there are sometimes secrets which need to be stored as environment variables (in the project or the context): API keys or passwords, for example. However, I still wanted to make them configurable by whoever uses the orb. Enter the  env_var_name parameter type. This type lets the user specify the name of the environment variable. If I set the password_env_var_name to AUTH_CHECK_PASS, then I need to make sure there is an AUTH_CHECK_PASS environment variable set somewhere in my project containing the password with which we’ll authenticate against FusionAuth. This lets the orb be both configurable and secure.

Finally, you can see that the first step of the command is posting login data to the authentication server. Again, if we see anything other than 202 we fail and the build stops. (You’ve seen that curl command before.)

Publishing the development orb

To be able to use the orb with a different project, I needed to publish the orb (I could have developed the orb inline to avoid this). The publishing instructions are here. The only issue I ran into was that I had to update my CircleCI organization settings and allow “Uncertified Orbs” before I could create a namespace. After that I was able to publish a development version of my orb:

circleci orb publish .circleci/config.yml mooreds/verifyauth@dev:testing

I was in the directory of my orb code and referenced my config. mooreds is my orb namespace, verifyauth is the orb name (which is arbitrary and not connected to the source repository name in any way) and dev:testing is the version of the orb. Note that there are two types of orb version: production, which strictly follow semantic versioning, and development, prefaced by dev: and after string that can contain “up to 1023 non whitespace characters”. Development orbs have other limitations: they are not public, are mutable and only last for 90 days. You’ll want to publish your orbs with production versions if you are using them for any purpose other than prototyping or exploration.

I published my orb via the command line, but the docs outline publishing via a CircleCI job.

Testing the orb

Now I wanted a second project to test the orb. Here’s the project source code. Here’s the interesting code:

...
orbs:
  verifyauth: mooreds/verifyauth@dev:testing
jobs:
  build:
    steps:
      - verifyauth/verifyauth: # when called from external job, command is namespaced to by the name of the orb
          username: "circlecimooreds"
          applicationid: "98113cee-d1a8-4abf-baf5-a6ea742f80a1"
  ...

You can see that I pull in the orb at the development version, which I’d previously published. Then I call the namespaced command with some parameters. For this command to work, I also needed to set up required environment variables (in this case, BUILDER_PASS and FUSION_AUTH_API because I didn’t pass in any of the env_var_name parameters). If you don’t set those environment variables (or, alternatively, set the parameters to different values, and then set those environment variables), the build will fail no matter what, as the API call won’t succeed.

I then pushed this sample project up to CircleCI and ran a few builds to make sure the parameters were being picked up.

Publishing the production orb

Now that I had an orb that is parameterized and exposed the command we want to share, I needed to publish it for everyone to use. Note that your configuration code is entirely exposed if you publish an orb. You can see the source of any orb via the circleci orb source command. circleci orb source mooreds/verifyauth@0.0.2 will show you the entire source of my sample orb. They warn you a number of times about this.

To promote to production an orb that you have published to development, update the dev version: circleci orb publish .circleci/config.yml mooreds/verifyauth@dev:testing to catch any changes and then promote it: circleci orb publish promote mooreds/verifyauth@dev:testing patch.

Note that the patch argument at the end of the promote command bumps the patch number (0.0.1 -> 0.0.2) but you can also bump the minor and major numbers. Any changes you make to a production orb require you to publish and promote it again; production orb versions are immutable. For instance, I wanted to update the description of some parameters, but had to publish an entirely new version.

After publishing, you’d want to update any projects that use the orb to use the production version.

Publiished orb listing

The listing of my published orb

Areas for further work

This was a slightly contrived example. I wanted to gain some experience both with FusionAuth and with CircleCI (I have friends who work at both companies). There are a number of areas where this could be improved:

  • authenticate against a different authentication server (LDAP, Okta, AWS IAM)
  • store additional information about the user in the authentication database (for instance, which projects they can build) and convert the authentication curl command into an authorization command
  • run the identity server over SSL (I just used HTTP because it was easier to get up and running, but that’s obviously a production no-no)
  • pull the user and password from the build environment. It’s pretty clear to me how you’d pull the user (there’s a CIRCLE_USERNAME environment variable) but I’m not sure how to pass the password. I can think of a couple of solutions:
    • don’t login at all, just allow the API key to pull user data and match on the username (this is probably the best option)
    • pass the password via a pipeline parameter, which means you’d have to set up an API call to build
    • have one common password for all users in the FusionAuth system, and use it only for access control to the build pipeline
    • make the password the same as the username in the FusionAuth system, and use it only for access control to the build pipeline

In conclusion

If you want to interact with external services from within CircleCI, check out the list of existing orbs.

If you have a service that you want to make it easier for CircleCI users to interact with and use, create an orb and publish it.

If you are working with CircleCI and have duplicate configuration that you want to share between projects, setting up your own orbs is a great idea. Orbs are flexible and easy to parameterize. If you’re OK with your configuration being public (it wasn’t clear to me if there was any way to have the configuration kept private), you can encapsulate your build and deploy best practices in an easy to consume manner.

APIs Devops FusionAuth | moore |

CircleCI shutting down version 1.0

I’ve been a happy user of CircleCI at multiple companies. Right now we pay them at The Food Corridor and they handle almost all our deployments. (I still deploy to production manually.)

We just got a note that they are shutting down their 1.0 offering and will not support it as of Aug 2018. The 2.0 offering was announced in 2016 and generally available in 2017. So, there will be about one year of overlap. Not too long.

I understand that desire to move forward.Trust me, I do.

I don’t know how much engineering effort it takes to support the two versions, but my guess is that they’ll see some significant customer loss from this. Why? CI is something that you just want to work. You don’t want to think about it. Which is why a SaaS solution makes so much sense. I am happy to just keep paying them month after month for their excellent product.

But, if I have to take some cycles to move from CircleCI 1.0 to CircleCI 2.0, why wouldn’t I take some time and evaluate other solutions too? I assume they’ve run the numbers and the amount of money it takes to support 1.0 must be more than the amount they will lose via churn.

AWS does a good job of this–they never deprecate anything (you can still set up SimpleDB if you want). They just hide it, make other offerings better, and make older offerings more expensive.

In fact, if I were CircleCI, I might offer a ‘legacy’ CircleCI 1.0 plan, where people with significant investments in the older infrastructure can pay more for access to that old codebase. Depending on the amount of support required, that might be some significant free money.

Relatedly, Amy Hoy has a great post on how to get your customers to pay you more money.

Devops Programming | moore | | Leave a comment

Great GitHub actions for techdocs sites

There are two main options for techdocs sites. You can use a SaaS solution like Readme or DeveloperHub. Or you can use a static site generator like Jekyll, Hugo or Docusaurus. There are strengths to each approach, but for this post I want to look at continuous integration/continuous deployment (CI/CD) tasks you can run on staticly generated tech documentation.

Why use CI/CD for your docs site? CI/CD tools let you automate code or documentation tasks. For documentation, you can run checks to ensure consistency and correctness which make your documentation better.

GitHub Actions is the CI/CD tool I’m going to refer to, but similar functionality is available with other CI/CD solutions such as CircleCI or Jenkins. Additionally, I’ll talk about the content living on GitHub in a repo, but you could use self hosted git, gitlab, bitbucket or other version control solutions.

This post has a couple of assumptions:

  • Your content is in version control.
  • Your content is in a format that needs to be processed before it is ready for publishing. In other words, you are editing text files in a certain format (markdown, asciidoc, etc) that are then processed, rather than using a WYSIWYG editor.
  • The people writing your content are comfortable using version control and tools like markdown.

Here are tasks you should automate:

Deploying The Documentation

This is the first one and the easiest win. You should have a workflow which automatically deploys changes when a PR is merged to your primary branch. This lowers the barrier to entry for contributing to documentation, because you only need to get a GitHub PR approved. Similar to continuous deployment of a SaaS application, it makes it easy to push changes regularly. Because it is a GitHub PR, it also is easy to roll back–you just revert the PR.

You may wonder at my including this, because it seems so obvious, but I’ve been on teams where the deploy process included sshing to a server, pulling the branch, and running a script.

Automating deployment was well worth it, because it changed “I found a typo, oh man, what a pain” to “I found a typo, I can push a PR and fix it in 3 min”.

Vale

Vale is an amazing, free linter for your prose. I haven’t witnessed it used at its full potential, but it is a spell checker, word police and living style guide all in one. It’s kinda imposing to start with since there are so many “nerd knobs”, but start simple and add to it as you learn it.

Vale runs fast enough you can have it run on every PR.

Spelling

Deploy a spell checker to run against your documentation to prevent misspellings and typos.

You have two choices here:

  • spell check just the changes (vale is good at this)
  • spell check the entire site (pyspelling is good at this)

The former lets you block a PR if there’s an issue, while the latter takes a lot longer and should run on a schedule. The latter is a good fit if you have non-technical docs (perhaps from a marketing site) that you also want to check for correctness.

You should plan to add a lot of words to the ‘known words’ file because most techdocs have jargon or names that aren’t in standard dictionaries.

Look For Dead Links

Running a link checker regularly will help your users avoid 404 pages. No matter how cute they are, they annoy folks because a 404 is a dead end. Techdocs are all about enabling end users to solve their problems and a 404 page doesn’t help.

You should run the checker periodically and fix busted links as soon as possible. I haven’t found a way to run it quickly enough to link checking on a PR. (Update July 7: someone on a slack pointed me to this open source partial link checker. I haven’t tested it, but they say they use it.) I like this fast link checker which catches 404s but also will error on page anchors that don’t exist. The latter are not as annoying as the former, but still impact developer experience.

Side note: link checkers only prevent internal linkrot. You can prevent external linkrot by being assiduous about your redirects. Never let an external link be sent to the home page or, worse, a 404 page.

Check For Closed Issues

Often documentation references an externally facing issue tracker, such as GitHub issues or Jira. After all, if there is a known problem that has a workaround or enhancement planned, adding a link to an external tracker lets the dev audience know and can help them accomplish their goals or know to wait for a future release. Often issues are closed but are still referenced in the docs, which means that the link is confusing.

This job mitigates that by iterating docs, looking for issue links, then checking to see the status of the issues. If the issue is closed, this task reports the problem that the doc can be updated.

Shrinking Images

Small images lead to a faster experience for your users, but it can be hard to remember to shrink the images as you are creating documentation. Automating such image shrinking using a tool like tinypng is an easy way to improve user experience.

This is best done on a pull request, but will require committing the changes using a tool like git-auto-commit.

Generic Content Checking

You can write a shell script using grep to look for known content issues. Examples:

  • Images without alt text.
  • Documents without a title or description tag.
  • Alt text that is not a full sentence.
  • Descriptions that are not a full sentence.
  • Enforcing that every blog post is in a known category.
  • Absolute URLs that point to your docs site. Everything should be relative so it’s easy to stand up locally.
  • Markdown syntax issues

In practice, this looks like: exit `find astro/src/content/ -type f -name "*.md*" | xargs grep ']()'| wc -l |sed 's/[ ]*//g'`

That one liner looks for a markdown link with an empty URL. The above GitHub action will fail if there is a non-zero exit code.

Custom Checks

Once you start automating content quality checks, you’ll find opportunities everywhere. Some ideas:

  • If you show example applications on your techdocs website, but store the code elsewhere in a repository, you can check to see that the numbers from each source are equivalent.
  • If you have handcrafted JSON examples, making sure they parse using a tool like jq. It can be easy to miss an errant comma.
  • Making sure every API page has example code on it.

As you look at your docs, I’m sure you’ll think of others.

How To Handle Errors

Many of these tasks will throw an error when something is incorrect, such as a misspelling or syntax error. There are two ways to handle these errors.

  1. If you can run the check quickly enough, run it on every push or every opened PR and provide feedback for fixing the issue. The doc author can then handle it immediately before the PR is merged.
  2. If the check takes a long time, like spell checking your entire site, then run it on a schedule. The last person to edit the workflow will get the notification, so it’s best to catch the error and have it send an email to a shared alias to capture the issue and then fix it.

Also, configure these tasks to be run manually (the workflow_dispatch event for GitHub actions). This helps with troubleshooting or testing when a fix has been made.

Conclusion

All of these tasks can help you remove some of the toil from creating an excellent techdocs site. You don’t have to do them all at once, but adding them will reduce your effort and increase your documentation quality.

Content Devrel | moore |

GitHub Actions Are Amazingly Easy

GitHub Workflows are automated jobs that can be triggered by various events against a GitHub repository. They are pretty awesome.

GitHub Actions are a way to encapsulate configuration and functionality in a way that can be easily reused in GitHub Workflows.

I was thinking it’d be fun to create some GitHub Actions (yes, I’m the life of the party), so I sat down a few mornings ago to do this. I was shocked at how easy it was.

I followed a few lines of this tutorial to create a workflow. Then I created an action by following this tutorial. Finally, I edited my workflow to use the new action. That was it.

It was amazingly simple and took me about 30 minutes. I ran into one unrelated issue (to set the executable bit on a shell script in windows, I had to modify the shell script contents in order to ensure the change was sent to the remote repo).

If you take a look, you’ll see these are both toy repositories, to be sure. However, the ability to write jobs which will be executed on a git push, pull request or other events is great and removes toil. Being able to extract common functionality to an action is even better. Finally, the ability to share the action publicly by adding it to the GitHub marketplace is fantastic.

I’ve liked CircleCI for a long time, but if I were them I’d be worried.

One issue I found is that the testing/release cycle is pretty tedious (I’ve mentioned that action debugging to be an issue for a while).

While I was troubleshooting my executable bit error, I had to do the following every time I wanted to test a change:

  • make a change in the action repository
  • create a new tag
  • push it to the remote
  • switch to the workflow repository
  • bump the action version
  • push to the remote
  • wait for the workflow to complete

Not horrific, but pretty tedious. I don’t know if there are other options such as local deployment which would reduce that cycle, but that would be swell.

Other than that, 10 out of 10, would write more actions.

Devops Programming Technology Tips Useful Tools | moore |

How To Start Improving a Legacy App

An interesting question appeared on HN recently: “Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?”

You can read that post and the answers there. I’m going to address a related, but different question in this post.

If you run encounter an application that has tremendous business value and yet is not following any modern software management processes, what are concrete steps you can take to help improve the application?

That is, what if you have (or are hired to be responsible  for) an app like the poster of the HN thread:

  • making plenty of money for the company
  • no version control
  • old school structure
  • a ball of mud architecture
  • no code deleted, just commented out
  • multiple versions of libraries on the front and back end
  • etc

First off, you need to convince someone that it is going to be worthwhile to invest in this process. If you can’t do that, you are dead in the water. So look for the pain points that occur when best practices are lacking:

  • slow delivery of features
  • catastrophic bugs which lose money, hurt the brand or impact data
  • talent hard to hire
  • hard to improve the application

If you can pinpoint pain caused by the app, you can start to build a case to improve it.

If you can’t, well then, maybe you shouldn’t touch it. If it ain’t broke, don’t fix it!

With that said, here’s my list of what to implement, in rough priority order. Don’t worry about best of breed for the tools, just pick what the company uses. If the tool isn’t in use at the company, pick something you and the team are familiar with. If there is nothing in that set, pick the industry standard. I include a recommendation for the latter.

1. Get the app under version control. Git is best if you don’t have any existing solution. GitHub or GitLab are great places to store your git repositories.

2. Start up a bug tracker. You have to have a place to keep track of all issues. GitHub issues is adequate, but there are a ton of options. This would be an awesome place to get buy-in from the existing team about whichever one they prefer. The truth it is doesn’t matter which particular bug tracker you use, just that you use one.

3. A way to get one click or zero click deploys. A SaaS tool like CircleCI, GitHub actions is fine. If you require “on prem”, Jenkins is a fine place to start. But you want to be able to deploy changes quickly.

4. Set up a staging environment. With one, you can manually test changes and debug issues without affecting production. Building this will also give you confidence that you understand how the system is deployed. Then you can can include that in the build tool process.

5. Unit and system/end to end testing. End to end testing can give you confidence in changes. However, it is overwhelming to add testing to an existing large, crufty codebase. I’d focus on two things: unit testing some of the weird logic; this is a relatively quick win. Second, setting up at least one or two end to end tests through core flows (login, purchase path, etc). In my experience, setting up the first instance of each of these is the toughest process, then it gets progressively easier. There’s usually an ‘xUnit’ framework in any language for unit testing. Look for that. I’m not sure what best practice is in end to end testing, but selenium or cypress are good for browser based applications.

6. Capture documentation. This might be a higher priority, depending on what your relationship with the existing team is. Few teams will say no to someone helping out with doc. Document high level architecture, deployment processes, key APIs, interfaces, data stores, and more. Capture this in google docs or a wiki if you don’t have an existing solution.

7. Start using data migrations. Having some way to automatically roll database changes forward and back is a huge help for moving faster.

None of these are about changing the code (except maybe the last one), but they all wrap the code in a blanket of safety.

After implementing one or more of these, the team should be able to move faster and with more confidence. This will build trust and allow you to suggest bigger changes, such as bringing in a framework or building abstraction layers.

Consulting Programming | moore |

GitHub actions and workflows

I recently wrote my first real GitHub action workflow at work. It was to publish our website after a merge or push to our main branch.

After this experience, I think these workflows are perfect for simple automation tasks. Things like:

  • Running a linter like rubocop on your code
  • Deploying a simple application (one or a few artifacts).
  • Running unit and integration tests.

I didn’t use self hosted actions, though that seems like a nice escape valve if you want to run things within your own network or run over limit. GitHub publishes the action and workflow limits (storage, runtime) and that’s definitely worth reviewing.

You also can easily stand up a couple of different service containers (right now only postgresql and resdis) for easy integration testing. You can also abstract out your commonly used workflow segments to versioned actions.

It was really a pain to write the workflow, however. I had to push repeatedly to our mainline branch, and there were times I screwed up the YAML or didn’t have my script correct. The feedback loop was slooow. Ouch. There are solutions to run them locally, but I didn’t try it yet.

Other than that, it was a positive experience. If you are using GitHub and have automation needs, take a look at GitHub actions. I am a big fan of CircleCI and have been for years. GitHub actions covers a lot of the same ground. GitHub actions are less sophisticated, but it seems like a definite “innovators dilemma” play. So I expect to see actions to get more and more sophisticated.

Programming Tips Useful Tools | moore |

The wonders of outsourcing devops

I have maintained a Jenkins server (actually, it was Hudson, the precursor). I’ve run my own database server.  I’ve installed a bug tracking system, and even extended it. I’ve set up web servers (apache and nginx).

And I’ll tell you what, if I never have to do any of these again, I’ll be happy as a clam. There are so many tools out there that let you outsource your infrastructure.  Often they start out free and end up charging when you reach a certain point.

By outsourcing the infrastructure to a service privder, you you let specialists focus on maintaining that infrastructure. They achieve scale that you’d be hard pressed to. They hire experts that you won’t be able to hire. They respond to vulnerabilities like it is their job (which it is).

Using one of these services also lets you punch above your weight. If you want, with AWS or GCP you can run your application in multiple data centers around the globe. With heroku, you can scale out during busy times, and you can scale in during slow times. With circleci or github or many of the other devtool offerings, you can have your ci/cd/source repository environment continually improved, without any effort on your part (besides paying the credit card bill).  Specialization wins.

What is the downside? You lose control–the ability to fine tune your infrastructure in ways that the service provider may not have thought of.  You have to conform to their view of the world.  You also may, depending on the service provider, have performance impacted.

At a certain scale, you may need that control and that performance.  But, you may or may not reach that scale.

It can be frustrating to have to workaround issues that, if you just had the appropriate level of access, you would be able to fix quickly.  It’s also frustrating having to come up to speed on the docs and environment that the service provider makes available.

That said, I try to remember all the other tasks that these services are taking off my plate, and the focus allowed on the unique business differentiators.

AWS Programming Useful Tools | moore | | Leave a comment