Skip to content

Using GPT to automate translation of locale messages files

At my current employer, FusionAuth, we have extracted out all the user facing messages to properties files. These files are maintained by the community, and cover over fifteen languages.

We maintain the English language version. Whenever new user facing messages are added, the properties file is updated. Sometimes, the community contributed messages files are out of date.

In addition, there are a number of common languages that we simply haven’t had a community member offer a translation for.

These include:

  • Korean (80M speakers)
  • Hindi (691M)
  • Punjabi (113M)
  • Greek (13.5M)
  • Many others

(All numbers from Wikipedia.)

While I have some doubts and concerns about AI, I have been using ChatGPT for personal projects and thought it would be interesting to use OpenAI APIs to automate translation of these properties files.

I threw together some ruby code, using ruby-openai, the ruby OpenAI community library that had been updated most recently.

I also used ChatGPT for a couple of programming queries (“how do I load a properties file into a ruby hash”) because, in for a penny, in for a pound.

The program

Here’s the results:


require "openai"
key = "...KEY..."

client = OpenAI::Client.new(access_token: key)

def properties_to_hash(file_path)
  properties = {}
  File.open(file_path, "r") do |f|
    f.each_line do |line|
      line = line.strip
      next if line.empty? || line.start_with?("#")
      key, value = line.split("=", 2)
      properties[key] = value
    end
  end
  properties
end

def hash_to_properties(hash, file_path)
  File.open(file_path, "w") do |file|
    hash.each do |key, value|
      file.write("#{key}=#{value}\n")
    end
  end
end

def build_translation(properties_in, properties_out, errkeys, language, client)
  properties_in.each do |key, value|
    sleep 1
# puts "# translating #{key}"
    message = value
    content = "Translate the message '#{message}' into #{language}"
    response = client.chat(
      parameters: {
        model: "gpt-3.5-turbo", # Required.
        messages: [{ role: "user", content: content}], # Required.
        temperature: 0.7,
      }
    )
    if not response["error"].nil?
      errkeys << key #puts response 
    end 

    if response["error"].nil? 
      translated_val = response.dig("choices", 0, "message", "content") 
      properties_out[key] = translated_val 
      puts "#{key}=#{translated_val}" 
    end 
  end 
end

# start the actual translation 
file_path = "messages.properties" 
properties = properties_to_hash(file_path) 
#puts properties.inspect 
properties_hi = {} 
language = "Hindi" 
errkeys = [] 

build_translation(properties, properties_hi, errkeys, language, client) 
puts "# errkeys has length: " + errkeys.length.to_s 

while errkeys.length > 0
# retry again with keys that errored before
  newprops = {}
  errkeys.each do |key|
    newprops[key] = properties[key]
  end

  # reset errkeys
  errkeys = []

  build_translation(newprops, properties_hi, errkeys, language, client)
  # puts "# errkeys has length: " + errkeys.length.to_s
end

# save file
hash_to_properties(properties_hi, "messages_hi.properties")

More about the program

This script translates 482 English messages into a different language. It takes about 28 minutes to run. 8 minutes of that are the sleep statement, of which more below. To run this, I signed up for an OpenAI key and a paid plan. The total cost was about $0.02.

I tested it with two languages, French and Hindi. I used French because we have a community provided French translation. Therefore, I was able to spot check messages against that. There was a lot of overlap and similarity. I also used Google Translate to check where they differed, and GPT seemed to be more in keeping with the English than the community translation.

I can definitely see places to improve this script. For one, I could augment it with a set of loops over different languages, letting me support five or ten more languages with one execution. I also had the messages file present in my current directory, but using ruby to retrieve them from GitHub or running this code in the cloned project would be easy.

The output occasionally needed to be reviewed and edited. Here’s an example:

[blank]=आवश्यक (āvaśyak)
[blocked]=अनुमति नहीं है (Anumati nahi hai)
[confirm]=पुष्टि करें (Pushṭi karen)

Now, I’m no expert on Hindi, but I believe I should remove the English/Latin letters above. One option would be to exclude certain keys or to refine the prompt I provided. Another would be to find someone who knows Hindi who could review it.

About that sleep call. I built it in because in my initial attempt, I saw error messages from the OpenAI API and was trying to slow down my requests so as not to trigger that. I didn’t dig too deep into the reason for the below exception; at first glance it appears to be a networking issue.


C:/Ruby31-x64/lib/ruby/3.1.0/net/protocol.rb:219:in `rbuf_fill': Net::ReadTimeout with #<TCPSocket:(closed)> (Net::ReadTimeout)
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/protocol.rb:193:in `readuntil'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/protocol.rb:203:in `readline'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http/response.rb:42:in `read_status_line'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http/response.rb:31:in `read_new'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:1609:in `block in transport_request'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:1600:in `catch'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:1600:in `transport_request'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:1573:in `request'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:1566:in `block in request'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:985:in `start'
        from C:/Ruby31-x64/lib/ruby/3.1.0/net/http.rb:1564:in `request'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/httparty-0.21.0/lib/httparty/request.rb:156:in `perform'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/httparty-0.21.0/lib/httparty.rb:612:in `perform_request'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/httparty-0.21.0/lib/httparty.rb:542:in `post'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/httparty-0.21.0/lib/httparty.rb:649:in `post'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/ruby-openai-3.7.0/lib/openai/client.rb:63:in `json_post'
        from C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/ruby-openai-3.7.0/lib/openai/client.rb:11:in `chat'
        from translate.rb:33:in `block in build_translation'
        from translate.rb:28:in `each'
        from translate.rb:28:in `build_translation'
        from translate.rb:60:in `

(Yes, I’m on Windows, don’t hate.)

Given this was a quick and dirty program, I added the sleep call, but then, later, added the while errkeys.length > 0 loop, which should help recover from any network issues. I’ll probably remove the sleep in the future.

I signed up for a paid account because I was receiving “quota exceeded” messages. To their credit, they have some great billing features. I was able to limit my monthly spend to $10, an amount I feel comfortable with.

As I mentioned above, translating every message into Hindi using GPT-3.5 cost about $0.02. Well worth it.

I used GPT-3.5 because GPT-4 was only in beta when I wrote this code. I didn’t spend too much time mulling that over, but it would be interesting to see if GPT4 is materially better at this task.

Worries

Translating these messages was a great exploration of the power of the OpenAI API, but I think it was also a great illustration of this tweet.

I had to determine what the problem was, and how to get the data into the model, and how to pull it out. As Reid Hoffman says in Impromptu, GPT was a great undergraduate assistant, but no professor.

Could I have dumped the entire properties file into ChatGPT and asked for a translation? I tried a couple of times and it timed out. When I shortened the number of messages, I was unable to figure out how to get it to ignore comments in the file.

One of my other worries is around licensing. I’m not alone. This is prototype code running on my personal laptop and the license for all the localization properties files is Apache2. But even with that, I’m not sure my company would integrate this process given the unknown legal ramifications of using OpenAI GPT models.

In conclusion

OpenAI APIs expose large language models and make them easy to integrate into your application. They are a super powerful tool, but I’m not sure where they fit into the legal landscape. Where have we heard that before?

Definitely worth exploring more.

Thoughts on managing a devtools forum

I’m one of the team members tasked with managing the FusionAuth community forum, where folks using FusionAuth who don’t have a paid support plan can find help and answers.

Here’s some advice for running such a forum. (I wrote previously about why you should use a forum rather than Slack/Discord/live chat.)
  • First, consider why are you going to run a forum? Lots of great reasons: ease a support burden, help with SEO, foster community, get product feedback. Get clear on what you are trying to build before committing, because it is a commitment.
  • Choose forum software carefully. Migration will be a pain. Common options include nodebb (what we use), discourse, and vanilla forums.
  • Seed the forum. This means gathering up questions as you see them pop up in other venues (support tickets, GitHub issues, customer calls). I did that religiously for a few months. I learned a lot about the product and the forum posts meant that folks were helped even when it was new. I’d recommend posting the question and then responding in-thread with an answer.
  • Forums will bubble up commonly asked questions. This can tell you where your docs should be improved.
  • You must groom the forum. It won’t be set and forget. You have to pay attention to it, answer questions, respond to responses. A forum full of unanswered questions is worse than no forum at all. Trust me, developers will notice (we’ve had customers mention that they appreciated how active our forum was).
  • Because we sell support, we don’t answer questions immediately or have engineering staff answer them. There are also questions that we can’t answer such as architecture recommendations. Immediate responses and answers requiring context and research are reserved for paying customers. This hurts my heart some times, but we are open about it. May not be applicable to in all cases.
  • Don’t be afraid to ban users. We ban anyone who spams, no questions asked. Delete the content and ban the user. We luckily haven’t had any abuse issues beyond spam.
  • Have a code of conduct. I grabbed GitHub’s (you can see ours here, and here’s GitHub’s)  but have something. We didn’t in the early days, but it’s a good thing to have out of the gate.
  • Don’t expect a lot of community to grow out of it. At least, I haven’t had that experience, most people just want their questions answered. May be because I’m extremely part time on it and haven’t fostered it, though. Slack/discord is much more likely to build community in my experience. But know what your users want: Google or Facebook?
  • At a certain point, I had to enable a post queue, where a team member approves every new user. We were getting a lot of spam accounts and then they’d post gambling ads and then direct a ton of traffic (1000s of pageviews) to the ads. I don’t know what the spammer endgame was, but approving each new post has solved the issue. I’d definitely look for that feature.
In general I love forums, and so do devs, but they do take some work.

The meetup opportunity

I’ve been presenting at a lot of meetups recently. With the move to virtual, if you want to spread the word about your project or company, you can speak to between five and 50 interested developers pretty easily. As a co-organizer of the Boulder Ruby Meetup, I can also tell you that I am always on the hunt for interesting speakers. Zoom makes it possible for speakers from around the world to talk to meetup attendees.

Why might you be interested in doing this? Well, if your product is aimed at developers, speaking at meetups is a great way to display that you and your company “get it” as well as get some brand awareness. It’s definitely retail rather than wholesale, but you’re still getting your name out there.

Organizers look for topics related to their meetup, from speakers who are responsive and know what they’re doing. As an organizer I’m looking for diversity and local representation, but am also happy to have folks from around the world present. We’re pretty open about the talk topic. Some of the talks are about ruby and rails, others are about generic technologies (machine learning, networking) and others are about the tech industry (ethics, interviewing).

Create a presentation

You want this to be on a topic that is technical, and related to your product, but not about your product. For FusionAuth, I present on JSON Web Tokens. If I was working for the Duckbill Group, I’d focus on the top 3 most expensive part of your AWS bill. For TimescaleDB, I’d talk about generic performance optimizations for PostgreSQL.

Do not hit the audience over their head with your product. Instead, display your awesomeness by teaching them something you’ve learned in a field adjacent to your product. Don’t worry, you’ll have your product name on every slide, and you can put in a slide or two about your company, but the focus should be on teaching something, not a pitch.

This is going to be time intensive, and the most effort, so see if you can spin off other useful artifacts. Can you make a screen cast on a similar topic? A blog post? Submit an article to Hackernoon?

You can also pitch a presentation to a meetup without fully writing it, but I’d suggest at least outlining it before reaching out to any organizers.

If possible, I like my presentation to be technology agnostic. Topics like authentication, databases, logging, etc all cut across different languages. Your presentation will be applicable to more meetups if the bulk of the explanatory slides be language agnostic. Feel free to tweak it by including a couple of slides or code examples in a particular language.

Find meetups

The meetup search tool is crap (most of the meetup UI needs some help, to be honest). I ended up creating a spreadsheet with the names of big cities in the USA in the left hand column and the names of various technologies (Ruby, JavaScript, etc) across the top. Then I built a URL searching for <city name>+<technology name>+meetup. Running this query on google was invariably better than trying to use meetup’s search tool.

Once you find a meetup, look at past events to see what kind of topics they cover, if they have online events, how big it was, and how active the meetup is. If it makes sense, send a message to the organizers. This is a good task to batch up and do weekly.

Reach out to organizers

Reach out to organizers via a meetup message. Some organizers aren’t doing meetups right now. You can scout their meetup page and see if they have online meetups. Sometimes I’ll ping an organizer even if they don’t because sometimes organizers will be interested in restarting a meetup or aren’t aware of the possibilities of remote speakers.

When you reach out, explain that you have a talk about such and such a subject and that you’re interested in doing a zoom meetup. If you’ve done it before, mention it, as that may make them more comfortable with you. Also, mention that you are interested in presenting to their meetup: “I’d like to present to the Boulder Ruby Group about JWTs”.

This serves two purposes. It shows that you aren’t a spammer because you took the time to customize the message at least a bit. And, when they reply to you (which can take days or weeks), you’ll know which meetup they run. There’s no way to go from a message to a meetup organizer to the home page of the meetup they help with. What? Yes, as mentioned, meetup has some UI flaws.

Follow up in a few weeks if they don’t respond. Messages get lost or forgotten. I also often ask if it is ok to take the conversation to email, because that’s less likely to be ignored or missed than meetup messages.

If they are interested, nail down the exact date and time (including timezone). Provide them with your bio, talk title, talk description and duration. Ask if they need anything else (pronouns, headshot, social media, the presentation for their review). I can tell you as a meetup organizer that it is quite nice to have these provided.

What you need from them is:

  • how you will connect to the meetup (zoom link, streamyard account, something else)
  • the length of the presentation
  • whether you can give anything away (electronic stuff is good, t-shirts cost a lot of money to ship but are good too)

I had one organizer want to review my talk, and a few others were happy to look at other presentations I’ve given, but other than that, the bar is pretty low. I also have a pretty extensive online presence; if you have less of one, you might need more proof that you will be a good speaker.

Remember, meetup organizers are volunteers and are looking to provide fun, useful content to their peers. Make it as easy for them as possible.

Conclusion

You won’t reach thousands of people when you do the Zoom meetup circuit, but you will get some great feedback on your presentation and help educate folks.

The largest audience I presented to was around 50 and the smallest was around five. I also got some great questions that led to additional slides and helped round out my presentation, which I was also able to submit to more traditional conferences.

Use Cameo for dev focused marketing

Recently, we used a Cameo for a developer focused announcement. If you are not familiar with this service, it lets you request a short video from an actor. You send the actor your idea, pay them, they send you the video, and you can use it for a limited number of purposes. If you, or someone you know, has a favorite actor, it can make for a real fun birthday message. But it also is fun for marketing messages and can help you stand out from the crowd.

My experience below is based on one business Cameo. We plan to do more, so there may be updates.

Why consider a Cameo

It is still relatively unique. I’ve seen a few celebrity endorsements of technical products via Cameo, but not that many. This means that it stands out in a fun way. Using Cameo also gets you easy access to a famous or semi-famous person. All you have to do is submit a form and pay some dollars. Compare this to any kind of commercial, which may involve a casting director, ad agency and other parties.

It also is relatively cheap. I looked at a few actors and none cost more than $2000 for commercial usage (more about that below). While this isn’t cheap, I also saw actors for a couple hundred bucks. We ended up choosing an actor who worked for $500.

Note that a Cameo is a pure brand marketing play. It is fun for shock or surprise value, rather than a CTA. It’s unlikely you’ll get deep technical analysis as well. This playful nature fit with our brand, but make sure it fits with yours.

How it works

You can check out the Cameo site FAQs, but here’s how the process worked for me.

  • Browse actors and come up with a shortlist.
  • Filter out actors who won’t do commercial messages. (Some actors won’t, so check before you get excited.)
  • Decide on a topic to be covered.
  • Review licensing terms for commercial use.
  • Sign up for an account.
  • Put in a credit card.
  • Submit a request on the website. This was limited to 250 characters. (Not 250 words. 250 characters. So the guidance was general.)
  • Install the application to get messaging. (The actor enabled free messaging so he could ask questions.)
  • Go back and forth with the actor and answer clarifying questions, maybe 2 rounds of qs. This had to be done on the Cameo app (boo!).
  • Accept the delivered video.
  • Promote and share it.

Note what wasn’t in there:

  • Any writing talent. I did talk to a number of writers and even selected one. However, after reviewing the constraints, we but mutually decided it didn’t make sense. There just isn’t a lot of room for a complex story line or even a funny line or two. That’s probably why Cameo has the limit.
  • A specific story line. I was able to convey one message to the actor, but otherwise it was in his hands.
  • A lot of back and forth or workshopping. I think I talked about this internally for maybe 15 or 30 minutes and definitely had a good idea of what we wanted to cover. But other than some questions, it wasn’t super collaborative. And, to be honest, that was fine. I believe any actor on Cameo is funnier and knows more about speaking to the camera than I do.

I do wonder whether all the actors would have the same devotion to detail. As mentioned above, the actor enabled free messaging and really dug into the topic. Everyone who watched the video was delighted.

After it is delivered

After it is delivered, it’s time to promote. At the time we bought the Cameo, you could put it on one social media platform or your website for 30 days. We chose Twitter. Then I realized that the actor had recorded 5+ minutes. You aren’t allowed to edit the videos, and the maximum length of a Twitter video is 2:20. So we posted it on an unlisted Youtube link and shared that. Check out the current terms (search for Business CAMEO Videos).

I submitted it to a few online communities, shared in social networks and basically did any other kind of promotion you’d do with an interesting video. It was shared to several email lists and slacks as well. We also bought some traffic.

It didn’t go viral, but it got ~10x the usual number of retweets and interactions as our normal tweets do. It’s unclear if any business came from it.

What I wish we had done differently

  • Understood my limits earlier. I spent a lot of time talking to writers before I realized that 250 characters meant sending over an idea and trusting in the actor. Would have been less stressful to have known that earlier.
  • Be a bit more familiar with the actor. One of the best parts of the Cameo was made in response to an offhand request from a co-worker who was more familiar with their work than I was. I should have done a bit more research.
  • While I focused on the topic and asked the actor to do it in character, I should have included the following in my pitch:
    • How to pronounce the brand.
    • Whether or not I should be mentioned (I was, but that was unnecessary).
    • The optimal length of time (2:20).

At the end of the day, this is a fun alternative (or complement) to the normal boring press release. If you have a character which is in line with your brand or product usage, do check it out.

Joining FusionAuth

I wanted to let y’all know that I’ve joined FusionAuth as a developer advocate. I’ll be working to help our customers succeed and promote the virtues of standards based user management systems. I get to write a lot of content and example applications against a full featured API.

I’ve built enough systems to know two things:

  • Users and their behavior are almost always a key part of any software application.
  • User management is difficult to get right, especially if you want to use secure best practices and standards such as OAuth.

FusionAuth wants to elevate everyone’s user identity management system. The community edition is free and will always be. (It’s important to note that it is free as in beer, not free as in speech, but almost all of the development happens in the open.) If you want to run FusionAuth on your own forever, that’s great! You get a secure user store that supports OAuth, SAML and two factor authentication, free forever. We’ll happily provide you “best effort” support in our forums and we’ve seen the community help each other out too (most notably in the creation of helm charts to run FusionAuth on Kubernetes).

If, on the other hand, you find value in FusionAuth and want guaranteed support, custom development, or hosting, we’re happy to sell that to you. The price is often a fraction of the other solutions out there. Another differentiator for FusionAuth is that you can host it wherever you want: in your data center, in your cloud, or on our cloud servers. Not every client needs that level of control, but many do.

I really love the business model of providing a ton of value to your end users and monetizing only a small percentage of them with unique needs. (I’ve been involved in this type of business before.) The business thrives and there’s a ton of consumer surplus generated.

I’m really excited about this opportunity. It’s a nimble company with a passionate team based in Denver. If you need a user identity management system built from the ground up for developer happiness, please check us out.

Creating a CircleCI orb to authenticate a user during a build

I’m a big fan of automating all the things and I also believe in the DRY principle. I’ve been using CircleCI for years and noticed that they had added a way to abstract away some repeated configuration called orbs. I recently built one and wanted to share my experience.

If you are thinking about building an orb, first take a look at the list of existing orbs. After all, it’s better to reuse code someone else will maintain if it does what you need.

In my case, I wanted to explore building an orb. I dreamed up a use case for which no one else had written an orb. The situation is that you store user data in FusionAuth. Each time a build runs, you want to verify that the user running the build is active andvalid before continuing the build. If the user can’t authenticate, fail the build.

Set up the authentication server

I set up a FusionAuth server using their instructions on EC2. It can’t be on localhost because CircleCI needs to communicate with it during the build. The server didn’t run well on a t2.micro size so I ended up springing for a t2.large, where it worked fine. I also had some difficulty installing mysql on the Amazon Linux AMI, but this SO answer helped me out. I added a FusionAuth application and user via the admin panel. I also got an API key. I limited the API key drastically; it could only post to the login endpoint. I tested access with curl like so (ALL_CAPS strings are placeholders you’d want to replace with real values):

curl -s -o /dev/null -w "%{http_code}" -XPOST -H 'Authorization: API_KEY' \
-H 'Content-Type: application/json' \
-d '{ "loginId": "USERNAME", "password": "PASSWORD", "applicationId": "APPLICATION_ID" }' \
http://FUSION_AUTH_SERVER_HOSTNAME:9011/api/login|grep 202 > /dev/null

FusionAuth returns a 404 status code if the user isn’t authenticated successfully (they don’t exist or have an incorrect password) and 202 if the user logs in successfully (more API information here). Note that this login process may skew some of the reporting (around DAUs, for example) so you’ll want to create a separate application just for CI/CD.

The curl + grep statement above exits with a value of 0 if the user successfully authenticates and with a value of 1 if authentication fails. The former will allow the build to continue and the latter will stop it.

The check I want to run worked. Now I just needed to build and publish the orb.

 

White orb

Building the orb

An orb is reusable CircleCI configuration. You have three kinds of configuration you can re-use:

  • jobs: a normal CircleCI job, but you can also pass in parameters. A great option if you have a job you want to use across different projects; something like ‘run this formatting tool’.
  • executors: an environment in which to execute code (docker container, vm, etc).
  • commands: a set of steps that can be reused. It is lower level than a job and can be used across different jobs.

I chose to create a single command. I also created a job to run while I was developing (I added it as a project in CircleCI). You can see the full source code here.

Here are the interesting bits where I define the command to be shared (ah, yaml):

commands:
  verifyauth:
    parameters:
      username:
        type: string
        default: "user"
        description: "FusionAuth username to try to validate"
      applicationid:
        type: string
        default: "appid"
        description: "FusionAuth application id"
      hostbaseurl:
        type: string
        default: "http://ec2-52-35-2-20.us-west-2.compute.amazonaws.com:9011/"
        description: "FusionAuth host base url"
      password_env_var_name:
        type: env_var_name
        default: BUILDER_PASS
        description: "The user's FusionAuth password is stored in this environment variable"
      fusionauth_api_key_env_var_name:
        type: env_var_name
        default: FUSION_AUTH_API
        description: "The FusionAuth API key is stored in this environment variable"
    steps:
      - run: |
         curl -s -o /dev/null -w "%{http_code}"   -XPOST -H "Authorization: ${<< parameters.fusionauth_api_key_env_var_name >>}"  -H 'Content-Type: application/json' -d '{ "loginId": "<< parameters.username >>", "password": "'${<< parameters.password_env_var_name >>}'", "applicationId": "<< parameters.applicationid >>" }' << parameters.hostbaseurl >>api/login |grep 202 > /dev/null
      - run: |
         echo User authorized

The command verifyauth takes parameters. These have defaults and descriptions. Anything that you wouldn’t mind seeing checked into a source code repository can be passed as a parameter. You then call the command in your job and pass parameters as needed (we’ll see that below).

However, there are sometimes secrets which need to be stored as environment variables (in the project or the context): API keys or passwords, for example. However, I still wanted to make them configurable by whoever uses the orb. Enter the  env_var_name parameter type. This type lets the user specify the name of the environment variable. If I set the password_env_var_name to AUTH_CHECK_PASS, then I need to make sure there is an AUTH_CHECK_PASS environment variable set somewhere in my project containing the password with which we’ll authenticate against FusionAuth. This lets the orb be both configurable and secure.

Finally, you can see that the first step of the command is posting login data to the authentication server. Again, if we see anything other than 202 we fail and the build stops. (You’ve seen that curl command before.)

Publishing the development orb

To be able to use the orb with a different project, I needed to publish the orb (I could have developed the orb inline to avoid this). The publishing instructions are here. The only issue I ran into was that I had to update my CircleCI organization settings and allow “Uncertified Orbs” before I could create a namespace. After that I was able to publish a development version of my orb:

circleci orb publish .circleci/config.yml mooreds/verifyauth@dev:testing

I was in the directory of my orb code and referenced my config. mooreds is my orb namespace, verifyauth is the orb name (which is arbitrary and not connected to the source repository name in any way) and dev:testing is the version of the orb. Note that there are two types of orb version: production, which strictly follow semantic versioning, and development, prefaced by dev: and after string that can contain “up to 1023 non whitespace characters”. Development orbs have other limitations: they are not public, are mutable and only last for 90 days. You’ll want to publish your orbs with production versions if you are using them for any purpose other than prototyping or exploration.

I published my orb via the command line, but the docs outline publishing via a CircleCI job.

Testing the orb

Now I wanted a second project to test the orb. Here’s the project source code. Here’s the interesting code:

...
orbs:
  verifyauth: mooreds/verifyauth@dev:testing
jobs:
  build:
    steps:
      - verifyauth/verifyauth: # when called from external job, command is namespaced to by the name of the orb
          username: "circlecimooreds"
          applicationid: "98113cee-d1a8-4abf-baf5-a6ea742f80a1"
  ...

You can see that I pull in the orb at the development version, which I’d previously published. Then I call the namespaced command with some parameters. For this command to work, I also needed to set up required environment variables (in this case, BUILDER_PASS and FUSION_AUTH_API because I didn’t pass in any of the env_var_name parameters). If you don’t set those environment variables (or, alternatively, set the parameters to different values, and then set those environment variables), the build will fail no matter what, as the API call won’t succeed.

I then pushed this sample project up to CircleCI and ran a few builds to make sure the parameters were being picked up.

Publishing the production orb

Now that I had an orb that is parameterized and exposed the command we want to share, I needed to publish it for everyone to use. Note that your configuration code is entirely exposed if you publish an orb. You can see the source of any orb via the circleci orb source command. circleci orb source mooreds/verifyauth@0.0.2 will show you the entire source of my sample orb. They warn you a number of times about this.

To promote to production an orb that you have published to development, update the dev version: circleci orb publish .circleci/config.yml mooreds/verifyauth@dev:testing to catch any changes and then promote it: circleci orb publish promote mooreds/verifyauth@dev:testing patch.

Note that the patch argument at the end of the promote command bumps the patch number (0.0.1 -> 0.0.2) but you can also bump the minor and major numbers. Any changes you make to a production orb require you to publish and promote it again; production orb versions are immutable. For instance, I wanted to update the description of some parameters, but had to publish an entirely new version.

After publishing, you’d want to update any projects that use the orb to use the production version.

Publiished orb listing

The listing of my published orb

Areas for further work

This was a slightly contrived example. I wanted to gain some experience both with FusionAuth and with CircleCI (I have friends who work at both companies). There are a number of areas where this could be improved:

  • authenticate against a different authentication server (LDAP, Okta, AWS IAM)
  • store additional information about the user in the authentication database (for instance, which projects they can build) and convert the authentication curl command into an authorization command
  • run the identity server over SSL (I just used HTTP because it was easier to get up and running, but that’s obviously a production no-no)
  • pull the user and password from the build environment. It’s pretty clear to me how you’d pull the user (there’s a CIRCLE_USERNAME environment variable) but I’m not sure how to pass the password. I can think of a couple of solutions:
    • don’t login at all, just allow the API key to pull user data and match on the username (this is probably the best option)
    • pass the password via a pipeline parameter, which means you’d have to set up an API call to build
    • have one common password for all users in the FusionAuth system, and use it only for access control to the build pipeline
    • make the password the same as the username in the FusionAuth system, and use it only for access control to the build pipeline

In conclusion

If you want to interact with external services from within CircleCI, check out the list of existing orbs.

If you have a service that you want to make it easier for CircleCI users to interact with and use, create an orb and publish it.

If you are working with CircleCI and have duplicate configuration that you want to share between projects, setting up your own orbs is a great idea. Orbs are flexible and easy to parameterize. If you’re OK with your configuration being public (it wasn’t clear to me if there was any way to have the configuration kept private), you can encapsulate your build and deploy best practices in an easy to consume manner.