AWS Questions: SQS

More questions, this time about SQS, the simple queue service that AWS provides.

  • What was the first AWS service?
  • Are there upper limits on limits on SQS in terms of message/second?
    • FIFO Queues have a limit (300/s), but I wasn’t able to file any hard limits for standard SQS.  In the developer guide they have some examples that reach 2500 messages/second.  I found some benchmarks from 2014, which were able to get to 108k messages/second.
  • Can you create alarms based on the number of messages in a queue?
    • Yes, that is a metric that Cloudwatch tracks: “NumberOfMessagesSent”.  You can use this in combination with an auto scaling group to handle batch processes in a dynamic manner (scale out when you have more work in the queue, scale in when you have less).
  • What is the maximum visibility timeout for SQS?

Interview with a boot camp grad

Lots of folks are moving into development and technical fields these days.  I remember that happening during the dot com book, but back then folks just read one of those “Learn Java in 24 Hours” books.

Nowadays there are a profusion of boot camps that help people gain the skills they need to be a developer. I have interacted with a few of these grads and was interested in learning more about their experience. Noel Worden, one of the organizers of Boulder.rb and a blogger, agreed to an interview.

– what was your background before you were a developer?

I got my degree in fine art photography, spent 5 years working in NYC as a Digital Technician in the photo industry, then moved to Colorado and was a cabinetmaker for 3 years.

– how did you land your first job?

I found the posting on the Denver Full Stack meetup whiteboard (

– what surprised you about the software industry?

How willing everyone is to help. It’s very different in the photo world, [which is] much more competitive, it’s hard to find guidance and mentorship.

– how did you pick your boot camp?

Bloc had one of the longest curriculums I could find in an online program. I figured when push came to shove, having more experience under my belt couldn’t hurt when competing with other junior [developers] looking for that first job. Bloc also has a part-time track, which allowed me to still work [a] full-time job while going to school.

– what is good about the job? What is challenging?

I appreciate the balance of being challenged but knowing I have the full support of any other engineer on the team if I need to reach out for assistance. Canvas United [ed: his current employer] has a lot of projects that I’ve been maintaining lately, and all are running different versions of Rails, which makes for interesting challenges. 

– what do you see current boot camp students doing that you’d advise against?

Not getting out and networking while going to school. This industry is all about networking and if you’re hoping to capitalize on the advantages of a good network you have to be building it while still in school.

– why did you want to transition into technology and development?

I wanted/needed a career path where the leaning curve wouldn’t plateau. I’m not a ‘cruise control’ kind of person, as soon as the challenge isn’t there for me anymore I lose interest.

– how can employers help boot camp grads in their first job?

Be ok with answering questions, and be transparent with the employee about the proper channels to ask those questions. Also, be upfront about the fact that it’s ok to fail (assuming that’s the case) a few time before you get to the best solution [ed: if it isn’t ok to fail, find a new job!]. Also also, a healthy balance of low hanging fruit and multi-day problems. Nothing kills my morale faster than ticket after ticket of problems that grind me into the ground.

– should employers have any different expectations of a boot camp grad vs someone who just graduated from college or high school?

I definitely think that it takes a particular type of management style to successfully level up a boot camp grad. If you’ve hired them you must have liked something about them, play to those strengths, but also sprinkle in challenges to help that developer evolve.

[This content has been edited for grammar and clarity.]

AWS Questions: Windows Servers

Windows servers are supported on AWS, but recently I had students ask a bevy of questions about them.  Here are some answers.  As a reminder, I speak solely for myself with these blog posts, not for AWS or any employer.

  • What versions of Windows are supported?
  • Can I create an AMI from an EBS snapshot of a Windows root volume?
    • Unlike with a linux EBS snapshot, you cannot create an AMI from a root volume.  You can create an AMI from a running instance, however.  The reason for the limitation is that sysprep must be run on the Windows server, and you can’t run sysprep on a EBS volume that is not running.
  • In order to take an accurate snapshot, I need to quiesce the disk.  How can I do so?
    • This is a thorny problem and I don’t think there’s a great answer. You want to shut down as many apps as you can. You also may find the Volume Shadow Copy Service useful. You may want to review the answers here on this reddit thread.
  • I have a Windows bastion host, and I want to allow more than two users to access this host at one time.  How can I do this?
    • You need to purchase additional Remote Desktop Services licenses.  From the FAQ: “Amazon EC2 instances come with two Remote Desktop Services (aka Terminal Services) licenses for administration purposes. If additional Remote Desktop Services licenses are needed, they should be purchased from Microsoft or a Microsoft license reseller. Remote Desktop Services licenses purchased with Software Assurance have license mobility benefits and can be brought to AWS multi-tenant environments.”
  • Is powershell a first class citizen with the same functionality as the CLI or the supported SDKs?
    • Nope.  From the Powershell page: “The AWS Tools for Windows PowerShell lets you perform many of the same actions available in the AWS SDK for .NET. You can use it from the command line for quick tasks, like controlling your Amazon EC2 instances.”  (Emphasis added.)
  • Do you have any example userdata scripts for Windows AMIs?

Amazon Machine Learning: An Introduction

From my book, Amazon Machine Learning: An Introduction:

Amazon Machine Learning, or AML, provides you access to widely applicable machine learning algorithms without having to run any servers.  This type of learning is useful for making predictions based on a set of data for which answers are known.  AML supports supervised learning with the stochastic gradient descent algorithm.  The end goal of AML is to create a model, which is what will allow you to make further predictions based on past data.

AML supports three different kinds of predictions.  For binary outcomes, where observations lead to a yes/no result, AML supports binary classification.  An example would be whether or not a prospect is likely to sign up for a new account, given their past interactions with your company.  For multi valued results, where observations lead to one of N results, AML supports multi class classification.  A good example of this would be which product to show a customer, given what they’ve looked at and bought in the past.  And, for numeric values, AML supports regression.  An example of that would be predicting house prices based on sales data and house attributes.

If you are not trying to use existing data and create predictions out of it using supervised learning, but are trying to instead recognize images or tease out patterns in text, you may want to consider alternatives to AML.

Boulder Startup Week

If you are into the tech scene in Boulder, Boulder Startup Week is a great set of events–it’s coming up May 15-19 this year.  This is a totally volunteer run set of events which highlight various aspects of startup and technology in the Boulder area.  You can learn more at the website.  It’s a great place to network and to learn about new things.

I’m lucky enough to be participating in two events this startup week.  I’ll be hanging out at the Engineering Leadership dinner.  And I’ll be presenting on bootstrapping a startup as a developer with a few other bootstrappers.  Most of my short presentation will cover lessons I’ve learned from joining The Food Corridor.  I’m especially looking forward to hearing about Brian and Inversoft that day, because I’ve been friends with him for a number of years and have followed along with some of his trials and triumphs.

Hope to see you there!

AWS Questions: EBS

So, more questions about AWS from students (and my own research/curiosity):

  • What happens when general purpose ebs volumes run out of credit because you’ve used too many IOPS?
    • Your disk performance reverts to the baseline performance: “If your gp2 volume uses all of its I/O credit balance, the maximum IOPS performance of the volume will remain at the baseline IOPS performance level (the rate at which your volume earns credits) and the volume’s maximum throughput is reduced to the baseline IOPS multiplied by the maximum I/O size.”
  • Before you take a snapshot you should quiesce the disk.  Can you do that via an AWS command?
    • You need to use whatever operating system or application command is recommended.  You can use EC2Run to execute that command, but you must determine what that command is.  From the backup and recovery whitepaper: “For this reason, you must quiesce the file system or database in order to make a clean backup. The way in which you do this depends on your database or file system. ”  Note also that the quiescing of the disk can be for only a few seconds, as the EBS snapshot process can be started quickly, though it may take a while to complete.
  • Can you be notified of snapshot completion via an event or do you have to poll?
  • Can you automate your EBS snapshots?
  • Does EBS encryption cost extra?
    • Nope: “This frequently requested feature provides you with seamless support for data encryption on block-level storage, at no additional cost.”

Getting the good content out of a Facebook group

I am astonished at how hard it is to get information out of Facebook groups.

The startup of which I am a part has created a Facebook group for dissemination of information of between commercial kitchen operators.  This was easier to get started than a forum and has the advantage of everyone being a “real person”, or at least real enough to get a Facebook account.  It also benefits from the ubiquity of Facebook–many many folks have it on their phones and get notifications about group activity.

However, it has the detriment of being a “walled garden”, with the content of the group being unavailable for searchers on the web.  Some might argue that privacy actually is a good thing, because it will encourage folks to be more honest, but really, anything you put on Facebook can be cut and pasted and made public, so I’m not sure I buy that argument.

Regardless, I wanted to find an easy, automated way to take the Facebook group content and pipe it elsewhere, where it could be reified and curated.  A human could do that, but I’d like an automated solution. And, other than the Facebook API, I haven’t found many.  Zapier (my go to integration choice) only recently released this as an option (I’m thr last few months.  IFTTT doesn’t have it.  There’s no commercial solution that I could find that does this.  There are, however, some open source solutions.

The Facebook API makes it fairly easy to grab the posts of a group, and from the posts, the comments, but frankly, I really want a solution that doesn’t require coding up the JSON parsing/pagination handling/Oauth access.  I just tried the facebook-export tool and it seems to work just fine (though I had ended up having to update the leveldown/levelup versions to 1.5/1.3 to get past a compile error: recipe for target 'Release/' failed). It gives you all your posts as JSON.

Amazon Machine Learning Video and Book

I’m working on a video series and an ebook about Amazon Machine Learning, or AML.

AML  is a great way to get started with machine learning, since you can focus on the key concepts of building and using a model and not worry about any infrastructure.  AWS takes care of provisioning all the underlying IT infrastructure–you just worry about getting your data to S3, choosing how to build the model, and then using the model.  Which, trust me, is quite enough to tackle if you are a machine learning newbie.

You can use the model to get predictions either in real time (with a default soft limit of 200 requests per second) or via batch processing, where you can upload up to 1TB of predictions to S3.  Like everything in AWS, you can control the entire process via a well documented API or from various SDKs.

AML isn’t a fit for all machine learning needs–it processes text that is in CSV format and supports only supervised learning.  There are other options on AWS (and other places as well).

The book is currently in progress, and I’ll be starting on the video soon.If you’d like to follow along as the book gets written, you can at leanpub: Amazon Machine Learning: An Introduction.

AWS Questions: Cloudformation

So, more questions from students.  This time about Cloudformation, the very cool way to built AWS infrastructure declaratively.  I would hate to have to pick a favorite AWS service, but if I had to, Cloudformation would be it.

  • By default a stack rolls back on failure.  You can also keep any successful stack elements by setting disable rollback to true.  Can you have some elements of a stack that must have successful creation, and others that may fail without rollback?
    • Nope.  I’d break this up into two stacks and chain them.
  • Why is YAML now supported for Cloudformation templates?
  • Can you a dry run of a cloudformation template?

The Four Types of Slacks

I have been using slack for a few years now, but have really noticed an uptick in the last year or so.  (If you aren’t familiar with slack, here’s an intro to slack usage, and if you are, here’s a great code of conduct for public slacks.)
It seems to me that there are four main types of slack groups.

The first is the company/department slack.  This slack is long lived, contains many channels, and is multi purpose.  There are channels for ops, marketing, etc.  This slack is typically limited to the employees of a company, though contractors are also given access.  The main purposes of this slack are an ad-hoc knowledge base and to reduce email.  Depending on IT, this slack may be under the radar and compete with other solutions like hipchat, wikis or internal mailing lists.

The next type is a project slack. This is related to the company slack, but is less long lived, and has fewer members and channels.  It is used for coordination amongst disparate people, often a set of contractors.  May be maintained by the client or prime contractor, also serves as an ad-hoc knowledge base, but is primarily a means for coordination of effort.

Both of the above slacks may have other integrations with systems (CI/CD, monitoring, etc).  These integrations with external systems can make the slack a one stop shop for corporate knowledge and memory, especially if the members are on paid accounts.

The above types are obviously limited in membership.  The next two types of slacks are more public.

Another type of slack is the event slack.  This slack replaces or augments Twitter as a way for people at a conference to communicate.  May exist between events, but is quiescent while events are not happening.  Here channels may be related to aspects of the event or tracks, and the slack is typically owned by the event coordinator and provided as a service to the conference attendees.

Slack can also be an email list replacement.  I have been a member of several email lists for user groups/meetups in the front range, and it serms much of the activity on some of them have been driven to slack (the BDNT meetup is a good example). In addition, I see a lot of new slacks being created that would, a decade ago been email groups.  (Facebook groups are also a replacement for email groups, depending on your audience, but I have found slack to be far superior in searchability.). The number of channels is typically related to member list size and length of existence.  I have found these slacks be on the free slack plan, with its limits. I have also heard of slacks of this type charging for membership.

What has been your slack experience?  What did I miss?

© Moore Consulting, 2003-2017 +