So, more questions about AWS from students (and my own research/curiosity):

• What happens when general purpose ebs volumes run out of credit because you’ve used too many IOPS?
  • Your disk performance reverts to the baseline performance: “If your gp2 volume uses all of its I/O credit balance, the maximum IOPS performance of the volume will remain at the baseline IOPS performance level (the rate at which your volume earns credits) and the volume’s maximum throughput is reduced to the baseline IOPS multiplied by the maximum I/O size.”
• Before you take a snapshot you should quiesce the disk.  Can you do that via an AWS command?
  • You need to use whatever operating system or application command is recommended.  You can use EC2Run to execute that command, but you must determine what that command is.  From the backup and recovery whitepaper: “For this reason, you must quiesce the file system or database in order to make a clean backup. The way in which you do this depends on your database or file system. ”  Note also that the quiescing of the disk can be for only a few seconds, as the EBS snapshot process can be started quickly, though it may take a while to complete.
• Can you be notified of snapshot completion via an event or do you have to poll?
  • You can use cloudwatch events to be notified.
• Can you automate your EBS snapshots?
  • Yes
• Does EBS encryption cost extra?
  • Nope: “This frequently requested feature provides you with seamless support for data encryption on block-level storage, at no additional cost.”

So, more questions from students.  This time about Cloudformation, the very cool way to built AWS infrastructure declaratively.  I would hate to have to pick a favorite AWS service, but if I had to, Cloudformation would be it.

• By default a stack rolls back on failure.  You can also keep any successful stack elements by setting disable rollback to true.  Can you have some elements of a stack that must have successful creation, and others that may fail without rollback?
  • Nope.  I’d break this up into two stacks and chain them.
• Why is YAML now supported for Cloudformation templates?
  • YAML is more readable for humans than JSON.
• Can you a dry run of a cloudformation template?
  • There is limited support (for change sets).  This has been a feature request for a long time (since 2013).

Here are some questions and answers about DynamoDB, Amazon’s managed NoSQL database offering.

• What are options for dynamically scaling DynamoDB provisioned throughput?
  • Hard to beat the options outlined in this StackOverflow post.  You can do it via scripting, the DynamicDynamoDB open source library, a lambda function, cloudwatch–lots of different ways.
• Do DynamoDB streams support multiple readers?
  • Yes, but you want to be careful to not have more than two readers per shard.
• How does optimistic concurrency control work?
  • Nicely outlined here but the long and the short of it is you need to make sure you associate a version with your items, read that version when you prepare to update, and then update if and only if the version is the same as the one you read.
• Do you have any insight into the internals of DynamoDB?
  • Here’s a PDF from 2007, here’s documentation about partitions, and here’s a Hacker News comment from someone who claims to be a former DynamoDB developer
• How do you connect to DynamoDB?  Is there an IP address?
  • You use the SDK or CLI which connect to an endpoint in a region that you know no further details of.
• What is the difference between eventual and strong consistency with respect to DynamoDB reads?
  • The docs talk about it and so does this excellent SO answer (which covers the general concept as opposed to DynamoDB’s specifics).
• Does DynamoDB have any automatic encrypt at rest options?
  • I couldn’t find any server side encryption option for DynamoDB, but did find this client side encryption post.

   

More question answered from an AWS course.

• Does the AWS ELB have the ability to throttle requests, to stop invalid/illegal traffic – if someone refreshes a page 10 times in 5 seconds and I want to block the unnecessary requests from the refreshes?
  • no throttling is available with the classic ELB.  However, check out the Application Load Balancer, which has a WAF integrated with it. 
• What is the availability of the ELB component?
  • I couldn’t find firm numbers, but here’s an interesting article about ELB best practices.
• In a DDOS attack, since there is a lot of traffic to your environment, do you get charged for the additional traffic?
  • Depending on the attack type, not if you are fronted by an ELB or set up your security groups/NACLs to discard the traffic.  From the DDOS whitepaper: “When [an ELB detects certain types of attacks], it will automatically scale to absorb the additional traffic but you will not incur any additional charges.”
• When an instance is decommissioned from an ASG, does the ELB know not to send new sessions to that ASG because the instance is getting ready to shut down?
  • Yes: “If connection draining is enabled for your load balancer, Auto Scaling waits for in-flight requests to complete.”

Cloudwatch is Amazon’s monitoring and alerting service.

Some questions and answers re: this awesome service.

• How can you create custom metrics?
  • Cloudwatch doesn’t limit you to the metrics it collects by default (at the hypervisor level).  You can push any metric that makes sense up the the statistics repository, using custom metrics.
• What protocol does cloudwatch use?  ICMP or SNMP?
  • I couldn’t find a definitive answer.  For custom metrics, use the api as you would for any other AWS service.  Cloudwatch also gathers metrics from the hypervisor and the host machine.
• What are cloudwatch logs?
  • A way for you to push logfiles from ec2 instances up to Cloudwatch, parse them, and create metrics out of them (“how many 404 errors has this application had in the last 30 minutes?”).  More here.
• Does cloudwatch allow you to setup different metric thresholds at different times? For example, set an alarm at 70% CPU on Wed night but 90% on Sat night?
  • No, but you could do this with custom metrics.  You could read the cloudwatch default metrics and have an ‘cpualarm’ metric which would be 1 or 0 depending on if certain parameters were set.  Then you could vary the parameters over time.  Then you could set an alarm on the ‘cpualarm’ metric.

Amazon VPC lets you create a virtual network in the cloud that you control–subnets, ip ranges, internet access, routing, etc.  At recent classes, I was asked some questions about VPC that I dug into to find answers.

• Does AWS VPC support multicast or broadcast?
  • No, per the FAQs.  But there are some projects to overlay multicast functionality on top of the unicast network within a VPC.
• Are vpc flow logs add on pricing?
  • There is no additional charge, but they go into Cloudwatch Logs and you are charged at the normal rate for that usage.
• Is the NAT Gateway (used to provide internet access to ipv4 private subnets) highly available?
  • It is redundant within an availability zone.  But, from the docs: “If you have resources in multiple Availability Zones and they share one NAT gateway, in the event that the NAT gateway’s Availability Zone is down, resources in the other Availability Zones lose Internet access. To create an Availability Zone-independent architecture, create a NAT gateway in each Availability Zone and configure your routing to ensure that resources use the NAT gateway in the same Availability Zone.”  See also the Egress-only Internet Gateway, if you are using ipv6.

• What happens if you push AWS Kinesis (a high volume managed streaming solution from AWS) past the provisioned shard limits (as specified here)?
  • You start getting exceptions if you are trying to write to or read from the stream.  You can back off or you can increase the number of shards, which increases your throughput.
• Any planned support for .NET with the Kinesis libraries (Kinesis Producer Library, Kinesis Client Library) which have some nice features?
  • I’m not aware of any future plans.  However both are available on github (KPL, KCL) and are open source(ish) under the Amazon Software License.  I say “ish” because of some concerns about section 3.3, limits of use.  So you could port the code to .NET.  In addition, there is support for running the KCL with other languages (Ruby, .NET, etc) but you still need to run a Java daemon.
• Can someone create an IAM group with more permissions than the group they are in?
  • Yes, if the IAM system is misconfigured.  If a user is in group A which has the attach group policy permission, and has no other limits, they can attach an arbitrary policy to group B.  As per of the AWS shared responsibility model, you are responsible for your IAM setup.

More questions from AWS course students.

• EC2 instances in auto scaling groups have a warmup period that you can specify (so that the EC2 instance can be fully ready to take traffic directed to it).  I retold a story from another consultant about the warmup period for an ASG increasing over time (due to increasing numbers of security patches against the base AMI) and one student asked: “Can you set an alarm on instances overrunning the the warmup period?”
  • Since you can create custom metrics in cloudwatch and create alarms on those, you can definitely capture the warmup period.  All you’d need to do is, as the last step before an EC2 instance was fully configured, subtract the current time from the launch time (obtained via the API).  Store that number as your ‘warmup’ metric and set an alert if it ever gets close to your ASG health check value, and you’ll avoid ASG thrashing.
  • Update 4/6/2017: Another instructor pointed out a flaw in the above statements.  Upon further research, warmup time settings  only apply if you are using step scaling, and cooldown periods only apply if you are using simple scaling.  They are both trying to solve similar problems–making sure that you don’t start up or shut down instances before the instances have a chance to affect the situation that triggered the Auto Scaling Group action.  More on policy types.
• “Can the minimum and maximum number of instances of an ASG be changed after initial configuration?”
  • Yes.
• “Can you point Amazon inspector at non aws resources?  In your own data center, for example?”
  • Amazon is a security tool that looks for vulnerabilities in your EC2 instances.  It requires installing an agent on the instances that it will be monitoring, and thus doesn’t work outside of AWS.

I have recently started a contract teaching AWS courses. (None of the following posts speak for my client.) AWS stands for Amazon Web Services.

During every course I teach I get questions that are not directly covered in the course material that I don’t know. I’m going to try to capture some of the questions asked by my students and post the answers.

• Does SQS have transactional messages akin to JMS?
  • No.  JMS has the idea of transactions over messages, so you can be sure that all or none of the messages were processed.  SQS has no such construct–each message is independent.  If I were going to have multiple units of work done, I’d use one message, perhaps pointing to different datastores if the message was too big for SQS.
• Can you push content to the AWS CDN, Cloudfront, ahead of use requests?
  • No, the content always has to be pulled by a requester.  You can of course configure a crawler to pull the data from the origins through Cloudfront (which will then store it).
• Can you configure Cloudfront to pull from origins over SSL/TLS?
  • Yes.