AWS Questions: DynamoDB

Here are some questions and answers about DynamoDB, Amazon’s managed NoSQL database offering.

  • What are options for dynamically scaling DynamoDB provisioned throughput?
    • Hard to beat the options outlined in this StackOverflow post.  You can do it via scripting, the DynamicDynamoDB open source library, a lambda function, cloudwatch–lots of different ways.
  • Do DynamoDB streams support multiple readers?
  • How does optimistic concurrency control work?
    • Nicely outlined here but the long and the short of it is you need to make sure you associate a version with your items, read that version when you prepare to update, and then update if and only if the version is the same as the one you read.
  • Do you have any insight into the internals of DynamoDB?
  • How do you connect to DynamoDB?  Is there an IP address?
    • You use the SDK or CLI which connect to an endpoint in a region that you know no further details of.
  • What is the difference between eventual and strong consistency with respect to DynamoDB reads?
  • Does DynamoDB have any automatic encrypt at rest options?

     


AWS Questions: Elastic Load Balancer

More question answered from an AWS course.

  • Does the AWS ELB have the ability to throttle requests, to stop invalid/illegal traffic – if someone refreshes a page 10 times in 5 seconds and I want to block the unnecessary requests from the refreshes?
  • What is the availability of the ELB component?
    • I couldn’t find firm numbers, but here’s an interesting article about ELB best practices.
  • In a DDOS attack, since there is a lot of traffic to your environment, do you get charged for the additional traffic?
    • Depending on the attack type, not if you are fronted by an ELB or set up your security groups/NACLs to discard the traffic.  From the DDOS whitepaper: “When [an ELB detects certain types of attacks], it will automatically scale to absorb the additional traffic but you will not incur any additional charges.”
  • When an instance is decommissioned from an ASG, does the ELB know not to send new sessions to that ASG because the instance is getting ready to shut down?

AWS Questions: Cloudwatch

Cloudwatch is Amazon’s monitoring and alerting service.

Some questions and answers re: this awesome service.

  • How can you create custom metrics?
    • Cloudwatch doesn’t limit you to the metrics it collects by default (at the hypervisor level).  You can push any metric that makes sense up the the statistics repository, using custom metrics.
  • What protocol does cloudwatch use?  ICMP or SNMP?
  • What are cloudwatch logs?
    • A way for you to push logfiles from ec2 instances up to Cloudwatch, parse them, and create metrics out of them (“how many 404 errors has this application had in the last 30 minutes?”).  More here.
  • Does cloudwatch allow you to setup different metric thresholds at different times? For example, set an alarm at 70% CPU on Wed night but 90% on Sat night?
    • No, but you could do this with custom metrics.  You could read the cloudwatch default metrics and have an ‘cpualarm’ metric which would be 1 or 0 depending on if certain parameters were set.  Then you could vary the parameters over time.  Then you could set an alarm on the ‘cpualarm’ metric.

 


AWS Questions: VPC

Amazon VPC lets you create a virtual network in the cloud that you control–subnets, ip ranges, internet access, routing, etc.  At recent classes, I was asked some questions about VPC that I dug into to find answers.

  • Does AWS VPC support multicast or broadcast?
    • No, per the FAQs.  But there are some projects to overlay multicast functionality on top of the unicast network within a VPC.
  • Are vpc flow logs add on pricing?
    • There is no additional charge, but they go into Cloudwatch Logs and you are charged at the normal rate for that usage.
  • Is the NAT Gateway (used to provide internet access to ipv4 private subnets) highly available?
    • It is redundant within an availability zone.  But, from the docs: “If you have resources in multiple Availability Zones and they share one NAT gateway, in the event that the NAT gateway’s Availability Zone is down, resources in the other Availability Zones lose Internet access. To create an Availability Zone-independent architecture, create a NAT gateway in each Availability Zone and configure your routing to ensure that resources use the NAT gateway in the same Availability Zone.”  See also the Egress-only Internet Gateway, if you are using ipv6.

AWS Questions: Kinesis and IAM

  • What happens if you push AWS Kinesis (a high volume managed streaming solution from AWS) past the provisioned shard limits (as specified here)?
    • You start getting exceptions if you are trying to write to or read from the stream.  You can back off or you can increase the number of shards, which increases your throughput.
  • Any planned support for .NET with the Kinesis libraries (Kinesis Producer Library, Kinesis Client Library) which have some nice features?
    • I’m not aware of any future plans.  However both are available on github (KPL, KCL) and are open source(ish) under the Amazon Software License.  I say “ish” because of some concerns about section 3.3, limits of use.  So you could port the code to .NET.  In addition, there is support for running the KCL with other languages (Ruby, .NET, etc) but you still need to run a Java daemon.
  • Can someone create an IAM group with more permissions than the group they are in?
    • Yes, if the IAM system is misconfigured.  If a user is in group A which has the attach group policy permission, and has no other limits, they can attach an arbitrary policy to group B.  As per of the AWS shared responsibility model, you are responsible for your IAM setup.

AWS Questions: ASGs and Amazon Inspector

More questions from AWS course students.

  • EC2 instances in auto scaling groups have a warmup period that you can specify (so that the EC2 instance can be fully ready to take traffic directed to it).  I retold a story from another consultant about the warmup period for an ASG increasing over time (due to increasing numbers of security patches against the base AMI) and one student asked: “Can you set an alarm on instances overrunning the the warmup period?”
    • Since you can create custom metrics in cloudwatch and create alarms on those, you can definitely capture the warmup period.  All you’d need to do is, as the last step before an EC2 instance was fully configured, subtract the current time from the launch time (obtained via the API).  Store that number as your ‘warmup’ metric and set an alert if it ever gets close to your ASG health check value, and you’ll avoid ASG thrashing.
    • Update 4/6/2017: Another instructor pointed out a flaw in the above statements.  Upon further research, warmup time settings  only apply if you are using step scaling, and cooldown periods only apply if you are using simple scaling.  They are both trying to solve similar problems–making sure that you don’t start up or shut down instances before the instances have a chance to affect the situation that triggered the Auto Scaling Group action.  More on policy types.
  • “Can the minimum and maximum number of instances of an ASG be changed after initial configuration?”
  • “Can you point Amazon inspector at non aws resources?  In your own data center, for example?”
    • Amazon is a security tool that looks for vulnerabilities in your EC2 instances.  It requires installing an agent on the instances that it will be monitoring, and thus doesn’t work outside of AWS.

AWS Questions: Cloudfront and SQS

I have recently started a contract teaching AWS courses. (None of the following posts speak for my client.) AWS stands for Amazon Web Services.

During every course I teach I get questions that are not directly covered in the course material that I don’t know. I’m going to try to capture some of the questions asked by my students and post the answers.

  • Does SQS have transactional messages akin to JMS?
    • No.  JMS has the idea of transactions over messages, so you can be sure that all or none of the messages were processed.  SQS has no such construct–each message is independent.  If I were going to have multiple units of work done, I’d use one message, perhaps pointing to different datastores if the message was too big for SQS.
  • Can you push content to the AWS CDN, Cloudfront, ahead of use requests?
    • No, the content always has to be pulled by a requester.  You can of course configure a crawler to pull the data from the origins through Cloudfront (which will then store it).
  • Can you configure Cloudfront to pull from origins over SSL/TLS?

Restoring a single table from an Amazon RDS backup

material-icon-1307676_640When you use SQL, how do you write delete statements at the database prompt?

A delete statement typically looks like this: delete from table_name where column_name = 'foo';. I usually write it in this order:

  1. delete
  2. delete where column_name = 'foo';
  3. delete from table_name where column_name = 'foo';

Even though this is a pain because you have to move back and forth (I really need to look into vi keybindings for mysql), it prevents you from making sending this command by accident: delete from table_name; which deletes all the data in your table.  (Another alternative is to never use the interactive client and always write out your delete statements in a file and run that file to delete data.)

But, recently, I did exactly that, because I forgot.  I deleted all the data from one table in our production database.  It was billing data, so rather important.  Luckily, I am using Amazon RDS and had set up backup retention.

I wanted to outline what I did to recover from this.

  • I took a deep breath.
  • I wrote a message on the slack channel documenting what had happened and the possible customer impact.
  • Depending on which data is removed, it’s possible you will want to put the application in maintenance mode and/or inform your customers of the issues.  What I deleted was used rarely enough that I didn’t have to take these steps.
  • I looked at how to restore an Amazon RDS backup.
  • I restored the missing data.
  • I communicated that things were back to normal to internal stakeholders.

Unfortunately, it wasn’t clear how to restore a single table.  I’m used to being able to download a .sql file and hand edit it, but that’s not an option.  Stackoverflow wasn’t super helpful.   But if there’s anytime you want clarity, it’s when you are restoring production data.  You don’t want to compound the problem by screwing up something else.

So, here’s how to restore a single table from an Amazon RDS backup:

  • Note the time just before you deleted the data.  (Another reason the slack message is nice.  chatops ftw.)
  • Start up another instance from that moment.  I named it something obvious like ‘has-data-from-tablename’.
  • Twiddle your thumbs anxiously while the new instance starts up.
  • The instance is put into your default security group (as of this writing) which probably doesn’t allow mysql access.  Make sure you modify this security group to allow access.
  • When the instance is up, do a dump of the table you need: mysqldump -t --ssl-ca=./amazon-rds-ca-cert.pem -u user -ppassword -h has-data-from-tablename.c1m7x25w24qor.us-east-1.rds.amazonaws.com -P3306 database_name tablename > restore-table_name.sql; (-t omits the create database/table statements.)
  • If your table is has had writes since you deleted everything, you may need to manually pull down the current data from the production system and merge it into restore-table_name.sql; I was able to avoid this step.
  • Load the data using mysql mysql --ssl-ca=./amazon-rds-ca-cert.pem -u user -ppassword -h production.c1m7x25w24qor.us-east-1.rds.amazonaws.com -P3306 database_name < restore-table_name.sql;
  • Review to make sure the data is correct.
  • Test the application.
  • Update the slack channel, and do any other notifications you need to (customers, internal contacts, etc).
  • Revoke the default security group access you allowed above.
  • Delete the ‘has-data-from-tablename’ instance.

Note this only works if you caught your mistake within the backup retention window. (Make sure you set that up.)  We aren’t multi AZ or clustered, so I’m not sure how that would affect things.

Happy deep breathing!



© Moore Consulting, 2003-2017 +