AWS CLI memorandum - Describe all Availability Zones for each region

I was creating CloudFormation template and wanted to list all of the Availability Zones for each regions to create the following Mappings in the template.

  "Mappings": {
    "AvailabilityZoneMap": {
       "ap-northeast-1": { "AZa": "ap-northeast-1a", "AZb": "ap-northeast-1b", "AZc": "ap-northeast-1c" }
      ...
       "ap-northeast-2": { "AZa": "ap-northeast-2a", "AZb": "ap-northeast-2b", "AZc": "ap-northeast-2c" 
    }
  },

Describing all the names of the regions

$ aws ec2 describe-regions | jq -r '.Regions[].RegionName'
eu-west-1
sa-east-1
us-east-1
ap-northeast-1
us-west-2
us-west-1
ap-southeast-1
ap-southeast-2

Describing all the AZs for each regions

$ for i in `aws ec2 describe-regions | jq -r '.Regions[].RegionName'` ; do
ec2-describe-availability-zones --region $i
done

AVAILABILITYZONE        eu-west-1a      available       eu-west-1       

AVAILABILITYZONE        eu-west-1b      available       eu-west-1       

AVAILABILITYZONE        eu-west-1c      available       eu-west-1       

AVAILABILITYZONE        sa-east-1a      available       sa-east-1       

AVAILABILITYZONE        sa-east-1b      available       sa-east-1       

AVAILABILITYZONE        us-east-1a      available       us-east-1       

AVAILABILITYZONE        us-east-1b      available       us-east-1       

AVAILABILITYZONE        us-east-1c      available       us-east-1       

AVAILABILITYZONE        ap-northeast-1a available       ap-northeast-1  

AVAILABILITYZONE        ap-northeast-1c available       ap-northeast-1  

AVAILABILITYZONE        us-west-2a      available       us-west-2       

AVAILABILITYZONE        us-west-2b      available       us-west-2       

AVAILABILITYZONE        us-west-2c      available       us-west-2       

AVAILABILITYZONE        us-west-1a      available       us-west-1       

AVAILABILITYZONE        us-west-1c      available       us-west-1       

AVAILABILITYZONE        ap-southeast-1a available       ap-southeast-1  

AVAILABILITYZONE        ap-southeast-1b available       ap-southeast-1  

AVAILABILITYZONE        ap-southeast-2a available       ap-southeast-2  

AVAILABILITYZONE        ap-southeast-2b available       ap-southeast-2  

I tried using AWS CLI to do the same, but AWS CLI only includes the zones for the region that you're currently using and it does not show the AZs for the other regions.

Describes one or more of the Availability Zones that are available to you. The results include zones only for the region you're currently using. If there is an event impacting an Availability Zone, you can use this request to view the state and any provided message for that Availability Zone.

$ for i in `aws ec2 describe-regions | jq -r '.Regions[].RegionName'` ; do
aws ec2 describe-availability-zones --zone-names $i
done
A client error (InvalidParameterValue) occurred: Invalid availability zone: [eu-west-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [sa-east-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [us-east-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [ap-northeast-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [us-west-2]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [us-west-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [ap-southeast-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [ap-southeast-2]

I need to think about how to convert the results into JSON format next.

AWS CLI memorandum - Delete S3 Buckets and all of the objects simultaneously

When we put some objects into S3 bucket to verify or validate a CloudFormation stack by uploading a template to S3 bucket, the buckets store thousands of objects that I don't have to keep anymore and we sometimes have to pay too much cost for that after receiving the billing.

Why don't we delete all of the objects and buckets simultaneously or you should consider of using Object Lifecycle Management in advance to automatically delete the objects that has been expired.

* List the buckets you want to remove and input them into a file

$ aws s3api list-buckets | jq -r '.Buckets[] | .Name' | egrep 'bucket-name-[1-3]' | tee bucket.txt
bucket-name-1
bucket-name-2
bucket-name-3

* Recursively delete all of the objects, the buckets and confirm that they are deleted

$ while read line ; do
echo "`date '+%x %T'` ## Removing $line"
aws s3 rm s3://$line --recursive
aws s3api delete-bucket --bucket $line
aws s3 ls s3://$line
echo -e "`date '+%x %T'` ## Finished removing $line\n"
done < bucket.txt

06/11/2014 17:22:33 ## Removing bucket-name-1
delete: s3://bucket-name-1/object_01.txt
...
delete: s3://bucket-name-1/object_10.txt
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:22:39 ## Finished removing bucket-name-1

06/11/2014 17:22:39 ## Removing bucket-name-2
delete: s3://bucket-name-2/object_01.txt
...
delete: s3://bucket-name-2/object_10.txt
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:22:44 ## Finished removing bucket-name-2

06/11/2014 17:22:44 ## Removing bucket-name-3
delete: s3://bucket-name-3/object_01.txt
...
delete: s3://bucket-name-3/object_10.txt
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:22:50 ## Finished removing bucket-name-3

* Use --quiet option if you want to quietly remove

$ while read line ; do
echo "`date '+%x %T'` ## Removing $line"
aws s3 rm s3://$line --recursive --quiet
aws s3api delete-bucket --bucket $line
aws s3 ls s3://$line
echo -e "`date '+%x %T'` ## Finished removing $line\n"
done < bucket.txt

06/11/2014 17:25:56 ## Removing bucket-name-1
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:26:01 ## Finished removing bucket-name-1

06/11/2014 17:26:01 ## Removing bucket-name-2
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:26:07 ## Finished removing bucket-name-2

06/11/2014 17:26:07 ## Removing bucket-name-3
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:26:12 ## Finished removing bucket-name-3

That's it!

AWS CLI memorandum - Describe the maximum and minumum price from the spot price history

I begin to use Amazon EC2 Spot Instances and feel like knowing the maximum and minimum price from the spot price history with a combination of AWS CLI and jq.
The sample below is supposed that Instance type is m1.small and Availability Zone is ap-northeast-1a in Tokyo region.

* Describe the maximum price

$ instance_type=m1.small

$ aws ec2 describe-spot-price-history \

--instance-types ${instance_type} \

--product-descriptions Linux/UNIX \(Amazon VPC\) \

--availability-zone ap-northeast-1a | jq ' .[] | max_by(.SpotPrice)'

{

  "AvailabilityZone": "ap-northeast-1a",

  "SpotPrice": "0.352000",

  "InstanceType": "m1.small",

  "ProductDescription": "Linux/UNIX",

  "Timestamp": "2014-02-01T05:01:34.000Z"

}

* Describe the minimum price

$ instance_type=m1.small

$ aws ec2 describe-spot-price-history \

--instance-types ${instance_type} \

--product-descriptions Linux/UNIX \(Amazon VPC\) \

--availability-zone ap-northeast-1a | jq ' .[] | min_by(.SpotPrice)'

{

  "AvailabilityZone": "ap-northeast-1a",

  "SpotPrice": "0.017100",

  "InstanceType": "m1.small",

  "ProductDescription": "Linux/UNIX",

  "Timestamp": "2014-04-01T09:35:08.000Z"

}

I tried, but I am not sure if it's possible to output both maximum and minimum in one line.

AWS CLI memorandum - Delete Available EBS Volumes

We sometimes need to delete or remove unnecessary resources which we forgot to do so when we are creating some resources on AWS, don't we? I'm making a note mainly with AWS CLI for those situations.

Today, we are deleting all of the available EBS volumes!

* Describe EBS Volumes

$ aws ec2 describe-volumes| jq '.Volumes[0]'
{
  "Size": 10,
  "AvailabilityZone": "ap-northeast-1a",
  "Attachments": [
    {
      "Device": "/dev/sda",
      "DeleteOnTermination": true,
      "State": "attached",
      "VolumeId": "vol-xxxxxxxx",
      "InstanceId": "i-xxxxxxxx",
      "AttachTime": "2013-09-30T02:16:37.000Z"
    }
  ],
  "VolumeType": "standard",
  "VolumeId": "vol-xxxxxxxx",
  "State": "in-use",
  "SnapshotId": "snap-xxxxxxxx",
  "CreateTime": "2013-09-30T02:16:37.000Z"
}

* Extract available EBS Volumes

$ aws ec2 describe-volumes| jq '.Volumes[] | select(.State == "available") | {VolumeId,State}'
{
  "State": "available",
  "VolumeId": "vol-xxxxxxxx"
}
...
{
  "State": "available",
  "VolumeId": "vol-xxxxxxxx"
}

$ aws ec2 describe-volumes| jq '.Volumes[] | select(.State == "available")' | jq '.VolumeId' | sed -e 's/"//g'
vol-xxxxxxxx
...
vol-xxxxxxxx

* Delete available EBS Volumes in dry-run mode

$ for i in `aws ec2 describe-volumes| jq '.Volumes[] | select(.State == "available")' | jq '.VolumeId' | sed -e 's/"//g'`
  do aws ec2 delete-volume --volume-id $i --dry-run
done
A client error (DryRunOperation) occurred: Request would have succeeded, but DryRun flag is set.

* Delete available EBS Volumes

$ for i in `aws ec2 describe-volumes| jq '.Volumes[] | select(.State == "available")' | jq '.VolumeId' | sed -e 's/"//g'`
  do aws ec2 delete-volume --volume-id $i
done
{
    "return": "true"
}
...
{
    "return": "true"
}

* Confirm available EBS Volumes deleted

$ aws ec2 describe-volumes| jq '.Volumes[] | select(.State == "available") | {VolumeId,State}'

Make sure that unnecessary resources have been released to save cost!