Configuring and Enabling Virutal MFA for AWS account

I tried enabling AWS MFA (Multi-Factor Authentication) for AWS root account, as it supports Virtual MFA applications such as Google Authenticator for iPhone or Android or Authenticator for Windows phone as for free and we can use AWS MFA itself without any charge as well.
This is a quick instruction to apply AWS MFA with Google Authenticator.

1. Install Google Authenticator in your smartphone.
   Cf. Virtual MFA applications
   
2. Enable virtual MFA application with the official instruction.
   
   * Sign in to the AWS Management Console and move to IAM section.
   * Click Manage MFA.
   
   * Select "A Virtual MFA device" and click "Next Step".
   
   
   * Just click "Next Step".
   
   
   * Scan the QR code on the screen by Google Authenticator and confirm the two codes and enter them.
   
   
   * Click "Finish" after successfully associating the MFA device.
   
   
3. Confirm that the AWS root account is available with Google Authenticator.
   
   * Sign in to the AWS management console with your AWS root account.
   
   * Confirm the authentication code with Google Authenticator and enter the code.
   
   
4. Delete your AWS account root access key.
   Security status, "Delete your root access keys" will be green after deleting it.
   

I checked the cloud providers that support Multi-Factor Authentication, but not so many providers support it?

• Windows Azure
• Rackspace ??

Failed Access Denied: S3 properties on us-east-1 region

When I created s3 buckets at each regions including us-east-1, us-west1, us-west-2, and ap-northeast-1 and applied the IAM policies below to a IAM group. I got some of the properties of other than us-east-1 region somehow.
I created an IAM group and attached two IAM policies with the group.

IAM policy to define the buckets to operate and access source IP address.

{
  "Statement": [
    {
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "xxx.xxx.xxx.xxx/32",
            "xxx.xxx.xxx.xxx/32"
          ]
        }
      },
      "Resource": [
        "arn:aws:s3:::bucket_name.ap-northeast-1",
        "arn:aws:s3:::bucket_name.ap-northeast-1/*",
        "arn:aws:s3:::bucket_name.us-west-1",
        "arn:aws:s3:::bucket_name.us-west-1/*",
        "arn:aws:s3:::bucket_name.us-west-1",
        "arn:aws:s3:::bucket_name.us-west-1/*",
        "arn:aws:s3:::bucket_name.us-west-2",
        "arn:aws:s3:::bucket_name.us-west-2/*"
      ],
      "Action": "s3:*",
      "Effect": "Allow"
    }
  ],
  "Version": "2012-10-17"
}

• IAM policy to list all the buckets for a s3 tool like S3 Browser

{
  "Statement": [
    {
      "Resource": "*",
      "Action": "s3:List*",
      "Effect": "Allow"
    }
  ],
  "Version": "2012-10-17"
}

I got the properties of us-west-1, us-west-2, ap-northeast-1, but couldn't get those of us-east-1, though the same IAM policies are applied.


I successfully got the properties of us-east-1 after adding the action as follows.

{
  "Statement": [
    {
      "Resource": "*",
      "Action": [
        "s3:List*",
        "s3:Get*"
      ],
      "Effect": "Allow"
    }
  ],
  "Version": "2012-10-17"
}

I am wondering if us-east-1 (US standard) region is different from other regions in how to apply IAM policy because it is the 1st region of AWS???

AWS CLI memorandum - Describe all Availability Zones for each region

I was creating CloudFormation template and wanted to list all of the Availability Zones for each regions to create the following Mappings in the template.

  "Mappings": {
    "AvailabilityZoneMap": {
       "ap-northeast-1": { "AZa": "ap-northeast-1a", "AZb": "ap-northeast-1b", "AZc": "ap-northeast-1c" }
      ...
       "ap-northeast-2": { "AZa": "ap-northeast-2a", "AZb": "ap-northeast-2b", "AZc": "ap-northeast-2c" 
    }
  },

Describing all the names of the regions

$ aws ec2 describe-regions | jq -r '.Regions[].RegionName'
eu-west-1
sa-east-1
us-east-1
ap-northeast-1
us-west-2
us-west-1
ap-southeast-1
ap-southeast-2

Describing all the AZs for each regions

$ for i in `aws ec2 describe-regions | jq -r '.Regions[].RegionName'` ; do
ec2-describe-availability-zones --region $i
done

AVAILABILITYZONE        eu-west-1a      available       eu-west-1       

AVAILABILITYZONE        eu-west-1b      available       eu-west-1       

AVAILABILITYZONE        eu-west-1c      available       eu-west-1       

AVAILABILITYZONE        sa-east-1a      available       sa-east-1       

AVAILABILITYZONE        sa-east-1b      available       sa-east-1       

AVAILABILITYZONE        us-east-1a      available       us-east-1       

AVAILABILITYZONE        us-east-1b      available       us-east-1       

AVAILABILITYZONE        us-east-1c      available       us-east-1       

AVAILABILITYZONE        ap-northeast-1a available       ap-northeast-1  

AVAILABILITYZONE        ap-northeast-1c available       ap-northeast-1  

AVAILABILITYZONE        us-west-2a      available       us-west-2       

AVAILABILITYZONE        us-west-2b      available       us-west-2       

AVAILABILITYZONE        us-west-2c      available       us-west-2       

AVAILABILITYZONE        us-west-1a      available       us-west-1       

AVAILABILITYZONE        us-west-1c      available       us-west-1       

AVAILABILITYZONE        ap-southeast-1a available       ap-southeast-1  

AVAILABILITYZONE        ap-southeast-1b available       ap-southeast-1  

AVAILABILITYZONE        ap-southeast-2a available       ap-southeast-2  

AVAILABILITYZONE        ap-southeast-2b available       ap-southeast-2  

I tried using AWS CLI to do the same, but AWS CLI only includes the zones for the region that you're currently using and it does not show the AZs for the other regions.

Describes one or more of the Availability Zones that are available to you. The results include zones only for the region you're currently using. If there is an event impacting an Availability Zone, you can use this request to view the state and any provided message for that Availability Zone.

$ for i in `aws ec2 describe-regions | jq -r '.Regions[].RegionName'` ; do
aws ec2 describe-availability-zones --zone-names $i
done
A client error (InvalidParameterValue) occurred: Invalid availability zone: [eu-west-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [sa-east-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [us-east-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [ap-northeast-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [us-west-2]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [us-west-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [ap-southeast-1]
A client error (InvalidParameterValue) occurred: Invalid availability zone: [ap-southeast-2]

I need to think about how to convert the results into JSON format next.

AWS CLI memorandum - Delete S3 Buckets and all of the objects simultaneously

When we put some objects into S3 bucket to verify or validate a CloudFormation stack by uploading a template to S3 bucket, the buckets store thousands of objects that I don't have to keep anymore and we sometimes have to pay too much cost for that after receiving the billing.

Why don't we delete all of the objects and buckets simultaneously or you should consider of using Object Lifecycle Management in advance to automatically delete the objects that has been expired.

* List the buckets you want to remove and input them into a file

$ aws s3api list-buckets | jq -r '.Buckets[] | .Name' | egrep 'bucket-name-[1-3]' | tee bucket.txt
bucket-name-1
bucket-name-2
bucket-name-3

* Recursively delete all of the objects, the buckets and confirm that they are deleted

$ while read line ; do
echo "`date '+%x %T'` ## Removing $line"
aws s3 rm s3://$line --recursive
aws s3api delete-bucket --bucket $line
aws s3 ls s3://$line
echo -e "`date '+%x %T'` ## Finished removing $line\n"
done < bucket.txt

06/11/2014 17:22:33 ## Removing bucket-name-1
delete: s3://bucket-name-1/object_01.txt
...
delete: s3://bucket-name-1/object_10.txt
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:22:39 ## Finished removing bucket-name-1

06/11/2014 17:22:39 ## Removing bucket-name-2
delete: s3://bucket-name-2/object_01.txt
...
delete: s3://bucket-name-2/object_10.txt
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:22:44 ## Finished removing bucket-name-2

06/11/2014 17:22:44 ## Removing bucket-name-3
delete: s3://bucket-name-3/object_01.txt
...
delete: s3://bucket-name-3/object_10.txt
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:22:50 ## Finished removing bucket-name-3

* Use --quiet option if you want to quietly remove

$ while read line ; do
echo "`date '+%x %T'` ## Removing $line"
aws s3 rm s3://$line --recursive --quiet
aws s3api delete-bucket --bucket $line
aws s3 ls s3://$line
echo -e "`date '+%x %T'` ## Finished removing $line\n"
done < bucket.txt

06/11/2014 17:25:56 ## Removing bucket-name-1
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:26:01 ## Finished removing bucket-name-1

06/11/2014 17:26:01 ## Removing bucket-name-2
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:26:07 ## Finished removing bucket-name-2

06/11/2014 17:26:07 ## Removing bucket-name-3
A client error (NoSuchBucket) occurred: The specified bucket does not exist
06/11/2014 17:26:12 ## Finished removing bucket-name-3

That's it!

AWS CLI memorandum - Describe the maximum and minumum price from the spot price history

I begin to use Amazon EC2 Spot Instances and feel like knowing the maximum and minimum price from the spot price history with a combination of AWS CLI and jq.
The sample below is supposed that Instance type is m1.small and Availability Zone is ap-northeast-1a in Tokyo region.

* Describe the maximum price

$ instance_type=m1.small

$ aws ec2 describe-spot-price-history \

--instance-types ${instance_type} \

--product-descriptions Linux/UNIX \(Amazon VPC\) \

--availability-zone ap-northeast-1a | jq ' .[] | max_by(.SpotPrice)'

{

  "AvailabilityZone": "ap-northeast-1a",

  "SpotPrice": "0.352000",

  "InstanceType": "m1.small",

  "ProductDescription": "Linux/UNIX",

  "Timestamp": "2014-02-01T05:01:34.000Z"

}

* Describe the minimum price

$ instance_type=m1.small

$ aws ec2 describe-spot-price-history \

--instance-types ${instance_type} \

--product-descriptions Linux/UNIX \(Amazon VPC\) \

--availability-zone ap-northeast-1a | jq ' .[] | min_by(.SpotPrice)'

{

  "AvailabilityZone": "ap-northeast-1a",

  "SpotPrice": "0.017100",

  "InstanceType": "m1.small",

  "ProductDescription": "Linux/UNIX",

  "Timestamp": "2014-04-01T09:35:08.000Z"

}

I tried, but I am not sure if it's possible to output both maximum and minimum in one line.

Monitoring tool for hyblid cloud and automating - installing Hyclops for Zabbix (updated)

I used to write about how to install zabbix+hyclops to automatically register zabbix hosts on AWS EC2 instance before. Recently, I tried installing the latest version of zabbix-2.2.2 and hyclops-0.2.
In fact, it's not working as expected so far because it seems that hyclops.connector.ec2 (ec2.py) python script is not working correctly. I will update this article when finishing to fix the problem (asking the engineers who created hyclops to debug the issue currently).
After uninstalling apache-libcloud from 0.14.1 to 0.13.2, the issue has been resolved.
Ikeda-san, thanks for your quick feedback and support!

OS, Middle ware, Libraries

• AMI ID (AWS EC2 Instance): ami-31e86030
• OS: CentOS 6.5(x86_64)
• Kernel: 2.6.32-358.6.1.el6.x86_64
• Apache: 2.2.15
• MySQL: 5.6.16
• PHP: 5.3.3
• Zabbix: 2.2.2
• Hyclops: 0.2.0
• ZeroMQ: 3.2.2
• GateOne: 1.1.1
• Python: 2.7.5
• Python Modules
• apache-libcloud (0.14.1) (0.13.2)
• boto (2.27.0)
• configobj (5.0.2)
• distribute (0.6.35)
• hyclops (0.2.1)
• ipython (1.2.1)
• lockfile (0.9.1)
• pip (1.5.4)
• psphere (0.5.2)
• python-daemon (1.6)
• PyYAML (3.10)
• pyzmq (14.1.1)
• setuptools (0.6c11)
• six (1.5.2)
• suds (0.4)
• tornado (2.4.1)
• wsgiref (0.1.2)
• zabbix-api (1.0)

Upgrade to CentOS-6.5 from 6.4

# yum -y install httpd httpd-devel mod_ssl

MySQL-5.6

# yum -y install http://repo.mysql.com/mysql-community-release-el6.rpm
# yum -y install mysql mysql-devel mysql-embedded-devel mysql-libs mysql-server

Apache-2.2

# yum -y install httpd httpd-devel mod_ssl

PHP-5.3

# yum -y install php php-bcmath php-gd php-mbstring php-xml php-mysql

Libraries for Zabbix

# yum -y install http://ftp-srv2.kddilabs.jp/Linux/distributions/fedora/epel/6/x86_64/epel-release-6-8.noarch.rpm
# yum -y install http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
# yum -y install --enablerepo=remi,epel fping iksemel-devel curl-devel openldap-devel OpenIPMI-devel net-snmp-devel unixODBC-devel git wget

Zabbix 2.2

# yum -y install http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm
# yum -y install --enablerepo=zabbix zabbix zabbix-server-mysql zabbix-web zabbix-web-mysql zabbix-web-japanese zabbix-agent zabbix-get zabbix-sender

ZeroMQ

# curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
# yum -y install gcc gcc-c++ zeromq zeromq-devel ipmitool detach

Createing Zabbix schema

# for i in schema images data
do mysql -u zabbix -p zabbix < /usr/share/doc/zabbix-server-mysql-2.2.2/create/$i.sql
done
# mysqladmin -u root -p create zabbix
# mysql -u root -p -e 'GRANT ALL PRIVILEGES on zabbix.* to zabbix@localhost IDENTIFIED BY "zabbix"; FLUSH PRIVILEGES;'
# mysql -u zabbix zabbix -e 'SHOW TABLES;'
+-----------------------+
| Tables_in_zabbix    |
+-----------------------+
| acknowledges          |
| actions                     |
| alerts                       |
...
| users_groups          |
| usrgrp                      |
| valuemaps               |
+-----------------------+
# service zabbix-server start

Python-2.7, Python packages

# wget -O- https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz | tar zxf -
# cd Python-2.7.5/
# /configure && make && make install
# wget -O- http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz | tar zxf -
# cd distribute-0.6.35/
# python2.7 setup.py install
# easy_install-2.7 pip

Python modules

# pip install zabbix-api pyzmq psphere python-daemon==1.6 configobj tornado==2.4.1 boto

Python module (apache-libcloud)
Make sure that apache-libcloud-0.13.2 should be compiled and installed from tar ball because it failed to install apache-libcloud-0.13.2 with python-2.7 via pip.

# wget -O- https://pypi.python.org/packages/source/a/apache-libcloud/apache-libcloud-0.13.2.tar.gz#md5=d3d127bb21ee73ca5b30bb367949c898 | tar zxf -
# cd apache-libcloud-0.13.2/
# python setup.py install
# pip list | grep apache-libcloud
apache-libcloud (0.13.2)

# pip install apache-libcloud==0.13.2
Downloading/unpacking apache-libcloud==0.13.2
  Downloading apache-libcloud-0.13.2.tar.bz2 (396kB): 396kB downloaded
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1197, in prepare_files
    do_download,
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1375, in unpack_url
    self.session,
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/download.py", line 582, in unpack_http_url
    unpack_file(temp_location, location, content_type, link)
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 625, in unpack_file
    untar_file(filename, location)
  File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 543, in untar_file
    tar = tarfile.open(filename, mode)
  File "/usr/local/lib/python2.7/tarfile.py", line 1678, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/local/lib/python2.7/tarfile.py", line 1744, in bz2open
    raise CompressionError("bz2 module is not available")
CompressionError: bz2 module is not available

Storing debug log for failure in /root/.pip/pip.log

Gateone

# yum -y install https://github.com/downloads/liftoff/GateOne/gateone-1.1-1.noarch.rpm
# cp /opt/gateone/tests/chat/keyfile.pem /opt/gateone/

Replacing some Zabbix dashboard files

# git clone https://github.com/tech-sketch/hyclops.git
# cd hyclops/
# python ./setup.py install
# cp -a ./misc/init.d/redhat/hyclops /etc/init.d/
# cp -a ./misc/init.d/ubuntu/hyclops.conf /etc/init/
# externalscripts=/usr/lib/zabbix/externalscripts
# mkdir -p ${externalscripts}
# cp -a ./misc/externalscripts/* ${externalscripts}
# chown -R zabbix:zabbix ${externalscripts}
# chmod u+x ${externalscripts}/*

I changed the python script for get_aws_charges.py and push_message.py scripts are changed.

# sed -i "s|#!.*|#!/usr/local/bin/python|" ${externalscripts}/*.py

# chkconfig hyclops on
# service hyclops start
# python ./setup.py replace -d /usr/share/zabbix --zabbix-version=2.2
running replace
backup original file /usr/share/zabbix/dashboard.php to /usr/share/zabbix/dashboard.org
create /usr/share/zabbix/dashboard.php
create /usr/share/zabbix/gateone.php
create /usr/share/zabbix/dashboard_scripts_exec.php
backup original file /usr/share/zabbix/include/menu.inc.php to /usr/share/zabbix/include/menu.inc.org
create /usr/share/zabbix/include/menu.inc.php
create /usr/share/zabbix/custom/additional_blocks_func.inc.php
create /usr/share/zabbix/custom/monitoring.dashboard.js.php
create /usr/share/zabbix/custom/additional_func.inc.php
create /usr/share/zabbix/custom/additional_blocks.inc.php

uid=500(hyclops) gid=500(hyclops) groups=500(hyclops)

Importing Zabbix templates,scripts and globalmacro data

# python ./setup.py import -u admin -p zabbix -f http://zabbix-server front end ip/zabbix/
running import
Import templates
Import global scripts
  Create 'Start EC2 instance'
  Create 'Stop EC2 instance'
  Create 'Reboot EC2 instance'
  Create 'Start vSphere instance'
  Create 'Stop vSphere instance'
  Create 'Suspend vSphere instance'
  Create 'Reboot vSphere instance'
  Create 'IPMI power on'
  Create 'IPMI power off'
Import global macros
  Create '{$GATEONE_URL}'
  Create '{$GATEONE_KEY}'
  Create '{$GATEONE_SECRET}'
  Create '{$HYCLOPS_SERVER}'
  Create '{$HYCLOPS_PORT}'
uid=500(hyclops) gid=501(hyclops) groups=501(hyclops)

After creating value mappings and creating zabbix hosts on zaxbbix dashboard, I got the error message below in fact. I need to dive into the logging...
This issue has been solved by downgrading apache-libcloud from 0.14.1 to 0.13.2.

# tail -f /opt/hyclops/logs/hyclops_server.log
[2014-03-25 16:30:17,416] hyclops.connector.ec2 (ec2-For AWS EC2-monitor) ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 46, in __call__
    result = self.run_command(hostname, params)
  File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 62, in run_command
    result = self.monitor(hostname, conn_params)
  File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 126, in monitor
    self.set_ami_info(node)
  File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 175, in set_ami_info
    images = node.driver.list_images(ex_image_ids=[node.extra["imageId"]])
KeyError: 'imageId'

See zabbix_server.log to confirm the both python scripts are working to get aws charges and push messages.

# tail -f /var/log/zabbix/zabbix_server.log 
t  3704:20140326:110510.922 item [For AWS EC2:get_aws_charges.py[{$KEY},{$SECRET}]] became supported
  3704:20140326:110510.922 item [For AWS EC2:push_message.py[{$HYCLOPS_SERVER},{$HYCLOPS_PORT},ec2,{HOST.HOST}]] became supported

See hyclops_server.log to confirm that polling and creating zabbix hosts have been successful.
* You need to change "log_level = DEBUG" on /opt/hyclops/hyclops.conf to debug.

# tail -f /opt/hyclops/logs/hyclops_server.log 
[2014-03-26 10:59:23,326] hyclops.queue (MainThread) INFO: Start message queue
[2014-03-26 10:59:23,326] hyclops (MainThread) INFO: Message queue is opened
[2014-03-26 10:59:23,326] hyclops.queue (MainThread) DEBUG: polling...
[2014-03-26 11:05:08,612] hyclops.queue (MainThread) DEBUG: recieved request: [['ec2', 'For AWS EC2']]
[2014-03-26 11:05:08,613] hyclops.queue (MainThread) DEBUG: run ec2-For AWS EC2-monitor thread
[2014-03-26 11:05:08,614] hyclops.queue (MainThread) DEBUG: running threads: [{'ec2-For AWS EC2-monitor': True}]
[2014-03-26 11:05:11,617] hyclops.queue (MainThread) DEBUG: polling...
...
[2014-03-26 11:05:43,452] hyclops.connector.ec2 (ec2-For AWS EC2-monitor) DEBUG: Create zabbix host i-xxxxxxx (For AWS EC2_i-xxxxxxx_i-xxxxxxx)
...

[2014-03-26 11:05:59,960] hyclops.connector.ec2 (ec2-For AWS EC2-monitor) DEBUG: success. thread finished.