AWS CLI memorandum - Getting CPU Metrics on CloudWatch

This is just memorandum to extract CPUUtilization with AWS CLI because I'm considering using Zabbix or Sensu to monitor the instances on AWS and thinking of how to extract each metrics, such as CPU usage, Memory usage, or Traffic.
Writing plugins will be helpful but the simplest way and using functions which they offer are easier than writing plugins, I believe.

Listing metrics of AWS/EC2

$ aws cloudwatch list-metrics --namespace AWS/EC2 --dimensions Name=InstanceType,Value=t1.micro
{
    "Metrics": [
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "NetworkOut"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskWriteOps"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskWriteBytes"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "CPUUtilization"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskReadOps"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskReadBytes"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "NetworkIn"
        }
    ]
}

Getting metrics of CPUUtilization

$ aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--dimensions Name=InstanceId,Value=i-xxx \
--statistics Average \
--start-time `date -u '+%FT%TZ' -d '10 mins ago'` \
--end-time `date -u '+%FT%TZ'` \
--period 60
{
    "Datapoints": [
        {
            "Timestamp": "2013-11-30T11:13:00Z", 
            "Average": 5.6659999999999995, 
            "Unit": "Percent"
        }, 
        {
            "Timestamp": "2013-11-30T11:18:00Z", 
            "Average": 6.6659999999999995, 
            "Unit": "Percent"
        }
    ], 
    "Label": "CPUUtilization"
}

Extracting average with jq command

$ aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--dimensions Name=InstanceId,Value=i-xxx \
--statistics Average \
--start-time `date -u '+%FT%TZ' -d '10 mins ago'` \
--end-time `date -u '+%FT%TZ'` \
--period 60 | jq '.Datapoints[0] | .Average'
6.6659999999999995

• Reference
• AWS Namespaces
• AWS CLI CloudWatch
• jq

Monitoring tool for hyblid cloud and automating - installing Hyclops for Zabbix

The update version on this article is here, composed of Zabbix-2.2 and Hyclops-0.2.

Automation is definitely necessary, and let's say common for the cloud or virtualization environment, especially it saves us lots of time by automatically setting up the servers to be monitored.
I believe that Zabbix is one of the most famous open source monitoring software which has been popular for the recent years.
I am pretty sure that there are other monitoring softwares to achieve it, such as Hinemos, Hyperic HQ, Sensu, and so on.
Here, I tried installing Hyclops for Zabbix to achieve automatically registering the EC2 instances to be monitored on AWS.

This is how to install Hyclops for Zabbix based on the official installation manual with some comments added.

OS/Middeware/Application

OS	CentOS 6.4
Kernel	2.6.32-358.6.1.el6.x86_64
Apache	2.2.15
MySQL	5.6.13
Zabbix	2.0.9
PHP	5.3.3
Hyclops	0.1.0
Python	2.6.6
python-setuptools	0.6.10
python-pip	1.3.1
ZeroMQ	3.2.2
GateOne	1.1.1
ipmitool	1.8.11
tornade	2.4.1

Python, Python packages

$ sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ sudo yum -y --enablerepo=epel install python python-libs python-devel python-setuptools python-pip dtach
$ sudo rpm -qa | egrep '^(python)' | sort
python-2.6.6-37.el6_4.x86_64
python-boto-2.13.3-1.el6.noarch
python-cheetah-2.4.1-1.el6.x86_64
python-configobj-4.6.0-3.el6.noarch
python-devel-2.6.6-37.el6_4.x86_64
python-ethtool-0.6-3.el6.x86_64
python-iniparse-0.3.1-2.1.el6.noarch
python-iwlib-0.1-1.2.el6.x86_64
python-libs-2.6.6-37.el6_4.x86_64
python-markdown-2.0.1-3.1.el6.noarch
python-pip-1.3.1-4.el6.noarch
python-pycurl-7.19.0-8.el6.x86_64
python-pygments-1.1.1-1.el6.noarch
python-setuptools-0.6.10-3.el6.noarch
python-urlgrabber-3.9.1-8.el6.noarch

ZeroMQ, ipmitools

$ curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
$ sudo yum -y install gcc gcc-c++ zeromq zeromq-devel ipmitool detach
$ rpm -qa | egrep '(zeromq|ipmitool)' | sort
ipmitool-1.8.11-14.el6_4.1.x86_64
zeromq-3.2.2-13.1.x86_64
zeromq-devel-3.2.2-13.1.x86_64

Python modules

$ sudo pip install apache-libcloud==0.13.2 zabbix-api pyzmq psphere python-daemon==1.6 configobj

ordereddict, argparse if phython is less than 2.7

$ sudo pip install futures ordereddict

tornade-2.4.1

$ sudo rpm -ivh https://github.com/downloads/liftoff/GateOne/tornado-2.4-1.noarch.rpm

tornade-2.4.1 is recommended because 3.x has some issues in attribute settings and it fails to run gateone with the message below. If pip is used to install tornade, tornade-3.x is installed.
Cf. https://github.com/liftoff/GateOne/issues/235

$ sudo /opt/gateone/gateone.py --auth=api --origins="http://zabbix-server-frontend/"
Traceback (most recent call last):
  File "/opt/gateone/gateone.py", line 289, in 
    tornado.options.enable_pretty_logging()
AttributeError: 'module' object has no attribute 'enable_pretty_logging'

PIL(Python Image Library)

$ sudo pip install PIL

boto if monitoring aws billing stats is needed

$ sudo pip install boto

installed python modules

$ pip list
apache-libcloud (0.13.2)
boto (2.5.2)
Cheetah (2.4.1)
cloud-init (0.6.3)
configobj (4.6.0)
distribute (0.6.10)
ethtool (0.6)
futures (2.1.5)
gateone (1.1)
hyclops (0.1.0)
iniparse (0.3.1)
iwlib (1.0)
lockfile (0.9.1)
Markdown (2.0.1)
ordereddict (1.1)
PIL (1.1.7)
psphere (0.5.2)
pycurl (7.19.0)
Pygments (1.1.1)
pygpgme (0.1)
python-daemon (1.6)
PyYAML (3.10)
pyzmq (14.0.0)
suds (0.4)
tornado (2.4)
urlgrabber (3.9.1)
yum-metadata-parser (1.1.2)
yum-presto (0.4.4)
zabbix-api (1.0)

GateOne

$ sudo rpm -ivh https://github.com/downloads/liftoff/GateOne/gateone-1.1-1.noarch.rpm
$ cp /opt/gateone/tests/chat/keyfile.pem /opt/gateone/

Modifying init script for GateOne to display the status function normally

$ sudo service gateone status
gateone.py is stopped
$ diff -u /etc/init.d/gateone{,.bak}
--- /etc/init.d/gateone 2013-11-20 16:31:22.585275659 +0900
+++ /etc/init.d/gateone.bak     2013-11-20 15:58:06.460275392 +0900
@@ -66,8 +66,7 @@
       stop
     ;;
   status)
-$    status gateone.py
-    status -p ${GATEONE_PID} python.py
+    status -p ${GATEONE_PID} ${0##*/}
     ;;
   restart)
       restart
$ sudo service gateone status
gateone (pid  3060) is running

Configuring GateOne Cf. http://liftoff.github.io/GateOne/About/index.html

$ sudo /opt/gateone/gateone.py --auth=api --origins="https://localhost;https://127.0.0.1;https://"
[I 131120 16:27:32 gateone:2917] Connections to this server will be allowed from the following origins: 'https://localhost https://127.0.0.1 https:// https://'
[I 131120 16:27:32 gateone:2305] Using api authentication
[I 131120 16:27:32 gateone:2404] Loaded plugins: bookmarks, convenience, example, help, logging, logging_plugin, mobile, notice, playback, ssh
[I 131120 16:27:32 gateone:3054] Listening on https://*:443/
[I 131120 16:27:32 gateone:3060] Process running with pid 3255
[I 131120 16:27:38 gateone:1063] WebSocket opened (ANONYMOUS).
[I 131120 16:27:41 web:1462] 200 GET /static/ubuntumono-normal.woff () 10.19ms
[I 131120 16:27:42 web:1462] 200 GET /static/bookmarks/images/star.svg () 1.08ms
Ctrl+c
[I 131120 16:28:52 gateone:3084] Caught KeyboardInterrupt.  Killing sessions...
[I 131120 16:28:52 gateone:3088] pid file removed.
[I 131120 16:28:52 termio:1491] Finalizing the log for pid 3260 (this can take some time).

$ sudo /opt/gateone/gateone.py --new_api_key
[I 131120 16:29:13 gateone:2880] A new API key has been generated: 
[I 131120 16:29:13 gateone:2881] This key can now be used to embed Gate One into other applications.
$ sudo chkconfig gateone on
$ sudo service gateone start

Installing & Setting up Hyclops

$ wget https://github.com/tech-sketch/hyclops/archive/0.1.0.tar.gz -O- | tar zxf -
$ cd hyclops-0.1.0/
$ sudo ./setup.py install
$ sudo cp -a ./misc/init.d/redhat/hyclops /etc/init.d/
$ sudo cp -a ./misc/init.d/ubuntu/hyclops.conf /etc/init/
$ sudo mkdir /etc/zabbix/externalscripts
$ sudo cp -a ./misc/externalscripts/* /etc/zabbix/externalscripts
$ sudo chown -R zabbix:zabbix /etc/zabbix/externalscripts
$ sudo chmod u+x /etc/zabbix/externalscripts/*
$ sudo chkconfig hyclops on
$ sudo service hyclops start

Replacing some Zabbix dashboard files

$ sudo ./setup.py replace -d /usr/share/zabbix --zabbix-version=2.0
running replace
backup original file /usr/share/zabbix/dashboard.php to /usr/share/zabbix/dashboard.org
create /usr/share/zabbix/dashboard.php
create /usr/share/zabbix/gateone.php
create /usr/share/zabbix/dashboard_scripts_exec.php
create /usr/share/zabbix/custom/additional_blocks.inc.php
create /usr/share/zabbix/custom/additional_blocks_func.inc.php
create /usr/share/zabbix/custom/additional_func.inc.php
create /usr/share/zabbix/custom/monitoring.dashboard.js.php
backup original file /usr/share/zabbix/include/menu.inc.php to /usr/share/zabbix/include/menu.inc.org
create /usr/share/zabbix/include/menu.inc.php
uid=500(hyclops) gid=501(hyclops) groups=501(hyclops)

It is possible to specify 2.2 with the command option, but it has not been actually implemented yet. When I installed Zabbix-2.2 and specified 2.0 to replace the php files, the dashboard did not show because of some permission error or something.
This issue has been solved as Hyclops begins to support Zabbix-2.2.

$ sudo ./setup.py replace -d /usr/share/zabbix --zabbix-version=2.4
running replace
Not supported version (Supported only 2.0 or 2.2)
uid=501(hyclops) gid=501(hyclops) groups=501(hyclops)
$ ls ./misc/zabbix-custom/    
2.0

Importing Zabbix templates,scripts and globalmacro data

$ sudo ./setup.py import -u admin -p zabbix -f http://54.238.236.194/zabbix/
running import
Import templates
Import global scripts
  Create 'Start EC2 instance'
  Create 'Stop EC2 instance'
  Create 'Reboot EC2 instance'
  Create 'Start vSphere instance'
  Create 'Stop vSphere instance'
  Create 'Suspend vSphere instance'
  Create 'Reboot vSphere instance'
  Create 'IPMI power on'
  Create 'IPMI power off'
Import global macros
  Create '{$GATEONE_URL}'
  Create '{$GATEONE_KEY}'
  Create '{$GATEONE_SECRET}'
  Create '{$HYCLOPS_SERVER}'
  Create '{$HYCLOPS_PORT}'
uid=500(hyclops) gid=501(hyclops) groups=501(hyclops)

Setting cron script

$ sudo crontab -u hyclops -e
*/5 * * * * python /opt/hyclops/cron_scripts/delete_not_exist_hosts.py

Zabbix Macro/Host settings

Follow the steps at the official site to setup the macros for AWS access and secret key,
you will be able to see the instances registered at the Zabbix dashboard.
I just only tried AWS EC2 setting, not vSphere ESXi.

I'm looking forward to Hyclops to be updated for Zabbix-2.2!