hashnao's Blog: 2013

Saturday, November 30, 2013

AWS CLI memorandum - Getting CPU Metrics on CloudWatch

This is just memorandum to extract CPUUtilization with AWS CLI because I'm considering using Zabbix or Sensu to monitor the instances on AWS and thinking of how to extract each metrics, such as CPU usage, Memory usage, or Traffic.
Writing plugins will be helpful but the simplest way and using functions which they offer are easier than writing plugins, I believe.

Listing metrics of AWS/EC2

$ aws cloudwatch list-metrics --namespace AWS/EC2 --dimensions Name=InstanceType,Value=t1.micro
{
    "Metrics": [
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "NetworkOut"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskWriteOps"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskWriteBytes"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "CPUUtilization"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskReadOps"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "DiskReadBytes"
        }, 
        {
            "Namespace": "AWS/EC2", 
            "Dimensions": [
                {
                    "Name": "InstanceType", 
                    "Value": "t1.micro"
                }
            ], 
            "MetricName": "NetworkIn"
        }
    ]
}

Getting metrics of CPUUtilization

$ aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--dimensions Name=InstanceId,Value=i-xxx \
--statistics Average \
--start-time `date -u '+%FT%TZ' -d '10 mins ago'` \
--end-time `date -u '+%FT%TZ'` \
--period 60
{
    "Datapoints": [
        {
            "Timestamp": "2013-11-30T11:13:00Z", 
            "Average": 5.6659999999999995, 
            "Unit": "Percent"
        }, 
        {
            "Timestamp": "2013-11-30T11:18:00Z", 
            "Average": 6.6659999999999995, 
            "Unit": "Percent"
        }
    ], 
    "Label": "CPUUtilization"
}

Extracting average with jq command

$ aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 \
--metric-name CPUUtilization \
--dimensions Name=InstanceId,Value=i-xxx \
--statistics Average \
--start-time `date -u '+%FT%TZ' -d '10 mins ago'` \
--end-time `date -u '+%FT%TZ'` \
--period 60 | jq '.Datapoints[0] | .Average'
6.6659999999999995

Reference

Thursday, November 21, 2013

Monitoring tool for hyblid cloud and automating - installing Hyclops for Zabbix

The update version on this article is here, composed of Zabbix-2.2 and Hyclops-0.2.

Automation is definitely necessary, and let's say common for the cloud or virtualization environment, especially it saves us lots of time by automatically setting up the servers to be monitored.
I believe that Zabbix is one of the most famous open source monitoring software which has been popular for the recent years.
I am pretty sure that there are other monitoring softwares to achieve it, such as Hinemos, Hyperic HQ, Sensu, and so on.
Here, I tried installing Hyclops for Zabbix to achieve automatically registering the EC2 instances to be monitored on AWS.

This is how to install Hyclops for Zabbix based on the official installation manual with some comments added.

OS/Middeware/Application

OS CentOS 6.4

Kernel 2.6.32-358.6.1.el6.x86_64

Apache 2.2.15

MySQL 5.6.13

Zabbix 2.0.9

PHP 5.3.3

Hyclops 0.1.0

Python 2.6.6

python-setuptools 0.6.10

python-pip 1.3.1

ZeroMQ 3.2.2

GateOne 1.1.1

ipmitool 1.8.11

tornade 2.4.1

Python, Python packages

$ sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ sudo yum -y --enablerepo=epel install python python-libs python-devel python-setuptools python-pip dtach
$ sudo rpm -qa | egrep '^(python)' | sort
python-2.6.6-37.el6_4.x86_64
python-boto-2.13.3-1.el6.noarch
python-cheetah-2.4.1-1.el6.x86_64
python-configobj-4.6.0-3.el6.noarch
python-devel-2.6.6-37.el6_4.x86_64
python-ethtool-0.6-3.el6.x86_64
python-iniparse-0.3.1-2.1.el6.noarch
python-iwlib-0.1-1.2.el6.x86_64
python-libs-2.6.6-37.el6_4.x86_64
python-markdown-2.0.1-3.1.el6.noarch
python-pip-1.3.1-4.el6.noarch
python-pycurl-7.19.0-8.el6.x86_64
python-pygments-1.1.1-1.el6.noarch
python-setuptools-0.6.10-3.el6.noarch
python-urlgrabber-3.9.1-8.el6.noarch

ZeroMQ, ipmitools

$ curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
$ sudo yum -y install gcc gcc-c++ zeromq zeromq-devel ipmitool detach
$ rpm -qa | egrep '(zeromq|ipmitool)' | sort
ipmitool-1.8.11-14.el6_4.1.x86_64
zeromq-3.2.2-13.1.x86_64
zeromq-devel-3.2.2-13.1.x86_64

Python modules

$ sudo pip install apache-libcloud==0.13.2 zabbix-api pyzmq psphere python-daemon==1.6 configobj

ordereddict, argparse if phython is less than 2.7

$ sudo pip install futures ordereddict

tornade-2.4.1

$ sudo rpm -ivh https://github.com/downloads/liftoff/GateOne/tornado-2.4-1.noarch.rpm

tornade-2.4.1 is recommended because 3.x has some issues in attribute settings and it fails to run gateone with the message below. If pip is used to install tornade, tornade-3.x is installed.
Cf. https://github.com/liftoff/GateOne/issues/235

$ sudo /opt/gateone/gateone.py --auth=api --origins="http://zabbix-server-frontend/"
Traceback (most recent call last):
  File "/opt/gateone/gateone.py", line 289, in 
    tornado.options.enable_pretty_logging()
AttributeError: 'module' object has no attribute 'enable_pretty_logging'

PIL(Python Image Library)

$ sudo pip install PIL

boto if monitoring aws billing stats is needed

$ sudo pip install boto

installed python modules

$ pip list
apache-libcloud (0.13.2)
boto (2.5.2)
Cheetah (2.4.1)
cloud-init (0.6.3)
configobj (4.6.0)
distribute (0.6.10)
ethtool (0.6)
futures (2.1.5)
gateone (1.1)
hyclops (0.1.0)
iniparse (0.3.1)
iwlib (1.0)
lockfile (0.9.1)
Markdown (2.0.1)
ordereddict (1.1)
PIL (1.1.7)
psphere (0.5.2)
pycurl (7.19.0)
Pygments (1.1.1)
pygpgme (0.1)
python-daemon (1.6)
PyYAML (3.10)
pyzmq (14.0.0)
suds (0.4)
tornado (2.4)
urlgrabber (3.9.1)
yum-metadata-parser (1.1.2)
yum-presto (0.4.4)
zabbix-api (1.0)

GateOne

$ sudo rpm -ivh https://github.com/downloads/liftoff/GateOne/gateone-1.1-1.noarch.rpm
$ cp /opt/gateone/tests/chat/keyfile.pem /opt/gateone/

Modifying init script for GateOne to display the status function normally

$ sudo service gateone status
gateone.py is stopped
$ diff -u /etc/init.d/gateone{,.bak}
--- /etc/init.d/gateone 2013-11-20 16:31:22.585275659 +0900
+++ /etc/init.d/gateone.bak     2013-11-20 15:58:06.460275392 +0900
@@ -66,8 +66,7 @@
       stop
     ;;
   status)
-$    status gateone.py
-    status -p ${GATEONE_PID} python.py
+    status -p ${GATEONE_PID} ${0##*/}
     ;;
   restart)
       restart
$ sudo service gateone status
gateone (pid  3060) is running

Configuring GateOne Cf. http://liftoff.github.io/GateOne/About/index.html

$ sudo /opt/gateone/gateone.py --auth=api --origins="https://localhost;https://127.0.0.1;https://"
[I 131120 16:27:32 gateone:2917] Connections to this server will be allowed from the following origins: 'https://localhost https://127.0.0.1 https:// https://'
[I 131120 16:27:32 gateone:2305] Using api authentication
[I 131120 16:27:32 gateone:2404] Loaded plugins: bookmarks, convenience, example, help, logging, logging_plugin, mobile, notice, playback, ssh
[I 131120 16:27:32 gateone:3054] Listening on https://*:443/
[I 131120 16:27:32 gateone:3060] Process running with pid 3255
[I 131120 16:27:38 gateone:1063] WebSocket opened (ANONYMOUS).
[I 131120 16:27:41 web:1462] 200 GET /static/ubuntumono-normal.woff () 10.19ms
[I 131120 16:27:42 web:1462] 200 GET /static/bookmarks/images/star.svg () 1.08ms
Ctrl+c
[I 131120 16:28:52 gateone:3084] Caught KeyboardInterrupt.  Killing sessions...
[I 131120 16:28:52 gateone:3088] pid file removed.
[I 131120 16:28:52 termio:1491] Finalizing the log for pid 3260 (this can take some time).

$ sudo /opt/gateone/gateone.py --new_api_key
[I 131120 16:29:13 gateone:2880] A new API key has been generated: 
[I 131120 16:29:13 gateone:2881] This key can now be used to embed Gate One into other applications.
$ sudo chkconfig gateone on
$ sudo service gateone start

Installing & Setting up Hyclops

$ wget https://github.com/tech-sketch/hyclops/archive/0.1.0.tar.gz -O- | tar zxf -
$ cd hyclops-0.1.0/
$ sudo ./setup.py install
$ sudo cp -a ./misc/init.d/redhat/hyclops /etc/init.d/
$ sudo cp -a ./misc/init.d/ubuntu/hyclops.conf /etc/init/
$ sudo mkdir /etc/zabbix/externalscripts
$ sudo cp -a ./misc/externalscripts/* /etc/zabbix/externalscripts
$ sudo chown -R zabbix:zabbix /etc/zabbix/externalscripts
$ sudo chmod u+x /etc/zabbix/externalscripts/*
$ sudo chkconfig hyclops on
$ sudo service hyclops start

Replacing some Zabbix dashboard files

$ sudo ./setup.py replace -d /usr/share/zabbix --zabbix-version=2.0
running replace
backup original file /usr/share/zabbix/dashboard.php to /usr/share/zabbix/dashboard.org
create /usr/share/zabbix/dashboard.php
create /usr/share/zabbix/gateone.php
create /usr/share/zabbix/dashboard_scripts_exec.php
create /usr/share/zabbix/custom/additional_blocks.inc.php
create /usr/share/zabbix/custom/additional_blocks_func.inc.php
create /usr/share/zabbix/custom/additional_func.inc.php
create /usr/share/zabbix/custom/monitoring.dashboard.js.php
backup original file /usr/share/zabbix/include/menu.inc.php to /usr/share/zabbix/include/menu.inc.org
create /usr/share/zabbix/include/menu.inc.php
uid=500(hyclops) gid=501(hyclops) groups=501(hyclops)

It is possible to specify 2.2 with the command option, but it has not been actually implemented yet. ~~When I installed Zabbix-2.2 and specified 2.0 to replace the php files, the dashboard did not show because of some permission error or something.~~
This issue has been solved as Hyclops begins to support Zabbix-2.2.

$ sudo ./setup.py replace -d /usr/share/zabbix --zabbix-version=2.4
running replace
Not supported version (Supported only 2.0 or 2.2)
uid=501(hyclops) gid=501(hyclops) groups=501(hyclops)
$ ls ./misc/zabbix-custom/    
2.0

Importing Zabbix templates,scripts and globalmacro data

$ sudo ./setup.py import -u admin -p zabbix -f http://54.238.236.194/zabbix/
running import
Import templates
Import global scripts
  Create 'Start EC2 instance'
  Create 'Stop EC2 instance'
  Create 'Reboot EC2 instance'
  Create 'Start vSphere instance'
  Create 'Stop vSphere instance'
  Create 'Suspend vSphere instance'
  Create 'Reboot vSphere instance'
  Create 'IPMI power on'
  Create 'IPMI power off'
Import global macros
  Create '{$GATEONE_URL}'
  Create '{$GATEONE_KEY}'
  Create '{$GATEONE_SECRET}'
  Create '{$HYCLOPS_SERVER}'
  Create '{$HYCLOPS_PORT}'
uid=500(hyclops) gid=501(hyclops) groups=501(hyclops)

Setting cron script

$ sudo crontab -u hyclops -e
*/5 * * * * python /opt/hyclops/cron_scripts/delete_not_exist_hosts.py

Zabbix Macro/Host settings

Follow the steps at the official site to setup the macros for AWS access and secret key,
you will be able to see the instances registered at the Zabbix dashboard.
I just only tried AWS EC2 setting, not vSphere ESXi.

I'm looking forward to Hyclops to be updated for Zabbix-2.2!

Tuesday, June 11, 2013

Monitoring tool - installing zabbix

I've got some opportunities to use zabbix not only for monitoring but mainly calculating system availability. It is just memorandum for me this time, but after introducing how to install zabbix-2.x on CentOS-6.x, I'm going to try some functions such as IT Services and Discovery.

Overview of System Structure/Middleware

OS	CentOS release 6.4 (Final)
Kernel	2.6.32-358.6.2.el6.x86_64
Language	ja_JP.UTF-8

role	package
WEB	httpd-2.2.15
DB	mysql-5.6.10
Monitoring	zabbix-2.0.11
Scripting Language	php-5.3.3

Note: I fixed to install zabbix-2.0.11 by referring to the URL where zabbix-2.0.11 have been stored because the latest version is being installed without specifying the version.

Installing zabbix

* install mysql

# wget http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-shared-5.6.10-1.el6.x86_64.rpm/from/http://cdn.mysql.com/
# wget http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-client-5.6.10-1.el6.x86_64.rpm/from/http://cdn.mysql.com/
# wget http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-server-5.6.10-1.el6.x86_64.rpm/from/http://cdn.mysql.com/
# wget http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-server-5.6.10-1.el6.x86_64.rpm/from/http://cdn.mysql.com/
# wget  http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-shared-compat-5.6.11-2.el6.x86_64.rpm/from/http://cdn.mysql.com/rpm -ivh MySQL-shared-compat-5.6.11-2.el6.x86_64.rpm
# rpm -ev mysql*
# rpm -ivh MySQL-client-5.6.10-1.el6.x86_64.rpm MySQL-server-5.6.10-1.el6.x86_64.rpm MySQL-shared-5.6.10-1.el6.x86_64.rpm

* MySQL-5.6 sets the initial password during the installation process, which is shown in "~/.mysql_secret"

# cat ~/.mysql_secret
# The random password set for the root user at Tue Jun  4 23:39:00 2013 (local time): <password>

* install php

# rpm -ivh http://ftp-srv2.kddilabs.jp/Linux/distributions/fedora/epel/6/i386/epel-release-6-8.noarch.rpm
# rpm -ivh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
# yum -y install php php-bcmath php-gd php-mbstring php-xml php-mysql

* install libraries

# yum -y install --enablerepo=remi,epel fping iksemel iksemel-devel curl curl-devel openldap openldap-devel OpenIPMI OpenIPMI-devel net-snmp net-snmp-devel unixODBC unixODBC-devel

* install zabbix

# rpm -ivh http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-release-2.0-1.el6.noarch.rpm
# yum -y install \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-2.0.11-1.el6.x86_64.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-server-2.0.11-1.el6.x86_64.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-server-mysql-2.0.11-1.el6.x86_64.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-web-2.0.11-1.el6.noarch.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-web-japanese-2.0.11-1.el6.noarch.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-web-mysql-2.0.11-1.el6.noarch.rpm
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-agent-2.0.11-1.el6.x86_64.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-get-2.0.11-1.el6.x86_64.rpm \
http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/zabbix-sender-2.0.11-1.el6.x86_64.rpm

* create database for zabbix

# service httpd start
# service mysql start

# mysqladmin -u root -p create zabbix
# mysql -u root -p -e 'GRANT ALL PRIVILEGES on zabbix.* to zabbix@localhost IDENTIFIED BY "zabbix"; FLUSH PRIVILEGES;'
# mysql -u zabbix -p zabbix < /usr/share/doc/zabbix-server-mysql-2.0.11/create/schema.sql
# mysql -u zabbix -p zabbix < /usr/share/doc/zabbix-server-mysql-2.0.11/create/images.sql
# mysql -u zabbix -p zabbix < /usr/share/doc/zabbix-server-mysql-2.0.11/create/data.sql
# mysql -u zabbix -p zabbix -e 'SHOW TABLES;'
+-----------------------+
| Tables_in_zabbix    |
+-----------------------+
| acknowledges          |
| actions                     |
| alerts                       |
...
| users_groups          |
| usrgrp                      |
| valuemaps               |
+-----------------------+

# service zabbix-agent start
# service zabbix-server start

* proceed the initial setup in the zabbix web interface, http://<zabbix server>/zabbix/

After completing the setup, enter the username and the password and you can see the dashboard of zabbix.

That's all today!

My bookshelf on Goodreads

Naoya's bookshelf: read

Building a Home Security System with BeagleBone

by Bill Pretty

Getting Started with Phalcon

by Stephan A Miller

I had known nothing about the Phalcon PHP framework before reading "Getting Started with Phalcon", but I learned the basic step by step with the following chapters: * Chapter 1: Installing Phalcon * Chapter 2: Setting Up a Phalcon Projec...

Mastering Zabbix

by Andrea Dalle Vacche

Mastering Zabbix is quite helpful for the beginners or the intermediate those who need to manage and monitor IT infrastructure especially for mission critical environment or with hundreds of nodes because it covers database-sizing, capac...

hashnao's Blog

ページ

Saturday, November 30, 2013

AWS CLI memorandum - Getting CPU Metrics on CloudWatch

Listing metrics of AWS/EC2

Getting metrics of CPUUtilization

Extracting average with jq command

Thursday, November 21, 2013

Monitoring tool for hyblid cloud and automating - installing Hyclops for Zabbix

OS/Middeware/Application

OS CentOS 6.4

Kernel 2.6.32-358.6.1.el6.x86_64

Apache 2.2.15

MySQL 5.6.13

Zabbix 2.0.9

PHP 5.3.3

Hyclops 0.1.0

Python 2.6.6

python-setuptools 0.6.10

python-pip 1.3.1

ZeroMQ 3.2.2

GateOne 1.1.1

ipmitool 1.8.11

tornade 2.4.1

Python, Python packages

ZeroMQ, ipmitools

Python modules

ordereddict, argparse if phython is less than 2.7

tornade-2.4.1

PIL(Python Image Library)

boto if monitoring aws billing stats is needed

installed python modules

GateOne

Modifying init script for GateOne to display the status function normally

Configuring GateOne Cf. http://liftoff.github.io/GateOne/About/index.html

Installing & Setting up Hyclops

Replacing some Zabbix dashboard files

Importing Zabbix templates,scripts and globalmacro data

Zabbix Macro/Host settings

Tuesday, June 11, 2013

Monitoring tool - installing zabbix

iJAWS@Doorkeeper

AWS Certified SysOps Administrator - Associate Level

AWS Certified Solutions Architect Associate Level

OS	CentOS 6.4
Kernel	2.6.32-358.6.1.el6.x86_64
Apache	2.2.15
MySQL	5.6.13
Zabbix	2.0.9
PHP	5.3.3
Hyclops	0.1.0
Python	2.6.6
python-setuptools	0.6.10
python-pip	1.3.1
ZeroMQ	3.2.2
GateOne	1.1.1
ipmitool	1.8.11
tornade	2.4.1

ページ

Saturday, November 30, 2013

AWS CLI memorandum - Getting CPU Metrics on CloudWatch

Listing metrics of AWS/EC2

Getting metrics of CPUUtilization

Extracting average with jq command

Thursday, November 21, 2013

Monitoring tool for hyblid cloud and automating - installing Hyclops for Zabbix

OS/Middeware/Application

OSCentOS 6.4 Kernel2.6.32-358.6.1.el6.x86_64 Apache2.2.15 MySQL5.6.13 Zabbix2.0.9 PHP5.3.3 Hyclops0.1.0 Python2.6.6 python-setuptools0.6.10 python-pip1.3.1 ZeroMQ3.2.2 GateOne1.1.1 ipmitool1.8.11 tornade2.4.1

Python, Python packages

ZeroMQ, ipmitools

Python modules

ordereddict, argparse if phython is less than 2.7

tornade-2.4.1

PIL(Python Image Library)

boto if monitoring aws billing stats is needed

installed python modules

GateOne

Modifying init script for GateOne to display the status function normally

Configuring GateOne Cf. http://liftoff.github.io/GateOne/About/index.html

Installing & Setting up Hyclops

Replacing some Zabbix dashboard files

Importing Zabbix templates,scripts and globalmacro data

Zabbix Macro/Host settings

Tuesday, June 11, 2013

Monitoring tool - installing zabbix

iJAWS@Doorkeeper

OS CentOS 6.4

Kernel 2.6.32-358.6.1.el6.x86_64

Apache 2.2.15

MySQL 5.6.13

Zabbix 2.0.9

PHP 5.3.3

Hyclops 0.1.0

Python 2.6.6

python-setuptools 0.6.10

python-pip 1.3.1

ZeroMQ 3.2.2

GateOne 1.1.1

ipmitool 1.8.11

tornade 2.4.1