hashnao's Blog: Monitoring tool for hyblid cloud and automating

I used to write about how to install zabbix+hyclops to automatically register zabbix hosts on AWS EC2 instance before. Recently, I tried installing the latest version of zabbix-2.2.2 and hyclops-0.2.
In fact, it's not working as expected so far because it seems that hyclops.connector.ec2 (ec2.py) python script is not working correctly. I will update this article when finishing to fix the problem (asking the engineers who created hyclops to debug the issue currently).
After uninstalling apache-libcloud from 0.14.1 to 0.13.2, the issue has been resolved.
Ikeda-san, thanks for your quick feedback and support!

OS, Middle ware, Libraries

AMI ID (AWS EC2 Instance): ami-31e86030
OS: CentOS 6.5(x86_64)
Kernel: 2.6.32-358.6.1.el6.x86_64
Apache: 2.2.15
MySQL: 5.6.16
PHP: 5.3.3
Zabbix: 2.2.2
Hyclops: 0.2.0
ZeroMQ: 3.2.2
GateOne: 1.1.1
Python: 2.7.5
Python Modules

apache-libcloud ~~(0.14.1)~~ (0.13.2)
boto (2.27.0)
configobj (5.0.2)
distribute (0.6.35)
hyclops (0.2.1)
ipython (1.2.1)
lockfile (0.9.1)
pip (1.5.4)
psphere (0.5.2)
python-daemon (1.6)
PyYAML (3.10)
pyzmq (14.1.1)
setuptools (0.6c11)
six (1.5.2)
suds (0.4)
tornado (2.4.1)
wsgiref (0.1.2)
zabbix-api (1.0)

Upgrade to CentOS-6.5 from 6.4

# yum -y install httpd httpd-devel mod_ssl

MySQL-5.6

# yum -y install http://repo.mysql.com/mysql-community-release-el6.rpm
# yum -y install mysql mysql-devel mysql-embedded-devel mysql-libs mysql-server

Apache-2.2

# yum -y install httpd httpd-devel mod_ssl

PHP-5.3

# yum -y install php php-bcmath php-gd php-mbstring php-xml php-mysql

Libraries for Zabbix

# yum -y install http://ftp-srv2.kddilabs.jp/Linux/distributions/fedora/epel/6/x86_64/epel-release-6-8.noarch.rpm
# yum -y install http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
# yum -y install --enablerepo=remi,epel fping iksemel-devel curl-devel openldap-devel OpenIPMI-devel net-snmp-devel unixODBC-devel git wget

Zabbix 2.2

# yum -y install http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm
# yum -y install --enablerepo=zabbix zabbix zabbix-server-mysql zabbix-web zabbix-web-mysql zabbix-web-japanese zabbix-agent zabbix-get zabbix-sender

ZeroMQ

# curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
# yum -y install gcc gcc-c++ zeromq zeromq-devel ipmitool detach

Createing Zabbix schema

# for i in schema images data
do mysql -u zabbix -p zabbix < /usr/share/doc/zabbix-server-mysql-2.2.2/create/$i.sql
done
# mysqladmin -u root -p create zabbix
# mysql -u root -p -e 'GRANT ALL PRIVILEGES on zabbix.* to zabbix@localhost IDENTIFIED BY "zabbix"; FLUSH PRIVILEGES;'
# mysql -u zabbix zabbix -e 'SHOW TABLES;'
+-----------------------+
| Tables_in_zabbix |
+-----------------------+
| acknowledges |
| actions |
| alerts |
...
| users_groups |
| usrgrp |
| valuemaps |
+-----------------------+
# service zabbix-server start

Python-2.7, Python packages

# wget -O- https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz | tar zxf -
# cd Python-2.7.5/
# /configure && make && make install
# wget -O- http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz | tar zxf -
# cd distribute-0.6.35/
# python2.7 setup.py install
# easy_install-2.7 pip

Python modules

# pip install zabbix-api pyzmq psphere python-daemon==1.6 configobj tornado==2.4.1 boto

Python module (apache-libcloud)
Make sure that apache-libcloud-0.13.2 should be compiled and installed from tar ball because it failed to install apache-libcloud-0.13.2 with python-2.7 via pip.

# wget -O- https://pypi.python.org/packages/source/a/apache-libcloud/apache-libcloud-0.13.2.tar.gz#md5=d3d127bb21ee73ca5b30bb367949c898 | tar zxf -
# cd apache-libcloud-0.13.2/
# python setup.py install
# pip list | grep apache-libcloud
apache-libcloud (0.13.2)

# pip install apache-libcloud==0.13.2
Downloading/unpacking apache-libcloud==0.13.2
Downloading apache-libcloud-0.13.2.tar.bz2 (396kB): 396kB downloaded
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1197, in prepare_files
do_download,
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1375, in unpack_url
self.session,
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/download.py", line 582, in unpack_http_url
unpack_file(temp_location, location, content_type, link)
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 625, in unpack_file
untar_file(filename, location)
File "/usr/local/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 543, in untar_file
tar = tarfile.open(filename, mode)
File "/usr/local/lib/python2.7/tarfile.py", line 1678, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/local/lib/python2.7/tarfile.py", line 1744, in bz2open
raise CompressionError("bz2 module is not available")
CompressionError: bz2 module is not available

Storing debug log for failure in /root/.pip/pip.log

Gateone

# yum -y install https://github.com/downloads/liftoff/GateOne/gateone-1.1-1.noarch.rpm
# cp /opt/gateone/tests/chat/keyfile.pem /opt/gateone/

Replacing some Zabbix dashboard files

# git clone https://github.com/tech-sketch/hyclops.git
# cd hyclops/
# python ./setup.py install
# cp -a ./misc/init.d/redhat/hyclops /etc/init.d/
# cp -a ./misc/init.d/ubuntu/hyclops.conf /etc/init/
# externalscripts=/usr/lib/zabbix/externalscripts
# mkdir -p ${externalscripts}
# cp -a ./misc/externalscripts/* ${externalscripts}
# chown -R zabbix:zabbix ${externalscripts}
# chmod u+x ${externalscripts}/*

I changed the python script for get_aws_charges.py and push_message.py scripts are changed.

# sed -i "s|#!.*|#!/usr/local/bin/python|" ${externalscripts}/*.py

# chkconfig hyclops on
# service hyclops start
# python ./setup.py replace -d /usr/share/zabbix --zabbix-version=2.2
running replace
backup original file /usr/share/zabbix/dashboard.php to /usr/share/zabbix/dashboard.org
create /usr/share/zabbix/dashboard.php
create /usr/share/zabbix/gateone.php
create /usr/share/zabbix/dashboard_scripts_exec.php
backup original file /usr/share/zabbix/include/menu.inc.php to /usr/share/zabbix/include/menu.inc.org
create /usr/share/zabbix/include/menu.inc.php
create /usr/share/zabbix/custom/additional_blocks_func.inc.php
create /usr/share/zabbix/custom/monitoring.dashboard.js.php
create /usr/share/zabbix/custom/additional_func.inc.php
create /usr/share/zabbix/custom/additional_blocks.inc.php

uid=500(hyclops) gid=500(hyclops) groups=500(hyclops)

Importing Zabbix templates,scripts and globalmacro data

# python ./setup.py import -u admin -p zabbix -f http://zabbix-server front end ip/zabbix/
running import
Import templates
Import global scripts
Create 'Start EC2 instance'
Create 'Stop EC2 instance'
Create 'Reboot EC2 instance'
Create 'Start vSphere instance'
Create 'Stop vSphere instance'
Create 'Suspend vSphere instance'
Create 'Reboot vSphere instance'
Create 'IPMI power on'
Create 'IPMI power off'
Import global macros
Create '{$GATEONE_URL}'
Create '{$GATEONE_KEY}'
Create '{$GATEONE_SECRET}'
Create '{$HYCLOPS_SERVER}'
Create '{$HYCLOPS_PORT}'
uid=500(hyclops) gid=501(hyclops) groups=501(hyclops)

~~After creating value mappings and creating zabbix hosts on zaxbbix dashboard, I got the error message below in fact. I need to dive into the logging...~~
This issue has been solved by downgrading apache-libcloud from 0.14.1 to 0.13.2.

# tail -f /opt/hyclops/logs/hyclops_server.log
[2014-03-25 16:30:17,416] hyclops.connector.ec2 (ec2-For AWS EC2-monitor) ERROR: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 46, in __call__
result = self.run_command(hostname, params)
File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 62, in run_command
result = self.monitor(hostname, conn_params)
File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 126, in monitor
self.set_ami_info(node)
File "/usr/local/lib/python2.7/site-packages/hyclops-0.2.1-py2.7.egg/hyclops/connector/ec2.py", line 175, in set_ami_info
images = node.driver.list_images(ex_image_ids=[node.extra["imageId"]])
KeyError: 'imageId'

See zabbix_server.log to confirm the both python scripts are working to get aws charges and push messages.

# tail -f /var/log/zabbix/zabbix_server.log
t 3704:20140326:110510.922 item [For AWS EC2:get_aws_charges.py[{$KEY},{$SECRET}]] became supported
3704:20140326:110510.922 item [For AWS EC2:push_message.py[{$HYCLOPS_SERVER},{$HYCLOPS_PORT},ec2,{HOST.HOST}]] became supported

See hyclops_server.log to confirm that polling and creating zabbix hosts have been successful.
* You need to change "log_level = DEBUG" on /opt/hyclops/hyclops.conf to debug.

# tail -f /opt/hyclops/logs/hyclops_server.log
[2014-03-26 10:59:23,326] hyclops.queue (MainThread) INFO: Start message queue
[2014-03-26 10:59:23,326] hyclops (MainThread) INFO: Message queue is opened
[2014-03-26 10:59:23,326] hyclops.queue (MainThread) DEBUG: polling...
[2014-03-26 11:05:08,612] hyclops.queue (MainThread) DEBUG: recieved request: [['ec2', 'For AWS EC2']]
[2014-03-26 11:05:08,613] hyclops.queue (MainThread) DEBUG: run ec2-For AWS EC2-monitor thread
[2014-03-26 11:05:08,614] hyclops.queue (MainThread) DEBUG: running threads: [{'ec2-For AWS EC2-monitor': True}]
[2014-03-26 11:05:11,617] hyclops.queue (MainThread) DEBUG: polling...
...
[2014-03-26 11:05:43,452] hyclops.connector.ec2 (ec2-For AWS EC2-monitor) DEBUG: Create zabbix host i-xxxxxxx (For AWS EC2_i-xxxxxxx_i-xxxxxxx)
...

[2014-03-26 11:05:59,960] hyclops.connector.ec2 (ec2-For AWS EC2-monitor) DEBUG: success. thread finished.