Devops trends from MongoDB, Grafana and spammed mailbox. Update #1

Hi all!

this is the first update from my new Big data experimental project…

How can I be updated with devops trend technologies without reading tons of blog?

I started to collect raw data from Twitter timeline and Gmail spammed mailbox that receives messages from websites for job seekers and others like Stack Exchange…

This is the result and the first day of data collecting…I am curious to see what happens with tons of data and some adjustments…

Link to Grafana dashboard

http://www.congruit.io/

screen-shot-2017-03-06-at-11-11-20

Advertisements

Install Chef Server on Suse Linux Enterprise 11

Hi Folks!

Today I dealt with a problem… and I found a solution because Chef is a great tool!

At moment there is not an RPM for Suse Linux available from the official website, but this does not matter 🙂

Problem: Install Chef Server, Chefdk, Chef-manage into a Suse Linux Enterprise 11 virtual machine without installing the rpm packages of RHEL systems.

Screen Shot 2016-08-02 at 19.02.17.png

This is what you can do:

  1. Dowload the following packages:
    • chef-server-core-12.8.0-1.el6.x86_64.rpm,
    • chefdk-0.16.28-1.el6.x86_64.rpm,
    • chef-manage-2.4.1-1.el6.x86_64.rpm
  2. Extract all stuff from RPM with:
    • rpm2cpio  chef-manage-2.4.1-1.el6.x86_64.rpm   | cpio -idmv

  3. Move content of the extraction to the correct folders: /opt/{chef,chef-manage,opscoode}
  4. Set PATH=”/opt/opscode/bin:/opt/chefdk/bin/:/data/opt/chef-manage/bin:$PATH” in your profile login script
  5. chef-server-ctl reconfigure
  6. chef-manage-ctl reconfigure
  7. again chef-server-ctl reconfigure

At the end all services are up and running

Screen Shot 2016-08-02 at 19.12.35.png

and my workstation too 🙂

Autoscaling with EC2 and Chef

Dear all,

It has been a long time since my last post and here I am with a new one, just to keep track of my current study case…

I would like to put in place an auto-scaling mechanism for my lab platform.

Currently I have one Ha-Proxy load balancer with 2 backends. I will perform stress test on my front-end  with Jmeter and create automatically a virtual machine joined to my Chef infrastructure in order to increase resources.

In this post I will describe just how to set  up an initial configuration of autoscaling-group + Chef ( today it is Friday… on Monday I will do the rest 😉

Let’s start  with the needed components:

  1. a Chef server
  2. one HaProxy load balancer
  3. two tomcat backend

Now I try the script for the unattended bootstrap. This script adds a new node under the Chef Server. I tried it on a simple virtual machine locally, using a Centos 7 running in Virtualbox.

[ ! -e /etc/chef ] && mkdir /etc/chef

cat <<EOF > /etc/chef/validation.pem
-----BEGIN RSA PRIVATE KEY-----
your super secret private key :)
-----END RSA PRIVATE KEY-----
EOF

cat <<EOF > /etc/chef/client.rb
log_location STDOUT
chef_server_url "https://mychefserver.goofy.goober/organizations/myorg"
ssl_verify_mode :verify_none
validation_client_name "myorg-validator"
EOF

cat <<EOF > /etc/chef/first-boot.json


{
 "run_list": ["role[tomcat_backend]"]
}


EOF

curl -L https://www.opscode.com/chef/install.sh | \
bash -s -- -v 12.9.41 &> /tmp/get_chef.log
chef-client -E amazon_demo -j /etc/chef/first-boot.json  \
&> /tmp/chef.log 


If things have done correctly you will see the new node into your Chef server dashboard..Check the logs on the new node in case of problems..

/tmp/chef.log
/tmp/get_chef.log

Now let’s create the autoscaling-group in Amazon EC2

Screen Shot 2016-05-06 at 13.43.35.png

Then select your preferred instance… I am using RHEL 7.2

Screen Shot 2016-05-06 at 13.44.48.png

Insert the bootstrap script “User data file” (the one we just created)

Screen Shot 2016-05-06 at 13.49.17.png

I have no instances running on my cloud, so the following configuration will generate a virtual machine due to the min required is 1.

Screen Shot 2016-05-06 at 17.42.56.png

After a minute I got an email saying:

Description: Launching a new EC2 instance: $my_id_istance
Cause: At 2016-05-06T15:10:17Z an instance was started in response to a 
difference between desired and actual

Finally I have a new configured node in my Chef server.. . which is the autoscaling_node01.

Screen Shot 2016-05-06 at 16.00.44.png

That’s all folks!

Bye for now…

Eugenio Marzo
DevOps Engineer at SourceSense

 

Build chef LWRP and manage OpenSSH server banner with Chef

Hi guys,
in this article we will se how to build a small LWRP Chef cookbook..The final result will be:

ssh_banner_banner “banner” do
banner_file _banner_file
sshd_config_file node[‘ssh_banner’][‘sshd_config_file’]
paranoic_mode true
action :create
notifies :restart, “service[sshd]”
end


If “paranoic mode” is true, chef will change configuration file and restart sshd, but after 20 seconds (by default) it will do a rollback of configuration

You can try it use Vagrant and Virtualbox..

1. clone git repo from github:

  git clone https://github.com/EugenioMarzo/cookbook-ssh-banner.git

2. show the new banner to copy:

 cat files/default/chef_ssh_banner

3. start vagrant virtual machine:

  vagrant up

4. once the deploy is completed:

Screen Shot 2014-09-01 at 16

Let’s see how to create a simple LWRP:

1. Declare variables in resources/banner.rb

actions :create, :delete

default_action :create

attribute :sshd_config_file, :kind_of => String

attribute :banner_file, :kind_of => String

attribute :paranoic_mode

2. create an action in providers/banner.rb.. Let’s see the :delete function :

action :delete do
#check if ssh banner file is present
check_banner_file new_resource.banner_file
#check if paranoic mode is enabled
paranoic_mode

if ::File.open(new_resource.sshd_config_file).grep(/Banner\ .*/).size >= 1
Chef::Log.info(“Deleting SSH Banner..”)
execute ” sed -i s/Banner\\\ .*//g #{new_resource.sshd_config_file}”

#the next function will inform that the state is changed, an action has been done. This is important because after this will be executed a notify action like a sshd restart

new_resource.updated_by_last_action(true)
else
Chef::Log.info(“SSH Banner not found … doing nothing..”)
new_resource.updated_by_last_action(false)

end

end

3. use it in a recipe.. Delete a banner:

ssh_banner_banner “banner” do
banner_file _banner_file
sshd_config_file node[‘ssh_banner’][‘sshd_config_file’]
paranoic_mode false
action :delete
notifies :restart, “service[sshd]”
end

4. for adding a banner use:

ssh_banner_banner “banner” do
banner_file _banner_file
sshd_config_file node[‘ssh_banner’][‘sshd_config_file’]
paranoic_mode false
action :create
notifies :restart, “service[sshd]”
end

Setting up Wildfly8 Cluster in 5 minutes with Chef and Vagrant

Hi! This is my first post of this blog and I would like start with my last cookbook “wildfly-clu”.   We will create a simple Wildlfy cluster(domain mode) composed by 3 servers. For this test I will use CentOS release 6.3 (Final).

Final result:

You can reach the app directly from the nodes:
http://myserver1:8080/helloworld/
http://myserver2:8080/helloworld/
http://myserver3:8080/helloworld/

WILDLFYCONS

haproxy

 

Quick HowTo:
check if the vagrant-berkshelf plugin is installed. If not, launch ” vagrant plugin install vagrant-berkshelf”
git clone https://github.com/EugenioMarzo/cookbook-wildfly-clu
cd wildfly-clu
vagrant up

Detailed Description:

Node1 (myserver1) => Domain controller –  Application Server – Reverse proxy

Node2 (myserver2) => Slave – Application Server

Node3 (myserver3) => Slave – Application Server

Prerequisites:  Virtualbox 4.3.10 ,Vagrant 1.4.3 , Ruby 1.9.3,Git

Let’s start..

1. Configure your /etc/hosts in order to resolve locally the name of all VMs
33.33.33.11 myserver1
33.33.33.13 myserver3
33.33.33.12 myserver2

2.  Clone the cookbook
`git clone https://github.com/EugenioMarzo/cookbook-wildfly-clu`

3. Quick overview of Vagrantfile
Vagrant can configure multiple virtual-machines. An example of ./mycookbook/Vagrantfile:
#This will be the configuration for myserver1. Being a cluster you will have the same configuration for myserver2 and myserver3

config.vm.define "myserver1" do |myserver1|

myserver1.vm.hostname = “myserver1”
myserver1.vm.network :private_network, ip: “33.33.33.11”
myserver1.vm.network :public_network
myserver1.vm.provision :chef_solo do |chef|
chef.json = {
:java => {:jdk_version => “7”}

}

chef.run_list = [
“recipe[java]”,  #to install java
“recipe[wildfly-clu::default]”,  #install wildfly
“recipe[wildfly-clu::logs]”, #log rotation
“recipe[wildfly-clu::domain]” # when is in run_list configure the domain mode

]
end
end

3. show the VMs configured in Vagrantfile

check if the vagrant-berkshelf plugin is installed. If not, launch ” vagrant plugin install vagrant-berkshelf”
root@myclient1:~/vagrantlab/wildfly-clu# vagrant status
Current machine states:

myserver1 not created (virtualbox)
myserver2 not created (virtualbox)
myserver3 not created (virtualbox)

 

4. a quick overview of the most important attributes of the cookbook

Version and URL of Wildfly8
default['wildfly-clu']['wildfly']['version'] = "8.0.0"
default['wildfly-clu']['wildfly']['url']="http://download.jboss.org/wildfly/8.0.0.Final/wildfly-8.0.0.Final.tar.gz"

#######################################################################
## Set the following variable to true if you want use the domain mode.
default[‘wildfly-clu’][‘mode’][‘domain’] = true
##
#######################################################################

#if you create this file the recipe will not change domain.xml,host.xml and mgmt-******.properties after the first installation
default[‘wildfly-clu’][‘wildfly’][‘lock’] = “/usr/local/#{node[‘wildfly-clu’][‘name’]}/conf.lock”

The cluster schema

default[‘wildfly-clu’][‘cluster_schema’] = {
“myserver1” => { :role => “domain-controller” ,
:ip => “33.33.33.11”,
:port_offset => “0” },
“myserver2” => { :role => “slave” ,
:ip => “33.33.33.12” ,
:master => “myserver1” ,
:port_offset => “0”},
“myserver3” => { :role => “slave” ,
:ip => “33.33.33.13” ,
:master => “myserver1” ,
:port_offset => “0” }

}

#set this to true in order to deploy an helloworld application
default[‘wildfly-clu’][‘wildfly’][‘deploy_hello_world’] = true

#set this to true in order to configure an haproxy with the slaves declared in the cluster_schema
default[‘wildfly-clu’][‘wildfly’][‘haproxy’] = true

#DEFAULT java options to use in all slaves and the master for run the application.
default[‘wildfly-clu’][‘java_opts’] = {
“heap-size” => “64m”,
“max-heap-size” => “64m”,
“permgen-size” => “64m”,
“max-permgen-size” => “64m” }

5. Setting up the test environment with Vagrant

root@myclient1:~/vagrantlab/wildfly-clu# vagrant up

Bringing machine ‘myserver1’ up with ‘virtualbox’ provider…
Bringing machine ‘myserver2’ up with ‘virtualbox’ provider…
Bringing machine ‘myserver3’ up with ‘virtualbox’ provider…

#downloading the virtualbox machine used as template
[myserver1] Importing base box ‘Berkshelf-CentOS-6.3-x86_64-minimal’…
Progress: 90%

[myserver1] Available bridged network interfaces:
1) eth0
2) virbr0
3) lxcbr0
4) virbr1
#Choose 1 if you want bridge the network cards of the VM to eth0

[myserver1] Booting VM…
[myserver1] Waiting for machine to boot. This may take a few minutes…
[myserver1] Machine booted and ready!
[myserver1] Configuring and enabling network interfaces.

Through Chef-Solo will be configured the VMs just created. Below the most important steps
Running chef-solo…
[2014-04-03T13:25:55+00:00] INFO: *** Chef 10.14.2 ***
[2014-04-03T13:26:01+00:00] INFO: Run List is

, recipe[wildfly-clu::default], recipe[wildfly-clu::logs], recipe[wildfly-clu::domain]]
[2014-04-03T13:26:08+00:00] INFO: package[java-1.7.0-openjdk] installing java-1.7.0-openjdk-1.7.0.51-2.4.4.1.el6_5 from upda
[2014-04-03T13:26:35+00:00] INFO: package[java-1.7.0-openjdk-devel] installing java-1.7.0-openjdk-devel-1.7.0.51-2.4.4.1.el6_5 from updates repository

#Downloading Wildfly..
[2014-04-03T13:26:46+00:00] INFO: user[wildfly] created
[2014-04-03T13:27:00+00:00] INFO: remote_file[wildfly] updated
[2014-04-03T13:27:00+00:00] INFO: remote_file[wildfly] owner changed to 502
[2014-04-03T13:27:00+00:00] INFO: remote_file[wildfly] group changed to 503
[2014-04-03T13:27:00+00:00] INFO: remote_file[wildfly] mode changed to 775
[2014-04-03T13:27:00+00:00] INFO: remote_file[wildfly] sending run action to bash[wildfly_extract] (immediate)
[2014-04-03T13:27:01+00:00] INFO: bash[wildfly_extract] ran successfully
[2014-04-03T13:27:01+00:00] INFO: bash[wildfly_extract] sending create action to link[/usr/local/wildfly] (immediate)
[2014-04-03T13:27:01+00:00] INFO: link[/usr/local/wildfly] created
[2014-04-03T13:27:01+00:00] INFO: link[/usr/local/wildfly] sending create action to template[/etc/default/wildfly.conf] (immediate)

# Copy configuration read by init script
[2014-04-03T13:27:01+00:00] INFO: template[/etc/default/wildfly.conf] updated content

#Copy configuration for domain mode
[2014-04-03T13:27:06+00:00] INFO: template[/usr/local/wildfly/domain/configuration/domain.xml] mode changed to 775

#installing Haproxy
[2014-04-03T13:27:06+00:00] INFO: package[haproxy] installing haproxy-1.4.24-2.el6 from base repository

#deploy Hello world!
[2014-04-03T13:27:15+00:00] INFO: cookbook_file[helloworld.war] sending run action to bash[deploy_helloworld] (delayed)
[2014-04-03T13:27:19+00:00] INFO: bash[deploy_helloworld] ran successfully
[2014-04-03T13:27:19+00:00] INFO: Chef Run complete in 77.838215277 seconds

The same operation will be for each virtual machines described in Vagrantfile.

 

6. a quick overview inside the virtual machines:

#use it to connect via SSH
root@myclient1:~/vagrantlab/wildfly-clu# vagrant ssh myserver1

[vagrant@myserver1 ~]$ sudo su
[root@myserver1 vagrant]# cat /etc/redhat-release
CentOS release 6.3 (Final)

[root@myserver1 vagrant]# /etc/init.d/wildfly status
wildfly is running (pid 5032)

#cron job for rotate logs
[root@myserver1 vagrant]# cat /var/spool/cron/root
# Chef Name: Wildfly log rotation 0
0 0 * * * find /usr/local/wildfly/domain/log -name ‘*’ -a ! -name ‘*.gz’ -mtime +1 -a ! -name ‘console.log’ -a ! -name ‘boot.log’ -exec gzip ‘{}’ ;
# Chef Name: Wildfly log rotation 1
0 0 * * * find /usr/local/wildfly/domain/log -name ‘*.txt.gz’ -mtime +30 -exec rm -f ‘{}’ ;

[root@myserver1 vagrant]# cat /etc/default/wildfly.conf
export JBOSS_USER=wildfly
export JBOSS_HOME=/usr/local/wildfly
export JBOSS_CONSOLE_LOG=/usr/local/wildfly/domain/log/console.log
export JBOSS_MODE=”domain”

Posted 3th April by Eugenio Marzo