Introduction to DevOps with Ansible

Matteo Magni

http://bit.ly/devops-ansible

In informatica DevOps (dalla contrazione inglese di development, "sviluppo", e operations, qui simile a "messa in produzione" o "deployment") è una metodologia di sviluppo del software che punta alla comunicazione, collaborazione e integrazione tra sviluppatori e addetti alle operations dell'information technology (IT)

Con il termine “Metodologie Agili” ci si riferisce ad una serie di metodologie di sviluppo software ispirate dal “Manifesto Agile”, impiegate per superare i limiti emersi dal modello tradizionale “a cascata” (waterfall).

Tra le pratiche Agili una è quella di avere frequenti rilasci di nuove versioni del software:

An Agile team frequently releases its product into the hands of end users, listening to feedback, whether critical or appreciative..

Rilasci

• In molte aziende però i rilasci applicativi sono eventi ad alto impatto e rischio, coinvolgendo più gruppi di lavoro.
• L’obiettivo è mettere in produzione nuove feature o correzioni di bug il più velocemente possibile e con il minor numero di rischi è molto difficile

• I dev spingono per mettere subito online le nuove feature
• i QA inoltrano
• gli operation sono molto prudenti perché sanno che toccare l’infrastruttura potrebbe creare problemi.

Come possiamo fare?

• Sviluppare e testare in ambienti simili a quelli di produzione
• Distribuire con processi automatici, ripetibili e affidabili
• Monitorare e convalidare la qualità operativa
• Amplificare e accelerare il ritorno di feedback da parte degli utenti

Continuous integration e testing

Continuous delivery e deployment

Continuous operations

Continuous assessment

L’utilizzo di pratiche DevOps spesso parte dal bisogno di poter creare, gestire, condividere e replicare configurazioni di ambienti complessi

Se pensiamo ad una infrastruttura web come ad esempio lo stack LAMP in una soluzione dove abbiamo più ambienti

• Development
• Testing
• Staging
• Production

• Quando devo aggiungere una nuova dipendenza di infrastruttura?
• Un nuovo pacchetto?
• Un nuovo servizio?

La metodologia DevOps aiuta le aziende nella gestione dei rilasci, standardizzando gli ambienti di sviluppo.

Le aziende con problemi di automazione dei rilasci solitamente hanno già un processo automatico in essere ma lo vorrebbero più flessibile e controllabile.

Idealmente tale automazione potrebbe essere utilizzata anche da risorse non operative (non appartenenti all'IT Operations) su ambienti non di produzione. In questo modo gli sviluppatori hanno a disposizione un maggiore controllo degli ambienti, dando all'infrastruttura una visione più incentrata sull'applicazione.[wikipedia]

Come sviluppatori siamo abituati a gestire diverse dipendenze tra componenti software (classi, package e librerie)

Ma anche le infrastrutture hanno delle dipendenze!

• File e Directory
• Utenti
• Software e loro configurazione

Possiamo usare un software per il controllo di versione e sfruttarne tutte le potenzialità.

• serve qualcuno che esegua le operazioni ripetitive
• 

Ansible

https://www.ansible.com

github.com/ansible/ansible

GNU GENERAL PUBLIC LICENSE
Version 3

$ deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main
$ sudo apt-get update
$ sudo apt-get install ansible

$ sudo dnf -y install ansible

$ pip install ansible

Dipendenze

• Python
• ssh

YAML

galaxy_info:
author: Matteo Magni
description: just another geek
license: license Apache 2.0
platforms:
  - name: Debian
    versions:
    - jessie
categories:
  - system
dependencies: []

Inventory

[databases]
stp.ilbonzo.net     ansible_ssh_user=scott

[webservers]
aic.ilbonzo.net     ansible_ssh_user=layne

Tasks

- name: add group devel
  group: name=devel state=present
- name: Apache | install apache
  apt: name=apache2 state=latest
  

Variables

user: vagrant
group: www-data
document_root: /var/www/officina-linux.it

Handlers

- name: restart postgresql
  service: name=postgresql state=restarted
- name: stop postgresql
  service: name=postgresql state=stopped
- name: start postgresql
  service: name=postgresql state=started
  

Roles

╰─➤  ls -l database
  total 0
  drwxr-xr-x  10 bonzo  staff  340 Dec  6 10:45 files
  drwxr-xr-x  10 bonzo  staff  340 Dec  6 10:45 handlers
  drwxr-xr-x  10 bonzo  staff  340 Dec  6 10:45 tasks
  drwxr-xr-x   5 bonzo  staff  170 Dec  6 10:45 templates
  

Playbook

- hosts: all
  sudo: true
  handlers:
    - include: handlers/main.yml
  pre_tasks:
    - name: update apt cache
      shell: apt-get update
    - name: upgrade apt
      shell: apt-get upgrade -y
  roles:
    - webserver
    - database
- include: nfs.yml

Ansible Galaxy

Galaxy role

Iniziamo a provare

Virtualizzare può essere d'aiuto per sperimentare e non solo

• Ci serve qualcuno che ci metta a disposizione semplicemente delle Virtual Machine.
• Potrebbe essere un qualsiasi sistema di cloud.
• In sviluppo possiamo usare Vagrant

Vagrant

Provider

• Virtualbox

• VMware

• Hyper-V

• Docker

commands

• $ vagrant init

• $ vagrant up

• $ vagrant halt

• $ vagrant reload

• $ vagrant destroy

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "base"
end

Vagrant Box

https://www.vagrantup.com/intro/getting-started/boxes.html

Vagrant Box

$ vagrant box list
$ vagrant box add debian/jessie64

Vagrant Box

Vagrant.configure("2") do |config|
  config.vm.box = "debian/jessie64"
end

Vagrant Networking

config.vm.hostname = "rocannon"
config.vm.network :forwarded_port, guest: 80, host: 8080
config.vm.network "private_network", ip: "192.168.33.66"

Vagrant ssh login

$ vagrant ssh

Testing port forwarding

$ vagrant ssh
$ echo pippo42 > index.html
$ sudo python -m SimpleHTTPServer 80

Attivare ssh "normale"

config.ssh.forward_agent

If true, agent forwarding over SSH connections is enabled. Defaults to false.

config.ssh.forward_agent = true

Debian login ssh con password

$ sudo vi /etc/ssh/sshd_config

PasswordAuthentication yes

$ sudo service ssh restart

ora possiamo entrare nella macchina in ssh "classico"

ssh vagrant@192.168.33.66

Provider configuration

Virtualbox

config.vm.provider "virtualbox" do |v|
  v.name = "rocannon-debian"
  v.memory = 512
  v.cpus = 2
end

Provisioning

Ansible

config.vm.provision "shell",
  inline: "sudo apt-get -y update && sudo apt-get -y install lynx"
  

Run provisioning with vagrant

$ vagrant up #first time when create a machines

$ vagrant up --provision

$ vagrant provision

Provisioning

Ansible

config.vm.provision :ansible do |ansible|
  ansible.groups = {
    "development" => ["default"],
    "all_groups:children" => ["development"]
    
  }
  ansible.playbook = "ansible/playbook.yml"
end

First playbook

- hosts: all
  become: true
  pre_tasks:
    - name: update apt cache
      command: apt-get -y update
      
  tasks:
    - name: install lynx
      command: apt-get install -y lynx
      
  post_tasks:
    - name: hostname
      command: hostname
      

Provisioning

with Ansible on windows

## if you don't want ansible on host
config.vm.provision :shell, :path => "ansible/provision.sh"

Provisioning

Ansible on windows

#!/bin/bash
sudo apt-get update
sudo apt-get install -y python-pip python-dev python-pycurl
sudo pip install ansible
sudo mkdir -p /etc/ansible
printf '[vagrant]\nlocalhost\n' | sudo tee /etc/ansible/hosts > /dev/null
echo Running provisioner: ansible
PYTHONUNBUFFERED=1 ansible-playbook -c local /workspace/tools/ansible/playbook.yml

Version Control System

(VCS)

Git for versioning

Our git repository

Inventory

• static inventory
• dynamic inventory

Static Inventory

[databases]
stp.ilbonzo.net     ansible_user=scott

[webservers]
aic.ilbonzo.net     ansible_user=layne

Inventory list

$ ansible --list-hosts web
  hosts (1):
    ilbonzo.net

# linux
$ /etc/ansible/hosts
# osx
$ /usr/local/etc/ansible/hosts

Local inventory

$ ansible -i hosts --list-hosts all
  hosts (4):
    web01
    web02
    db01
    db02
    

Subset

$ ansible -i hosts --list-hosts webserver
  hosts (4):
    web01
    web02
    
$ ansible -i hosts --list-hosts webserver:dbserver
  hosts (4):
    web01
    web02
    db01
    db02
    

Local inventory

ansible_user=vagrant
ansible_port=2606
ansible_ssh_private_key_file=~/.ssh/id_rsa

Vagrant local inventory

# Generated by Vagrant
default ansible_ssh_host=127.0.0.1
  ansible_ssh_port=2222
  ansible_ssh_user='vagrant'
  ansible_ssh_private_key_file=
    '~/.vagrant/machines/default/virtualbox/private_key'
    
[development]
default

[all_groups:children]
development

Running command

$ ansible -i ansible/hosts -m ping all -k
    SSH password:
    192.168.33.66 | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    
$ ansible -i ansible/hosts -m command -a "hostname" all -k
    SSH password:
    192.168.33.66 | SUCCESS | rc=0 >>
    rocannon
    
$ ansible -i ansible/hosts -a "hostname" all -k
    SSH password:
    192.168.33.66 | SUCCESS | rc=0 >>
    rocannon
    

Vagrantfile

config.vm.provision :ansible do |ansible|
  ansible.groups = {
    "development" => ["default"],
    "all_groups:children" => ["development"]
    
  }
  ansible.playbook = "ansible/playbook.yml"
end

Playbook

---
- hosts: all
  tasks:
    - name: install lynx
      command: apt-get install -y linux
    - command: hostname
    

Running playbook

$ ansible-playbook ansible/playbook.yml -i hosts --ask-sudo-pass --ask-pass

Modules

Modules (also referred to as “task plugins” or “library plugins”) are the ones that do the actual work in ansible, they are what gets executed in each playbook task. But you can also run a single one using the ‘ansible’ command.

Packaging: apt

Manages apt packages (such as for Debian/Ubuntu).

module apt

pre_tasks:
  - name: update apt cache
    apt: update_cache=yes
    
tasks:
  - name: install lynx
    apt: name=lynx state=latest
    

apt

tasks:
  - name: install apache2
    apt: name=apache2 state=latest update_cache=yes
    

become

privilege escalation

Ansible allows you to ‘become’ another user, different from the user that logged into the machine (remote user). This is done using existing privilege escalation tools, which you probably already use or have configured, like sudo, su, pfexec, doas, pbrun, dzdo, ksu and others.

become

tasks:
  - name: Run a command as the apache user
    command: somecommand
    become: true
    become_user: apache
    

Loops

tasks:
  - name: install lynx
    apt: name=lynx state=latest
  - name: install vim
    apt: name=vim state=latest
    

Loops

tasks:
  - name: install packages
    apt: name={{ item }} state=latest
    with_items:
    - lynx
    - vim
    

Jinja2

Templating

- name: touch files with an optional mode
  file: dest={{item.path}} state=touch mode={{item.mode|default(omit)}}
  with_items:
    - path: /tmp/foo
    - path: /tmp/bar
    - path: /tmp/baz
      mode: "0444"
      

Service

Controls services on remote hosts. Supported init systems include BSD init, OpenRC, SysV, Solaris SMF, systemd, upstart.

Service

- name: install apache2
  apt: name=apache2 state=latest update_cache=yes
- name: apache2 is running
  service: name=apache2 state=started enabled=yes
  

Include Statements

"single responsibility principle"

- include: webserver.yml

Handlers

Handlers are lists of tasks, not really any different from regular tasks, that are referenced by a globally unique name, and are notified by notifiers. If nothing notifies a handler, it will not run. Regardless of how many tasks notify a handler, it will run only once, after all of the tasks complete in a particular play.

- handlers:
  - name: restart apache2
    service: name=apache2 state=restarted
    

Notify to Handlers

- name: check mod rewrite
  apache2_module:
    state: present
    name: rewrite
  notify:
  - restart apache2
  

Demo App

Flask framework

webserver.yml

---
- hosts: all
  become: true
  tasks:
  
    - name: install apache2 and other components
      apt: name={{ item }} state=present update_cache=yes
      with_items:
        - apache2
        - libapache2-mod-wsgi
        - python-pip
        - python-virtualenv
        
    - name: ensure apache is started
      service: name=apache2 state=started enabled=yes
      
    - name: ensure mod_wsgi enabled
      apache2_module: state=present name=wsgi
      notify: restart apache2
      
  handlers:
    - name: restart apache2
      service: name=apache2 state=restarted
      

Copy modules

source copied from control machine to server

- name: copy demo app source
  copy: src=../demo/app/ dest=/var/www/demo mode=755
  notify: restart apache2
  

Copy modules

apache virtual hosts

- name: copy virtual host config
  copy: src=../demo/demo.conf dest=/etc/apache2/sites-available mode=755
  notify: restart apache2
  

Pip modules

- name: setup python virtualenv
  pip:
    requirements=/var/www/demo/requirements.txt
    virtualenv=/var/www/demo/.venv
  notify: restart apache2
  

Handlers

RUNNING HANDLER [restart apache2] ******************
changed: [default]

Flush Handlers

- meta: flush_handlers

Files module

symlink

- name: de-activate default apache site
  file: path=/etc/apache2/sites-enabled/000-default.conf state=absent
  notify: restart apache2
- name: activate demo apache site
  file:
    src=/etc/apache2/sites-available/demo.conf
    dest=/etc/apache2/sites-enabled/demo.conf
    state=link
  notify: restart apache2
  

View demo app

MySql db module

MySql user module

webserver.yml

---
- hosts: all
  become: true
  tasks:
  - name: install apache2 and other components
    apt: name={{ item }} state=present update_cache=yes
    with_items:
      ...
      - python-mysqldb
      

database.yml

---
- hosts: all
  become: true
  tasks:
  
    - name: install mysql-server
      apt:
        name=mysql-server
        state=present
        update_cache=yes
        
    - name: ensure mysql is started
      service:
        name=mysql
        state=started
        enabled=yes
        
  handlers:
    - name: restart mysql
      service: name=mysql state=restarted
      

line in line module

- name: ensure mysql listen on all ports
  lineinfile:
    dest=/etc/mysql/my.cnf
    regexp=^bind-address
    line="bind-address = 0.0.0.0"
  notify: restart mysql
  

Mysql create db

- name: create demo database
  mysql_db: name=demo state=present
  

Mysql create user

- name: create demo user
  mysql_user:
    name=demo
    password=demo
    priv=demo.*:ALL host='%'
    state=present
    

View demo app db

Db down

$ sudo service mysql stop

Nginx - loadbalancer

loadbalancer.yml

---
- hosts: loadbalancer
  become: true
  tasks:
  
    - name: install nginx
      apt: name={{ item }} state=present update_cache=yes
      with_items:
        - nginx
        
    - name: ensure nginx is started
      service: name=nginx state=started enabled=yes
      
  handlers:
    - name: restart nginx
      service: name=nginx state=restarted
      

Vagrant multi machine

Vagrantfile - multi machine

config.vm.define "web" do |web|
  web.vm.box = "debian/jessie64"
  web.vm.hostname = "rocannon-web"
end

config.vm.define "db" do |db|
  db.vm.box = "debian/jessie64"
  db.vm.hostname = "rocannon-db"
end

config.vm.define "lb" do |lb|
  lb.vm.box = "debian/jessie64"
  lb.vm.hostname = "rocannon-lb"
end

Multi machine provisioning

config.vm.provision :ansible do |ansible|
  ansible.groups = {
    "webserver" => ["web"],
    "dbserver" => ["db"],
    "loadbalancer" => ["lb"],
    "all_groups:children" => ["development"]
    
  }
  ansible.playbook = "ansible/playbook.yml"
end

Inventory for multi machine

[webserver]
web1

[dbserver]
db

[loadbalancer]
lb

Vagrant command for multi machine

$ vagrant ssh [machine]
$ vagrant provision [machine]

Change playbook hosts

- hosts: webserver
- hosts: dbserver
- hosts: loadbalancer

Verify

App

Nginx

Web to db connection

# demo.wsgi
os.environ['DATABASE_URI'] = 'mysql://demo:demo@192.168.33.67/demo'

# or
os.environ['DATABASE_URI'] = 'mysql://demo:demo@192.168.33.67/demo'

/etc/hosts

- name: edit etc hosts
  lineinfile:
    path: /etc/hosts
    state: present
    line: '192.168.33.67 db'
    

Templates

templates/nginx.conf.j2

upstream web {
  {% for server in groups.webserver %}
      server {{ server }};
  {% endfor %}
}

server {
    listen 80;
    
    location / {
        proxy_pass http://web;
    }
}

Templates

loadbalancer.yml

- name: edit /etc/hosts
  lineinfile:
    path: /etc/hosts
    state: present
    line: '192.168.33.66 web'
    
- name: config nginx site
  template:
    src=templates/nginx.conf.j2
    dest=/etc/nginx/sites-available/demo
    mode=0644
  notify: restart nginx
  
- name: de-activate default nginx site
  file: path=/etc/nginx/sites-enabled/default state=absent
  notify: restart nginx
  
- name: activate demo nginx site
  file:
    src=/etc/nginx/sites-available/demo.conf
    dest=/etc/nginx/sites-enabled/demo
    state=link
  notify: restart nginx
  

Vagrantfile - multi machine

config.vm.define "web1" do |web1|
  web1.vm.box = "debian/jessie64"
  web1.vm.hostname = "rocannon-web-1"
end
config.vm.define "web2" do |web2|
  web2.vm.box = "debian/jessie64"
  web2.vm.hostname = "rocannon-web-2"
end

Vagrantfile - multi machine

provisioning

config.vm.provision :ansible do |ansible|
  ansible.groups = {
    "webserver" => ["web1" , "web2"],
    "dbserver" => ["db"],
    "loadbalancer" => ["lb"],
    "all_groups:children" => ["development"]
    
  }
  ansible.playbook = "ansible/playbook.yml"
end

loadbalancer.yml

- name: edit /etc/hosts
  lineinfile:
    path: /etc/hosts
    state: present
    line: "{{ item.ip }} {{ item.name }}"
  with_items:
    - { ip: 192.168.33.65, name: web1 }
    - { ip: 192.168.33.66, name: web2 }
    

Roles

reuse the code

init role

╰─➤  ansible-galaxy init apache
    - apache was created successfully
    

Roles

╰─➤  ls -l apache
  total 0
  drwxr-xr-x  10 bonzo  staff  340 Dec  6 10:45 files
  drwxr-xr-x  10 bonzo  staff  340 Dec  6 10:45 handlers
  drwxr-xr-x  10 bonzo  staff  340 Dec  6 10:45 tasks
  drwxr-xr-x   5 bonzo  staff  170 Dec  6 10:45 templates
  

main.yml - default file

╰─➤  ls apache/tasks
     main.yml
     

Converting file to role

copy task from webserver file to main.yml

Converting file to role

webserver.yml

---
- hosts: webserver
  become: true
  roles:
    - apache
    

site.yml

pleybook with include of other playbook

Variables facts

$ ansible -m setup db

Variables default

defaults/main.yml

db_user: demo
db_name: demo
db_password: demo

Variables vars

group_vars/database

db_user: demo
db_name: demo
db_password: demo

end

fonti

• ansible.com
• devops.com
• github.com
• devopsreactions.tumblr.com
• losviluppatore.it
• buildings.it
• mastering ansible

Questions?

Thanks to

officina-linux.it