Docker? VMs? EC2?

Yes! With Packer.io

A Code4Lib 2015 presentation by Kevin S. Clarke  (+KevinClarke | @ksclarke)

Digital Library Programmer, UCLA Library

https://github.com/ksclarke

Identical machine images

For multiple platforms

From a single source configuration

Have IT (DevOps) Your Way      


Machine Provisioners System Platforms
Puppet (Server) VMWare
Puppet (Masterless) VirtualBox
Chef (Client) Amazon (AWS)
Chef (Solo) Digital Ocean
Ansible OpenStack
Salt Docker
Good Ole' Bash Google Compute Engine

Islandora!

Drupal    MySQL

Apache    Fedora

Tomcat    Drupal Servlet Filter

Solr    GSearch

Lots and lots of Islandora modules...

Djatoka    Kakadu

ExifTool    ImageMagick

Lame    SWF Tools

Tesseract    Video Tools (ffmpeg, etc.)

Not. So. Much.

  • What happens when you get a new laptop?
  • Do you want co-developers to share setups?
  • Do you ever use external contractors?
  • What's your strategy for system upgrades?
  • How do you experiment with other provisioners?
  • How do you experiment with other platforms?

The Packer Basics

  • Templates
  • Variables
  • Builders
  • Provisioners
  • Post-Processors

The Packer.io JSON Template


{
  "variables": {
    "NAME_GOES_HERE": "VALUE_GOES_HERE"
  },
  "builders": [
    { "type": "amazon-ebs" },
    { "type": "docker" }
  ],
  "provisioners": [
    { "type": "file" },
    { "type": "shell" },
    { "type": "ansible-local" }
  ],
  "post-processors": [
    { "type": "docker-tag" }
  ]
}
        

Template Variables


{
  "variables": {
    "docker_registry_user": "ksclarke",
    "aws_access_key": "{{env `AWS_ACCESS_KEY`}}",
    "aws_secret_key": "{{env `AWS_SECRET_KEY`}}",
    "keystore_password": "",
    "server_admin_email": "null"
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
  }]
}
		

Setting Variables


  packer build -var 'aws_access_key=foo' \
    -var 'aws_secret_key=bar' template.json
        

or


  packer build -var-file=variables.json template.json
        

or you can mix and match


  packer build -var-file=variables.json \
    -var 'server_admin_email=ksclarke@gmail.com' template.json
        

Packer.io Builders

  • Amazon EC2 (AMI)
  • DigitalOcean
  • Docker
  • VirtualBox
  • VMWare
  • OpenStack
  • Parallels
  • Google Compute Engine
  • QEMU
  • Something Custom
  • Null

One or More Builders


packer build -only=amazon-ebs fcrepo4.json
        

or


packer build -only=amazon-ebs,docker fcrepo4.json
        

or


packer build -except=digitalocean fcrepo4.json
        

AWS EC2 (AMI) Builder


{
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "security_group_id": "{{user `aws_security_group_id`}}",
    "region": "{{user `aws_region`}}",
    "associate_public_ip_address": "true",
    "source_ami": "{{user `aws_source_ami`}}",
    "instance_type": "{{user `aws_instance_type`}}",
    "ami_virtualization_type": "{{user `aws_virtualization_type`}}",
    "ssh_username": "ubuntu",
    "tags": {
      "OS_Version": "Ubuntu",
      "Release": "Fedora Repository {{user `fcrepo4_version`}}"
    },
    "ami_name": "Fedora Repository ({{user `fcrepo4_version`}}) [{{timestamp}}]",
    "ami_description": "A Fedora Repository ({{user `fcrepo4_version`}})"
  }]
}
        

DigitalOcean Builder


{
  "builders": [{
    "type": "digitalocean",
    "api_token": "{{user `digitalocean_api_token`}}",
    "image": "{{user `digitalocean_image`}}",
    "region": "{{user `digitalocean_region`}}",
    "size": "{{user `digitalocean_size`}}",
    "droplet_name": "fcrepo4-{{user `server_host_name`}}",
    "snapshot_name": "Fedora Repository ({{user `fcrepo4_version`}})"
  }]
}
        

Docker Builder


{
  "builders": [{
    "type": "docker",
    "image": "{{user `docker_image`}}",
    "commit": "true",
    "pull": "true"
  }]
}
        

Provisioning

  • Shell Scripts
  • File Uploads
  • Puppet (Masterless)
  • Puppet (Server)
  • Chef (Solo)
  • Chef (Client)
  • Salt (Masterless)
  • Ansible (Local Mode)
  • A Mix (e.g., Shell and Puppet)
  • A Custom Option

Bash Shell Provisioner


{
  "provisioners": [{
    "type": "shell",
    "environment_vars": [
      "SERVER_ADMIN_EMAIL={{user `server_admin_email`}}",
      "GRAPHITE_SECRET_KEY={{user `graphite_secret_key_password`}}",
      "GRAPHITE_ADMIN_PASSWORD={{user `graphite_admin_password`}}"
    ],
    "scripts": [
      "scripts/setup-base-system.sh",
      "scripts/install-graphite.sh",
      "scripts/configure-apache.sh"
    ]
  }]
}
        

Targeted Provisioning


{
  "provisioners": [{
    "type": "shell",
    "only": ["docker"],
    "script": "scripts/install-supervisor.sh"
  }]
}
        

File Upload Provisioner


{
  "provisioners": [{
    "type": "file",
    "except": ["docker"],
    "source": "scripts/carbon-cache-initd.sh",
    "destination": "/tmp/carbon-cache"
  }]
}
        

Ansible Provisioner


{
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 30",
      "sudo apt-get install -y ansible"
    ]
  },
  {
    "type": "ansible-local",
    "playbook_file": "playbooks/freelib-djatoka.yml"
  }]
}
        

Post-Processors


{
  "post-processors": [{
    "type": "docker-tag",
    "only": ["docker"],
    "repository": "{{user `docker_user`}}/packer-fcrepo4",
    "tag": "{{user `packer_fcrepo4_version`}}"
  }]
}
        

FcRepo4 and Graphite

Graphite Packer Build
https://github.com/ksclarke/packer-graphite
Fedora 4 Packer Build
https://github.com/ksclarke/packer-fcrepo4

Final and Future Thoughts

  • Docker output without a Dockerfile
  • Coordinating between separate services
  • Vagrant to orchestrate development use
  • How do you make your outputs shareable
PACKERIZE ALL THE THINGS !!!

Interested in Learning More?

Packer.io Documentation
https://packer.io/docs
Graphite Packer Build
https://github.com/ksclarke/packer-graphite
Fedora 4 Packer Build
https://github.com/ksclarke/packer-fcrepo4
Fedora 3 Packer Build
https://github.com/ksclarke/packer-fcrepo3
Other Packer Builds
https://github.com/search?q=user%3Aksclarke+packer

Thank You!