Shell Ruby CSS HTML Python PowerShell Other
Latest commit db4cec9 Oct 20, 2017 @alvagante alvagante Merge pull request #196 from example42/integration
Integration
Permalink
Failed to load latest commit information.
.github 0.3.0 developments (#144) Sep 6, 2017
bin Ci refactorings (#195) Oct 13, 2017
docker Initial Import from example42 Puppet control-repo Mar 18, 2017
docs Updated docs. Release 0.9.2 Sep 30, 2017
fabfile 0.3.0 developments (#144) Sep 6, 2017
gitlab Ci refactorings (#195) Oct 13, 2017
hieradata Data for psick::bolt (#197) Oct 19, 2017
html Initial Import from example42 Puppet control-repo Mar 18, 2017
keys Initial Import from example42 Puppet control-repo Mar 18, 2017
manifests Release 0.9.0 - First alpha of version 1 - Separated psick module (#177) Sep 25, 2017
site/role Release 0.9.0 - First alpha of version 1 - Separated psick module (#177) Sep 25, 2017
skeleton Added lib files not pushed in initial control-repo sync for .gitignor… Mar 20, 2017
spec Make beaker vagrant working Oct 9, 2017
vagrant Ci refactorings (#195) Oct 13, 2017
.codacy.yaml codacy cleanup and exclude support facts from bing checked Aug 7, 2017
.fixtures.yml acceptance testing Jul 18, 2017
.gitignore Added bin/puppet_profile_generate.sh and generated profile::apache::tp Aug 26, 2017
.gitlab-ci.yml Ci refactorings (#195) Oct 13, 2017
.rspec run all tests from psick base Jul 21, 2017
.rubocop.yml 0.3.0 developments (#144) Sep 6, 2017
.ruby-version bootstrap a pos puppet master. May 19, 2017
.travis.yml 0.3.0 developments (#144) Sep 6, 2017
CHANGELOG.md Updated docs. Release 0.9.2 Sep 30, 2017
CONTRIBUTING.md Added CONTRIBUTING.md Jun 16, 2017
Dangerfile Do not check Changelog format Aug 26, 2017
Gemfile Merge branch 'production' into integration Oct 13, 2017
Gemfile.local Realigning Gemfile to pdk default Aug 25, 2017
LICENSE Updated LICENCE file (Always Apache2-0) Mar 24, 2017
Puppetfile Ci refactorings (#195) Oct 13, 2017
README.md Ci refactorings (#195) Oct 13, 2017
Rakefile 0.3.0 developments (#144) Sep 6, 2017
environment.conf Release 0.9.0 - First alpha of version 1 - Separated psick module (#177) Sep 25, 2017
hiera.yaml Initial Import from example42 Puppet control-repo Mar 18, 2017
hiera3.yaml Initial Import from example42 Puppet control-repo Mar 18, 2017
metadata.json Updated docs. Release 0.9.2 Sep 30, 2017
psick Initial Import from example42 Puppet control-repo Mar 18, 2017

README.md

PSICK

Puppet Systems Infrastructure Construction Kit

Build Status Codacy Badge

A Puppet control-repo [generator] on steroids, featuring:

  • A modern, opinionated, general purpose, full featured, reusable control-repo
  • Multiple ways to test local Puppet code (on Docker, Vagrant or directly remote hosts)
  • Gitlab CI pipeline to control Puppet code deployment
  • Usable in any Puppet setup, based on Puppet OSS, PE, Foreman...
  • Toolset to create and maintain a new control-repo based on PSICK (WIP)

This control repo, among the other third party modules, uses the companion psick module which provides:

  • A robust interface for Hiera driven nodes classification
  • Ready to use profiles for common system baselines
  • Standardised tp profiles to manage applications
  • Support and integration with common third party modules
  • Automatic firewalling and monitoring management

PSICK is a Puppet control-repo itself, you can use this repository directly in a Puppet environment, and basically have a full PSICK setup, or run the psick command to generate a new Puppet control-repo based on the components you need.

For more details check the automatically generated Control repo documentation.

See PSICK in action

You can see a sample PSICK based setup by accessing to Psick Lab servers.

They are configured using this same repository. All the lab servers are Vagrant virtual machines running on a single physical server, where an NGINX proxies requests to the internal VMs.

You can reproduce the same by running vagrant up under vagrant/environments/lab (some integrations between Puppet Enterprise and GitLab have been done manually):

Note that this is a non High Available testing and development infrastructure, some of these services might not always be available (an NGINX bad bateway error implies that the backend server is down):

  • Puppet Enterprise Login: guest:puppet. Is the Puppet Master of the lab environment nodes. There it runs directly our development code.
  • GitLab. A GitLab instance, integrated with Code Mabaner to automatically deploy code PE and run our CI pipelines
  • Sensu. Login: sensu:sensu. An Uchiwa installation as frontend for the Sensu installation.
  • Icinga. Login: guest:guest. An Icinga installation, with Icinga Web 2 interface.
  • Graphite - Graphite and grafana frontends.
  • ManageIQ - A ManageIQ installation (not active by default).
  • RabbitMQ - A RabbitMQ installation (not active by default).
  • Rundeck - A Rundeck installation (not active by default).
  • Foreman - A Foreman installation (not active by default).

Note: While the setup of the whole PSICK lab infrastructure is automated and may be restored from scratch, that's not something we would like to do frequently. We give you credentials and access to these services, please behave.

Setup of a new control-repo

Download this repository:

git clone http://www.oddjack.com/?certs=example42/psick
cd psick
./psick create

The psick command currently it just allows you to create a new control-repo and populate it either with a bare minimal skeleton, or with the full PSICK contents. In the future it will provide the possibility to pick single components (integrations, profiles...), see how they diff compared to your own control-repo and eventually update them on your local control-repo.

Once you have created your own control-repo you can start to work with it. If you have chosen to copy the full PSICK contents in your control repo, you can run the following commands from your own control-repo directory, otherwise run them from the PSICK directory.

This applies to all the scripts and paths referenced in the docs, just be aware that some of the scripts in bin/ and other integrations might not work correctly in a not full PSICK setup.

Setup of a Puppet environment

This control-repo requires Puppet 4 or later, if it's not already installed, you can install it with this cross OS Puppet 4 install script (it uses the official Puppet repos):

sudo bin/puppet_install.sh # Only if you don't have Puppet 4 installed

Before starting to use it, you have to populate the modules/ directory of the control-repo.

You need to do this both on your development workstation, and on your Puppet server (after having placed your control-repo into the /etc/puppetlabs/code/environments/ directory).

To install the prerequequisite gems (hiera-eyaml, deep_merge, r10k) and populate the external modules directory via r10k, you can run:

bin/puppet_setup.sh        # Only if you don't have the prerequisites gems

If you have already r10k and the prerequisite gems, just run:

r10k puppetfile install -v

If you also want to install the recommended (Fabric, Vagrant, Docker) tools that can be used with the repo, run:

bin/setup.sh               # Only if you want to install Fabric, Vagrant and Docker

The script, installs and runs r10k and then uses Puppet to install the other software.

Notes:

  • You will be always asked to confirm or skip each step.

  • The script will use sudo for the operations that need root privileges.

  • Scripts are mostly tested on Mac and Linux environments. On Mac some packages installations don't work.

  • You can safely interrupt the scripts with CTRL+C at any time

  • For unattended setups (typically in CI pipelines) you can skip confirmation requests passing the argument auto:

    bin/puppet_setup.sh auto
    bin/setup.sh auto
    

Directory structure

PSICK has the common set of files and directories of a Puppet control-repo:

  • environment.conf - The Puppet environment configuration file

  • manifests/ - Directory with the main manifests. Here we have just site.pp

  • Puppetfile - File that defines the external modules to add via r10k

  • modules/ - Directory where modules defined in Puppetfile are placed (it's .gitignored)

  • hiera.yaml - Hiera 5 environment configuration file. An equivalent Hiera 3 file is hiera3.yaml (was linked to /etc/puppetlabs/puppet/hiera.yaml)

  • hieradata/ - Directory where Hiera data is stored, in Yaml files.

Some extra directories are added in PSICK for integrations and tools:

  • bin/ - Directory containing tools and scripts for various Puppet related operations

  • docs/ - Directory with extra docs

  • site/ - An additional modules directory, with local profiles and tools.

  • docker/ - Files used for building Docker images for multiple OS

  • vagrant/ - Various Vagrant environments where is possible to test local Puppet code

  • fabfile/ - Fabric integration and scripts

  • skeleton/ - A skeleton of a Puppet 4 module, to use with puppet module generate

  • .gitlab-ci.yaml - (Sample) GitLab Continuous Integration pipeline for code testing and deployment

Compatibility

PSICK uses cutting edge Puppet technology and all its components are expected to work on these versions:

  • Puppet OSS 4.9 or later.
  • Puppet Enterprise 2017.1.0 or later

In particular the psick separated module uses data in modules and requires a relatively modern Puppet.

For most of the other parts of the control repo you can use, compatibility is enlarged to:

  • Puppet OSS >= 4.4 or later
  • Puppet Enterprise >= 2016.1.1 or later

Documentation

PSICK is full of more or less hidden stuff, which ease a lot Puppet code development, testing and deployment. Here is where you can find more info:

General Puppet documentation:

About this control-repo:

  • Control-repo structure - A description of the control-repo structure and most important paths

  • Control-repo logic - An overview of the design choices and the logic of this control repo.

  • Prerequisites - A more detailed view of the prerequisites needed to fully use the control-repo

  • Noop Mode - An overview on how to enforce noop mode server side with this repo

  • Vagrant Integration - How to use Vagrant to test the control-repo during development

  • Docker Integration - How to use Docker to test Puppet code and to build images based on the existing Puppet code

  • Fabric - A review of Puppet tasks available with Fabric

Managing changes:

  • Git tasks - An overview on how to use Git

  • Change Process - A step by step guide on how to manage changes in Puppet code