In 2015, Puppet celebrated its 10th anniversary. At Camptocamp, this open source software configuration management tool has totally revolutionized our way of doing systems administration. We adopted Puppet in the course of 2007, but this year marks a special milestone for us, as our four Puppet git control repositories all date back to early 2008! It’s then time to take a look back at 10 years of Puppet practice at Camptocamp, what we’ve tried, and what we’ve learned.
Let us first have a look at our control repositories. We use four of them to manage four different Puppet infrastructure (our own and three main clients). They are all separate git repositories and they share modules.
In each repository, we set up local modules (not shared) residing in a site-modules directory, so they don’t get mixed up with the shared modules pulled by r10k.
Before we moved to r10k in the beginning of 2015, we used git subtree to manage modules in the repositories. We had a crontab with a script (githubsync) responsible for pulling module repositories inside the control repositories every night. We could also pull changes manually with git subtree. The solution was quite flexible, albeit quite error-prone and hard to maintain (we often had to fix failing automatic merges).
Switching to r10k in 2015 cut our codebase by half by removing the shared modules from the control repositories.
We have historically used two main branches in our repositories: stable and staging. The stable branch is used by production-critical machines while the staging branch is pointed to by less critical nodes. New features are added to the staging branch where they live for a few days until they are merged into the stable branch.
Since 2015, the Puppet Catalog Diff Viewer has helped us visualize changes between the two branches before deciding to merge.
As our code base grew and we started distributing our modules at large on the Forge, it became necessary to set up a good base for testing our code. In the course of 2013, we set up testing on most of our modules and plugged it to Travis CI for automatic testing of our modules.
In order to add acceptance tests to our public modules, we started to dynamically provision VMs on our Openstack internal cloud using beaker, and then switched to using Docker containers when they became available in Travis CI build environments.
The experience of setting up tests on our modules led to a series of blog posts on Puppet Test Driven Development in 2015. We have applied this method not only to public modules, but to our internal modules as well.
If there is one thing that Camptocamp has become famous for in the Puppet community, it’s the modules. While we maintain about a hundred of them on GitHub, we have published roughly half of them on the Forge, some of which are approved modules.
Over the years, our Puppet stack has changed a lot.
As most early adopters, we started with a Puppet Master running on Mongrel, which was pretty slow. Then we went on to use Apache with mod_passenger. The release of Thin as a replacement for Mongrel was a good opportunity for us to get rid of Apache and set up Thin-based Puppet Masters behind an Nginx proxy. At this point, the Nginx proxy was used as the SSL termination point and allowed to route requests to specific Puppet Masters depending on the environment. We typically had a basic machine for production environments (stable and staging) and a bigger one for testing (features + users) ones. This allowed to have reactive runs when humans were involved.
When Puppet 2.6 was released in 2010, our code was not ready for the migration due to many incompatibilities. We migrated directly from 0.25 to 2.7 in 2012, after testing extensively using catalog diffs on Jenkins.
In 2013 we adopted Foreman, mainly to be able to visualize the state of our fleet. Until then, we had only stored Puppet reports on disk and sent reports as notifications on an IRC channel. Foreman gave us great insights on our platform.
We spent a good part of 2013 preparing our modules to migrate to Puppet 3. This was a very large task because lots of our historical modules used dynamic scoping (which was deprecated in Puppet 3) with no class parameters (we had used class inheritance extensively in Puppet 0.2x to make up for this). This porting was accompanied by a lot of work on tests (both unit and acceptance) and CI/CD. The switch from Puppet 2.7 to 3 was done in the beginning of 2014.
Having to maintain tens of modules was an important task. We started using modulesync to ease that task during the summer 2014, and adopted it to synchronize our Puppet control repositories in December. Since then, we have been maintaining our Puppetfiles using modulesync, using a static Puppetfile as a template and Augeas to prune entries per control repository. We also developed puppetfile-updater to ease the grooming of our reference Puppetfile.
By the end of 2014, the Puppetserver (Puppet Master on the JVM) was released. We adopted it quickly as performance had always been a problem we faced with the Puppet Master, and set it up behind our proxified setup.
All the setup we put together during our migration to Puppet 3 clearly paid off for the Puppet 4 migration, as everything was already in place to automate testing. Since the main change in Puppet 4 was the switch to the future parser, we were able to integrate tests early using Puppet 3 with the future parser in the test matrix. But what really helped us with this language migration was the development of puppet-lint plugins to automate the detection and fixing of language deprecations in Puppet 4. We were able to run these not only on our public modules but also on our private codebase, and quickly get our code ready for Puppet 4.
When Puppet 4 was released, the AIO packages and changes in configuration paths encouraged us to try a new way to deploy the Puppet stack. By that time, we had decided we preferred to classify nodes in code instead of using a database, so we were ready to switch Foreman for Puppetboard. We started building a Puppet stack on Docker, and after a few months moved it to Rancher.
As more and more of our internal applications are moving to Docker (Rancher and Openshift), it has become more and more important to clean up unused Puppet code. To this end, we developed puppet-ghostbuster, a tool using the PuppetDB to find unused Puppet classes, defines, templates and files. This has helped us cleaning up lots of code. We also donated several modules to Voxpupuli for community maintenance (including the puppetserver module and Puppet 4 puppet-lint plugins).
Today, we are really satisfied with this Dockerized Puppet setup, and are in the process of migrating our infrastructure to a Puppet 5 stack based on Openshift.
Camptocamp has been involved in the Puppet community since the early days of the tool.
In 2012, we co-organized a Puppetcamp in Geneva where we presented the Augeas and Augeasproviders projects. We have since gotten involved in many other conferences such as Puppetconf in 2014 and most of the cfgmgmtcamp events.
Partnerships & Services
Camptocamp became a Puppet Partner in 2013 and our team has since been teaching dozens of official Puppet Training sessions all over Europe.
In 2018, Camptocamp is a Puppet Inc. Gold Partner for Switzerland, France, and Germany.