Puppet Server, aka the Puppet Master on the JVM, is a great project to improve the performance of the Puppet Master. We recently tested it as a replacement for the Master on Passenger, but we faced an issue when it came to scaling it.
At Camptocamp, we currently load balance all of our Puppet traffic through an Nginx proxy. This is not only for scaling reasons, but also for logical reasons. The Nginx proxy is used as the SSL termination (and for this reason as the Puppet CA, too). It redirects to various Puppet Master backends, depending on the incoming URL. This way, we have Puppet Masters for our development, staging and stable environments, which ensure both scaling and a clean separation of concerns. However, this means that, as of now, we only encrypted our Puppet stream up to the proxy, and the communication between the proxy and the backends was not encrypted. As a result, only the HTTP headers were used to authenticate the agents on the Puppet Masters sitting behind the proxy.
This became an issue when we switched to Puppet Server, since we could not get it to accept non-SSL connections. Scaling using the `ca_server` parameter and an external CA was apparently not an option (then) either. So we had to think of another solution.
The solution came as a hardening of our current method, replacing Nginx with Apache.
The HTTP proxy now runs Apache with `mod_proxy_http`. It still works as a CA, with a local Puppet Master running on port 8139 and acting as the CA for the whole fleet. The proxy is the SSL termination for Puppet streams. It redirects to various backends based on the requested URL, and it reencrypts the stream with its own key for the Puppet Masters.
The Apache setup looks like this:
The Puppet Masters
Since the Puppet Servers still use SSL and do not recognize the `ca_server` option, they must believe they are CA servers and accept incoming connections.
In order to achieve this, we leave the standard CA settings for the Puppet Servers, and simply copy several files the CA certificates and keys in `/etc/puppet/ssl/ca/ca_*` from the proxy.