open source basecamp

For several reasons, the need to centralize some metadata comes relatively quickly when you work everyday with Puppet.

For example:

• the data that you wish to use are already present in another system/service
• you want to share data between different systems/services
• you don’t manage these data, you can just read them
• etc.

In the example below, I use CouchDB [1], a simple document-oriented database which has the great advantage of providing a RESTful JSON HTTP API. To start, simply install CouchDB. For example, on a recent Debian/Ubuntu you can simply do:

$ apt-get install couchdb

A simple request to check that CouchDB works:

$ curl http://localhost:5984
{"couchdb":"Welcome","version":"0.11.0"}

Now, I want to create a database named “user” and insert in it some documents (users). First of all, I create a file (user1.json) that contains a document that is actually a JSON object:

{
   "_id": "user1",
   "ensure": "present",
   "firstname": "foo",
   "lastname": "bar",
   "uid": 1001,
   "password": "$6$rXn/lNjS$CRdQQ05D7ZV2xgg...",
   "managehome": true,
   "groups": ["admin","sudo"],
   "shell": "/bin/bash",
   "email": "foo@bar.com",
   "ssh_type": "ssh-rsa",
   "ssh_pub_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCzc..."
}

Then, I create the database and insert in the document:

$ curl -X PUT http://localhost:5984/user
{"ok":true}
$ curl -X PUT http://localhost:5984/user/user1 -d @user1.json
{"ok":true,"id":"user1","rev":"1-b82efa560b9699fd4d5898bfdaa95277"}

This last step may be repeated to create other users of course with different values especially the username which is the “_id” of the document. Once the users are inserted in the CouchDB database “user”, we can easily test their presence:

$ curl -s http://localhost:5984/user/_all_docs |python -m simplejson.tool
{
    "offset": 0,
    "rows": [
        {
            "id": "user1",
            "key": "user1",
            "value": {
                "rev": "10-aed60887f53030470891c158c424a44b"
            }
        },
        {
            "id": "user2",
            "key": "user2",
            "value": {
                "rev": "3-628d7cc78888a9344273fa42bd95dca5"
            }
        },
        {
            "id": "user3",
            "key": "user3",
            "value": {
                "rev": "3-35e523b183ad70ab4532374f423931b2"
            }
        },
        {
            "id": "user4",
            "key": "user4",
            "value": {
                "rev": "1-b28b7aeea77170126b83d7c7f470ea4f"
            }
        }
    ],
    "total_rows": 4
}

If you want to see the contents of each document, you can use the following query:

$ curl http://localhost:5984/user/_all_docs?include_docs=true

Now, our CMDB is ready to use with Puppet! To start, we will simply retrieve the parameters of resources via a small Puppet function named couchdblookup available inside our module puppet-couchdb [2]. Here is a simple definition which wraps the creation of a user-based on metadata contained in CouchDB:

define couchdb-user () {

  $couchdb_bind_address = "localhost"
  $couchdb_port = "5984"
  $couchdb_baseurl = "http://${couchdb_bind_address}:${couchdb_port}"
  $tinyurl = "${couchdb_baseurl}/user/${name}"
  $email = couchdblookup($tinyurl, "email")
  $firstname = couchdblookup($tinyurl, "firstname")
  $lastname = couchdblookup($tinyurl, "lastname")

  user {$name:
    ensure     => couchdblookup($tinyurl, "ensure"),
    uid        => couchdblookup($tinyurl, "uid"),
    password   => couchdblookup($tinyurl, "password"),
    groups     => couchdblookup($tinyurl, "groups"),
    shell      => couchdblookup($tinyurl, "shell"),
    managehome => couchdblookup($tinyurl, "managehome"),
    comment    => "${firstname} ${lastname}",
  }

  ssh_authorized_key {"${email} on ${name}":
    ensure  => couchdblookup($tinyurl, "ensure"),
    type    => couchdblookup($tinyurl, "ssh_type"),
    user    => $name,
    key     => couchdblookup($tinyurl, "ssh_pub_key"),
    require => User[$name],
  }

}

Finally, you can declare where you want in your recipes one or more users:

couchdb-user {["user1","user2","user3","user4"]:}

The example above is based on a standard approach using the external Puppet DSL which means having to explicitly declare the resources. A great trick would be to loop through all documents in the database “user” to create resources dynamically. Since version 2.6, it’s been possible but for this we must use the Puppet’s Ruby DSL [3].

Here is a simple piece of Ruby (hostclass couchdbusers) which loops through all documents and creates the users:

require 'json'
require 'open-uri'

hostclass :couchdbusers do
  url="http://localhost:5984/user/_all_docs?include_docs=true"
  result = JSON.parse(open(URI.parse(url)).read)
  result['rows'].each do |x|
    user x['doc']['_id'],
      :ensure     => x['doc']['ensure'],
      :uid        => x['doc']['uid'],
      :shell      => x['doc']['shell'],
      :password   => x['doc']['password'],
      :managehome => x['doc']['managehome']

    ssh_authorized_key "#{x['doc']['email']} on #{x['doc']['_id']}",
      :ensure  => x['doc']['ensure'],
      :user    => x['doc']['_id'],
      :type    => x['doc']['ssh_type'],
      :key     => x['doc']['ssh_pub_key'],
      :require => "User[#{x['doc']['_id']}]"
  end
end

Then, this class can be used directly in your manifests (for example, in your site.pp) like this:

import "classes/couchdbusers.rb"
include couchdbusers

This last example does exactly the same as the previous one except that, in this case, we can add documents in our database without having to explicitly declare the new resources. Finally, here’s what you should get by running Puppet:

$ sudo puppetd -t --verbose
info: ...
...
info: Applying configuration version '1305882280'
notice: /Stage[main]/Couchdbusers/User[user2]/ensure: created
notice: /Stage[main]/Couchdbusers/Ssh_authorized_key[foo2@bar2.com on user2]/ensure: created
notice: /Stage[main]/Couchdbusers/User[user1]/ensure: created
notice: /Stage[main]/Couchdbusers/Ssh_authorized_key[foo@bar.com on user1]/ensure: created
notice: /Stage[main]/Couchdbusers/User[user3]/ensure: created
notice: /Stage[main]/Couchdbusers/Ssh_authorized_key[foo3@bar3.com on user3]/ensure: created
notice: /Stage[main]/Couchdbusers/User[user4]/ensure: created
notice: /Stage[main]/Couchdbusers/Ssh_authorized_key[foo4@bar4.com on user4]/ensure: created
notice: Finished catalog run in 1.01 seconds

$ sudo grep "^user" /etc/passwd
user2:x:1001:1001::/home/user2:/bin/bash
user1:x:1002:1002::/home/user1:/bin/bash
user3:x:1003:1003::/home/user3:/bin/bash
user4:x:1004:1004::/home/user4:/bin/bash

Hope someone finds this useful!

[1] http://couchdb.apache.org/
[2] https://github.com/camptocamp/puppet-couchdb
[3] http://projects.puppetlabs.com/projects/1/wiki/Ruby_Dsl