Camptocamp – Actualités

An automated Redis Sentinel deployment on Rancher

21 septembre 2018

At Camptocamp, we have been using Rancher to manage our Odoo platforms for over 2 years, as it helps us create fast, reliable and reproducible deployments. Also it makes it possible to scale our services by having multiple running Docker containers which can be distributed over multiple hosts.

Our deployments don’t need any local storage to store data and it is very easy to have redundancy in our containers. This allows us to easily scale up by spawning multiple containers of the same project so we can handle more requests. It also makes the system more robust as we can deal with the loss of a container or a host.

When we started deploying Odoo on our platforms using Rancher, this desired scalability was not achievable yet because Odoo stores user sessions in the local filesystem of the container. To solve this, we developed an Odoo addon (which will be published soon) that makes it possible to save the sessions in a Redis database, therefore allowing different containers to share the same sessions.

At this point, we only moved the single point of failure. Although our Odoo instances were now fully redundant, our Redis was not. Hardly ever but still, it could happen that Redis would go down because of an host failure, network error or hardware migration. Rancher was smart enough to detect that Redis was down and it would then start a new instance on an available host, but it would still take a couple of minutes for the new container to be up again and of course we also would loose all the session data that were originally saved in Redis, forcing users to reconnect.

We had to find a solution to this which was Redis Sentinel, the high availability solution for Redis. It works by having multiple Redis instances: one master and any amount of replicas. Sentinel instances monitor the Redis instances and elect one of the replicas as master when the current master is down. We could then also handle the loss of a Redis instance with a smooth failover.

We soon discovered that deploying a master with replicas and sentinels on Rancher needed automation. The services had to be linked together by their addresses and we could not consider to do it manually in such a setup. We then added entrypoint scripts in the containers, so that new replicas or new Sentinels could look for the current master from the Rancher’s metadata and configure themselves accordingly (as soon as a containers starts, it joins the cluster).

We now have a fully redundant environment which can be easily deployed, in a reproducible way. The source code for our Redis Sentinel container can be found here and the built image here.

  1. *
  2. *
  3. *
  4. *