Mapproxy - the customizable tile proxy
This article shall provide a short feedback of the journey into the internals of mapproxy
We have recently been contacted by a French client about some small functional improvements in mapproxy. Without extensive prior experience with this software we were able to provide the customer with a complete solution in a short period of time.
Well isn’t this a frequent situation ? We want to set up a tile server for a new project and have to choose among those available. In the end, we always choose the one we are most familiar with, and always hope it will never be phased out, right ?
Let’s check some options we often encounter at camptocamp:
- Mapcache: great along with mapserver, somehow the reference, well supported, feature-rich, reliable
- GeoWebCache (Integrated with GeoServer or available as stand-alone application): yet another “good-old-trusty” tool with limited cache drivers
- ArcGIS: complete server solution with every imaginable configuration, but proprietary, of course we prefer the open-source solutions
- QGIS server with cache plugin: feature-rich and easy to set up, but with a rather big overhead
Enter mapproxy, a tile server with multiple backends written in python.
|Source data backends
WMS sources (1.0.0–1.3.0),
If you have ever tried to do some source code modifications in mapserver, the first thing to do is brew a huge thermos of coffee. Digging into highly optimized C code feels like a trip back in time and I have never felt comfortable about potential side effects of a small code change.
On the contrary, making a change in mapproxy has proved to be a rather joyful experience. The software is entirely written in python which makes a readable and maintainable code base. The source code is well structured and has good test coverage.
Tile cache setup
A tile cache is placed between a geodata source and potential clients.
- It will take load off the geodata generation server for resource-hungry data
- Recurrent computation of identical data is avoided
- It can be used for advanced operations like watermarking, reprojection or generation of data sets in multiple spatial reference systems.
It has 2 main methods of operation: seeding and serving.
- Seeding is the process of loading a defined selection of data from a source and storing it in an optimized way as a cache in storage.
- Seeding consists of generating an initial data set as well as refreshing existing data in cache
- Seeding can be done as a one-off manual operation or as a recurrent automated job.
- Serving means listening to clients and serving requests in a given protocol from cached data. Optionally, this service may fetch source data on the fly in case of failed cache lookup.
Mapproxy mainly uses two distinct configuration files:
Many aspects of cache operations are specified in these configuration files:
- Data sources
- Grid definition
- Cache types and location
- Cache ages
- Default data (empty tiles)
Performance of the mapproxy server can be improved by deploying the app in a robust web server (apache + mod_wsgi, NGINX, gunicorn, etc.)
A pre-generated tile pyramid can also be accessed locally by a front-end application or can be served directly as static files by a high performance file server.
Our code modification in detail
Our wish was to add some more detail to cache handling, with additional options for stale tile handling and re-seeding.
The main design drivers of the parent project were:
- Reduction of both load on the source server
- Minimisation of storage space for cache
- Handling unreliable source servers
The code is clearly separated into main functions, each part having usually a virtual parent class with basic shared functionality and child classes implementing a precise function (eg. parent = MapLayer; children = [WMSSource, TiledSource, ArcGISSource, etc.])
Main functions are separated into:
- Cache management
- Source Handling
- Published services
This modular design means that it is rather easy to propagate an option through the different consumer classes.
- Refresh stale tiles while serving
- Seeding improvements
- Refresh only existing tiles (already cached through client queries)
- Keep stale tiles in case of source error
These were very precise changes, requiring a deep dive into the code structure. However, the design of mapproxy proved robust so that only minimal code changes were needed without any major refactoring.
For point 1, the modifications were well localized:
- changes in the configuration system
- changes in the cache -> tile internal service
- minor accessory modifications for blank tile source on error
2 only required cross-wiring of existing options, so that some default values can be overridden from the command line.
Without going further into detail, this showed that changes in very different parts of the software were feasible without any side effects on other parts of the code.
Cache handling in mapproxy is already extremely fine-grained. But in case the current configuration options are not sufficient, it has proved to be easy to change the source code in order to take into account additional options.
Mapproxy is a good choice if you have high requirements on the cache configuration and if you enjoy tweaking a tool’s internals. The mapproxy community is also open for new developments, so new functionalities can be made available to a large number of users.
The tool will definitely enter our company’s toolbox, since the experience of working with mapproxy was very positive.
In summary: highly recommended!
For more information,
do not hesitate to get in contact with us!
Interested in working in an inspiring environment and joining our motivated and multicultural teams?
- Odoo Developer Internship (m/f/d) - Chambéry
- Java Developer Engineer (m/f/d) - Paris/Chambéry
- Odoo Developer (m/f/d) - Olten/Zurich/Munich
- ERP Consultant (m/f/d) - Zürich/Olten
- ERP Project Manager (m/f/d) - France
- Sales Manager France Infrastructure Solutions (m/f/d) - Paris
- Marketing Specialist - Munich/Olten