We have recently been contacted by a French client about some small functional improvements in mapproxy. Without extensive prior experience with this software we were able to provide the customer with a complete solution in a short period of time.

Mapproxy - the customizable tile proxy | © Omniscale GmbH & Co. KG - Apache License - C2C

Tool selection

Well isn’t this a frequent situation ? We want to set up a tile server for a new project and have to choose among those available. In the end, we always choose the one we are most familiar with, and always hope it will never be phased out, right ?

Let’s check some options we often encounter at camptocamp:

  • Mapcache: great along with mapserver, somehow the reference, well supported, feature-rich, reliable
  • GeoWebCache (Integrated with GeoServer or available as stand-alone application): yet another “good-old-trusty” tool with limited cache drivers
  • ArcGIS: complete server solution with every imaginable configuration, but proprietary, of course we prefer the open-source solutions
  • QGIS server with cache plugin: feature-rich and easy to set up, but with a rather big overhead


Enter mapproxy, a tile server with multiple backends written in python.

 Source data backendsCache backends
Compatibility

WMS sources (1.0.0–1.3.0),
WMTS/TMS sources,
Mapserver and Mapnik configurations,
any TileCache, Google Maps or Bing compatible source,
ArcGIS REST servers and compact cache files

Filesystem,
MBTiles/SQLite,
ArcGIS Compact Cache,
S3,
Redis,
Riak,
CouchDB

 

If you have ever tried to do some source code modifications in mapserver, the first thing to do is brew a huge thermos of coffee. Digging into highly optimized C code feels like a trip back in time and I have never felt comfortable about potential side effects of a small code change.

On the contrary, making a change in mapproxy has proved to be a rather joyful experience. The software is entirely written in python which makes a readable and maintainable code base. The source code is well structured and has good test coverage.

 

 

Tile cache setup


Purpose

A tile cache is placed between a geodata source and potential clients.

  • It will take load off the geodata generation server for resource-hungry data
  • Recurrent computation of identical data is avoided
  • It can be used for advanced operations like watermarking, reprojection or generation of data sets in multiple spatial reference systems.

It has 2 main methods of operation: seeding and serving.

  • Seeding is the process of loading a defined selection of data from a source and storing it in an optimized way as a cache in storage.
    • Seeding consists of generating an initial data set as well as refreshing existing data in cache
    • Seeding can be done as a one-off manual operation or as a recurrent automated job.
  • Serving means listening to clients and serving requests in a given protocol from cached data. Optionally, this service may fetch source data on the fly in case of failed cache lookup.

 

Configuration

Mapproxy mainly uses two distinct configuration files:

  • mapproxy.yaml
  • seed.yaml

 

Many aspects of cache operations are specified in these configuration files:

  • Data sources
  • Grid definition
  • Cache types and location
  • Cache ages
  • Default data (empty tiles)
  • etc.

 

Performance tuning

Performance of the mapproxy server can be improved by deploying the app in a robust web server (apache + mod_wsgi, NGINX, gunicorn, etc.)

A pre-generated tile pyramid can also be accessed locally by a front-end application or can be served directly as static files by a high performance file server.

 

Our code modification in detail

Our wish was to add some more detail to cache handling, with additional options for stale tile handling and re-seeding.

The main design drivers of the parent project were:

  • Reduction of  both load on the source server
  • Minimisation of storage space for cache
  • Handling unreliable source servers

Code organization

The code is clearly separated into main functions, each part having usually a virtual parent class with basic shared functionality and child classes implementing a precise function (eg. parent = MapLayer; children = [WMSSource, TiledSource, ArcGISSource, etc.])

Main functions are separated into:

  • Cache management
  • Source Handling
  • Published services
  • Seeding
  • Etc.

This modular design means that it is rather easy to propagate an option through the different consumer classes.
 

Mapproxy - the customizable tile proxy | © Camptocamp

New options

  1. Refresh stale tiles while serving
  2. Seeding improvements
    1. Refresh only existing tiles (already cached through client queries)
    2. Keep stale tiles in case of source error


These were very precise changes, requiring a deep dive into the code structure. However, the design of mapproxy proved robust so that only minimal code changes were needed without any major refactoring.

For point 1, the modifications were well localized:

  • changes in the configuration system
  • changes in the cache -> tile internal service
  • minor accessory modifications for blank tile source on error

https://github.com/mapproxy/mapproxy/pull/518

2 only required cross-wiring of existing options, so that some default values can be overridden from the command line.

https://github.com/mapproxy/mapproxy/pull/515

Without going further into detail, this showed that changes in very different parts of the software were feasible without any side effects on other parts of the code.

 

Conclusion

Cache handling in mapproxy is already extremely fine-grained. But in case the current configuration options are not sufficient, it has proved to be easy to change the source code in order to take into account additional options.

Mapproxy is a good choice if you have high requirements on the cache configuration and if you enjoy tweaking a tool’s internals. The mapproxy community is also open for new developments, so new functionalities can be made available to a large number of users.

The tool will definitely enter our company’s toolbox, since the experience of working with mapproxy was very positive.

In summary: highly recommended!

For more information,

do not hesitate to get in contact with us!

HOW DID YOU HEAR ABOUT US? *
By submitting this form, I accept that the information entered will be used for the purposes described in the privacy policy.
This site is protected by reCAPTCHA and the Google Privacy Policy & Terms of Service apply