Elasticsearch, Logstash and Kibana
The software suite comprising Elasticsearch, Logstash, and Kibana allows to implement advanced visualizations from data generated by an IT infrastructure. The attention to these software is increasing and a lot of information is available on the internet.
In a few words:
Logstash extracts relevant information of events (log) produced by different software components, systems or applications.
Elasticsearch indexes and provides an advanced search interface on information processed by Logstash.
Kibana provides a graphical interface that allows you to build visualizations of data indexed in ElasticSearch.
Finally, Kibana visualizations show information filtered, sorted and aggregated from a stream of technical logs. These visualizations allow non-technical business users to make sense of the data and draw the relevant business information they need.
WMS and WMTS OGC protocols
Two HTTP protocols defined by the Open Geospatial Consortium (OGC) are frequently used in the geospatial world : WMS and WMTS. To Summarize and simplify, these protocols define URL formats that allow to specify which geographic information must be produced (information type, geographic area, zoom level, projection, etc.). Generally, the server response is an image, a map, generated from the query.
WHen Logstash becomes spatial
Camptocamp is strongly involved in the implementation and management of complex geospatial infrastructure as well as in Open Source GIS. At the intersection of these two worlds, it was natural for Camptocamp to use these new tools to analyze the data produced by specialized IT infrastructure in the provision of geospatial information.
Logstash can extract information from the log messages sent by a server database, or the standard fields of an HTTP request logs (host, protocol, etc.) But Logstash does not know the OGC WMS and WMTS protocols. So it can not extract useful information from these queries, such as the geographic area or the accessed layer data.
The extraction and the enhancement of these business data allow, for instance, our customers to know what areas of a map served by one of the two filtered services ( WMS / WMTS ) are popular, at what period of the day, or which layers are the most looked at. They can then make informed decisions on what to do (data pre-generation or data cache, for example) to ensure an optimum distribution. From a mountain of several gigabytes of logs emerge aggregated business information allowing a fine decision analysis by non technical users.
Camptocamp has developed a Logstash module for the analysis of WMS and WMTS geospatial queries. Following these developments and their validation on the infrastructure of one of our clients, we have put together these improvements on the Logstash project; they should be made available in the upcoming version 1.4 of the software.
These filters can be set as required. It is for example possible to make on-the-fly reprojections of geographical coordinates of requested tiles in the case of WMTS, or bounding box in the case of WMS. This is made possible via the use of Geoscript (Ruby ) and Geotools (Java) libraries, that we know well and often use in our projects.
Camptocamp is convinced that the management and the enhancement of raw data generated by an infrastructure distributing geospatial information is an area with many opportunities both for infrastructure administrators and for data managers. Leading our customers in this direction is part of our mission.