Logging

Architecture

+------------- rs236235 --------------------
|
| [[Kibana]]
|     |
|     v
| [[Elasticsearch]]
|     ^
|     |
| [Fluentd] --+ <file:aggregation.current>
|     +
|     |
+-----|-------------------------------------
      |
+-----|------- Service host ----------------
|     |
|  [Fluentd] --> <file:aggregation.current>
|     ^ ^
|     | |
|     | +------------------+                       
|     |                    |
|  [Docker daemon]   [Host service]
|        ^
|        |
| +---<stdout>-- [[Docker container]] -------
| |      ^
| |      |
| | [Fluentd]
| |      ^
| |      |
| | <Application log>
| |      ^
| |      |
| | [Application]
| |
| +-----------------------------------------
|
+-------------------------------------------

Log aggregation

Service hosts collect log information from their services (typically through the docker daemon) using Fluentd. These logs are aggregated locally and sent to the central log aggregation host (rs236235). Here, the logs are processed and sent to two targets:

• The file system: /var/log/fluent/aggregation.current which is flushed and rotated periodically
• A local Elasticsearch instance (using the ​Elasticsearch output plugin)

A Kibana instance is also running and connected to Elasticsearch and available (through the proxy) via ​https://logs.clarin.eu. This can be used to view, query and visualise the indexed log messages and the data therein.

Data processing via logs

Using Kibana logs cannot only be viewed and queried, they can also be used for data visualisation, for example the evolution of a certain value included in regular log messages over time. For this, the logs have to be parsed into fields, which can happen in various places. Ideally this is done at the primary source of the log collection, i.e. in the Fluentd enabled application container.

Log parsing

Examples

Solr

This uses the ​regexp parser to extract a number of fields, and also specifies field types for those fields that should not be interpreted as string.

<source>
  @type tail
  path /opt/solr/server/logs/solr.log
  pos_file /opt/solr/server/logs/solr.log.pos
  tag solr
  emit_unmatched_lines true
  <parse>
    @type regexp
    expression /^(?<log_time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+)\s+(?<log_level>([^\s]+))\s+((\([^\)]*\)\s+\[\s*(?<solr_index>[^\]]+)\s*\].*(webapp=(?<solr_webapp>[^\s]+)).*(path=(?<solr_path>[^\s]+)).*(params=(?<solr_params>{[^\}]+}))(.*hits=(?<solr_hits>\d+)(.*status=(?<solr_status>\d+)(.*QTime=(?<solr_qtime>\d+)?)?)?)?)?(?<message>.+)?)$/
    types solr_hits:integer,solr_status:integer,solr_qtime:integer
  </parse>
</source>