Logging for distributed applications

Central Logging Service for Distributed Applications

With the release of our great new presentation layer for the Goettingen Digitization Centre we faced new problems. The Application consists of many different separate services (and servers). For example a Symfony frontend application, separate applications to deliver and process IIIF manifests and images. One service is responsible to get and process newly scanned works, one genereates PDF files, and so on.

Each service may fail and behave somehow unexpected, but these errors are not visible to the user, but logged inside the application.

Slowly we get to the point, but one may imagine, that looking for errors in each of these services and handling with different logging formats is a lot of work and very error prone.

So, using a central logging server for all these applications is the solution to our problem. We decided to use Graylog2 that is quite easy to setup in our docker-based environment. It’s nothing more than

docker run --name mongo -d mongo:3

docker run --name elasticsearch \
    -e "http.host=0.0.0.0" -e "xpack.security.enabled=false" \
    -d docker.elastic.co/elasticsearch/elasticsearch:5.6.6

docker run --link mongo --link elasticsearch \
    -p 9000:9000 -p 8441:8441/udp -p 12201:12201 -p 514:514 -p 53160:53160/udp -p 7946:7946/udp -p 8084:8084 \
    -e GRAYLOG_WEB_ENDPOINT_URI="https://logging.example.com:9000/api" \
    -d graylog/graylog:2.4.3-1

and the service is up.

Next up, we configure our Symfony Apps to use the Gelf LogHandler (the default format for logging to Graylog2).

composer require graylog2/gelf-php adds the formatter and adding a service and add the handler to the monolog configuration like that:

In services.yml:

  monolog.gelf_handler:
      class: Monolog\Handler\GelfHandler
      arguments: ["@gelf.publisher", 200]

  gelf.validator:
      class: Gelf\MessageValidator

  gelf.transport:
      class: Gelf\Transport\TcpTransport
      arguments: ['%env(LOG_DOMAIN)%', '%env(LOG_PORT)%']

  gelf.publisher:
      class: Gelf\Publisher
      arguments: ["@gelf.transport", "@gelf.validator"]

and finally the monolog configuration in config.yml:

monolog:
  handlers:
    main:
      type:         fingers_crossed
      level:        info
      handler:      gelf
    gelf:
      type: service
      id: monolog.gelf_handler
      level: debug

With these small steps we have a fully working, centralized logging for our services (at least our Symfony applications).