DOCKER CONTAINER LOG COLLECTOR

Zebrium’s docker container log collector collects container logs and sends logs to Zebrium for automated Incident detection. Our github repository is located here.

Note: A non-containerized Docker log collector based on Fluentd is also avalaible here

ze-docker-log-collector

Getting Started

Docker

Use the following command to create a docker log collector container:

sudo docker run -d --name="zdocker-log-collector" --restart=always \
                -v=/var/run/docker.sock:/var/run/docker.sock \
                -e ZE_LOG_COLLECTOR_URL="<ZE_LOG_COLLECTOR_URL>" \
                -e ZE_LOG_COLLECTOR_TOKEN="<ZE_LOG_COLLECTOR_TOKEN>" \
                -e ZE_HOSTNAME="<HOSTNAME>" \
                -e ZE_DEPLOYMENT_NAME="YOUR_DEPLOYMENT_NAME_HERE" \
                zebrium/docker-log-collector:latest

NOTE:

The ZE_DEPLOYMENT_NAME label essentially defines a failure domain boundary for anomaly correlation. This allows you to collect logs from multiple applications and isolate the logs of one application from another application so as not to mix these in a Root Cause Report. This is referred to as Service Groups in the Zebrium UI.

Docker Compose

Use the following configuration file to deploy via docker-compose command:

version: '3.5'

services:
  zdocker-log-collector:
    image: zebrium/docker-log-collector:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      ZE_LOG_COLLECTOR_URL: "<ZE_LOG_COLLECTOR_URL>"
      ZE_LOG_COLLECTOR_TOKEN: "<ZE_LOG_COLLECTOR_TOKEN>"
      ZE_DEPLOYMENT_NAME: "<YOUR_DEPLOYMENT_NAME_HERE>"
      ZE_HOSTNAME: "<HOSTNAME>"

AWS Elastic Container Service (ECS)

Add the following service to ECS on EC2 cluster configuration.

services:
  zdocker-log-collector:
    image: zebrium/docker-log-collector:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      ZE_LOG_COLLECTOR_URL: "<ZE_LOG_COLLECTOR_URL>"
      ZE_LOG_COLLECTOR_TOKEN: "<ZE_LOG_COLLECTOR_TOKEN>"
      ZE_DEPLOYMENT_NAME: "<YOUR_DEPLOYMENT_NAME_HERE>"

To collect container logs from all nodes in an ECS cluster, zdocker-log-collector service must be configured to run as an ECS daemon task. Please follow the steps below to configure the daemon task:

1. Log in to the AWS console and navigate to the ECS Clusters section. Click into your cluster you run the Agent on.
2. Choose Service tab, click on the Create button.
3. For launch type, select EC2, for service type, select DAEMON, type a service name, and click on Next step.
4. For Load balance type option, select None, and click on Next step. On next page, select Next step without configuring Auto Scaling.
5. Review and click on Create Service.

Please note ECS tasks must be configured to use ‘json-file’ Log Driver for Zebrium log collector to receive container logs. If there is special log configuration on ECS instances, for example, using UserData section on instance to set log configuration, those configurations may need to be modified or deleted.

Environment Variables

The following environment variables are supported by the collector:

Testing your installation

Once the docker log collector software has been deployed in your environment, your container logs and incident detection will be available in the Zebrium UI.

Contributors

• Brady Zuo (Zebrium)