DOCKER FLUENTD COLLECTOR DETAILS

Zebrium’s fluentd output plugin is used to send logs from your docker containers and docker host to Zebrium for automated Anomaly detection. Our github repository is located here.

ze-fluentd-plugin for docker environments

Getting Started

Installing

Get Zebrium API server URL and authentication token from Zebrium.
Determine what deployment name to use.
Run the following command in a shell on host:
```
curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=<ZAPI_URL> ZE_LOG_COLLECTOR_TOKEN=<AUTH_TOKEN> ZE_HOST_TAGS="ze_deployment_name=<deployment_name>" /bin/bash
```
The default system log file paths are defined by the ZE_LOG_PATHS environment variable. Its default value is
```
"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure"
```
The ZE_USER_LOG_PATHS environment variable can be used to add more user specific log file paths. For example, to add app log files at /app1/log/app1.log and /app2/log/\*.log, you can set ZE_USER_LOG_PATHS to:
```
"/app1/log/app1.log,/app2/log/*.log"
```
NOTE:

The ZE_DEPLOYMENT_NAME label essentially defines a failure domain boundary for anomaly correlation. This allows you to collect logs from multiple applications and isolate the logs of one application from another application so as not to mix these in a Root Cause Report. This is referred to as Service Groups in the Zebrium UI.

Upgrading

The upgrade command is similar to the installation command:

curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=<ZAPI_URL> ZE_LOG_COLLECTOR_TOKEN=<AUTH_TOKEN> ZE_HOST_TAGS="ze_deployment_name=<deployment_name>" OVERWRITE_CONFIG=1 /bin/bash

Please note setting OVERWRITE_CONFIG to 1 will cause /etc/td-agent/td-agent.conf to be upgraded to latest version.

Uninstalling

curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_OP=uninstall /bin/bash

Configuration

The configuration file for td-agent is at /etc/td-agent/td-agent.conf. The following parameters must be configured for your instance:

Parameter	Description	Note
ze_log_collector_url	Zebrium log host URL	Provided by Zebrium once your account has been created.
ze_log_collector_token	Authentication token	Provided by Zebrium once your account has been created.
path	Log files to read ferom the Docker Host (if desired)	Both files and file patterns are allowed. Files should be separated by comma. The default value is `"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure"`
ze_host_tags	Host meta data	This parameter is optional. You can pass meta data in key-value pairs, the format is: "key1=value1,key2=value2". We suggest at least set one tag for deployment name: "ze_deployment_name=<your_deployment_name>"

Docker Container Log Collection

When Zebrium’s log collector is installed and started on the Docker host, the service monitors Docker containers using the docker inspect command to find the container log file path automatically. No further configuration is necessary.

User Log Paths for Docker Host Log Collection

User log paths can be configured via /etc/td-agent/log-file-map.conf. During log collector service startup, if /etc/td-agent/log-file-map.conf exists, log collector service script writes log paths defined in /etc/td-agent/log-file-map.conf to /etc/td-agent/conf.d/user.conf. Please note any user log paths configured at installation time via ZE_USER_LOG_PATHS must be added to /etc/td-agent/log-file-map.conf to avoid being overwritten.

{
  "mappings": [
    {
      "file": "/app1/log/error.log",
      "alias": "app1_error"
    },
    {
      "file": "/app2/log/error.log",
      "alias": "app2_error"
    },
    {
      "file": "/var/log/*.log",
      "exclude": "/var/log/my_debug.log,/var/log/my_test.log"
    }
  ]
}

Filtering Specific Log Events

If you wish to exclude certain sensitive or noisy events from being sent to Zebrium, you can filter them at the source collection point by doing the following:

Add the following in /etc/td-agent/td-agent.conf after other “@include”:
```
@include conf.d/log_msg_filters.conf
```

Create a config file /etc/td-agent/conf.d/log_msg_filters.conf containing:

<filter TAG_FOR_LOG_FILE>
  @type grep
  <exclude>
    key message
    pattern /<PATTERN_FOR_LOG_MESSAGES>/
</exclude>
</filter>

Restart td-agent: sudo systemctl restart td-agent

Example

Below is an example log_msg_filters.conf for filtering out specific messages from a Vertica log file at /fast1/vertica_catalog/zdb/v_zdb_node0001_catalog/vertica.log

In this example, the Fluentd tag for file is node.logs.<FILE_NAME_REPLACE_/_WITH_DOT> (i.e replace all slashes with dots in the file path).

<filter node.logs.fast1.vertica_catalog.zdb.v_zdb_node0001_catalog.vertica.log>
  @type grep
  <exclude>
    key message
    pattern /^[^2]|^.[^0]|TM Merge|Authenticat|[Ll]oad *[Bb]alanc[ei]|\[Session\] <INFO>|\[Catalog\] <INFO>|\[Txn\] <INFO>|Init Session.*<LOG>/
  </exclude>
</filter>

Environment Variables

None

Usage

Start/stop Fluentd on CentOS 7/Ubuntu 16.04/18.04

Fluentd agent can be started or stopped with the command:

sudo systemctl <start | stop> td-agent

Start/stop Fluentd on CentOS 6

On CentOS 6, Fluentd agent can be started or stopped with the command:

sudo /etc/init.d/td-agent <start | stop>

Testing your installation

Once the collector has been deployed in your environment, your logs and anomaly detection will be available in the Zebrium UI.

Troubleshooting

In the event that Zebrium requires the collector logs for troubleshooting, logs are located here:

Collector installation log: /tmp/zlog-collector-install.log.*
Collector runtime log: /var/log/td-agent/td-agent.log

Please contact Zebrium at support@zebrium.com if you need any assistance.

Contributors

Brady Zuo (Zebrium)