You can pipe system and application logs from the nodes in a DC/OS cluster to an Elasticsearch server. These instructions are based on CentOS 7 and might differ substantially from other Linux distributions.

What this document does and does not cover

This document describes how to send Filebeat output from each node to a centralized Elasticsearch instance. This document describes how to directly stream from Filebeat into Elasticsearch. Logstash is not used in this architecture. If you are interested in filtering, parsing and understanding the logs with an intermediate Logstash stage, see the Logstash documentation and the example in Filtering logs with ELK.

This document does not explain how to set up and configure an Elasticsearch server. This document does not describe how to set up secure TLS communication between the Filebeat instances and Elasticsearch. For details on how to achieve this, see the Filebeat and Elasticsearch documentation.

Prerequisites

An existing Elasticsearch installation that can ingest data for indexing
All DC/OS nodes must be able to connect to your Elasticsearch server on the port used for communication between Elasticsearch and Filebeat (9200 by default)

Step 1: Install Filebeat

For all nodes in your DC/OS cluster:

Install Elastic’s Filebeat.

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.0-x86_64.rpm
sudo rpm -vi filebeat-5.0.0-x86_64.rpm

Create the /var/log/dcos directory:
```
sudo mkdir -p /var/log/dcos
```

Move the default Filebeat configuration file to a backup copy:

sudo mv /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.BAK

Populate a new filebeat.yml configuration file, including an additional input entry for the file /var/log/dcos/dcos.log. The additional log file will be used to capture the DC/OS logs in a later step. Remember to substitute the variables $ELK_HOSTNAME and $ELK_PORT below for the actual values of the host and port where your Elasticsearch is listening on.

filebeat.prospectors:
- input_type: log
  paths:
    - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stdout*
    - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr*
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
tail_files: true
output.elasticsearch:
  hosts: ["$ELK_HOSTNAME:$ELK_PORT"]

Important: The agent node Filebeat configuration expects tasks to write logs to `stdout` and `stderr`. Some DC/OS services, including Cassandra and Kafka, do not write logs to `stdout` and `stderr`. If you want to log these services, you must customize your agent node Filebeat configuration.

Step 2: Set up service for parsing the journal

For all nodes in your DC/OS cluster:

Create a script /etc/systemd/system/dcos-journalctl-filebeat.service that parses the output of the DC/OS master journalctl logs and funnels them to /var/log/dcos/dcos.log.

This script can be used with DC/OS and Enterprise DC/OS. Log entries that do not apply are ignored.

sudo tee /etc/systemd/system/dcos-journalctl-filebeat.service<<-EOF
[Unit]
Description=DCOS journalctl parser to filebeat
Wants=filebeat.service
After=filebeat.service

[Service]
Restart=always
RestartSec=5
ExecStart=/bin/sh -c '/bin/journalctl  --since="5 minutes ago" --no-tail --follow --unit="dcos*.service" >> /var/log/dcos/dcos.log 2>&1'

[Install]
WantedBy=multi-user.target
EOF

Step 3: Start and enable Filebeat

For all nodes, start and enable the Filebeat log parsing services created above:

sudo chmod 0755 /etc/systemd/system/dcos-journalctl-filebeat.service
sudo systemctl daemon-reload
sudo systemctl start dcos-journalctl-filebeat.service
sudo systemctl enable dcos-journalctl-filebeat.service
sudo systemctl start filebeat
sudo systemctl enable filebeat

ELK node notes

The ELK stack will receive, store, search and display information about the logs parsed by the Filebeat instances configured above for all nodes in the cluster.

This document describes how to directly stream from Filebeat into ElasticSearch. Logstash is not used in this architecture. If you are interested in filtering, parsing and understanding the logs with an intermediate Logstash stage, please check the Logstash documentation.

You must modify the default parameter values to prepare ElasticSearch to receive information. For example, edit the ElasticSearch configuration file (typically /etc/elasticsearch/elasticsearch.yml):

network.host = [IP address from the interface in your ElasticSearch node connecting to the Filebeat instances]

Other parameters in the file are beyond the scope of this document. For details, please check the ElasticSearch documentation.

Step 4: Configure log rotation

You should configure logrotate on all of your nodes to prevent the file /var/log/dcos/dcos.log from growing without limit and filling up your disk. Your logrotate config should contain copytruncate because otherwise the journalctl pipe remains open and pointing to the same file even after it has been rotated. When using copytruncate, there is a very small window between copying the file and truncating it, so some logging data might be lost - you should balance pros and cons between filling up the disk and losing some lines of logs.

For example, your logrotate configuration should look like this:

/var/log/dcos/dcos.log {    
  size 100M
  copytruncate
  rotate 5
  compress
  compresscmd /bin/xz
}

What’s Next

For details on how to filter your logs with ELK, see Filtering DC/OS logs with ELK.

Log Management with ELK