In my previous post, I went through the process of installing Fluentd using the command line starting method. In this post, I will describe the mechanism for using the td-agent mode of Fluentd.
With the td-agent mode of installation, everything is supplied for running Fluentd as a service. The tricky part with this install is that there is no official support for Ubuntu Trusty. Digging deeper into Treasure's web site, they do have files for Trusty, and so requires a tiny bit extra work to get things installed.
There are no changes in how elasticsearch and kibana are installed. So this article carries on from that point in the previous.
packages.treasure.com points to the more recent packages, which, at the time of this writing, refer to the experimental version 2.x of Treasure Agent.
aptitude install build-essential ruby-dev make libcurl4-gnutls-dev curl http://packages.treasuredata.com/GPG-KEY-td-agent | apt-key add - echo "deb http://packages.treasuredata.com/2/ubuntu/trusty/ trusty contrib" > \ /etc/apt/sources.list.d/treasure-data.list apt-get update apt-get install -y --force-yes td-agent # the following step appears to take a while: /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
Before starting up td-agent, a directory has to be set up for maintaining persisted file pointers:
mkdir /var/log/td-agent/tmp chown td-agent.td-agent /var/log/td-agent/tmp adduser td-agent adm
The /etc/td-agent/td-agent.conf configuration file can be configured in a manner similar to the /etc/fluent/fluent.conf from the previous article. Then td-agent can be started with
/etc/init.d/td-agent start
Some system tuning parameters:
In /etc/security/limits.conf: root soft nofile 65536 root hard nofile 65536 * soft nofile 65536 * hard nofile 65536
In /etc/sysctl.conf: net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 10240 65535 sysctl -p
deimos is a good site for a discussion of these configuration tasks.
From Go Ahead is a reference to using fluent-cat for sending json text to Fluentd's 'in_forward' plugin:
% echo '{"message":"Logging Data"}' | fluent-cat es.event % echo '{"message":"Elasticsearch"}' | fluent-cat es.event # es.event is a tag. es.event matches es.** of <match>
There is a recommendation of using jemalloc for larger installations. graylog2 is mentioned as an alternative log analysis package. logstash is another logging framework.