Getting Logstash to run as a daemon

Currently evaluating Logstash and its centralised log management facilities. So far its looking like a good candidate for replacement of far more expensive utilities and seems very nippy when searching. The search query syntax is well documented so its very easy to use once set up.

As ever, using the nice and simple approach to get it working in the first instance as a standalone server. Then needing to keep the engine running for long term testing, its obviously better to run as a daemon. This was a bit of a cludge so noted for ref.

Cribbing and creating the following:

sudo vi /etc/init.d/logstash

Copy the contents of the github link. A few small changes are required:

DAEMON=/usr/bin/java #obviously!
ulimit -n 32000 #allows elastic search to hold more than the default 1000 files open
ARGS="-Xmx$JAVAMEM -Xms$JAVAMEM -jar /opt/logstash/logstash-monolithic.jar agent --config ${CONFIG_DIR} --log ${LOGFILE} --grok-patterns-path ${PATTERNSPATH} -- web --backend elasticsearch:///?local"

This last sections tells the engine to start a web server (port 9292 default) and also use the local server as the elasticsearch provider. Everything is therefore running on one server in this instance which also maintains syslog for all other devices on the network.

Create the new user to run as:

sudo adduser --system --disabled-password --no-create-home --group --quiet logstash

Allow the user to read syslog files (probably a far better way to do this but I am not concerned about security at the moment)

sudo usermod -a -G adm logstash

Create all the directories mentioned on the script:

mkdir /opt/logstash/
mkdir /etc/logstash.d/
mkdir /var/log/logstash/
mkdir /opt/logstash/patterns/

CHOWN the directories as per the following example:

sudo chown -R logstash:logstash /etc/logstash.d/
sudo chown -R logstash:logstash /var/log/logstash/
sudo chown -R logstash:logstash /opt/logstash/

Download the jar file and rename:

sudo wget
sudo mv logstash-1.1.0-monolithic.jar logstash-monolithic.jar

Add the config below to /etc/logstash.d/mylogstash.conf

input {
 file {
 type => "linux-syslog"
# Wildcards work, here
 path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
# file {
# type => "apache-access"
# path => "/var/log/apache2/access.log"
# }
# file {
# type => "apache-error"
# path => "/var/log/apache2/error.log"
# }
output {
 # Emit events to stdout for easy debugging of what is going through
 # logstash.
 stdout { }
# This will use elasticsearch to store your logs.
 # The 'embedded' option will cause logstash to run the elasticsearch
 # server in the same process, so you don't have to worry about
 # how to download, configure, or run elasticsearch!
 elasticsearch { embedded => true }

As you can see, all we look to do in the first instance is syslog search only. I have removed the apache logging for the moment (basically since its not installed on my test server!)

Make the logstash script executable and start the engine

sudo chmod +x /etc/init.d/logstash
sudo /etc/init.d/logstash start

If all is good, you can

 tail -f /var/log/logstash/logstash.log

And hopefully see the engine start and the fact its logging syslog content as it arrives at the server.

Then, give it a few minutes, open your browser, then hopefully you will have a log crawler and searcher! Ill be leaving this running for a few weeks to see how it scales so will report back on performance and any additional tweaks required.