Getting Logstash to run as a daemon

Currently evaluating Logstash and its centralised log management facilities. So far its looking like a good candidate for replacement of far more expensive utilities and seems very nippy when searching. The search query syntax is well documented so its very easy to use once set up.

As ever, using the nice and simple approach to get it working in the first instance as a standalone server. Then needing to keep the engine running for long term testing, its obviously better to run as a daemon. This was a bit of a cludge so noted for ref.

Cribbing https://gist.github.com/1121786 and creating the following:

sudo vi /etc/init.d/logstash

Copy the contents of the github link. A few small changes are required:

DAEMON=/usr/bin/java #obviously!
ulimit -n 32000 #allows elastic search to hold more than the default 1000 files open
ARGS="-Xmx$JAVAMEM -Xms$JAVAMEM -jar /opt/logstash/logstash-monolithic.jar agent --config ${CONFIG_DIR} --log ${LOGFILE} --grok-patterns-path ${PATTERNSPATH} -- web --backend elasticsearch:///?local"

This last sections tells the engine to start a web server (port 9292 default) and also use the local server as the elasticsearch provider. Everything is therefore running on one server in this instance which also maintains syslog for all other devices on the network.

Create the new user to run as:

sudo adduser --system --disabled-password --no-create-home --group --quiet logstash

Allow the user to read syslog files (probably a far better way to do this but I am not concerned about security at the moment)

sudo usermod -a -G adm logstash

Create all the directories mentioned on the script:

mkdir /opt/logstash/
mkdir /etc/logstash.d/
mkdir /var/log/logstash/
mkdir /opt/logstash/patterns/

CHOWN the directories as per the following example:

sudo chown -R logstash:logstash /etc/logstash.d/
sudo chown -R logstash:logstash /var/log/logstash/
sudo chown -R logstash:logstash /opt/logstash/

Download the jar file and rename:

sudo wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
sudo mv logstash-1.1.0-monolithic.jar logstash-monolithic.jar

Add the config below to /etc/logstash.d/mylogstash.conf

input {
 file {
 type => "linux-syslog"
# Wildcards work, here
 path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
 }
# file {
# type => "apache-access"
# path => "/var/log/apache2/access.log"
# }
# file {
# type => "apache-error"
# path => "/var/log/apache2/error.log"
# }
}
output {
 # Emit events to stdout for easy debugging of what is going through
 # logstash.
 stdout { }
# This will use elasticsearch to store your logs.
 # The 'embedded' option will cause logstash to run the elasticsearch
 # server in the same process, so you don't have to worry about
 # how to download, configure, or run elasticsearch!
 elasticsearch { embedded => true }
}

As you can see, all we look to do in the first instance is syslog search only. I have removed the apache logging for the moment (basically since its not installed on my test server!)

Make the logstash script executable and start the engine

sudo chmod +x /etc/init.d/logstash
sudo /etc/init.d/logstash start

If all is good, you can

 tail -f /var/log/logstash/logstash.log

And hopefully see the engine start and the fact its logging syslog content as it arrives at the server.

Then, give it a few minutes, open your browser, then hopefully you will have a log crawler and searcher! Ill be leaving this running for a few weeks to see how it scales so will report back on performance and any additional tweaks required.

9 thoughts on “Getting Logstash to run as a daemon”

  1. This script is not working in my scenario.
    do you have any update on this
    my logstash daemon is get start and after few second it will automatically get stop.
    I dont know the reason can you please solve my query.

    –Ram

  2. yes all paths are correct and i have java installed in my system.
    Now what will be the error.

    Its get started properly but after few second it’s get stop automatically and no logs are generated for this.

    Can you please help me…

    Thanks
    Ram

  3. hi my query is resolved.

    Please add 2 more line in this blog i.e.
    sudo chown -R logstash:logstash /var/log/logstash
    sudo chown -R logstash:logstash /opt/logstash

    thats it.

    Thanks
    Ranjit

  4. Thank you for this post. You mentioned there might be better ways to handle the file permissions on the logs, and one better way would be to use the set file access control lists (`setfacl`) command to add the logstash group with read permissions to only the log files you are importing. This can be done with a command similar to:

    sudo setfacl -m g:logstash:r /var/log/syslog

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.