Currently evaluating Logstash and its centralised log management facilities. So far its looking like a good candidate for replacement of far more expensive utilities and seems very nippy when searching. The search query syntax is well documented so its very easy to use once set up.
As ever, using the nice and simple approach to get it working in the first instance as a standalone server. Then needing to keep the engine running for long term testing, its obviously better to run as a daemon. This was a bit of a cludge so noted for ref.
Cribbing https://gist.github.com/1121786 and creating the following:
sudo vi /etc/init.d/logstash
Copy the contents of the github link. A few small changes are required:
DAEMON=/usr/bin/java #obviously! ulimit -n 32000 #allows elastic search to hold more than the default 1000 files open
ARGS="-Xmx$JAVAMEM -Xms$JAVAMEM -jar /opt/logstash/logstash-monolithic.jar agent --config ${CONFIG_DIR} --log ${LOGFILE} --grok-patterns-path ${PATTERNSPATH} -- web --backend elasticsearch:///?local"
This last sections tells the engine to start a web server (port 9292 default) and also use the local server as the elasticsearch provider. Everything is therefore running on one server in this instance which also maintains syslog for all other devices on the network.
Create the new user to run as:
sudo adduser --system --disabled-password --no-create-home --group --quiet logstash
Allow the user to read syslog files (probably a far better way to do this but I am not concerned about security at the moment)
sudo usermod -a -G adm logstash
Create all the directories mentioned on the script:
mkdir /opt/logstash/ mkdir /etc/logstash.d/ mkdir /var/log/logstash/ mkdir /opt/logstash/patterns/
CHOWN the directories as per the following example:
sudo chown -R logstash:logstash /etc/logstash.d/ sudo chown -R logstash:logstash /var/log/logstash/ sudo chown -R logstash:logstash /opt/logstash/
Download the jar file and rename:
sudo wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar sudo mv logstash-1.1.0-monolithic.jar logstash-monolithic.jar
Add the config below to /etc/logstash.d/mylogstash.conf
input { file { type => "linux-syslog" # Wildcards work, here path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ] } # file { # type => "apache-access" # path => "/var/log/apache2/access.log" # } # file { # type => "apache-error" # path => "/var/log/apache2/error.log" # } } output { # Emit events to stdout for easy debugging of what is going through # logstash. stdout { } # This will use elasticsearch to store your logs. # The 'embedded' option will cause logstash to run the elasticsearch # server in the same process, so you don't have to worry about # how to download, configure, or run elasticsearch! elasticsearch { embedded => true } }
As you can see, all we look to do in the first instance is syslog search only. I have removed the apache logging for the moment (basically since its not installed on my test server!)
Make the logstash script executable and start the engine
sudo chmod +x /etc/init.d/logstash sudo /etc/init.d/logstash start
If all is good, you can
tail -f /var/log/logstash/logstash.log
And hopefully see the engine start and the fact its logging syslog content as it arrives at the server.
Then, give it a few minutes, open your browser, then hopefully you will have a log crawler and searcher! Ill be leaving this running for a few weeks to see how it scales so will report back on performance and any additional tweaks required.
This script is not working in my scenario.
do you have any update on this
my logstash daemon is get start and after few second it will automatically get stop.
I dont know the reason can you please solve my query.
–Ram
You are going to have to give me a smidge more info on this. You have installed Java to /usr/bin/java and Ill assume all the paths are correct?
yes all paths are correct and i have java installed in my system.
Now what will be the error.
Its get started properly but after few second it’s get stop automatically and no logs are generated for this.
Can you please help me…
Thanks
Ram
hi my query is resolved.
Please add 2 more line in this blog i.e.
sudo chown -R logstash:logstash /var/log/logstash
sudo chown -R logstash:logstash /opt/logstash
thats it.
Thanks
Ranjit
Thanks for the feedback. Updated.
Thank you for this post. You mentioned there might be better ways to handle the file permissions on the logs, and one better way would be to use the set file access control lists (`setfacl`) command to add the logstash group with read permissions to only the log files you are importing. This can be done with a command similar to:
sudo setfacl -m g:logstash:r /var/log/syslog
I have a similar walkthrough , some slightly different changes, works on my centos vm
https://sites.google.com/site/bladelogicwiki/logstash_elasticsearch/guides/install-elastic_logstash_kibana/run-logstash-as-daemon
how compatible is the logstash-1.1.0-monolithic.jar file with logstash version 1.4.0?
No idea! To be fair the post is a bit outdated!