As a DevOps, it is very difficult to identify, collect and process the logs which could of System logs, Web server logs etc.
The one of the solution of above problem could be ELK stack.
ELK stack mainly composed of 3 components:
- Elasticsearch: Store the logs collected from different servers
- Logstash: Collect all types of logs at centralized location
- Kibana: Runs the searchs and generate visual dashboard
How ELK works ?
Logstash is an open source tool for collecting, parsing, and storing logs for future use. Kibana is a web interface that can be used to search and view the logs that Logstash has indexed. Both of these tools are based on Elasticsearch. All these together are called ELK Stack.
Using Elasticsearch as a backend datastore, and kibana as a frontend reporting tool, logstash as pipeline for storing, querying and analyzing your logs.
ELK Stack Setup has mainly four components:
- Logstash: Server component which collects all the incoming logs
- Elasticsearch: stores all the logs
- Kibana: Web interface for searching and analysing logs
- Logstash Forwarder: Installed on server from which logs need to be collected. It acts as an agent uses lumberjack networking protocol to communicate with logstash.
First requirement in order to setup ELK stack is to have java installed on the system.
It is recommended to run a recent version of Java in order to ensure the greatest success in running Logstash.
It’s fine to run an open-source version such as OpenJDK:http://openjdk.java.net/
Or you can use the official Oracle version:http://www.oracle.com/technetwork/java/index.html
It’s fine to run an open-source version such as OpenJDK:http://openjdk.java.net/
Or you can use the official Oracle version:http://www.oracle.com/technetwork/java/index.html
Install Elasticsearch
Run the below command to get repo from apt
sudo wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
Create the Elasticsearch source list
sudo echo 'deb http://packages.elasticsearch.org/elasticsearch/1.1/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
Update your apt package database:
sudo apt-get update
Now, we have the source to download elasticsearch, give the below command to install it
sudo apt-get -y install elasticsearch=1.1.1
Once, it is installed, we need to add below line to /etc/elasticsearch/elasticsearch.yml, to disable dynamic scripts.
script.disable_dynamic: true
In order to restrict the outside access to Elasticsearch instance(9200) by specifing the below line in configuration file.
network.host: localhost
If you want to access you Elasticsearch instance from clients on a different IP address via Javascript, add the following inside elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-origin: "*"
Once you are done with changes, save the file and restart the service
sudo service elasticsearch restart
In order to boot up the Elasticsearch
sudo update-rc.d elasticsearch defaults 95 10
Now your Elasticsearch is all set to store for your log from multiple servers.
Testing Elasticsearch:
Use the below commands to test the working of elasticsearch
ps aux | grep elasticsearch
curl -X GET 'http://localhost:9200'
curl 'http://localhost:9200/_search?pretty'
In Browser: http://localhost:9200/
Elasticsearch Kopf Plugin
The kopf plugin provides an admin GUI for Elasticsearch. It helps in debugging and managing clusters and shards. It’s really easy to install:
sudo /usr/share/elasticsearch/bin/plugin -install lmenezes/elasticsearch-kopf
Below is the screenshot for your understanding:
Install Kibana
Use the below link to download the build to install kibana
sudo wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gz
Extract the build,
sudo tar xvf kibana-3.0.1.tar.gz
Now, we need to open the kibana configuration to specify the port for elasticsearch from 9200 to 80, so we will later setup proxy server which could be nginx or squid etc. to redirect all request from port 80 to 9200. See below line in location ~/kibana/config.js
elasticsearch: "http://"+window.location.hostname+":80",
This is necessary as kibana will be running on 80 port.
Since, we are going to install nginx, we need to specify particular location for kibana say.
/opt/kibana3/
Copy all the contents to this location:
cp -R ~/kibana/* /opt/kibana3/
Now, we hava setup the kibana, as discussed we need to install nginx, which is pretty easy.
Install Nginx
Use below command to install nginx
sudo apt-get install nginx
Next, download the Nginx configuration from kibana's github repositories to your home dir.
sudo wget https://gist.githubusercontent.com/thisismitch/2205786838a6a5d61f55/raw/f91e06198a7c455925f6e3099e3ea7c186d0b263/nginx.conf
In above configuration we need to specify the FQDN (Fully Qualified Domain Name), and Kibana's root directory.
server_name FQDN;
root /root/kibana3;
root /root/kibana3;
Once you are done with changes move the file to nginx configuration location
sudo cp nginx.conf /etc/nginx/sites-available/default
In order to improve your security to access nginx, you can apache2-utils to use htpasswd to generate username and password.
sudo apt-get install apache2-utils
sudo htpasswd -c /etc/nginx/conf.d/kibana.htpasswd user
Now, you can restart the nginx service for the changes to take effect.
Kibana is now accessible via your FQDN or the public IP address of your Logstash Server i.e. http://logstash_server_public_ip/. If you go there in a web browser, you should see a Kibana welcome page which will allow you to view dashboards but there will be no logs to view because Logstash has not been set up yet. Let's do that now.
Install Logstash
The Logstash package is available from the same repository as Elasticsearch. As public key is already setup, let's create the source list
sudo echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list
Update your apt package database
sudo apt-get update
Install the logstash using below command:
sudo apt-get install logstash=1.4.2-1-2c0f5a1
Generate SSL certificates:
Since we are going to use Logstash Forwarder to ship logs from our Servers to our Logstash Server, we need to create an SSL certificate and key pair. The certificate is used by the Logstash Forwarder to verify the identity of Logstash Server. Create the directories that will store the certificate and private key with the following commands:
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
If you have a DNS setup with your private networking, you should create an A record that contains the Logstash Server's private IP address—this domain name will be used in the next command, to generate the SSL certificate. Use the below command to generate it:
sudo cd /etc/pki/tls; sudo openssl req -subj '/CN=logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
Since, ssl certificate is generated, we need to transfer it to server from which we are going to ships logs to logstash.
Configure Logstash
Now, lets configure the logstash so that kibana can get the log from it.
Logstash configuration files are in the JSON-format, and reside in /etc/logstash/conf.d. The configuration consists of three sections: inputs, filters, and outputs.
Let's create a configuration file called 01-lumberjack-input.conf and set up our "lumberjack" input (the protocol that Logstash Forwarder uses):
Create a file at location /etc/logstash/conf.d/01-lumberjack-input.conf
which contains
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
Now let's create a configuration file called 10-syslog.conf, where we will add a filter for syslog messages at location /etc/logstash/conf.d/10-syslog.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "% {SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
if [type] == "syslog" {
grok {
match => { "message" => "% {SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
This filter looks for logs that are labeled as "syslog" type (by a Logstash Forwarder), and it will try to use "grok" to parse incoming syslog logs to make it structured and query-able at location /etc/logstash/conf.d/30-lumberjack-output.conf
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
This output basically configures Logstash to store the logs in Elasticsearch.
With this configuration, Logstash will also accept logs that do not match the filter, but the data will not be structured (e.g. unfiltered Nginx or Apache logs would appear as flat messages instead of categorizing messages by HTTP response codes, source IP addresses, served files, etc.).
Now Restart the logstash to make changes take effect:
sudo service logstash restart
Setup Logstash Forwarder Package:
Get the SSL certificate which we generated previously to server from which we get the logs
sudo scp /etc/pki/tls/certs/logstash-forwarder.crt user@server_private_IP:/tmp
Install Logstash-forwarder Package:
On Server, create the Logstash Forwarder source list:
sudo echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list
It also uses the same GPG key as Elasticsearch, which can be installed with this command:
sudo wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
Then install the Logstash Forwarder package:
sudo apt-get install logstash-forwarder
Next, you will want to install the Logstash Forwarder init script, so it starts on bootup:
sudo cd /etc/init.d/; sudo wget https://raw.githubusercontent.com/elasticsearch/logstash-forwarder/a73e1cb7e43c6de97050912b5bb35910c0f8d0da/logstash-forwarder.init -O logstash-forwarder
sudo chmod +x logstash-forwarder
sudo update-rc.d logstash-forwarder defaults
Now copy the SSL certificate into the appropriate location (/etc/pki/tls/certs):
sudo mkdir -p /etc/pki/tls/certs
sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/
Configure Logstash Forwarder:
We need to specify in logstash forwarder configuration details of the logstash server so that it can send the details to server.
Now add the following lines into the file, substituting in your Logstash Server's private IP address for logstash_server_private_IP at location /etc/logstash-forwarder:
{
"network": {
"servers": [ "logstash_server_private_IP:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/syslog",
"/var/log/auth.log"
],
"fields": { "type": "syslog" }
}
]
}
"network": {
"servers": [ "logstash_server_private_IP:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/syslog",
"/var/log/auth.log"
],
"fields": { "type": "syslog" }
}
]
}
Now, restart the logstash forwarder using below command
sudo service logstash-forwarder restart
Now, we have setup ELK stack, we just need to check the kibana dashboard for more details.
Below are the useful screenshots which represents the data which we got from server.
Conclusion:-
The ELK stack is very useful in different manners as in this case you will find almost all the logs that could be application log, webserver log, system log etc. Using kibana, will have a visualize report which shows the analysis of the logs which is very helpful to extract the required information.
The ELK stack is easy to maintain on the large scale systems and and large scale infrastructure.
No comments:
Post a Comment