Setting Upwards Elk As Well As Pushing Relational Information Using Logstash Jdbc Input Plugin As Well As Secrets Keystore

Introduction

In this tutorial, nosotros volition become over the installation of Elasticsearch. We volition also demo you lot how to configure it to assemble in addition to visualize information from a database.

Logstash is an opened upwards source tool for collecting, parsing in addition to storing logs/data for hereafter use.

Kibana is a spider web interface that tin live used to search in addition to persuasion the logs/data that Logstash has indexed. Both of these tools are based on Elasticsearch, which is used for storing logs.

Our Goal

The destination of this tutorial is to ready Logstash to assemble records from a database in addition to ready Kibana to practise a visualization.

Our ELK stack setup has 3 principal components:

• Logstash: The server cistron of Logstash that processes database records

• Elasticsearch: Stores all of the records

• Kibana: Web interface for searching in addition to visualizing logs.

Install Elasticsearch

Using the MSI Installer package. The packet contains a graphical user interface (GUI) to guides you lot through the installation process.

Then double-click the downloaded file to launch the GUI. Within the commencement screen, conduct the deployment directories:

Then conduct whether to install every bit a service or start Elasticsearch manually every bit needed. Choose install every bit a service:

For configuration, precisely leave of absence the default values:

Uncheck all plugins to non install whatsoever plugins:

After clicking the install button, Elasticsearch volition live installed:

To depository fiscal establishment lucifer if the Elasticsearch is running, open ascendancy prompt in addition to type "services.msc" in addition to expect for Elasticsearch. You should reckon the condition that it is 'Running'.

Or precisely download the zipped file from https://www.elastic.co/downloads/elasticsearch.

Install Kibana

You tin download the Kibana https://www.elastic.co/downloads/kibana.

After downloading Kibana in addition to unzipping the file you lot volition reckon a folder construction every bit below.

The runnable file is located at bin\kibana.bat.

To test, start kibanat.bat in addition to betoken your browser at http://localhost:5601 in addition to you lot should reckon a spider web page like below:

Install Logstash

You tin download the Logstash https://www.elastic.co/products/logstash.

After downloading Logstash in addition to unzipping the file you lot volition reckon a folder construction every bit below.

The runnable file is located at bin\logstash.bat.

Inserting Data Into Logstash By Using Select Data from Database

Elastic.co has a proficient weblog regarding this topic that you lot tin catch https://www.elastic.co/blog/logstash-jdbc-input-plugin.

Secrets keystore

When you lot configure Logstash, you lot mightiness quest to specify sensitive settings or configuration, such every bit passwords. Rather than relying on file organization permissions to protect these values, you lot tin role the Logstash keystore to securely shop clandestine values for role inwards configuration settings.

Create a keystore

To practise a secrets keystore, role the create:

bin/logstash-keystore create

Add keys

To shop sensitive values, such every bit authentication credentials, role the add together command:

bin/logstash-keystore add together PG_PWD

When prompted, encounter a value for the key.

We could role keystore to shop values jdbc_connection_string, jdbc_user, jdbc_password, etc.

For simplicity, an underscore was added for referencing the keys. See below for sample config file.

Let's state that a Postgresql tabular array was changed afterwards pushing the information to Elasticsearch. Those changes volition non live introduce inwards Elasticsearch. To run out on Elasticsearch updated nosotros quest to update it past times running logstash alongside the configuration below. 

In this configuration nosotros are running logstash every second, of course, you lot wouldn't practise that :-) Normally nosotros run per day, week, month, etc. Can live configure depending on your needs.

References:

• Secrets keystore
• Insert Into Logstash Select Data From Database
• Pushing Relational Data to Elasticsearch using Logstash JDBC input plugin