Using Metrics Toolkit with Bundled ELK Dashboards
This document covers steps to setup and integrate ELK with Metrics-Toolkit (formerly metrics accelerator/framework) in the local Windows environment. To be more specific, we will be installing Elasticsearch to capture the metrics and Kibana to visualize that data on dashboards. We’ll be making use of the pre-built dashboards that come bundled with the Metrics-Toolkit.
Now, we can either install and use Elasticsearch & Kibana directly in the local environment or we can also deploy them in Docker containers. For this guide, we will be using the Docker option.
Also, since the Metrics-Toolkit repository has not been updated in a while, for this guide, we are using version 8.4.1 of Elasticsearch and Kibana for better compatibility. However, you should be able to use the latest versions just as fine.
Let’s begin...
Pre-requisites
- Installed Docker Desktop - Link
If you are using WSL 2 Based Engine in Docker instead of Hyper-V backend, then make sure that your .wslconfig file (Refer - Link) has an entry for allocating atleast 4 GB memory to WSL 2 VM.
Press Windows + R and type %UserProfile% then hit Enter. If there is no file named .wslconfig here, then create a text file and change its name (with extension) to .wslconfig
Type following in that text file and save it -
- Imported Metrics-Toolkit Code from GitHub in Anypoint Studio - Link
- Mulesoft Anypoint Platform Account and Connected Apps Credentials set up as mentioned in the Metrics-Toolkit Documentation
- Anypoint Studio and Java (Be sure to verify that AnypointStudio.ini file’s -vm path is pointing to the JDK you are using).
For this demo, I am using JDK 11.0.17 - Link. Alternatively, you can also use Amazon Corretto 11 - Link. But MuleSoft recommends JDK 8 for best compatibility in Anypoint Studio and Cloudhub deployments. I have also correctly configured %JAVA_HOME% and Path system environment variables in Windows.
Anypoint Studio being used is version 7.14 and its AnypointStudio.ini file has following entry -
Installing Elasticsearch & Kibana
Open Windows Command Prompt (Admin).
Type this command and hit enter to pull Elasticsearch 8.4.1 image -
After it finishes, type the next command to pull Kibana 8.4.1 image and hit enter -
After that is done, type the following command to verify that you have successfully downloaded the two images in your local environment -
Next, we type the following command to create a docker network named elastic -
Now, before we proceed further, please note that as the Elasticsearch container gets created, you will receive some initial credentials & tokens which expire within 30 Minutes. So, from this point, please do not get distracted by some other work or you will have to generate a new password and token manually if you stay idle for more than 30 Minutes.
Type the following command to create Elasticsearch container named esdemo -
Do not use -d argument in the above command or else you may not see the credentials in Docker logs.
As this command completes, you will receive your credentials and tokens in the Command Prompt window like this (You will have to scroll up a little to find them) -
Copy and paste them somewhere safe in a notepad window as we’ll need them. Do not close the Command Prompt yet.
Let’s test our Elasticsearch installation before proceeding further. In your browser go to this address (https is important)-
It will ask for Username & Password. Type elastic as username and password from the notepad. Hit enter and you should be able to see a JSON payload.
This confirms that Elasticsearch is up and running fine.
Leave the Command Prompt window open in the background. If you close it now, the Elasticsearch container may exit, but going forward you can start it from the Docker Desktop again.
One more important step here is to copy the HTTP CA certificate file from the Elasticsearch container in your local Windows environment. After that, we will have to import it in our Java Dev Kit installation.
Open a new Windows Command Prompt (Admin), then type the following command and hit enter -
This command assumes that your Elasticsearch container name is esdemo and will copy the certificate into your current Windows user’s Documents folder.
Now that you have copied the certificate to your local Windows environment, you can use the command below to import it into your JDK installation.
The command prompt will ask for confirmation, type Y and hit enter.
This command assumes that you copied the certificate file in the Documents folder and your JDK is installed in the location mentioned in the pre-requisites section.
Now, we will set up Kibana.
Open a new Windows Command Prompt (Admin).
Type the following command and hit enter -
After few seconds, you will get following message and a URL in the Command Prompt window -
A browser window will open too. If it doesn’t then simply open your browser and go to that address (Remember to change 0.0.0.0 to localhost) (Also, note that your code will be different) -
The webpage will ask for a token. Enter the token which you copied from the Elasticsearch installation in the notepad window.
Next, enter the Elasticsearch credentials. Username will be elastic and password is the one you copied in notepad earlier. You will enter the Kibana portal as you login successfully.
If you close the command prompt window now, the Kibana container may exit, but going forward you can start it from the Docker Desktop again.
This completes the installation part for Elasticsearch and Kibana.
Configuration
We can now proceed with the configuration part. Assuming that the Elasticsearch and Kibana containers are up and running, open the Kibana portal -
Login with elastic credentials and click on the hamburger ≡ icon on the top-left corner of the page, then scroll down to the Management heading and click on Dev Tools.
Clear the left-hand side console window and type in the following 2 commands -
Run these two commands one by one. This will create two indexes named metrics and platformbenefits in Kibana.
Now, let’s send data from Metrics-Toolkit and test if we are able to receive something or not.
Load the metrics-toolkit project in Anypoint Studio and open the following file -
Here, we make changes to specific fields as following -
This configuration change translates as following – Metrics-Toolkit will fetch data from Anypoint Platform every minute (CRON format - Link) using the Connected App credentials and feed that data into ELK.
Now, go to this path -
Make a copy of _template.yaml file in the same folder and refactor its name to app-dev.yaml
In this newly created file, put down the respective credentials but encrypt them first using Secure Properties Tool - Link
Don’t forget to provide your encryption key in Run configuration as environment variables – mule.key
Run the application and wait. If all goes well, then the Console View in Anypoint Studio should log this message every minute – Metrics Aggregated Successfully.
Now, let’s get back into Kibana portal, click on the hamburger ≡ icon, scroll down to the Management heading and open the Dev Tools section again. Clear the console and this time type -
Run them one by one and you will see Anypoint Platform data sent by Metrics-Toolkit in JSON format. You may not find any data when you send a GET request to /platformbenefits. Perhaps this exists for some other purpose.
Until this point, you were successful in using Metrics-Toolkit to fetch data from Anypoint Platform and then sending it to ELK. Now, we make use of the pre-built dashboards in the Metrics-Toolkit repository.
In the Kibana portal, click on the hamburger ≡ icon, then scroll down to the Management heading and click on Stack Management. Now, in this new menu, scroll down to the Kibana heading and click on Data Views. On the right side, click on the Create Data View button.
Give the data view a name of your choice and then in the Index pattern field, type the following -
This will ensure that the data view we are creating only fetches data from these 2 indexes. Then choose the Timestamp field as date.
If you want to, click on Show advanced settings and give your data view a Custom Data View ID. We’ll need the data view ID in the next steps.
Now just click on the Save Data View to Kibana button and your newly created data view will show up with all the fields sent by Metrics-Toolkit.
If you didn’t create a Custom Data View ID or forgot the name you entered, then click on the Edit button on the upper-right portion of this screen. Next, click on Show advanced settings, and you will see the auto-generated Data View ID or your Custom Data View ID. Copy this Data View ID.
Now, go to this path-
You will find a bunch of *.ndjson files here. These are pre-built dashboards that come bundled with Metrics-Toolkit. Open any one of them in a text editor like notepad and then Find & Replace - <YOUR-INDEX-PATTERN-ID> with the Data View ID mentioned previously. Save the changes done to this *.ndjson file.
We are almost done now, let’s get back to the Kibana portal. While still being in the Stack Management section, under the Kibana heading, click on Saved Objects. Here on the upper-right side, click on the Import link, then browse and select the *.ndjson file you just edited. Click on the Import button and then Done.
That’s it. Click on the hamburger ≡ icon, then under the Analytics heading, click on Dashboards. Here you have the dashboard corresponding to the *.ndjson file you uploaded. Click on it and you’ll see the pre-built charts but with live data from Anypoint Platform (will have to click on the Refresh button manually though).
Troubleshooting Common Issues
Known Issue #1
Next time when you start Elasticsearch container from Docker Desktop if the container stops itself after few seconds while throwing the following error in the logs -
Then, open a new Windows Command Prompt (Admin). Type this command and hit enter -
After that, type this command and hit enter -
Now, you can start the Elasticsearch container from Docker Desktop. Note that this is a temporary solution and when you restart your machine, you will have to do it again. You can find a permanent solution somewhere online. But for this demo purpose, I’m leaving this as is.
Known Issue #2
If you get the following error in Anypoint Studio -
This means that there is a problem with the certificate installation. You may have imported it into some other JDK installation while your Anypoint Studio is pointing to its internal JDK.
Comments
Post a Comment