Skip to main content

Elasticsearch - OpenTelemetry Collector

Thumbnail icon Thumbnail icon

The Elasticsearch app is a unified logs and metrics app that helps you monitor the availability, performance, health, and resource utilization of your Elasticsearch clusters. Preconfigured dashboards provide insight into cluster health, resource utilization, sharding, garbage collection, and search, index, and cache performance.

We use the OpenTelemetry collector to collect Elasticsearch metrics and logs.

The diagram below illustrates the components of the Elasticsearch collection for each database server. OpenTelemetry collector runs on the same host as Elasticsearch, and uses the Elasticsearch Receiver to obtain Elasticsearch metrics, and the Sumo Logic OpenTelemetry Exporter to send the metrics to Sumo Logic. Elasticsearch logs are sent to Sumo Logic through a filelog receiver.

Schematics

Fields Create in Sumo Logic for Elasticsearch​

Following are the Fields which will be created as part of Elasticsearch App install, if not already present:

  • db.cluster.name. User configured. Enter a name to identify this Elasticsearch cluster. This cluster name will be shown in the Sumo Logic dashboards.
  • db.system. Has a fixed value of elasticsearch.
  • sumo.datasource. Has a fixed value of elasticsearch.
  • db.node.name. Has the value of host name of the machine which is being monitored.

Prerequisites​

This receiver supports Elasticsearch versions 7.9+.

If Elasticsearch security features are enabled, you must have either the monitor or manage cluster privilege. See the Elasticsearch docs for more information on authorization and Security privileges.

Configure logging in Elasticsearch. Elasticsearch supports logging via local text log files. Elasticsearch logs have four levels of verbosity. To select a level, set loglevel to one of:

  • debug: a lot of information, useful for development/testing
  • verbose: includes information not often needed, but logs less than debug
  • notice (default value): moderately verbose, ideal for production environments
  • warning: only very important/critical messages are logged

All logging settings are located in Elasticsearch.conf. By default, Elasticsearch logs are stored in /var/log/elasticsearch/ELK-<Clustername>.log. The default directory for log files is listed in the Elasticsearch.conf file.

Collecting Logs, Metrics, and installing Elasticsearch app​

Here are the steps for collecting logs, metrics, and installing the app.

Step 1: Set up Collector​

note

If you want to use an existing OpenTelemetry Collector, you can skip this step by selecting the Use an existing Collector option.

To create a new Collector:

  1. Select the Add a new Collector option.
  2. Select the platform where you want to install the Sumo Logic OpenTelemetry Collector.

This will generate a command that you can execute in the machine environment you need to monitor. Once executed, it will install the Sumo Logic OpenTelemetry Collector.

Collector

Step 2: Configure integration​

In this step, you will configure the yaml required for Elasticsearch Collection.

Below are the inputs required:

  • Endpoint. Enter the url of the server you need to monitor. Example: http://localhost:9200.
  • User Name. Enter the Elasticsearch Username.
  • Elasticsearch cluster log path. By default, Elasticsearch logs are stored in /var/log/elasticsearch/ELK-<Clustername>.log.
  • Tags. db.cluster.name.

You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the Download YAML File button to get the yaml file.

For Linux platform, click on Download Environment Variables File button to get the file with the password which is supposed to be set as environment variable.

YAML

Step 3: Send logs and metrics to Sumo​

Once you have downloaded the yaml file as described in the previous step, follow the below steps based on your platform.

  1. Copy the yaml file to /etc/otelcol-sumo/conf.d/ folder in the Elasticsearch instance which needs to be monitored.
  2. Place Env file in the following directory:
    /etc/otelcol-sumo/env/
  3. Testart the collector using:
     sudo systemctl restart otelcol-sumo

After successfully executing the above command, Sumo Logic will start receiving data from your host machine.

Click Next. This will install the app (dashboards and monitors) to your Sumo Logic Org.

Dashboard panels will start to fill automatically. It's important to note that each panel fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but within 20 minutes, you'll see full graphs and maps.

Sample Log Messages​

{
   "type":"server",
   "timestamp":"2021-07-12T11:42:25,862+07:00",
   "level":"INFO",
   "component":"o.e.x.s.a.s.FileRolesStore",
   "cluster.name":"elasticsearch",
   "node.name":"v103-157-218-134.3stech.vn",
   "message":"parsed [0] roles from file [/etc/elasticsearch/roles.yml]"
}

Sample OpenTelemetry metrics​

{
    "queryId":"A",
    "_source":"sumo_hosted_collector_otel_elasticsearch",
    "state":"completed",
    "thread_pool_name":"analyze",
    "elasticsearch.node.name":"ip-172-31-86-95",
    "elasticsearch.cluster.name":"elasticsearch",
    "metric":"elasticsearch.node.thread_pool.tasks.finished",
    "db.cluster.name":"elastic_otel_cluster",
    "_collectorId":"000000000C5B7100",
    "deployment.environment":"otel_elastic_dev",
    "_sourceId":"0000000000000000",
    "unit":"{tasks}",
    "db.system":"elasticsearch",
    "_sourceHost":"sumoOtelelasticsearch",
    "_collector":"sumo_hosted_collector_otel_elasticsearch",
    "max":0,
    "min":0,
    "avg":0,
    "sum":0,
    "latest":0,
    "count":2
}

Sample Queries​

Logs​

Sample log query from the Errors panel.

db.system=elasticsearch %"deployment.environment"={{deployment.environment}} db.cluster.name={{db.cluster.name}} ERROR | json "log" as _rawlog nodrop 
| if (isEmpty(_rawlog), _raw, _rawlog) as _raw
| json field=_raw "timestamp" as timestamp
| json field=_raw "level" as level
| json field=_raw "component" as es_component
| json field=_raw "message" as message
| where level = "ERROR"
| count

Metrics​

Metrics query from the JVM Memory Used (MB) panel.

deployment.environment=* metric=jvm.memory.heap.used db.cluster.name=* db.node.name=* | sum by db.cluster.name, db.node.name

Viewing Elasticsearch Dashboards​

Overview​

The Elasticsearch - Overview dashboard provides the health of Elasticsearch clusters, shards analysis, resource utilization of Elasticsearch host and clusters, search and indexing performance.

Overview

Total Operations Stats​

The Elasticsearch - Total Operations stats dashboard provides information on the operations of the Elasticsearch system.

Total Operations Stats

Thread Pool​

The Elasticsearch - Thread Pool dashboard analyzes thread pools operations to manage memory consumption of nodes in the cluster.

Thread Pool

Resource​

The Elasticsearch - Resource dashboard monitors JVM Memory, Network, Disk, Network and CPU of Elasticsearch node.

Resource

Performance Stats​

The Elasticsearch - Performance Stats dashboard performance statistics such as latency and Translog operations and size.

Performance Stats

Indices​

The Elasticsearch - Indices dashboard monitors Index operations, size and latency. It also provides analytics on doc values, fields, fixed bitsets, and terms memory.

Indices

Documents​

The Elasticsearch - Documents dashboard provides analytics and monitoring on Elasticsearch documents.

Documents

Caches​

The Elasticsearch - Caches dashboard allows you to monitor query cache size, evictions and field data memory size.

Caches

Errors And Warnings​

The Elasticsearch - Errors And Warnings dashboard shows errors and warnings by Elasticsearch components.

Errors And Warnings

Garbage Collection​

The Elasticsearch - Garbage Collector dashboard provides information on the garbage collection of the Java Virtual Machine.

Garbage Collection

Login And Connections​

The Elasticsearch - Login And Connections dashboard shows geo location of client connection requests, failed connection logins and count of failed login attempts.

Login And Connections

Operations​

The Elasticsearch - Operations dashboard allows you to monitor server stats and events such as node up/down, index creation/deletion. It also provides disk usage and cluster health status.

Operations

Queries​

The Elasticsearch - Queries dashboard shows Elasticsearch provides analytics on slow queries, and query shards.

Resource Usage
Legal
Privacy Statement
Terms of Use

Copyright © 2023 by Sumo Logic, Inc.