Stay up-to-date on the health of your Elasticsearch cluster, from its overall status down to JVM heap usage and everything in between. Get notified when you need to revive a replica, add capacity to the cluster, or otherwise tweak its configuration. After doing so, track how your cluster metrics respond.

The Datadog Agent's Elasticsearch check collects metrics for search and indexing performance, memory usage and garbage collection, node availability, shard statistics, disk space and performance, pending tasks, and many more. The Agent also sends events and service checks for the overall status of your cluster.

The Elasticsearch check is included in the Datadog Agent package, so you don't need to install anything else on your Elasticsearch nodes, or on some other server if you use a hosted Elasticsearch (e.g. Elastic Cloud).

1. Edit the elastic.d/conf.yaml file, in the conf.d/ folder at the root of your Agent's configuration directory to start collecting your Elasticsearch metrics and logs. See the sample elastic.d/conf.yaml for all available configuration options.

2. Restart the Agent

• Add this configuration block to your elastic.yaml file to start gathering your ElasticSearch metrics:

init_config:

instances:
  - url: http://localhost:9200 # or wherever your cluster API is listening
    cluster_stats: false # set true ONLY if you're not running the check on each cluster node
    pshard_stats: true # the agent sends primary shard metrics
    index_stats: true # the agent sends index level metrics
    pending_task_stats: true # the agent sends cluster-wide pending task metrics

Note:

• If you're collecting Elasticsearch metrics from just one Datadog Agent running outside the cluster - e.g. if you use a hosted Elasticsearch - set cluster_stats to true.

• To use the Agent's Elasticsearch integration for the AWS Elasticsearch services, set the url parameter to point to your AWS Elasticsearch stats URL.

See the sample elastic.yaml for all available configuration options, including those for authentication to and SSL verification of your cluster's API url.

Finally, Restart the Agent to begin sending Elasticsearch metrics to Datadog.

Available for Agent >6.0

• Collecting logs is disabled by default in the Datadog Agent, enable it in the datadog.yaml file with:

  logs_enabled: true
  

• Then Add this configuration block to your elastic.d/conf.yaml file to start collecting your Elasticsearch logs:

    logs:
        - type: file
          path: /var/log/elasticsearch/*.log
          source: elasticsearch
          service: myservice

  Change the path and service parameter values and configure them for your environment.

• Restart the Agent to begin sending Elasticsearch logs to Datadog.

Learn more about log collection in the log documentation

Run the Agent's status subcommand and look for elastic under the Checks section.

By default, not all of the following metrics are sent by the Agent. To send all metrics, configure flags in elastic.yaml as shown above.

• pshard_stats sends elasticsearch.primaries.* and elasticsearch.indices.count metrics
• index_stats sends elasticsearch.index.* metrics
• pending_task_stats sends elasticsearch.pending_* metrics

For version >=6.3.0, set xpack.monitoring.collection.enabled configuration to true in your Elasticsearch configuration in order to collect all elasticsearch.thread_pool.write.* metrics. See Elasticsearch release notes - monitoring section.

See metadata.csv for a list of metrics provided by this integration.

The Elasticsearch check emits an event to Datadog each time the overall status of your Elasticsearch cluster changes - red, yellow, or green.

elasticsearch.cluster_health:

Returns OK if the cluster status is green, Warn if yellow, and Critical otherwise.

elasticsearch.can_connect:

Returns Critical if the Agent cannot connect to Elasticsearch to collect metrics.

• Agent can't connect
• Why isn't Elasticsearch sending all my metrics?

To get a better idea of how (or why) to integrate your Elasticsearch cluster with Datadog, check out our series of blog posts about it.