Connect Kafka to Datadog to:

• Visualize the performance of your cluster in real time.
• Correlate the performance of Kafka with the rest of your applications.

This check has a limit of 350 metrics per instance. The number of returned metrics is indicated on the info page. Specify the metrics you are interested in by editing the configuration below. To learn how to customize the metrics to collect visit the JMX Checks documentation for more detailed instructions.

To collect Kafka consumer metrics, see the kafka_consumer check.

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

The Agent's Kafka check is included in the Datadog Agent package, so you don't need to install anything else on your Kafka nodes.

The check collects metrics via JMX, so you need a JVM on each kafka node so the Agent can fork jmxfetch. You can use the same JVM that Kafka uses.

Edit the kafka.d/conf.yaml file, in the conf.d/ folder at the root of your Agent's configuration directory.

Kafka bean names depend on the exact Kafka version you're running. Use the example configuration file that comes packaged with the Agent as a base since it is the most up-to-date configuration. Note: the Agent version in the example may be for a newer version of the Agent than what you have installed.

After you've configured kafka.yaml, restart the Agent to begin sending Kafka metrics to Datadog.

Available for Agent >6.0

1. Kafka uses the log4j logger by default. To activate logging to a file and customize the format edit the log4j.properties file:

     # Set root logger level to INFO and its only appender to R
     log4j.rootLogger=INFO, R
     log4j.appender.R.File=/var/log/kafka/server.log
     log4j.appender.R.layout=org.apache.log4j.PatternLayout
     log4j.appender.R.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
   

2. By default, our integration pipeline supports the following conversion patterns:

     %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
     %d [%t] %-5p %c - %m%n
     %r [%t] %p %c %x - %m%n
   

   Clone and edit the integration pipeline if you have a different format.

3. Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml file:

     logs_enabled: true

4. Add the following configuration block to your kafka.d/conf.yaml file. Change the path and service parameter values based on your environment. See the sample kafka.d/conf.yaml for all available configuration options.

     logs:
       - type: file
         path: /var/log/kafka/server.log
         source: kafka
         service: myapp
         #To handle multi line that starts with yyyy-mm-dd use the following pattern
         #log_processing_rules:
         #  - type: multi_line
         #    name: log_start_with_date
         #    pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])

5. Restart the Agent.

Run the Agent's status subcommand and look for kafka under the JMXFetch section:

========
JMXFetch
========
  Initialized checks
  ==================
    kafka
      instance_name : kafka-localhost-9999
      message :
      metric_count : 46
      service_check_count : 0
      status : OK

See metadata.csv for a list of metrics provided by this check.

The Kafka check does not include any events.

kafka.can_connect:
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored Kafka instance, otherwise returns OK.

• Troubleshooting and Deep Dive for Kafka
• Agent failed to retrieve RMIServer stub
• Producer and Consumer metrics don't appear in my Datadog application

• Monitoring Kafka performance metrics
• Collecting Kafka performance metrics
• Monitoring Kafka with Datadog