KEY FEATURES | PREREQUISITES | JOURNAL SOURCES | JOURNAL FIELDS | VISUALIZATION | PLAY MODE | FULL TEXT SEARCH | QUERY PERFORMANCE | PERFORMANCE AT SCALE | BEST PRACTICES | CONFIGURATION | FAQ | HOW TO TROUBLESHOOT COMMON ISSUES | HOW TO VERIFY SETUP

The systemd journal plugin provides an efficient way to view, explore, and analyze systemd journal logs directly from the Netdata dashboard. It combines powerful filtering, real-time updates, and visual analysis tools to help you troubleshoot system issues effectively.

• Unified view of logs from multiple sources (system, user, namespace, remote)
• Real-time streaming with PLAY mode for continuous monitoring
• Powerful filtering on all journal fields with counters showing matching entries
• Full-text search with wildcard and pattern matching
• Visual analysis with interactive histograms showing log frequency
• Enriched field display for improved readability (priorities, UIDs, timestamps)
• High performance with intelligent sampling for large datasets
• Zero configuration for immediate use on supported systems
• Multi-node support for centralized log analysis
• UI-based exploration without needing to learn complex journalctl syntax
• Integrated with Netdata's dashboard for correlation with metrics

:::tip

This plugin is a Netdata Function Plugin. A free Netdata Cloud account is required. See the Netdata Functions discussion.

:::

The plugin is designed for native package installations, source installations, and Docker installations (Debian-based). If using Docker, make sure you're using the Debian-based containers.

The plugin automatically detects available journal sources based on the journal files in /var/log/journal (persistent logs) and /run/log/journal (volatile logs).

By default, all sources merge into a unified view of log messages.

:::tip

Select a specific source before analyzing logs in depth to improve query performance.

:::

Default journals on all systemd-based systems. Includes:

• Kernel log messages (kmsg)
• Audit records
• Syslog messages via systemd-journald
• Output from service units
• Native journal API messages

• Shows journal files for all users, not just the current user
• Each regular user (UID > 999) typically has their own user journal
• Merged into remote journals on centralization servers

• Isolate log streams per project or service
• Set with LogNamespace= in systemd unit files
• Requires special setup to propagate to central servers

• Created by systemd-journal-remote
• Typically named by sender IP, then resolved to hostname

systemd journals support dynamic fields per log entry. All fields and values are indexed for fast querying.

Fields are enriched for readability:

:::tip

Enrichments are visual only and not searchable. UID/GID values are based on the system where the plugin runs.

:::

Use the ⚙️ icon above the table to select fields to display as columns.

The table view provides a powerful way to analyze logs with:

• Scrollable list of log entries with customizable columns
• Color-coded PRIORITY levels for quick identification of issues
• Clickable entries for detailed viewing
• Time-ordered display (newest first by default)
• Pagination controls to navigate large datasets

Click a log entry to open the right-hand info panel showing all fields for that entry.

The sidebar shows:

• Every field present in the selected journal entry
• Raw and enriched field values
• Copyable text for sharing or further analysis
• Quick filtering options for any field value

The plugin offers select fields as filters with counters. Field allowlists and blocklists protect performance.

:::tip

"Full data queries" mode enables negative/empty matches but may slow performance.

:::

Key filter features:

• Real-time counters showing matching entry counts
• Multi-select capability for each field
• Toggleable inclusion/exclusion mode
• Persistent filter selections across page reloads

Histograms visualize log frequency per field value over time. Supports:

• Zoom
• Pan
• Click-to-navigate

The plugin offers several visualization features to help you understand and navigate your logs effectively.

The timeline at the top of the interface shows:

• Log frequency distribution over time
• Interactive zoom and pan controls
• Time selection capabilities
• Anomaly highlighting

Field-specific histograms provide:

• Visual breakdown of log entries by field value
• Color-coded frequency indicators
• Click-to-filter capability
• Time correlation with the main timeline

The plugin uses color to enhance readability:

• Priority levels (emergency, alert, critical, etc.) have distinct colors
• Selected filters are highlighted
• Active elements use consistent color indicators
• Error states and warnings have clear visual differentiation

The interface offers several ways to navigate logs:

• Scroll through paginated results
• Jump to specific timeframes
• Click on histogram bars to focus on specific values
• Use filter panels to narrow down results
• Toggle between data views

The plugin supports PLAY mode for real-time log streaming. Click the ▶️ button at the top of the dashboard to activate it.

• Continuously updates the screen with newly received logs
• Works for both single nodes and centralized log servers

:::tip

PLAY mode offers a similar experience to journalctl -f, but with visual enhancements.

:::

The plugin supports full-text search using flexible pattern matching:

:::tip

Full-text search applies across all fields. Combine with filters for precise results.

:::

The plugin reads journal files directly using libsystemd, supporting concurrent readers and one writer.

Query performance depends on several factors:

For best performance:

• Keep the visible timeframe short
• Limit the number of rows displayed
• Apply filters to reduce the dataset
• Use specific sources instead of querying across all journals

The plugin handles large datasets efficiently using a sampling algorithm, ensuring responsive queries even on busy log servers.

The plugin uses a sophisticated algorithm that prioritizes newer logs while maintaining reasonable estimates for historical data. This approach ensures that even with terabytes of journal data, the interface remains responsive and usable.

At scale, this plugin achieves up to 25–30x faster query performance compared to journalctl, especially on multi-journal queries.

:::tip

The sampling algorithm is designed to be resilient to large datasets. Even if you see [unsampled] or [estimated] indicators, the results remain statistically representative of the full dataset.

:::

Netdata's sampling budget evaluates up to 1,000,000 log entries before it ever marks rows as [unsampled]. The proportion of the dataset we examine is:

evaluated_entries = min(total_entries, 1_000_000)
evaluated_ratio   = evaluated_entries / total_entries

Because the sampling set is so large, percentage breakdowns stay tight even on massive datasets. For a 10 M–entry window where 60 % of logs share a value, the 95 % confidence interval around that percentage is:

standard_error ≈ sqrt(p * (1 - p) / evaluated_entries)
CI95 ≈ 1.96 * standard_error = 1.96 * sqrt(0.6 * 0.4 / 1_000_000) ≈ ±0.9 %

By contrast, evaluating only 5,000 entries (a small-sample approach typical of many log explorers when speed is prioritized) would yield:

CI95 ≈ 1.96 * sqrt(0.6 * 0.4 / 5_000) ≈ ±8.7 %

The result is that even at extreme scale, mainly because Netdata samples 200x more data, it can provide significantly more accurate estimations on value distributions, at comparable performance.

systemd-journal is designed for reliability first and performance second. It uses deduplication, field linking, and compression to minimize disk footprint, but the structure of journal files can still result in higher disk I/O during queries.

:::tip

Journal data is cached by the operating system. The more RAM available for caching, the faster your queries will be.

:::

The Netdata systemd journal plugin is designed to work out of the box with minimal configuration.

:::tip

No additional configuration is required for this plugin to operate on supported systems.

:::

Use the following solutions to resolve common issues with the systemd journal plugin.

:::tip

Sampling ensures responsiveness at scale, but selecting sources and filters remains the best way to optimize performance.

:::

sudo netdata -W plugins

Check that the systemd-journal plugin is listed as active.

1. Open the Logs tab in the Netdata UI
2. Use the Sources filter on the right sidebar
3. Ensure you can see your expected sources (e.g., system, user, remote, or specific namespaces)

:::tip

If sources are missing, check the journal file locations and verify symlinks if needed.

:::

• Apply a simple filter (like PRIORITY=3) and confirm logs are returned
• Use full-text search (e.g., search for error) and verify results populate correctly
• Toggle PLAY mode to confirm live logs are streaming

Check the Netdata Agent logs for plugin startup messages:

sudo journalctl -u netdata | grep journal

Look for lines confirming the journal plugin started successfully and detected sources.