tsdb: Find the last series ID on startup from the last series id file and WAL scan by RushabhMehta2005 · Pull Request #18333 · prometheus/prometheus

RushabhMehta2005 · 2026-03-22T09:59:57Z

This PR builds directly on top of the state file created and maintained as in PR #18303
Now on startup, the Head.Init (...) method checks for this file,reads it, and if the shutdown is clean, trusts the last series ID value in it. If it is an unclean shutdown, we scan the WAL segments backwards, up until the last WAL segment as told by the file.
Added unit tests for this.

Which issue(s) does the PR fix:

NA

Does this PR introduce a user-facing change?

NONE

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-03-24T12:16:34Z

The failing CI linter test is due to a recent push by somebody else to a file I have not touched.

Signed-off-by: Rushabh Mehta <[email protected]>

codesome

Good work! Most of the comments are just styling nits.

codesome · 2026-03-26T00:05:50Z

+		state, err := h.readSeriesState()
+		if err != nil {
+			h.logger.Warn("Failed to read series state file, falling back to slow ID initialization", "err", err)
+		} else if state != nil {


Don't need the else

Suggested change

} else if state != nil {

}

if state != nil {

codesome · 2026-03-26T00:09:29Z

+	if h.opts.EnableFastStartup {
+		state, err := h.readSeriesState()
+		if err != nil {
+			h.logger.Warn("Failed to read series state file, falling back to slow ID initialization", "err", err)


Let's keep it simple and only do fast startup if this file exists. Let's not keep an option to scan all files for series ID for the initial implementation of this feature.

Suggested change

h.logger.Warn("Failed to read series state file, falling back to slow ID initialization", "err", err)

h.logger.Warn("Failed to read series state file, skipping the fast startup", "err", err)

I am assuming once this feature is ready to go, we will change this up so that a new instance can infact make use of it.

codesome · 2026-03-26T00:12:33Z

+				h.lastSeriesID.Store(state.LastSeriesID)
+				h.logger.Info("Fast startup: clean shutdown detected, restored last series ID", "id", state.LastSeriesID)
+			} else {
+				h.logger.Info("Fast startup: unclean shutdown detected, performing bounded reverse scan", "from_segment", endAt, "to_segment", state.LastWALSegment)


Suggested change

h.logger.Info("Fast startup: unclean shutdown detected, performing bounded reverse scan", "from_segment", endAt, "to_segment", state.LastWALSegment)

h.logger.Info("Fast startup: unclean shutdown detected, performing WAL scan", "from_segment", endAt, "to_segment", state.LastWALSegment)

codesome · 2026-03-26T00:14:53Z

+			} else {
+				h.logger.Info("Fast startup: unclean shutdown detected, performing bounded reverse scan", "from_segment", endAt, "to_segment", state.LastWALSegment)
+				if err := h.findLastSeriesID(state, endAt); err != nil {
+					h.logger.Warn("Bounded reverse scan failed, falling back to slow ID initialization", "err", err)


Suggested change

h.logger.Warn("Bounded reverse scan failed, falling back to slow ID initialization", "err", err)

h.logger.Error("Fast startup: WAL scan failed, skipping fast startup", "err", err)

codesome · 2026-03-26T00:15:20Z

+				if err := h.findLastSeriesID(state, endAt); err != nil {
+					h.logger.Warn("Bounded reverse scan failed, falling back to slow ID initialization", "err", err)
+				} else {
+					h.logger.Info("Fast startup: bounded reverse scan completed", "id", h.lastSeriesID.Load())


Suggested change

h.logger.Info("Fast startup: bounded reverse scan completed", "id", h.lastSeriesID.Load())

h.logger.Info("Fast startup: WAL scan completed", "last_series_id", h.lastSeriesID.Load())

codesome · 2026-03-27T01:48:53Z

+	require.Equal(t, uint64(2), head.lastSeriesID.Load(), "Bounded scan should find the highest ID (2) from segment 1")
+}
+
+func TestHead_FastStartup_CleanShutdown(t *testing.T) {


The WAL replay is still running so we can't really test if we are loading the right series id using the new mechanism. We can drop this test for now and come back to it later to test the mechanism end-to-end.

That is true, this test is better for later.

codesome · 2026-03-27T02:00:30Z

+	// Get the current max segment number.
+	endSegment, _, err := head.wal.LastSegmentAndOffset()
+	require.NoError(t, err)
+	require.Equal(t, 1, endSegment)


I am not sure if the first file is 0 or 1. How about use wlog.Segments to get the first and last segment?

codesome · 2026-03-27T02:02:34Z

+	require.Equal(t, expectedState, *state, "read state should match written state")
+}
+
+func TestHead_FindLastSeriesIDBounded(t *testing.T) {


I'd add another case here: before adding metric="B", add another sample for metric="A" in the second segment and then findLastSeriesID() on two files should give 1, and after adding metric=B the same call (scan on both files) should give 2 as the last ID.

Suggested change

func TestHead_FindLastSeriesIDBounded(t *testing.T) {

func TestHead_FindLastSeriesID(t *testing.T) {

codesome · 2026-03-27T02:03:14Z

 	require.Equal(t, 0, state.LastWALSegment, "LastWALSegment should remain 0")
 }
+
+func TestHead_ReadSeriesState(t *testing.T) {


Suggested change

func TestHead_ReadSeriesState(t *testing.T) {

func TestHead_ReadSeriesStateFile(t *testing.T) {

…move test Signed-off-by: Rushabh Mehta <[email protected]>

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-03-27T07:41:59Z

@codesome I think I have fixed all the nits + the changes you requested, have another look. This is looking good so far, thanks for your detailed reviews.

codesome

LGTM! Just a couple of nits and one more test case and we are ready to merge.

codesome · 2026-04-01T03:25:44Z

+				h.lastSeriesID.Store(state.LastSeriesID)
+				h.logger.Info("Fast startup: clean shutdown detected, restored last series ID", "last_series_id", state.LastSeriesID)
+			} else {
+				h.logger.Info("Fast startup: unclean shutdown detected, performing WAL scan", "from_segment", endAt, "to_segment", state.LastWALSegment)


Suggested change

h.logger.Info("Fast startup: unclean shutdown detected, performing WAL scan", "from_segment", endAt, "to_segment", state.LastWALSegment)

h.logger.Info("Fast startup: unclean shutdown detected, performing WAL scan", "from_segment", state.LastWALSegment, "to_segment", endAt)

codesome · 2026-04-01T03:29:06Z

+	// Scanning both files should now return 2
+	id, err = head.findLastSeriesID(mockState, last)
+	require.NoError(t, err)
+	require.Equal(t, uint64(2), id, "Should find ID 2 after new series was created in segment 2")


Let's add one more case duplicating these lines but with mockState.LastWALSegment = last. It should still return 2 because on unclean shutdown it should still scan the last file.

codesome · 2026-04-01T03:31:30Z

+		var highestID chunks.HeadSeriesRef
+		var found bool
+
+		// Read the segment forwards


Suggested change

// Read the segment forwards

// Read the segment forwards.

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-04-01T04:52:11Z

Thanks for the merge! Onto the final PR of this idea.

RushabhMehta2005 requested review from a team, bwplotka, codesome, jesusvazquez and krajorama as code owners March 22, 2026 09:59

RushabhMehta2005 commented Mar 22, 2026

View reviewed changes

Comment thread tsdb/head_test.go

RushabhMehta2005 added 2 commits March 24, 2026 17:36

Add logic to Head.Init(...) for fast startup

f7b81fb

Signed-off-by: Rushabh Mehta <[email protected]>

Add unit tests

fc1fca8

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 force-pushed the read-state-file branch from 98ce6d0 to fc1fca8 Compare March 24, 2026 12:07

Empty commit to retrigger CI

16a9fbc

Signed-off-by: Rushabh Mehta <[email protected]>

Empty commit to retrigger CI

85cb94c

Signed-off-by: Rushabh Mehta <[email protected]>

codesome reviewed Mar 27, 2026

View reviewed changes

RushabhMehta2005 added 2 commits March 27, 2026 12:17

Make readSeriesStateFile return a struct directly, fix small nits, re…

727d3e8

…move test Signed-off-by: Rushabh Mehta <[email protected]>

Fix test for readSeriesStateFile function

cc070ef

Signed-off-by: Rushabh Mehta <[email protected]>

codesome reviewed Apr 1, 2026

View reviewed changes

Fix some more nits, add extra testcase

c901c21

Signed-off-by: Rushabh Mehta <[email protected]>

codesome approved these changes Apr 1, 2026

View reviewed changes

codesome changed the title ~~tsdb: Fast startup via bounded WAL reverse scan~~ tsdb: Find the last series ID on startup from the last series id file and WAL scan Apr 1, 2026

codesome merged commit a2172f9 into prometheus:main Apr 1, 2026
37 checks passed

This was referenced Apr 3, 2026

tsdb: Refactor Head.Init(...) for fast startup #18438

Closed

tsdb: Refactor Head.Init(...) for fast startup #18442

Open

	h.logger.Warn("Failed to read series state file, falling back to slow ID initialization", "err", err)
	h.logger.Warn("Failed to read series state file, skipping the fast startup", "err", err)

	h.logger.Info("Fast startup: unclean shutdown detected, performing bounded reverse scan", "from_segment", endAt, "to_segment", state.LastWALSegment)
	h.logger.Info("Fast startup: unclean shutdown detected, performing WAL scan", "from_segment", endAt, "to_segment", state.LastWALSegment)

	h.logger.Warn("Bounded reverse scan failed, falling back to slow ID initialization", "err", err)
	h.logger.Error("Fast startup: WAL scan failed, skipping fast startup", "err", err)

	h.logger.Info("Fast startup: bounded reverse scan completed", "id", h.lastSeriesID.Load())
	h.logger.Info("Fast startup: WAL scan completed", "last_series_id", h.lastSeriesID.Load())

	func TestHead_FindLastSeriesIDBounded(t *testing.T) {
	func TestHead_FindLastSeriesID(t *testing.T) {

	func TestHead_ReadSeriesState(t *testing.T) {
	func TestHead_ReadSeriesStateFile(t *testing.T) {

Conversation

RushabhMehta2005 commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

Uh oh!

RushabhMehta2005 commented Mar 24, 2026

Uh oh!

codesome left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RushabhMehta2005 commented Mar 27, 2026

Uh oh!

codesome left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RushabhMehta2005 commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

RushabhMehta2005 commented Mar 22, 2026 •

edited

Loading