Skip to content

Normalize runtime by number of forecasts#75

Merged
shchur merged 6 commits intomainfrom
normalize-inference-time
Jan 7, 2026
Merged

Normalize runtime by number of forecasts#75
shchur merged 6 commits intomainfrom
normalize-inference-time

Conversation

@shchur
Copy link
Contributor

@shchur shchur commented Jan 5, 2026

Issue #, if available: Fixes #68

Description of changes:

  • Store num_forecasts in the evaluation summary, equal to num_items * num_windows * num_targets
  • Add an option to normalize the runtime for each task by num_forecasts in fev.leaderboard
  • Bump version to 0.7.0
  • Fix notebook test

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@shchur shchur requested a review from abdulfatir January 7, 2026 09:23
Copy link
Collaborator

@abdulfatir abdulfatir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall. Left some minor comments.

@shchur shchur merged commit fde040f into main Jan 7, 2026
3 checks passed
@shchur shchur deleted the normalize-inference-time branch January 7, 2026 13:29
@shchur shchur restored the normalize-inference-time branch January 7, 2026 14:34
@shchur shchur deleted the normalize-inference-time branch January 14, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace median runtime with a better aggregate metric

2 participants