Skip to content

Fix CSVLogger: read existing learning_stats.csv file when resuming previous training#3177

Merged
C-Achard merged 9 commits intoDeepLabCut:mainfrom
deruyter92:jaap/fix_csvlogger_resume_training
Feb 13, 2026
Merged

Fix CSVLogger: read existing learning_stats.csv file when resuming previous training#3177
C-Achard merged 9 commits intoDeepLabCut:mainfrom
deruyter92:jaap/fix_csvlogger_resume_training

Conversation

@deruyter92
Copy link
Copy Markdown
Collaborator

@deruyter92 deruyter92 commented Jan 13, 2026 •
edited
Loading

Previously, when resuming training the CSVLogger created a new learning_stats.csv file for the second training run, overwriting previously saved training metrics.

This commit addresses feature request #3176 and changes the behaviour to reading any previously saved log metrics and appending new epochs to the learning_stats.csv file.

  • added _load_existing_data method to CSVLogger
  • added unit test for _load_existing_data

@deruyter92
Fix CSVLogger reads existing file for resumed training
a78f75d 
Previously, when resuming training the CSVLogger created a new learning_stats.csv file for the second training run, overwriting previously saved training metrics.

This commit addresses feature request DeepLabCut#3176 and changes the behaviour to reading any previously saved log metrics and appending new epochs to the learning_stats.csv file.
@deruyter92 deruyter92 requested a review from Copilot January 13, 2026 09:49
Copilot AI reviewed Jan 13, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@deruyter92 deruyter92 changed the title Fix CSVLogger reads existing file for resumed training Fix CSVLogger: read existing learning_stats.csv file when resuming previous training Jan 13, 2026
@deruyter92 deruyter92 requested a review from Copilot January 13, 2026 09:50
Copilot AI reviewed Jan 13, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@MMathisLab MMathisLab closed this Jan 13, 2026
@MMathisLab
Copy link
Copy Markdown
Member

MMathisLab commented Jan 13, 2026

closed in favor of #3179

@deruyter92
Reverting PR DeepLabCut#3179 which breaks csv logging.
ce2e260 
The header and previous rows are appended for every save call. This in unintended behavior.

- changing back to 'write' mode instead of 'append' mode
@deruyter92 deruyter92 mentioned this pull request Jan 16, 2026
Merged
@deruyter92 deruyter92 reopened this Jan 16, 2026
@deruyter92 deruyter92 mentioned this pull request Jan 16, 2026
Merged
@deruyter92 deruyter92 marked this pull request as draft January 16, 2026 09:25
@deruyter92 deruyter92 marked this pull request as ready for review January 16, 2026 16:03
@deruyter92 deruyter92 requested a review from C-Achard January 16, 2026 16:03
@C-Achard
Copy link
Copy Markdown
Collaborator

C-Achard commented Jan 16, 2026 •
edited
Loading

Looks good ! Should we consider adding some more validation in case the previous output is somehow broken, interrupted or to try and account for potential formatting changes in the future ? Or do you feel that would be redundant for now/the current solution is enough ?

@C-Achard C-Achard requested a review from Copilot January 16, 2026 17:06
Copilot AI reviewed Jan 16, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@C-Achard C-Achard added the enhancement New feature or request label Jan 16, 2026
@deruyter92
Copy link
Copy Markdown
Collaborator Author

deruyter92 commented Jan 19, 2026

Should we consider adding some more validation in case the previous output is somehow broken, interrupted or to try and account for potential formatting changes in the future ?
Good thought!

  • The new _load_existing_data method is agnostic to the specific columns in the existent CSV. If they are different from the new training run, this doesn't break the CSV file (missing fields are just kept empty)
  • The only required field is step, containing an integer value. There are no further constraints on what the existing data should look like.
  • I have adjusted the behavior now, based on your suggestion: In case any exception occurred during reading (e.g. corrupted CSV, or CSV that misses 'step' column), the reader now starts with an empty CSV instead of keeping the succeeded rows from the existent CSV.

I don't think it would be a good idea to put specific constraints on the values or data types that should be in the existent CSV, except for the 'step' index. The CSVLogger is currently quite flexible and even logs different metrics for different epoch types (e.g. validation epochs). I think the benefits are limited when we drop this flexible/column-agnostic behavior of the CSVLogger.

deruyter92 and others added 2 commits January 19, 2026 12:03
@deruyter92
CSVLogger: change _load_existing_data handling of partly valid CSVs
dfa3c3f 
In case an exception occurrs during reading (e.g. corrupted CSV, or CSV that misses 'step' column), the reader now starts with an empty CSV instead of keeping the succeeded rows from the existent CSV.
@deruyter92
Merge branch 'main' into jaap/fix_csvlogger_resume_training
bb073b9
@deruyter92 deruyter92 requested a review from C-Achard January 19, 2026 14:54
@C-Achard
Copy link
Copy Markdown
Collaborator

C-Achard commented Jan 20, 2026

@deruyter92 Sounds good, thanks !

@deruyter92
Copy link
Copy Markdown
Collaborator Author

deruyter92 commented Jan 20, 2026

Thanks @C-Achard! Ok, @MMathisLab, please have a look if you agree. In that case these changes can be merged!

@deruyter92 deruyter92 added the bug Something isn't working label Jan 21, 2026
@deruyter92 deruyter92 requested a review from MMathisLab February 2, 2026 09:56
@deruyter92
Copy link
Copy Markdown
Collaborator Author

deruyter92 commented Feb 4, 2026

@MMathisLab let me know what you think! I think it could be merged.

@C-Achard C-Achard merged commit ca9cfd2 into DeepLabCut:main Feb 13, 2026
10 checks passed
@deruyter92 deruyter92 mentioned this pull request Feb 28, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@C-Achard C-Achard C-Achard approved these changes

@MMathisLab MMathisLab Awaiting requested review from MMathisLab

Assignees

No one assigned

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@deruyter92 @MMathisLab @C-Achard