Skip to content

Reverting PR #3179 which breaks csv logging.#3183

Merged
MMathisLab merged 1 commit intoDeepLabCut:mainfrom
deruyter92:revert_append_csv_logger
Jan 16, 2026
Merged

Reverting PR #3179 which breaks csv logging.#3183
MMathisLab merged 1 commit intoDeepLabCut:mainfrom
deruyter92:revert_append_csv_logger

Conversation

@deruyter92
Copy link
Copy Markdown
Collaborator

A quick revert for PR #3179 that breaks csv logging. The PR addressed issue #3176 about overwriting logs when resuming training. However the solution introduces another issue.

Problem
The header and previous rows are appended for every save call. This in unintended behavior. This is what the csv looks like after 5 epochs of training + 5 epochs after resuming training:

step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
1,0.002396461321040988,0.5657591819763184,0.015342211350798607,0.015342211350798607
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
1,0.002396461321040988,0.5657591819763184,0.015342211350798607,0.015342211350798607
2,0.0011502221459522843,0.4335229694843292,0.01141318492591381,0.01141318492591381
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
1,0.002396461321040988,0.5657591819763184,0.015342211350798607,0.015342211350798607
2,0.0011502221459522843,0.4335229694843292,0.01141318492591381,0.01141318492591381
3,0.000915985438041389,0.2670913636684418,0.007135278545320034,0.007135278545320034
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
1,0.002396461321040988,0.5657591819763184,0.015342211350798607,0.015342211350798607
2,0.0011502221459522843,0.4335229694843292,0.01141318492591381,0.01141318492591381
3,0.000915985438041389,0.2670913636684418,0.007135278545320034,0.007135278545320034
4,0.0006651507574133575,0.20210027694702148,0.005385081749409437,0.005385081749409437
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
1,0.002396461321040988,0.5657591819763184,0.015342211350798607,0.015342211350798607
2,0.0011502221459522843,0.4335229694843292,0.01141318492591381,0.01141318492591381
3,0.000915985438041389,0.2670913636684418,0.007135278545320034,0.007135278545320034
4,0.0006651507574133575,0.20210027694702148,0.005385081749409437,0.005385081749409437
5,0.00048237110604532063,0.138005331158638,0.003691319143399596,0.003691319143399596
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
6,0.0004938317579217255,0.15738332271575928,0.0041814991272985935,0.0041814991272985935
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
6,0.0004938317579217255,0.15738332271575928,0.0041814991272985935,0.0041814991272985935
7,0.000375963601982221,0.10041985660791397,0.0026984785217791796,0.0026984785217791796
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
6,0.0004938317579217255,0.15738332271575928,0.0041814991272985935,0.0041814991272985935
7,0.000375963601982221,0.10041985660791397,0.0026984785217791796,0.0026984785217791796
8,0.0003080902970395982,0.07751169055700302,0.0020918375812470913,0.0020918375812470913
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
6,0.0004938317579217255,0.15738332271575928,0.0041814991272985935,0.0041814991272985935
7,0.000375963601982221,0.10041985660791397,0.0026984785217791796,0.0026984785217791796
8,0.0003080902970395982,0.07751169055700302,0.0020918375812470913,0.0020918375812470913
9,0.0002659921592567116,0.07983513921499252,0.0021288746502250433,0.0021288746502250433
step,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss
6,0.0004938317579217255,0.15738332271575928,0.0041814991272985935,0.0041814991272985935
7,0.000375963601982221,0.10041985660791397,0.0026984785217791796,0.0026984785217791796
8,0.0003080902970395982,0.07751169055700302,0.0020918375812470913,0.0020918375812470913
9,0.0002659921592567116,0.07983513921499252,0.0021288746502250433,0.0021288746502250433
10,0.00028138142079114914,0.08120895177125931,0.0021709143184125423,0.0021709143184125423
step,losses/eval.bodypart_heatmap,losses/eval.bodypart_locref,losses/eval.bodypart_total_loss,losses/eval.total_loss,losses/train.bodypart_heatmap,losses/train.bodypart_locref,losses/train.bodypart_total_loss,losses/train.total_loss,metrics/test.mAP,metrics/test.mAR,metrics/test.rmse,metrics/test.rmse_pcutoff
6,,,,,0.0004938317579217255,0.15738332271575928,0.0041814991272985935,0.0041814991272985935,,,,
7,,,,,0.000375963601982221,0.10041985660791397,0.0026984785217791796,0.0026984785217791796,,,,
8,,,,,0.0003080902970395982,0.07751169055700302,0.0020918375812470913,0.0020918375812470913,,,,
9,,,,,0.0002659921592567116,0.07983513921499252,0.0021288746502250433,0.0021288746502250433,,,,
10,0.0004623084969352931,0.12547896802425385,0.0033681283239275217,0.0033681283239275217,0.00028138142079114914,0.08120895177125931,0.0021709143184125423,0.0021709143184125423,82.93069306930694,85.0,5.353951772054036,5.353951772054036

Proposed solution

The header and previous rows are appended for every save call. This in unintended behavior.

- changing back to 'write' mode instead of 'append' mode
@deruyter92 deruyter92 requested review from AlexEMG, C-Achard and MMathisLab and removed request for C-Achard January 16, 2026 09:04
@MMathisLab MMathisLab merged commit f729f62 into DeepLabCut:main Jan 16, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants