@juan-cobos, wouldn't this solution cause duplicate headers to be written to the CSV file? I think it might be better to handle the existing file first no?

@MMathisLab, what do you think?

@deruyter92 Yep, it stores headers each time. It depends if you want to keep a reference of last save. If not, in case file exists, we should append [1:] (without headers). Thank you for spotting it!

PS: WandbLogger is also overwriting at the moment :)

We should probably revert this merge!

See attached the CSV file generated with 5 epochs of training + another 5 epochs after resuming from the saved weights.
learning_stats.csv
The header is saved each time. When more metrics are added (e.g. during evaluation), the CSV completely breaks, because of mismatching columns.

@deruyter92 wow that looks really bad, sorry. Could you try without appending headers? I cannot test training at the moment. Otherwise, what would you suggest? Creating directly another file in case headers mismatch? Thank you for your follow-up!

No problem, would have been a good suggestion if the CSVLogger didn't write all metrics every save call, but it actually keeps a store of all previous runs as well. I opened a PR to revert the change, so should be recovered quickly.

My suggestion is to read the previously saved metrics in existing CSV into the loggers metric store, e.g. something in the line of #3177.