add support for backing up modules with --module-only via --backup-modules#2134
add support for backing up modules with --module-only via --backup-modules#2134boegel merged 28 commits intoeasybuilders:developfrom
Conversation
|
@damianam This needs more work (e.g. review & tests), and since neither of us will have time to work on this in the coming days, let's bump the milestone to |
|
@boegel take a look when you can. I'll see if tomorrow I can write some tests for it. |
| if module_only and os.path.exists(mod_filepath): | ||
| warning_msg = "Old module file found. Backing it up in %s. Diff is:\n%s" | ||
| mod_bck_filepath = back_up_file(mod_filepath, backup_extension="bck") | ||
| (mod_diff, _) = run_cmd("diff %s %s" % (mod_bck_filepath, mod_filepath)) |
| module_only = build_option('module_only') | ||
| if module_only and os.path.exists(mod_filepath): | ||
| warning_msg = "Old module file found. Backing it up in %s. Diff is:\n%s" | ||
| mod_bck_filepath = back_up_file(mod_filepath, backup_extension="bck") |
There was a problem hiding this comment.
we should make this a hidden module, no?
also, this should be configurable, people should be able to disable the backup if they're confident enough (but I guess having it enabled by default makes sense)
There was a problem hiding this comment.
Making it a hidden module makes sense just for TCL modules. I had lua modules in mind, but since there are still people out there with TCL modules I agree with you.
Regarding making it configurable..... I am not so sure.
Pros:
- Flexibility.
Cons:
- Yet another option cluttering the
--help - More code to maintain
However, if you are using TCL modules with Lmod, these will be recognized as modules that can be loaded, which is not what you want. Any other combination (TCL modules with Tmod, or lua modules with Lmod), will result in an invisible (to the modules tool) backup. So yes, I guess I should also prepare an option for it.....
| _log.info("Moved existing log file %s to %s" % (new_log_path, oldlog_backup)) | ||
| back_up_file(new_log_path) | ||
|
|
||
| # remove first to ensure portability |
|
|
||
|
|
||
| def back_up_file(src_file, backup_extension=""): | ||
| """Backs up a file appending a backup extension and a number to it. Returns the name of the backup""" |
There was a problem hiding this comment.
clarify: the number is optional (only when existing backups are already there)
use :param: to document arguments
also support hide named argument to backup to a hidden file
|
@damianam For functions you touch/change, please check whether they have a dedicated unit test already (and maybe enhance it while you're at it). When adding new functions, add a dedicated test for them. |
…arify stuff in comments.
|
@boegel I addressed all your comments. The code hasn't been tested yet, and the tests are still missing, but most of the bits and pieces should be there already. |
…dule. Written full tests for that functionality.
|
Tests are complete and code seems to be working. @easybuilders/easybuild-framework-maintainers please review. |
|
|
||
| # if --backup-modules is used without --module-only print a warning | ||
| if self.options.backup_modules and not self.options.module_only: | ||
| print_warning("--backup-modules can be used just together with --module-only. Ignoring it...") |
| ]: | ||
| self.assertEqual(ft.guess_patch_level([patched_file], self.test_buildpath), correct_patch_level) | ||
|
|
||
| def test_back_up_file(self): |
There was a problem hiding this comment.
missing additional tests for find_backup_name_candidate and move_file?
| # remove existing module file under --force (but only if --skip and --backup-modules are not used) | ||
| elif build_option('force') or build_option('rebuild'): | ||
| if os.path.exists(self.mod_filepath): | ||
| if os.path.exists(self.mod_filepath) and not build_option('backup_modules'): |
There was a problem hiding this comment.
rather than simply not removing, shouldn't we be renaming the module here already to its backup name?
| else: | ||
| self.log.info(warning_msg + ' No differences found.', mod_bck_filepath) | ||
| print_warning(warning_msg % mod_bck_filepath + ' No differences found.') | ||
| self.log.info("Module file %s written: %s", mod_filepath, txt) |
There was a problem hiding this comment.
this whole block is rather big to just shove it into make_module_step like this, it's better to move it into a separate method named back_up_existing_module_file or something like that?
| warning_msg = "Old module file found. Backing it up in %s." | ||
| hidden = False | ||
| if isinstance(self.module_generator, ModuleGeneratorTcl): | ||
| hidden = True |
There was a problem hiding this comment.
indent is off here?
Also, why only do hidden for Tcl modules?
The backup should always be a hidden module imho, you don't want to expose the backup module to users?
Maybe we should also back up outside of the module tree, e.g. in the current directory (or at a location specified via --backup-modules?
There was a problem hiding this comment.
Because lua modules have to end in .lua, otherwise they aren't treated as modules by Lmod, and don't show up with ml av. So there is no need to keep them hidden since the backup extension avoids that file ends in .lua.
Tcl modules on the other need to be hidden. Which is not that nice, since Lmod can still show them with --show-hidden. But that's the best compromise IMO. The alternative would be to remove the module header.
Backing up outside of the module tree kind of defeats the purpose of the backup. In my opinion the backup should be used for quick verification of the changes and for doing a "rollback" if necessary. If I have the current module and the backup in different files it slightly makes it more complicated and cumbersome, and more difficult to directly see if there was a previous version at any point with a simple ls
| stderr = self.get_stderr() | ||
| self.mock_stderr(False) | ||
| self.assertTrue(os.path.exists(toy_mod)) | ||
| self.assertTrue(os.path.exists(os.path.join(os.path.dirname(toy_mod), '.'+os.path.basename(toy_mod)+'.bck'))) |
There was a problem hiding this comment.
use os.path.split & os.path.join, spaces around +
| self.eb_main(args + ['--backup-modules'], do_build=True, raise_error=True, verbose=True) | ||
| stderr = self.get_stderr() | ||
| self.mock_stderr(False) | ||
| self.assertTrue(os.path.exists(os.path.join(os.path.dirname(toy_mod), '.'+os.path.basename(toy_mod)+'.bck_0'))) |
There was a problem hiding this comment.
use os.path.split & os.path.join, spaces around +
| warning = "WARNING: Old module file found. Backing it up in .* Diff is:" | ||
| diff = "some difference\n" | ||
| with open(toy_mod, "a") as fh: | ||
| fh.write(diff) |
There was a problem hiding this comment.
use write_file (it has an append mode)
| ] | ||
| args = common_args + ['--module-syntax=Tcl'] | ||
|
|
||
| warning = "WARNING: Old module file found. Backing it up in .* No differences found" |
There was a problem hiding this comment.
Hmm, should this be a warning? It's just an informative message, no?
| if isinstance(self.modtool, Lmod): | ||
| args = common_args + ['--module-syntax=Lua'] | ||
|
|
||
| toy_mod = os.path.join(self.test_installpath, 'modules', 'all', 'toy', '0.0-deps.lua') |
There was a problem hiding this comment.
rather than copy-pasting all the checks, we should hoist them in a helper function, and simplify the test to
def test_backup_modules(self):
"""Test use of --backup-modules."""
def run_checks():
...
...
toy_mod_fn = '0.0-deps'
run_checks()
if isinstance(self.modtool, Lmod):
toy_mod_fn = '0.0-deps.lua'
run_checks()clean up back_up_file implementation & test
cleanup & tests for find_backup_name_candidate & move_file
reimplement find_backup_name_candidate to use timestamp rather than increasing number
a6e052d to
9f4d813
Compare
fix broken move_logs test
…use diff_files function in make_module_step to print module file diff, trash useless module_generator.prepare method
…gure step to ensure identical module under --module-only
…s under --module-only & --skip + enhance/fix test_backup_modules
rework support for --backup-modules
|
Going in, thanks @damianam! |
With this PR easybuild will create a backup of existing modules when using
--modules-only. 2 extra comments:back_up_filefunction to rotate logsdiffof modules instdout. Thoughts?