Archive easyblocks along with easyconfig by ocaisa · Pull Request #2653 · easybuilders/easybuild-framework

ocaisa · 2018-11-05T16:43:34Z

Fix broken check for generic easyblock

boegel

@ocaisa A bit of logging wouldn't hurt here imho...

Be more strict on where to stop easyblock dumping

ocaisa · 2018-11-06T11:18:01Z

        errormsg = "build failed (first %d chars): %s" % (first_n, err.msg[:first_n])
        _log.warning(errormsg)
        result = False
-    app.close_log()


Logging was being shut down too early, meaning all logging below this was lost

ocaisa · 2018-11-06T15:28:00Z

@boegel Cleaned up and good to go now

ocaisa · 2018-11-12T14:29:05Z

    return (success, application_log, errormsg)


+def reproduce_build(app, reprod_dir_root):


@boegel I need some help with how to unit test this, not sure what's required to get a proper app instance

...or perhaps what is already tested is good enough?

@ocaisa Once you have a parsed easyconfig (for example for toy-0.0.eb in the tests, you can just create an easyblock instance like MyEasyBlock(parsed_easyconfig) (e.g. Toy(toy_ec)).

But I guess there's no strong need to test this function in isolation, since it's always being called. Just make sure whatever it produces it being checked, which you're already doing I think.

boegel · 2018-11-13T08:44:11Z

@ocaisa I'll check why @boegelbot isn't picking up on this, but there's a failing test:

ERROR: Test toy build produces expected reproducability files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/453993240/lib/python2.6/site-packages/easybuild_framework-3.7.2.dev0-py2.6.egg/test/framework/toy_build.py", line 1557, in test_reproducability
    for framework_easyblock in FRAMEWORK_EASYBLOCK_MODULES:
NameError: global name 'FRAMEWORK_EASYBLOCK_MODULES' is not defined

ocaisa · 2018-11-13T09:19:27Z

To address easybuilders/easybuild-easyblocks#1574 , should I be doing the reproducability dump before app.run_all_steps()? That should save me from changes made to the instance by the easyblocks while preserving everything else. I'll have to do it in the temporary space and then move it if we succeed with the build.

ocaisa · 2018-11-13T09:39:15Z

@boegel After a quick test, the approach in the last comment looks like it would solve easybuilders/easybuild-easyblocks#1574 but I would like your opinion before I implement.

…d then copy it over after success

ocaisa · 2018-11-13T11:05:37Z

Ok, I've implemented the move. This covers the case where modifications come from the command line (or configuration) and dependency resolution but doesn't cover what hooks might do. I think I'd also like to archive those but I don't know how to access them.

ocaisa · 2018-11-13T11:08:27Z

Urgh, I already see a problem with the hook example https://easybuild.readthedocs.io/en/latest/Hooks.html?highlight=hooks#example-hook-to-inject-a-custom-patch-file
I guess patch files should also be copied over to the reprod folder?

ocaisa · 2018-11-16T14:18:07Z

I've reverted this back to simply also dumping the used easyblocks. Fixing the deeper problems of this approach (as in easybuilders/easybuild-easyblocks#1574) is for another PR.

boegel · 2018-11-21T20:08:36Z

lgtm, thanks @ocaisa!

Alan O'Cais added 4 commits November 5, 2018 17:41

Archive easyblocks along with easyconfig

8c3aa66

Be more careful

337e282

Add a test for easyblock in reprod dir

fbe92ad

Weaken class origin checking for developers

afc0f93

Fix broken check for generic easyblock

boegel requested changes Nov 5, 2018

View reviewed changes

Comment thread easybuild/framework/easyblock.py Outdated

Comment thread easybuild/framework/easyblock.py Outdated

Comment thread test/framework/toy_build.py Outdated

boegel added the enhancement label Nov 5, 2018

boegel added this to the 3.8.0 milestone Nov 5, 2018

Alan O'Cais added 3 commits November 6, 2018 09:41

Make sure logging is carried out

79b9ea0

Be more strict on where to stop easyblock dumping

Add check to ensure framework easyblocks are not dumped

e69da33

Revert breaking change

fb83cb7

houndci-bot reviewed Nov 6, 2018

View reviewed changes

Comment thread test/framework/sandbox/easybuild/easyblocks/t/toytoy.py Outdated

Add more complete reproducability tests

d9335a1

easybuilders deleted a comment from boegelbot Nov 6, 2018

Fix broken test

b7738cc

ocaisa commented Nov 6, 2018

View reviewed changes

Comment thread test/framework/toy_build.py

Add additional test for --easyblock

6e24acd

ocaisa added the bug fix label Nov 6, 2018

ocaisa commented Nov 6, 2018

View reviewed changes

Use positive logic

e249579

boegel requested changes Nov 8, 2018

View reviewed changes

Comment thread easybuild/framework/easyblock.py Outdated

Comment thread easybuild/framework/easyblock.py Outdated

Comment thread easybuild/framework/easyblock.py Outdated

Alan O'Cais added 2 commits November 12, 2018 15:23

Separate out the reproducability dumping

f4fa36a

Remove unused global

2778635

ocaisa commented Nov 12, 2018

View reviewed changes

Remove outdated import

7e3800c

ocaisa mentioned this pull request Nov 13, 2018

reprod easyconfig does not work for VMD, Tau, MrBayes (and possibly others) easybuilders/easybuild-easyblocks#1574

Open

Replace outdated import in test

5f61048

easybuilders deleted a comment from boegelbot Nov 13, 2018

Move population of reprod dir to before beginning the installation an…

2792c5c

…d then copy it over after success

Alan O'Cais added 3 commits November 13, 2018 12:45

Check reprod dir does not exist before attempting a copy

9f9b4e7

REvert changes to restrict PR to only saving easyblocks

dc45f02

REmove unneeded import

c7e433c

easybuilders deleted a comment from boegelbot Nov 15, 2018

ocaisa mentioned this pull request Nov 16, 2018

fix problems with easyconfig file saved in 'reprod' directory by copying it before running any installation steps #2664

Merged

boegel approved these changes Nov 21, 2018

View reviewed changes

boegel merged commit 1c70aef into easybuilders:develop Nov 21, 2018

boegel mentioned this pull request Nov 21, 2018

Add ability to archive easyblocks #2633

Closed

		return (success, application_log, errormsg)


		def reproduce_build(app, reprod_dir_root):

Conversation

ocaisa commented Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

ocaisa commented Nov 6, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocaisa Nov 12, 2018

Choose a reason for hiding this comment

Uh oh!

ocaisa Nov 12, 2018

Choose a reason for hiding this comment

Uh oh!

boegel Nov 13, 2018

Choose a reason for hiding this comment

Uh oh!

boegel commented Nov 13, 2018

Uh oh!

ocaisa commented Nov 13, 2018

Uh oh!

ocaisa commented Nov 13, 2018

Uh oh!

ocaisa commented Nov 13, 2018

Uh oh!

ocaisa commented Nov 13, 2018

Uh oh!

ocaisa commented Nov 16, 2018

Uh oh!

boegel commented Nov 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ocaisa commented Nov 5, 2018 •

edited

Loading