Skip to content

Also dump environment to reprod directory#3374

Merged
akesandgren merged 16 commits intoeasybuilders:developfrom
ocaisa:dumpenv_reprod
Nov 2, 2021
Merged

Also dump environment to reprod directory#3374
akesandgren merged 16 commits intoeasybuilders:developfrom
ocaisa:dumpenv_reprod

Conversation

@ocaisa
Copy link
Copy Markdown
Member

@ocaisa ocaisa commented Jun 29, 2020 •
edited by boegel
Loading

fixes #2111, related to #2284 (but doesn't really fixes it, according to @boegel)

@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Jun 29, 2020

Urgh, this triggers new logging which is breaking some tests that don't expect it.

@boegel boegel added the bug fix label Jun 30, 2020
@boegel boegel added this to the next release (4.2.2?) milestone Jun 30, 2020
@boegel
Copy link
Copy Markdown
Member

boegel commented Jun 30, 2020

@ocaisa So you should fix up those tests then?

@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Jun 30, 2020

@boegel I need some advice on this one, placing the env dump in the reprod dir saves me from having to do any bookkeeping but because it creates a nested easyblock instance there is a bit of a logging mess that is kicking errors in the tests

@easybuilders easybuilders deleted a comment from boegelbot Jun 30, 2020
boegel
boegel previously requested changes Jul 4, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ocaisa The env script is dumped too early, the build environment isn't set up yet when you create it:

$ cat $EASYBUILD_PREFIX/software/bzip2/1.0.8-GCCcore-9.3.0/easybuild/reprod/*env
#!/bin/bash
# script to set up build environment as defined by EasyBuild v4.2.2.dev0 for bzip2-1.0.8-GCCcore-9.3.0.eb
# usage: source bzip2-1.0.8-GCCcore-9.3.0.env

# toolchain & dependency modules
# (no modules loaded)

# build environment
# (no build environment defined)

@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Jul 6, 2020

@boegel I switched it to be done after the easyblock has been used and it seems to work. I haven't capture that in the tests though yet.

This is not foolproof I suspect since it doesn't handle multi-deps or iterating builds

Add test for dumping of buildenv (to check if it includes module load…
c2d4361 
… statements)

Include Extension as an easyblock that should not be dumped to the reprod dir (since it is part of the framework)
@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Jul 6, 2020

@boegel I've added an additonal test that at least captures the module load for a dependency. Is it possible to trigger a build environment variable in the tests?

I've also addressed the comment you made in #3375 (comment)

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 14, 2020

@boegel I've added an additonal test that at least captures the module load for a dependency. Is it possible to trigger a build environment variable in the tests?

@ocaisa What do you mean by "build environment variable"?

To change the EasyBuild configuration used in a test, you can use init_config.
To define one or more environment variables, just use os.environ or setvar from easybuild.tools.environment

@boegel boegel modified the milestones: next release (4.3.1), 4.x Oct 14, 2020
@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Oct 14, 2020

Hard to remember, but I think I was talking about populating the lines

# build environment
# (no build environment defined)

such as CC, CXX etc.

@ocaisa ocaisa mentioned this pull request Feb 22, 2021
Open
@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Sep 17, 2021

@boegel I'm kicking the tyres on this because it is actually really useful to have for EESSI, for example if someone wants build their own custom version of a package (I'm thinking about application maintainers using it in CI)

@easybuilders easybuilders deleted a comment from boegelbot Sep 20, 2021
@boegel boegel modified the milestones: 4.x, next release (4.5.0?) Sep 29, 2021
@boegel boegel modified the milestones: 4.5.0, release after 4.5.0 Oct 13, 2021
@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Nov 1, 2021

@ocaisa This looks sane to me.
Is it finished and ready for merging?
@boegel any remaining issues from your side?

@ocaisa
Copy link
Copy Markdown
Member Author

ocaisa commented Nov 1, 2021

I think it works fine, there is potentially an issue with multideps but we don't really have those any more.

Example:

[ocaisa@login1 easybuild-framework]$ eb OSU-Micro-Benchmarks-5.7.1-gompi-2021b.eb --force
...
[ocaisa@login1 easybuild-framework]$ cat /home/ocaisa/.local/easybuild/software/OSU-Micro-Benchmarks/5.7.1-gompi-2021b/easybuild/reprod/OSU-Micro-Benchmarks-5.7.1-gompi-2021b.env
#!/bin/bash
# script to set up build environment as defined by EasyBuild v4.4.3.dev0 for OSU-Micro-Benchmarks-5.7.1-gompi-2021b.eb
# usage: source OSU-Micro-Benchmarks-5.7.1-gompi-2021b.env

# toolchain & dependency modules
module load gompi/2021b

# build environment
export CC='mpicc'
export CC_SEQ='gcc'
export CFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno'
export CXX='mpicxx'
export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno'
export CXX_SEQ='g++'
export F77='mpifort'
export F77_SEQ='gfortran'
export F90='mpifort'
export F90FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno'
export F90_SEQ='gfortran'
export FC='mpifort'
export FCFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno'
export FC_SEQ='gfortran'
export FFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno'
export FLIBS='-lgfortran'
export LDFLAGS='-L/project/boegelbot/Rocky8/haswell/software/GCCcore/11.2.0/lib64 -L/project/boegelbot/Rocky8/haswell/software/GCCcore/11.2.0/lib'
export LIBS='-lm -lpthread'
export MPICC='mpicc'
export MPICXX='mpicxx'
export MPIF77='mpifort'
export MPIF90='mpifort'
export MPIFC='mpifort'
export MPI_INC_DIR='/project/boegelbot/Rocky8/haswell/software/OpenMPI/4.1.1-GCC-11.2.0/include'
export MPI_LIB_DIR='/project/boegelbot/Rocky8/haswell/software/OpenMPI/4.1.1-GCC-11.2.0/lib64'
export MPI_LIB_SHARED='/project/boegelbot/Rocky8/haswell/software/OpenMPI/4.1.1-GCC-11.2.0/lib64/libmpi.so'
export MPI_LIB_STATIC=''
export OMPI_CC='gcc'
export OMPI_CXX='g++'
export OMPI_F77='gfortran'
export OMPI_F90='gfortran'
export OMPI_FC='gfortran'
export OPTFLAGS='-O2 -ftree-vectorize -march=native'
export PRECFLAGS='-fno-math-errno'

@akesandgren akesandgren dismissed boegel’s stale review November 2, 2021 07:24

Requested changes have been made

akesandgren
akesandgren approved these changes Nov 2, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@akesandgren akesandgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Nov 2, 2021

Going in, thanks @ocaisa!

@akesandgren akesandgren merged commit ce9b66f into easybuilders:develop Nov 2, 2021
@ocaisa ocaisa deleted the dumpenv_reprod branch November 2, 2021 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@akesandgren akesandgren akesandgren approved these changes

@boegel boegel boegel left review comments

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

4.5.1

Development

Successfully merging this pull request may close these issues.

RFF: dump a copy of --dump-env-script output in the %(installdir)/easybuild directory as an .env file for each build

3 participants

@ocaisa @boegel @akesandgren