make unit tests more robust (REVIEW) by boegel · Pull Request #1167 · easybuilders/easybuild-framework

boegel · 2015-02-11T10:25:43Z

things to fix:

make sure the test suite totally ignores the user's EasyBuild configuration: unset any $EASYBUILD_ environment variables
- example: the unit test suite now goes down hard if a different module naming scheme is used
skip tests that require external dependencies, if those deps are not available
skip tests that have to be run online, if the tests are being run offline
skip tests that require a GitHub token, if no token is available (for the easybuild_test 'user')
make sure the framework unit tests can run without having easybuild-easyblocks or easybuild-easyconfigs around
make sure no sources are being downloaded by the unit tests

hpcugentbot · 2015-02-11T10:38:15Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1298/
Test FAILed.

geimer · 2015-02-11T11:01:08Z

As requested by @boegel, here is the output of two failing tests when buildpath is set in the config.cfg file:

FAIL: test_default_config (test.framework.config.EasyBuildConfigTest)
Test default configuration.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/framework/config.py", line 99, in test_default_config
    self.assertEqual(config_options['buildpath'], os.path.join(eb_homedir, 'build'))
AssertionError: '/tmp/eb' != '/home/geimer/.local/easybuild/build'

======================================================================
FAIL: test_generaloption_config (test.framework.config.EasyBuildConfigTest)
Test new-style configuration (based on generaloption).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/framework/config.py", line 145, in test_generaloption_config
    self.assertEqual(build_path(), os.path.join(prefix, 'build'))
AssertionError: '/tmp/eb' != '/tmp/tmpWdtu4H/test1/build'

…function in options.py

…ile with all options set to default

… remove unused dump_cfgfile_using_defaults

hpcugentbot · 2015-02-20T20:38:46Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1391/
Test FAILed.

…ources

hpcugentbot · 2015-02-20T21:53:07Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1393/
Test FAILed.

hpcugentbot · 2015-02-20T22:24:58Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1394/
Test FAILed.

hpcugentbot · 2015-02-21T10:05:47Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1395/
Test FAILed.

boegel · 2015-02-21T10:05:50Z

Jenkins: test this please

hpcugentbot · 2015-02-21T10:55:28Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1396/
Test PASSed.

wpoely86 · 2015-02-21T18:38:57Z

With new patches: https://gist.github.com/c24222df954cf1d40323

Conflicts: setup.py

…ig.py tests

boegel · 2015-02-21T20:01:25Z

@wpoely86: thanks for sharing your test results;

one test is failing because you were using an older version of easybuild-easyconfigs, which doesn't have easyblock = 'ConfigureMake' yet in TEMPLATE.eb; that's something you need to fix

another issue was caused by you having $EASYBUILD_TEST_TMPDIR defined, which is translated to $EASYBUILD_TMPDIR by the tests; this is fixed now, the tests will be immune to this where needed (in test/framework/config.py

I'm not sure what's going wrong with the last one, which seems to be an issue with the default 0022 umask that is being set in test_toy_permissions not being honored...

======================================================================
FAIL: test_toy_permissions (test.framework.toy_build.ToyBuildTest)
Test toy build with custom umask settings.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/framework/toy_build.py", line 448, in test_toy_permissions
    self.assertEqual(perms, correct_perms, msg)
AssertionError: Path /home/ward/tmp/easybuild-framework/tmp/easybuild-Winupc/tmpkJpJ00/modules has 0755 permissions: 0775

What's especially weird here is that the test is complaining about the modules subdirectory, but not about the software subdir, so the latter must have the correct permissions...

boegel · 2015-02-21T20:27:33Z

@wpoely86: I'm also seeing the permission/umask issue on another system I have access to, but I'm unable to reproduce the problem with eb itself:

$ umask
0002
$ eb --umask=022 bzip2-1.0.6.eb --installpath=$HOME/newtestdir -f
...
$ ls -ld newtestdir/*
drwxr-xr-x 4 vsc40023 vsc40023 512 Feb 21 21:24 newtestdir/modules
drwxr-xr-x 3 vsc40023 vsc40023 512 Feb 21 21:24 newtestdir/software

Will look into this further sometime later.

wpoely86 · 2015-02-21T20:29:26Z

Debian 7:

======================================================================
FAIL: test_toy_permissions (test.framework.toy_build.ToyBuildTest)
Test toy build with custom umask settings.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/framework/toy_build.py", line 448, in test_toy_permissions
    self.assertEqual(perms, correct_perms, msg)
AssertionError: Path /home/ward/tmp/easybuild-D6hwzN/tmpxXy3QU/modules has 0755 permissions: 0775

----------------------------------------------------------------------
Ran 231 tests in 1245.473s

FAILED (failures=1)
ERROR: Not all tests were successful.

hpcugentbot · 2015-02-21T22:06:54Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1405/
Test PASSed.

hpcugentbot · 2015-02-21T23:35:34Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1407/
Test PASSed.

hpcugentbot · 2015-02-22T09:32:35Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1408/
Test PASSed.

wpoely86 · 2015-02-22T18:07:18Z

Okay, it now fully works!

hpcugentbot · 2015-02-22T22:29:36Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1409/
Test PASSed.

stdweird · 2015-02-23T06:43:57Z

if you do a and not key.startswith(test_env_var_prefix) you don't need teh eb_test_env_vars nor the last for loop

ok, good suggestion

on second thought, no, that won't work, since I'm redefining $EASYBUILD_TEST_FOO as $EASYBUILD_FOO, the $EASYBUILD_TEST_FOO shouldn't stay around, it should be undefined to make sure the tests are isolated from any haphazard config settings

hpcugentbot · 2015-02-23T13:15:38Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1411/
Test PASSed.

…ests to configure EasyBuild

boegel · 2015-02-24T13:37:28Z

Jenkins: test this please

hpcugentbot · 2015-02-24T14:53:24Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1416/
Test PASSed.

boegel · 2015-02-24T16:05:55Z

This should be good to go in now, and fixing the minor remarks @stdweird had.

Thanks a lot for the feedback, @wpoely86 and @stdweird, the unit tests should be a lot more robust now, which is a nice change to go along with EasyBuild v2.0.

Going in!

make unit tests more robust (REVIEW)

skip tests that require a GitHub token if no token is available

48cceb3

boegel mentioned this pull request Feb 11, 2015

EasyBuild 2.0 #1000

Closed

27 tasks

boegel mentioned this pull request Feb 12, 2015

tests do not complete successfully #1175

Closed

boegel added 2 commits February 20, 2015 12:48

Merge branch 'develop' into robust_unit_tests

94dec42

Merge branch 'develop' into robust_unit_tests

63565f2

boegel mentioned this pull request Feb 20, 2015

add support for (un)setting options back to None in configuration files hpcugent/vsc-base#157

Closed

boegel added 9 commits February 20, 2015 16:01

fix small issues in cmdline options, add dump_cfgfile_using_defaults …

0979fda

…function in options.py

isolate tests from non-default configuration by using configuration f…

8a0d855

…ile with all options set to default

fix issues in github unit tests

9301b35

use $EASYBUILD_IGNORECONFIGFILES to ignore any existing EB cfg files,…

48de142

… remove unused dump_cfgfile_using_defaults

revert changes made in options.py tests

cb89dfd

re-add parallelbuild unit test(s), and fix the broken one

3638748

remove print statements in test utilities module

12927ee

Merge branch 'vsc_base_dep' into robust_unit_tests

fb01e02

bump vsc-base dep version

3d5794a

boegel added 2 commits February 20, 2015 22:12

fix test for build_easyconfigs_in_parallel, avoid that it downloads s…

962e47c

…ources

make test_fetch_easyconfigs_from_pr robust against offline testing

3869d27

boegel added 3 commits February 20, 2015 22:58

make test_obtain_file robust against offline testing

aa8a189

make test_gitrepo robust against offline testing

f05c69b

include '.' in $PYTHONPATH when testing generate_software_list script

b178cdb

use GITHUB_TEST_ACCOUNT constant in options.py tests

2545e3d

wrap 'import keyring' in try-except in suite.py

65b3d6a

boegel mentioned this pull request Feb 21, 2015

bump version of vsc-base dep to 2.0.3 #1186

Merged

boegel added 2 commits February 21, 2015 20:34

Merge branch 'develop' into robust_unit_tests

2c50616

Conflicts: setup.py

unset *all* $EASYBUILD_X env vars in purge_environment method in conf…

1387793

…ig.py tests

Merge branch 'develop' into robust_unit_tests

ecef213

empty test install dir before running test to check permissions

a1f2600

set $EASYBUILD_ROBOT_PATHS in setUp, unset it where needed

e334beb

stdweird reviewed Feb 23, 2015
View reviewed changes

Merge branch 'develop' into robust_unit_tests

ff6d24d

use TEST_EASYBUILD_ prefix for environment variables picked by unit t…

d0a8c5b

…ests to configure EasyBuild

tiny style fix

1a20ea6

boegel added a commit that referenced this pull request Feb 24, 2015

Merge pull request #1167 from boegel/robust_unit_tests

f2d7b3f

make unit tests more robust (REVIEW)

boegel merged commit f2d7b3f into easybuilders:develop Feb 24, 2015

boegel deleted the robust_unit_tests branch February 24, 2015 16:06

boegel mentioned this pull request Feb 24, 2015

Problem using eb on POWER architecture easybuilders/easybuild#32

Closed

boegel mentioned this pull request Aug 14, 2015

Failing test due to config.cfg #833

Closed

geimer mentioned this pull request Mar 10, 2017

Properly isolate easyblocks unit tests easybuilders/easybuild-easyblocks#1130

Open

Conversation

boegel commented Feb 11, 2015

Uh oh!

hpcugentbot commented Feb 11, 2015

Uh oh!

geimer commented Feb 11, 2015

Uh oh!

hpcugentbot commented Feb 20, 2015

Uh oh!

hpcugentbot commented Feb 20, 2015

Uh oh!

hpcugentbot commented Feb 20, 2015

Uh oh!

hpcugentbot commented Feb 21, 2015

Uh oh!

boegel commented Feb 21, 2015

Uh oh!

hpcugentbot commented Feb 21, 2015

Uh oh!

wpoely86 commented Feb 21, 2015

Uh oh!

boegel commented Feb 21, 2015

Uh oh!

boegel commented Feb 21, 2015

Uh oh!

wpoely86 commented Feb 21, 2015

Uh oh!

hpcugentbot commented Feb 21, 2015

Uh oh!

hpcugentbot commented Feb 21, 2015

Uh oh!

hpcugentbot commented Feb 22, 2015

Uh oh!

wpoely86 commented Feb 22, 2015

Uh oh!

hpcugentbot commented Feb 22, 2015

Uh oh!

stdweird Feb 23, 2015

Choose a reason for hiding this comment

Uh oh!

boegel Feb 23, 2015

Choose a reason for hiding this comment

Uh oh!

boegel Feb 23, 2015

Choose a reason for hiding this comment

Uh oh!

hpcugentbot commented Feb 23, 2015

Uh oh!

boegel commented Feb 24, 2015

Uh oh!

hpcugentbot commented Feb 24, 2015

Uh oh!

boegel commented Feb 24, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants