fix modpath_extensions_for method: take into account modules in Lua syntax by boegel · Pull Request #1766 · easybuilders/easybuild-framework

boegel · 2016-05-17T09:54:46Z

(replacement for #1763)

Updating the regex within the modules modpath_extensions_for function to find the module dependencies when using lua module files.

…l_modules for tests

…e to extract $MODULEPATH extensions

…e in Lua syntax

…e_tree

…ase notes

damianam · 2016-05-17T16:40:18Z

+            r'^\s*prepend-path\s+MODULEPATH\s+"?([^"\s]+)"?',  # prepend to $MODULEPATH in Tcl modules
+            r'^\s*prepend_path\(\"MODULEPATH\",\s*\"(\S+)\"',  # prepend to $MODULEPATH in Lua modules
+        ])
+        modpath_ext_regex = re.compile(modpath_ext_regex, re.M)


Doesn't it make sense to determine beforehand which kind of module are we reading, and apply the appropriate regex for each kind, instead of packing them all together?

Maybe, but what extra would that give us?

If we detect a Tcl module (which is easy, via the required %#Module header), then we still need to try two regex's: module use and prepend-path MODULEPATH...

It's not like this is going to result in a noticeable speedup: module files are quite small, and regex search in Python is quite fast. It may even be faster to do what we do now than figure out first which type of module we're dealing with and then searching with a regex...

It wasn't performance what bothered me, but keeping things clean and well structured. But admittedly this is splitting hairs in this particular case.

…s_for

…ath_to_top_of_module_tree

…n Lua syntax

boegel · 2016-05-18T12:11:15Z

Thanks for the review @wpoely86 and @damianam!

I ran into another bug when testing this thoroughly with Python-2.7.11-intel-2016a.eb, using HierarchicalMNS, Lmod + modules in Lua syntax and with --recursive-module-unload enabled. The path_to_top_of_module_tree function was not taking into account the .lua extension.

This has been fixed in here too now, and a dedicated unit test has been added.

I've retested this with the setup mentioned above, works like a charm now:

$ find . -type f | xargs ls -1rt
./Core/M4/1.4.17.lua
./Core/Autoconf/2.69.lua
./Core/Automake/1.15.lua
./Core/libtool/2.4.6.lua
./Core/flex/2.5.39.lua
./Core/Autotools/20150215.lua
./Core/Bison/3.0.4.lua
./Core/zlib/1.2.8.lua
./Core/binutils/2.25.lua
./Core/GCCcore/4.9.3.lua
./Compiler/GCCcore/4.9.3/M4/1.4.17.lua
./Compiler/GCCcore/4.9.3/flex/2.5.39.lua
./Compiler/GCCcore/4.9.3/Bison/3.0.4.lua
./Compiler/GCCcore/4.9.3/zlib/1.2.8.lua
./Compiler/GCCcore/4.9.3/binutils/2.25.lua
./Core/icc/2016.1.150-GCC-4.9.3-2.25.lua
./Core/ifort/2016.1.150-GCC-4.9.3-2.25.lua
./Core/iccifort/2016.1.150-GCC-4.9.3-2.25.lua
./Compiler/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150.lua
./Core/iimpi/8.1.5-GCC-4.9.3-2.25.lua
./Core/intel/2016a.lua
./Core/iimpi/2016.01-GCC-4.9.3-2.25.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/imkl/11.3.1.150.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/bzip2/1.0.6.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/zlib/1.2.8.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/ncurses/6.0.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/libreadline/6.3.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Tcl/8.6.4.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/SQLite/3.9.2.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Tk/8.6.4-no-X11.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/M4/1.4.17.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Autoconf/2.69.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Automake/1.15.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/libtool/2.4.6.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Autotools/20150215.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/GMP/6.1.0.lua
./MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Python/2.7.11.lua

$ cat MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Python/2.7.11.lua
help([[Python is a programming language that lets you work more quickly and integrate your systems
 more effectively. - Homepage: http://python.org/]])

whatis([[Description: Python is a programming language that lets you work more quickly and integrate your systems
 more effectively. - Homepage: http://python.org/]])

local root = "/tmp/software/MPI/intel/2016.1.150-GCC-4.9.3-2.25/impi/5.1.2.150/Python/2.7.11"

conflict("Python")

load("imkl/11.3.1.150")

load("bzip2/1.0.6")

load("zlib/1.2.8")

load("libreadline/6.3")

load("ncurses/6.0")

load("SQLite/3.9.2")

load("Tk/8.6.4-no-X11")

load("GMP/6.1.0")

prepend_path("CPATH", pathJoin(root, "include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("MANPATH", pathJoin(root, "share/man"))
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("PKG_CONFIG_PATH", pathJoin(root, "lib/pkgconfig"))
setenv("EBROOTPYTHON", root)
setenv("EBVERSIONPYTHON", "2.7.11")
setenv("EBDEVELPYTHON", pathJoin(root, "easybuild/MPI-intel-2016.1.150-GCC-4.9.3-2.25-impi-5.1.2.150-Python-2.7.11-easybuild-devel"))

-- Built with EasyBuild version 2.8.0.dev0
setenv("EBEXTSLISTPYTHON", "setuptools-18.7.1,pip-7.1.2,nose-1.3.7,numpy-1.10.1,scipy-0.16.1,blist-1.3.6,mpi4py-1.3.1,paycheck-1.0.2,argparse-1.4.0,pbr-1.8.1,lockfile-0.12.2,Cython-0.23.4,six-1.10.0,dateutil-2.4.2,deap-1.0.2,decorator-4.0.4,arff-2.1.0,pycrypto-2.6.1,ecdsa-0.13,paramiko-1.16.0,pyparsing-2.0.6,netifaces-0.10.4,netaddr-0.7.18,funcsigs-0.4,mock-1.3.0,pytz-2015.7,pandas-0.17.1,enum34-1.1.2,bitstring-3.1.3,virtualenv-14.0.5")

t-brown and others added 9 commits November 13, 2015 13:00

Bug fix for modpath_extensions_for function.

70d17c3

Updating the regex within the modules modpath_extensions_for function to find the module dependencies when using lua module files.

Merge branch 'develop' into 1473_modpath_extensions_for_lua

5c2793e

fix implementation of modpath_extensions_for via 'fake' load

ed3b3eb

make sure path for 'module use' statements exist in setup_hierarchica…

ded88fd

…l_modules for tests

also test modpath_extensions_for using icc/ifort modules

207f401

fix long lines in test_exclude_path_to_top_of_module_tree

cc84cce

fix implementation of modpath_extensions_for, parse actual module fil…

5baa60f

…e to extract $MODULEPATH extensions

fix and extend test for modpath_extensions_for, also check test modul…

ba94b0c

…e in Lua syntax

also test modpath_extensions_for in test_exclude_path_to_top_of_modul…

97db72c

…e_tree

boegel added this to the v2.8.0 milestone May 17, 2016

boegel mentioned this pull request May 17, 2016

fix modpath_extensions_for method: take into account modules in Lua syntax and conditional $MODULEPATH extensions (WIP) #1763

Closed

boegel added 2 commits May 17, 2016 13:20

add modules header in test module file for test_modpath_extensions_for

981a625

flesh out docstring for modpath_extensions_for method

f8b80a8

boegel added a commit to boegel/easybuild-framework that referenced this pull request May 17, 2016

include easybuilders#1729+easybuilders#1766-easybuilders#1768 in rele…

a606da4

…ase notes

damianam reviewed May 17, 2016
View reviewed changes

boegel added 5 commits May 17, 2016 19:20

use more descriptive names in list comprehension in modpath_extension…

ce0cc5f

…s_for

Merge branch 'develop' into 1473_modpath_extensions_for_lua

955ebcd

take into account possible .lua extension of path to module file in p…

1fa848b

…ath_to_top_of_module_tree

add dedicated test for path_to_top_of_module_tree with module files i…

a381ec4

…n Lua syntax

be more careful when stripping of .lua extension from module file path

5530a38

boegel merged commit ee9647f into easybuilders:develop May 18, 2016

boegel deleted the 1473_modpath_extensions_for_lua branch May 18, 2016 12:11

boegel mentioned this pull request May 18, 2016

Bug fix for modpath_extensions_for function. #1474

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix modpath_extensions_for method: take into account modules in Lua syntax#1766

fix modpath_extensions_for method: take into account modules in Lua syntax#1766
boegel merged 16 commits intoeasybuilders:developfrom
boegel:1473_modpath_extensions_for_lua

boegel commented May 17, 2016

Uh oh!

damianam May 17, 2016

Uh oh!

boegel May 17, 2016

Uh oh!

damianam May 17, 2016

Uh oh!

boegel commented May 18, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

boegel commented May 17, 2016

Uh oh!

damianam May 17, 2016

Choose a reason for hiding this comment

Uh oh!

boegel May 17, 2016

Choose a reason for hiding this comment

Uh oh!

damianam May 17, 2016

Choose a reason for hiding this comment

Uh oh!

boegel commented May 18, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants