I've successfully tested this feature with one of our internal Git packages. In my EasyConfig file, I have an extensions block:

exts_defaultclass = 'PythonPackage'
exts_list = [
    ('pyCAP', 'master', {
        'filename': 'pyCAP-master.tar.gz',
        'git_config': {
            'url': local_git + '/Analysis',
            'repo_name': 'pyCAP',
            'tag': 'master',
        }
    }),
]

When I run with the production eb (EasyBuild/2020a-sprint), I get the expected failure:

== fetching files...
== FAILED: Installation ended unsuccessfully (build directory: /scratch/group/mi
tchcomp/eb/tmp/build/cdmsPyGlobal/0.5/system-system-Python-3.6.6test):
build failed (first 300 chars): Couldn't find file pyCAP-master.tar.gz anywhere,
and downloading it didn't work either... Paths attempted (in order): /scratch/gr
oup/mitchcomp/eb/eb-files/c/cdmsPyGlobal/extensions/pyCAP-master.tar.gz, /scratc
h/group/mitchcomp/eb/eb-files/c/cdmsPyGlobal/packages/pyCAP-master.tar.gz, /scra
tch/group/m (took 0 sec)

but when I run with my personal checkout, it succeeds:

== fetching files...
== creating build dir, resetting environment...
[...]
== taking care of extensions...
== installing extension pyCAP master (1/1)...
[...]

Is there documentation for how to create a new unit test? I'd love to exercise this in some way as part of the pull request.

@boegel Suggested that a unit test could be added to either the easyblock.py or toy_build.py tests.

I think I can do this, but neither one appears to actually perform a download of sources. What I would want to test is a dictionary entry like

exts_list = [('toy', '1.0', {
    'filename':'toy-build.tar.gz', 
    'git_config': {
        'url':'https://github.com/something-something',
        'repo_name':'toy', 'tag':'1.0',
    }
})]

and see that either the download actually occurs (ideal) or at least that the contents of 'git_config' is seen and is validated.

I don't think I need to actually " see that ... the download actually occurs (ideal)." The filetools.py tests already include test_get_source_tarball_from_git(). That exercises both the different components of git_config as well as the downloading procedure.

For this situation, I just need to test whether having a 'git_config' dictionary in exts_list does not generate an error. This should be easier for me to figure out how to implement.

Update. Well, that didn't go so well. The test failed (which I expected), but then spewed hundreds of lines of DEBUG output. I'm sure an explanation for the failure is in there somewhere...

I just noticed that test_obtain_file() in the easyblock.py test does not exercise the git_config option. Should I try to add that as a unit test as well?

@boegel @kelseymh Shouldn't the format for this be

exts_list = [
  ('pyCAP', 'master', {
    'sources': [{
      'filename':...,
      'git_config'
    }],
  }),
]

To make it clear that it's the source specification, and make it behave like non-ext-list sources, allowing multiple sources files...

@akesandgren That's a good question! The allowed content of exts_list is undocumented, so I constructed that example block by trial and error.

In the code of this PR, I didn't want to try to change how exts_list parsed the extra_options dictionary, just added extraction of a git_config element.

The EasyConfigs which make use of 'sources' in a dictionary all seem to have it either at the top level or in default_component_specs (see, e.g., SpliceMap).

My write up of exts_list behaviour (see Framework PR #617 follows this searc of existing examples. If putting a whole 'sources': dictionary into extra_options does work, then I should update that attempt at documentation accordingly.

@boegel wrote:

An enhanced test for obtain_file to cover the use of git_config makes sense too, but that can also be tackled in a separate PR I think, up to you...

No need. I got most of the way through writing it, and discovered that test/framework/filetools.py already contains test_test_get_source_tarball_from_git(). I've closed the issue (#3322) I created for this.

This isn't working. If I use a sources dictionary within exts_list, and do what I described above, putting the name of the fetched source file into an ext_sources entry with a src key, then the extension-building step fails:

The command that tries to get executed is (this is from running test.framework.toy_build):

gcc exts-git.tar.gz.c -o exts-git.tar.gz

where it's plugged in the name of the tarball itself as the thing to be compiled!

Solved! Instead of the whole data-structure output from the new fetch_source(), ext_source really only wants the downloaded filename put into the src key in ext_src. With this change the toy_build_extension_sources test runs successfully.

However, toy_build_download_sources is failing (for me, when I specify just the "sources" tests) because it doesn't find EASYBUILD_BUILDPATH in the environment variables.

Close & re-open to try and get Travis to wake up... You're running on fumes here Travis, seriously...