add support for 'eb --inject-checksums' by boegel · Pull Request #2286 · easybuilders/easybuild-framework

boegel · 2017-08-16T12:01:15Z

collect checksums for all files/patches, incl. for extensions
construct lines for checksums and exts_list easyconfig parameter to be injected into easyconfig file
replace existing value for checksums under --force
replace existing value for exts_list with value that includes checksums
inject checksums in easyconfig file where no checksums are provided yet
~~make checksum type required parameter~~
- (not needed, proper error message is printed when --inject-checksums accidentally tries to consume an easyconfig filename thanks to choice option type)
add test for --inject-checksums
update docs: document --inject-checksums easybuild#369

…ums per file

…sources/patches specs

…s for extensions

wpoely86 · 2017-08-21T14:36:30Z

+
+        # back up easyconfig file before injecting checksums
+        ectxt = read_file(ec['spec'])
+        ec_backup = os.path.join(ec['spec'] + '.bck')


isn't bak the generally accepted suffix?

I'm not sure there's a generally accepted standard... See https://fileinfo.com/filetypes/backup

Well, I would take bak 😉

changed to .bak after checking with @damianam, since he chose .bck initially via #2134

wpoely86 · 2017-08-21T14:49:18Z

+        checksums_txt = '\n'.join(checksum_lines)
+
+        if app.cfg['checksums']:
+            regex = re.compile(r'^checksums(?:.|\n)*?\]\s*$', re.M)


unless you're going to use it in many places, there is no point in compiling first?

and shouldn't it be +? instead of *??

good points, fixed

Actually, there is a technical reason for splitting this up with re.compile: using re.M in re.sub requires Python 2.7... (see also 4f41328)

wpoely86 · 2017-08-21T14:53:43Z

+        if app.cfg['checksums']:
+            regex = re.compile(r'^checksums(?:.|\n)*?\]\s*$', re.M)
+            ectxt = regex.sub(checksums_txt, ectxt)
+        elif app.src:


just else?

no, it's possible there are no sources, and hence no patches/checksums

for example: bundles, like intel-2017a.eb easyconfig

typo: it is

wpoely86 · 2017-08-21T14:54:31Z

+        else:
+            checksum_lines = ['checksums = [']
+            for fn, checksum in checksums:
+                checksum_lines.append("    '%s',  # %s" % (checksum, fn))


below you do: ' ' * 4, be consistent.

I'll use INDENT_4SPACES consistently instead...

wpoely86 · 2017-08-21T15:27:29Z

+                                 'choice', 'store_or_None', CHECKSUM_TYPE_SHA256, CHECKSUM_TYPES),
+        })
        self.log.debug("easyconfig_options: descr %s opts %s" % (descr, opts))
-        self.add_group_parser(opts, descr, prefix='easyconfig')


why change the prefix?

Otherwise the option would be --easyconfig-inject-checksums, which is overkill imho.

There were no options in this category yet, so this change is fine.

wpoely86 · 2017-08-21T15:27:46Z

+
+                if ext.keys() == ['name']:
+                    exts_list_lines.append("%s'%s'," % (' ' * 4, ext['name']))
+


less empty lines?

…d needless re.compile

…M in re.sub requires Python 2.7

boegel · 2017-08-21T21:20:03Z

@wpoely86 remarks fixed, please rereview?

wpoely86 · 2017-08-22T07:16:50Z

+
+        # back up easyconfig file before injecting checksums
+        ectxt = read_file(ec['spec'])
+        ec_backup = os.path.join(ec['spec'] + '.bck')


Well, I would take bak 😉

wpoely86 · 2017-08-22T07:20:28Z

+        if app.cfg['checksums']:
+            regex = re.compile(r'^checksums(?:.|\n)*?\]\s*$', re.M)
+            ectxt = regex.sub(checksums_txt, ectxt)
+        elif app.src:


typo: it is

…sums is used

add initial (incomplete) support for --inject-checksums

1713fa8

boegel added this to the 3.4.0 milestone Aug 16, 2017

boegel added 6 commits August 16, 2017 14:41

fix injecting checksums in easyconfig file without 'checksums' spec

8f1abf4

make checksum type required argument for --inject-checksums

0e9653e

don't blindly overwrite existing .eb.bck files, print computed checks…

51585f1

…ums per file

enhance injecting of checksums to deal with out-of-order source_urls/…

a151f85

…sources/patches specs

make SHA256 default checksum type for --inject-checksums

43ae9d8

use single line format if there's only one checksum

5149321

boegel mentioned this pull request Aug 16, 2017

{data][foss/2017a] h5py 2.7.0 w/ Python 2.7.13 & HDF5 1.8.19 + 1.10.1, and bump Szip dep to 2.1.1 easybuilders/easybuild-easyconfigs#5021

Merged

boegel added 3 commits August 16, 2017 22:18

fix regex for replacing of existing checksums, fix injecting checksum…

8289b41

…s for extensions

fix catching of SystemExit in eb_main

83d0285

add test for --inject-checksums

06ab976

boegel changed the title ~~add support for 'eb --inject-checksums' [WIP]~~ add support for 'eb --inject-checksums' Aug 16, 2017

boegel requested a review from a team August 16, 2017 20:29

boegel mentioned this pull request Aug 21, 2017

{bio}[foss/2016b] vsearch v2.4.3 easybuilders/easybuild-easyconfigs#4485

Closed

wpoely86 reviewed Aug 21, 2017

View reviewed changes

boegel added 5 commits August 21, 2017 20:27

Merge branch 'develop' into inject_checksums

cdf60fa

use back_up_file, also log printed messages, use INDENT_4SPACES, avoi…

0e9d184

…d needless re.compile

minor cleanup

638a87a

split up re.sub into regex=re.compile & regex.sub because passing re.…

4f41328

…M in re.sub requires Python 2.7

take into account that no options may be specified for extension

2fb22a6

wpoely86 previously approved these changes Aug 22, 2017

View reviewed changes

boegel added 3 commits August 23, 2017 19:08

skip verifying of listed checksums for extensions when --inject-check…

d8b0116

…sums is used

use .bak as (default) extension in back_up_file to make @wpoely86's day

eb0565f

fix typo

32809fb

boegel dismissed wpoely86’s stale review via 32809fb August 23, 2017 17:20

boegel mentioned this pull request Aug 24, 2017

document --inject-checksums easybuilders/easybuild#369

Merged

wpoely86 approved these changes Aug 26, 2017

View reviewed changes

wpoely86 merged commit 12e9671 into easybuilders:develop Aug 26, 2017

boegel deleted the inject_checksums branch August 28, 2017 07:57

boegel mentioned this pull request Aug 29, 2017

avoid introducing duplicate entry for 'checksums' in exts_list with --inject-checksums #2293

Merged

boegel mentioned this pull request Sep 8, 2017

fix function signature for fetch_extension_sources in OCaml easyblock easybuilders/easybuild-easyblocks#1240

Merged

boegel mentioned this pull request Sep 4, 2018

add test to enforce SHA256 checksums in touched files in pull requests to develop (REVIEW) easybuilders/easybuild-easyconfigs#5005

Merged

2 tasks


		if ext.keys() == ['name']:
		exts_list_lines.append("%s'%s'," % (' ' * 4, ext['name']))

Conversation

boegel commented Aug 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boegel Aug 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boegel commented Aug 21, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

boegel commented Aug 16, 2017 •

edited

Loading

boegel Aug 21, 2017 •

edited

Loading