Skip to content

[yeb] YAML format for ebfiles_repo#1822

Merged
boegel merged 23 commits intoeasybuilders:developfrom
Caylo:yeb-fix-1695
Jul 8, 2016
Merged

[yeb] YAML format for ebfiles_repo#1822
boegel merged 23 commits intoeasybuilders:developfrom
Caylo:yeb-fix-1695

Conversation

@Caylo
Copy link
Copy Markdown
Contributor

@Caylo Caylo commented Jun 28, 2016

Fixes #1695
Includes #1826

@boegel boegel added this to the v2.9.0 milestone Jun 29, 2016
@boegel boegel changed the title YAML format for ebfiles_repo [yeb] YAML format for ebfiles_repo Jun 29, 2016
YAML_DIR = r'%YAML'
YAML_SEP = '---'
YEB_FORMAT_EXTENSION = '.yeb'
EB_FORMAT_EXTENSION = '.eb'
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this belongs in format/one.py since it has nothing to do with the .yeb format?

@boegel
Copy link
Copy Markdown
Member

boegel commented Jun 29, 2016

@Caylo you should look into a test for this too, i.e. build & install toy easyconfig, and check whether the dumped easyconfig can be parsed again, both for .eb and .yeb?

if set(YAML_SPECIAL_CHARS).intersection(val):
return "'%s'" % val

return val
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use 4-space indent :)

@boegel boegel modified the milestones: v2.9.0, v2.8.2 Jul 4, 2016
@boegel
Copy link
Copy Markdown
Member

boegel commented Jul 5, 2016

@Caylo please sync with develop now #1826 is merged?

"""
if isinstance(val, basestring):
if YAML_SPECIAL_CHARS.intersection(val):
return "'%s'" % val
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't return inline, it makes code harder to read

so, do something like

if if isinstance(val, basestring) and YAML_SPECIAL_CHARS.intersection(val):
    val = "'%s'" % val

return val

@boegel
Copy link
Copy Markdown
Member

boegel commented Jul 6, 2016

@Caylo still a broken test:

======================================================================
ERROR: Test dumping of file in eb_filerepo in both .eb and .yeb format
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/hpcugent/easybuild-framework/test/framework/toy_build.py", line 1149, in test_toy_dumped_easyconfig
    ec = EasyConfig(args[0])
  File "/home/travis/build/hpcugent/easybuild-framework/easybuild/framework/easyconfig/easyconfig.py", line 313, in __init__
    auto_convert_value_types=auto_convert_value_types)
  File "/home/travis/build/hpcugent/easybuild-framework/easybuild/framework/easyconfig/parser.py", line 105, in __init__
    self._set_formatter(filename)
  File "/home/travis/build/hpcugent/easybuild-framework/easybuild/framework/easyconfig/parser.py", line 196, in _set_formatter
    self._formatter.parse(self.rawcontent)
  File "/home/travis/build/hpcugent/easybuild-framework/easybuild/framework/easyconfig/format/yeb.py", line 96, in parse
    self.parsed_yeb = yaml.load(txt)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 111, in compose_sequence_node
    node.value.append(self.compose_node(node, index))
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/parser.py", line 543, in parse_flow_mapping_key
    if not self.check_token(FlowMappingEndToken):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/scanner.py", line 252, in fetch_more_tokens
    return self.fetch_plain()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/scanner.py", line 680, in fetch_plain
    self.tokens.append(self.scan_plain())
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/yaml/scanner.py", line 1309, in scan_plain
    "Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.")
ScannerError: while scanning a plain scalar
  in "<string>", line 107, column 208:
     ... configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' -- ... 
                                         ^
found unexpected ':'
  in "<string>", line 107, column 256:
     ... 6.3-1ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.6/README ... 
                                         ^
Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.
----------------------------------------------------------------------
Ran 419 tests in 234.224s
FAILED (errors=1)
ERROR: Not all tests were successful.

"""
if isinstance(val, basestring):
if "'" in val:
val = '"%s"' % val
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if it also has "? :)

val = '"%s"' % val

elif YAML_SPECIAL_CHARS.intersection(val):
val = "'%s'" % val
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Caylo shouldn't this be the first option you try?

if the string includes both ' and any special YAML charachter, you'll still be in trouble since then you'll take the first option which quotes with "..." but doesn't escape YAML chars?

Copy link
Copy Markdown
Contributor Author

@Caylo Caylo Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel YAML special chars are escaped by quoting the string (as YAML doesn't usually require quoting for regular strings). If the string contains both ' and any other special char, the string will be quoted with ".." anyway, so special chars don't matter anymore

now that im typing this, maybe a better solution is to add ' to YAML_SPECIAL_CHARS and always escape with double quotes

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, but there was a difference between single-quoting and double-quoted strings in YAML, right?

Include a comment and reference w.r.t. that in this function please...

if isinstance(val, basestring):
if "'" in val or YAML_SPECIAL_CHARS.intersection(val):
val = val.replace("'", "''")
val = "'%s'" % val
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks a bit silly, let's collapse it to

val = "'%s'" % val.replace("'", "''")

@boegel
Copy link
Copy Markdown
Member

boegel commented Jul 8, 2016

Going in, thanks @Caylo!

@boegel boegel merged commit 8dcc5f1 into easybuilders:develop Jul 8, 2016
boegel added a commit to boegel/easybuild-framework that referenced this pull request Jul 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants