Skip to content

force building of ccmake for CMake 3.12.1 + fix deps#6967

Merged
boegel merged 4 commits intoeasybuilders:developfrom
smoors:20181005163623_new_pr_CMake3121
Oct 12, 2018
Merged

force building of ccmake for CMake 3.12.1 + fix deps#6967
boegel merged 4 commits intoeasybuilders:developfrom
smoors:20181005163623_new_pr_CMake3121

Conversation

@smoors
Copy link
Copy Markdown
Contributor

@smoors smoors commented Oct 5, 2018

(created using eb --new-pr)

@akesandgren
Copy link
Copy Markdown
Contributor

Why would you need this? ccmake gets built by without this.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 5, 2018

+1 on @akesandgren's remark, can you clarify why you added this @smoors?

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 7, 2018

@akesandgren @boegel ccmake does not get built on our clusters, its not entirely clear to me why.
I get this message in the logs: Checking for curses support - Failed.
it looks like conda-forge had the same problem (and they seem to have solved it in the same way):
conda-forge/cmake-feedstock#62
it was attributed to this:
https://github.com/conda-forge/cmake-feedstock/pull/64/files
if you think the problem is site-specific, I'll close the PR.

@akesandgren
Copy link
Copy Markdown
Contributor

That then suggests that we need to make a better job of telling it where ncurses is.
Can you pick out the actual problem it has with finding curses?

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 7, 2018

@akesandgren with -DBUILD_CursesDialog=1, cmake executes find_package(Curses), which successfully finds ncurses. without that option find_package is not executed.
so either we add -DBUILD_CursesDialog=1 or we patch CMake.
why do you dislke adding the extra option?

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 8, 2018

@smoors Can we explicitly specify the location where ncurses is installed rather than letting CMake hunt for it (since it that case it may pick up the system ncurses, who knows).

I agree with @akesandgren that it's pretty strange this hasn't popped up before, I guess that's why he was asking (not because he dislikes adding the option)

@akesandgren
Copy link
Copy Markdown
Contributor

If that's what it's doing then I don't have any objections as such. But as @boegel says, we should take a look at how it detects curses so we can make sure it picks up the EB one even on systems which happen to have ncurses system-installed. I know that it's often done in a way that doesn't work the way we'd like it to. (Not talking just about curses here but Findxxx functions in general)

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 8, 2018

@akesandgren @boegel the updated patch specifies the path to the curses libs.
although I am confident that my previous patch was also ok, because -DCMAKE_PREFIX_PATH=$EBROOTNCURSES ensures that find_package(Curses) looks for the EB-installed curses first.
The fact that it worked on your clusters without the patch suggests to me that a system-installed curses was used. would be interesting to check that?

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Oct 8, 2018

Curses is indeed installed but not curses-dev.
And Cmake (at least 3.12.1 for GCCcore 7.3.0) finds the EB installed curses.

Found Curses: /hpc2n/eb/software/Compiler/GCCcore/7.3.0/ncurses/6.1/lib/libncurses.so

And more to the point, ccmake is linked against libncurses.so.6 and our system installed one is libncurses.so.5

So I think there is something else going on here.

configopts = '-- -DCMAKE_USE_OPENSSL=1 -DCMAKE_PREFIX_PATH=$EBROOTNCURSES'
configopts = '-- -DCMAKE_USE_OPENSSL=1'
configopts += ' -DCURSES_CURSES_LIBRARY=$EBROOTNCURSES/lib/libcurses.%s' % SHLIB_EXT
configopts += ' -DCURSES_NCURSES_LIBRARY=$EBROOTNCURSES/lib/libncurses.%s' % SHLIB_EXT
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smoors We need to specify both? Also, shouldn't we leave the CMAKE_PREFIX_PATH in place?

Maybe I'm too overly careful here, but I just want to avoid introducing regressions.

Also, it's still not clear to me why it didn't work for you without the -DCURSES_*, it would be good to know...

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 9, 2018

@boegel it is not clear to me either, but this cmake thing is not so easy for me to understand and I don't have too much time to spend on it. I was actually hoping to get some help from experts like you guys..

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Oct 9, 2018

@smoors No risk of that. Oh, now i see what you mean. Sorry about that, bad formatting of my answer.
Fixing that.... I didn't look at the result after clicking the comment button...

I keep forgetting that some things are considered "formatting" characters...

@akesandgren
Copy link
Copy Markdown
Contributor

@smoors Could you please pick out the actual error you're getting from the build, not just the stdout message that if doesn't find curses but the config.log text that says exactly why it fails to find it.
If you don't know what I mean, please mail me the complete log from easybuild (using --debug) and i'll figure out what file to look in.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 9, 2018

@akesandgren That would be CMakeError.log (and maybe also CMakeOutput.log) rather than config.log, right?

@akesandgren
Copy link
Copy Markdown
Contributor

akesandgren commented Oct 9, 2018

No, CMake bootstraps with configure if i read my build log correctly

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 9, 2018

Oh, right, I got things mixed up here... Sorry.

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 9, 2018

@akesandgren that's a relief! I am way too sensitive for these things 😃
I could not find a config.log file, but the CMakeError.log says this:

Change Dir: /tmp/smoors/build/CMake/3.12.1/GCCcore-7.3.0/cmake-3.12.1/Source/Checks/Curses-build

Run Build Command:"/usr/bin/gmake" "CheckCurses"
gmake: *** No rule to make target `CheckCurses'.  Stop.

there are several other errors, not sure if they are related, see here:
https://gist.github.com/smoors/580eb9b7d46c94a99c96e03be6bcc537

@akesandgren
Copy link
Copy Markdown
Contributor

I'm busy in a deep learning course (my first) so i'll try to look at stuff somewhat later.
But can you start by mailing me the build log from EB (using --debug), i'd like to see that one first.

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 9, 2018

i sent the log via slack.

@akesandgren
Copy link
Copy Markdown
Contributor

The building of CMake is done in two steps, first using configure/make to create an initial cmake binary, then using that binary to run cmake on itself.

The problem is that the second step doesn't pick up on lib(n)curses and libflow correctly using the current CMake easyconfig files.

What we have to do is to use this configopt setting:

configopts = '-- -DCMAKE_USE_OPENSSL=1 '
configopts += '-DCURSES_INCLUDE_PATH=$EBROOTNCURSES/include '
configopts += '-DCURSES_CURSES_LIBRARY=$EBROOTNCURSES/lib/libcurses.%s ' % SHLIB_EXT
configopts += '-DCURSES_FORM_LIBRARY=$EBROOTNCURSES/lib/libform.%s ' % SHLIB_EXT
configopts += '-DCURSES_NCURSES_LIBRARY=$EBROOTNCURSES/lib/libncurses.%s ' % SHLIB_EXT

(Tell CMake about all it searches for that is in our ncurses so it doesn't pick any of that up from the system.)

There is no need for the -DCMAKE_PREFIX_PATH=$EBROOTNCURSES which can be dropped.

And since CMake also uses bzip2, zlib, curl we could add the following, but if not it will build those internally so there is probably no risk of them getting picked up from any system installed versions.

configopts += '-DCMAKE_USE_SYSTEM_BZIP2=1 '
configopts += '-DBZIP2_INCLUDE_DIR=$EBROOTBZIP2/include '
configopts += '-DBZIP2_LIBRARY_RELEASE=$EBROOTBZIP2/lib/libbz2.%s ' % SHLIB_EXT
configopts += '-DCMAKE_USE_SYSTEM_ZLIB=1 '
configopts += '-DZLIB_INCLUDE_DIR=$EBROOTZLIB/include '
configopts += '-DZLIB_LIBRARY_RELEASE=$EBROOTZLIB/lib/libz.%s ' % SHLIB_EXT
configopts += '-DCMAKE_USE_SYSTEM_CURL=1 '
configopts += '-DCURL_INCLUDE_DIR=$EBROOTCURL/include '
configopts += '-DCURL_LIBRARY=$EBROOTCURL/lib/libcurl.%s ' % SHLIB_EXT

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 11, 2018

@smoors Are you up for making these changes to the most recent CMake easyconfig(s)?

@akesandgren
Copy link
Copy Markdown
Contributor

And of course for that later part we need

    ('zlib', '1.2.11'),
    ('bzip2', '1.0.6'),
    ('cURL', '7.60.0'),

in the dependency section

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 12, 2018

@boegel yes, though I am not sure if we should do this for older CMake versions as well, since we did not have problems with them, and I don't want to introduce regressions to them. what do you think?

@smoors
Copy link
Copy Markdown
Contributor Author

smoors commented Oct 12, 2018

ah you probably meant only CMake-3.12.1 but with other toolchains as well.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 12, 2018

@smoors I'm fine with not doing it for older CMake versions, but it should certainly be consistent across all CMake 3.12.1 easyconfigs + more recent versions (if any)

@boegel boegel changed the title CMake-3.12.1-GCCcore-7.3.0 force build ccmake force building of ccmake for CMake 3.12.1 + fix deps Oct 12, 2018

dependencies = [
('ncurses', '6.0'),
('ncurses', '6.1'),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smoors Please stick to ncurses 6.0 here, bumping the version is going to lead to headaches..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops..

@boegel boegel added this to the 3.7.1 milestone Oct 12, 2018
@easybuilders easybuilders deleted a comment from boegelbot Oct 12, 2018
@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 12, 2018

Test report by @boegel
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in this PR)
node2566.golett.os - Linux centos linux 7.5.1804, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, Python 2.7.5
See https://gist.github.com/f333dbeafa2717db35a7fde584e5f9ca for a full test report.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 12, 2018

Test report by @boegel
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in this PR)
node2086.delcatty.os - Linux centos linux 7.5.1804, Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, Python 2.7.5
See https://gist.github.com/011763b2ad9a4f384f1e71dc556a3760 for a full test report.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 12, 2018

Test report by @boegel
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in this PR)
node3126.skitty.os - Linux centos linux 7.5.1804, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Python 2.7.5
See https://gist.github.com/6537d4ea9a88ba35346d8c3238728724 for a full test report.

@boegel
Copy link
Copy Markdown
Member

boegel commented Oct 12, 2018

Going in, thanks @smoors!

@boegel boegel merged commit 75e1a3a into easybuilders:develop Oct 12, 2018
@smoors smoors deleted the 20181005163623_new_pr_CMake3121 branch January 27, 2020 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants