Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1703/
Test FAILed.

Not out of the woods yet:

Traceback (most recent call last):
  File "/var/lib/jenkins/.local/lib/python2.6/logging/handlers.py", line 77, in emit
    self.doRollover()
  File "/var/lib/jenkins/.local/lib/python2.6/logging/handlers.py", line 137, in doRollover
    self.stream = self._open()
  File "/var/lib/jenkins/.local/lib/python2.6/logging/__init__.py", line 846, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 2] No such file or directory: '/var/tmp/eb-SFKPKF/eb-RIx5GT/eb-MZB8sc/eb-26IEHo/eb-cCFCzg/easybuild-toy-0.0-20150514.162546.AjYeE.log'

Traceback (most recent call last):
  File "/var/lib/jenkins/.local/lib/python2.6/logging/handlers.py", line 76, in emit
    if self.shouldRollover(record):
  File "/var/lib/jenkins/.local/lib/python2.6/logging/handlers.py", line 150, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
ValueError: I/O operation on closed file

Hmm, problems mentioned above seemed to be caused by a known bug, see http://bugs.python.org/issue10949 (Jenkins is running Python 2.6).

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1704/
Test FAILed.

OK, found the actual culprit of the log files not being removed: the log handlers were not being cleaned up...

Simple fix, huge impact.

with this patch: Ran 284 tests in 313.337s

without this patch: Ran 284 tests in 907.145s

looks fine

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1705/
Test PASSed.

@boegel can you explain why this makes such a huge difference in performance?!

@pforai: Not removing the log handlers means that they stay in place, and also keep getting used. Every time main() is called from the tests, one or two handlers are added.
If these don't get removed, they will be used every time a log method is called (even though the log file they are supposed to write to is already gone, or at least the inode pointing to the data in the file is).

Near the end of the test suite, over 100 handlers were still active, all of which were still writing to disk (even though the inode of the log file was already removed via shutil.rmtree(self.test_prefix) in tearDown).

Before this patch, about 3GB of disk space was required in /tmp because the log files of previous tests were still being written to. Now, about 100MB is more than enough...

That's actually was caused us diving into this: /tmp on the Jenkins server is quite limited (~6GB free space), and two framework test suites running at the same was enough to fill up /tmp and let all things go to hell. Quite annoying (since all subsequent tests, including PR tests, failed hard then), and I was (finally) fed up with it, which motivated me to get to the bottom of this. ;-)

Some other changes were also done in this PR, i.e. not reconfiguring in tearDown, only to then reconfigure again in setUp of the next step. That helps too, but just a little bit.

Time to merge this in, and forget about it.

Thanks for the review @wpoely86, and @stdweird for helping to figure this out (a bit ;-)).

@boegel nice!