Looks good

@boegel Failed again. Are you sure that remove is needed?

@wpoely86: yes, because the subsequent subtest generates a module file with the same name, otherwise the exists test is useless

the failure is deeper down now though... I'm starting to suspect this is caused by premature garbage collection of some sort, triggering a removed of the tmp dir created via tempfile.mkdtemp() (i.e. self.test_installdir)

The test_toy_hierarchical test runs for a while (over 1m), so it's likely, and it would explain the random failures...

Jenkins: test this please

@wpoely86: latest run of tests failed again, in yet another subtest of toy_build.py... Weird stuff.

Jenkins: test this please

Checked the tmp cleanup cron job again that's running on Jenkins...

Before diving into this, files in /tmp and /var/tmp were being removed by that cron job based on modification times of files (mtime). That isn't correct, since in the tests some files are being copied without being modified (e.g. module files to test the hierarchy), and so it was possible that those were being removed mid-test.
I then changed this to removing based on access time (atime), but that also turns out the be incorrect, since the access time is not bumped for a file that has just been copied (that doesn't make 100% sense to me, since creating the file means accessing it imho, but fine):

$ cp -a test/framework/easyconfigs/toy-0.0.eb /tmp 
$ stat -x /tmp/toy-0.0.eb
  File: "/tmp/toy-0.0.eb"
  Size: 651          FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/ kehoste)  Gid: (   20/   staff)
Device: 1,2   Inode: 17024890    Links: 1
Access: Thu Jul 31 14:14:08 2014
Modify: Thu Jul  3 21:32:24 2014
Change: Fri Aug  1 08:44:34 2014

However, change time (ctime) is being bumped, and is bumped again when the file gets changed (duh):

$ echo "foo" >> /tmp/toy-0.0.eb 
$ stat -x /tmp/toy-0.0.eb      
  File: "/tmp/toy-0.0.eb"
  Size: 655          FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/ kehoste)  Gid: (   20/   staff)
Device: 1,2   Inode: 17024890    Links: 1
Access: Thu Jul 31 14:14:08 2014
Modify: Fri Aug  1 08:46:35 2014
Change: Fri Aug  1 08:46:35 2014

So, I've changed the cron job to remove files based on ctime now. This should resolve the randomly failing tests...

This issue only started occurring lately since I recently made the cleanup cron job a lot more aggressive, to deal with the full disk issues that have been happening when several tests were failing in rapid succession.

So, this PR actually doesn't resolve any 'unstable' tests, but since it includes quite a bit of cleanup and an additional test for read_file and (the enhanced) write_file, it can/should still go in.

@wpoely86: can you give this another quick review before I merge this in?

Jenkins: test this please

I'll go ahead and merge this in since @wpoely86 already reviewed the bulk of it, and it mostly concerns the tests.