Skip to content

add create_unused_dir function to create a directory which does not yet exist#3551

Merged
boegel merged 4 commits intoeasybuilders:developfrom
Flamefire:create_unused_dir
Apr 5, 2021
Merged

add create_unused_dir function to create a directory which does not yet exist#3551
boegel merged 4 commits intoeasybuilders:developfrom
Flamefire:create_unused_dir

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

This will append _%d until a new folder could be created.
If the base folder name does not exists, it is used directly, so the simple, most common case is the cleanest

Note that I didn't use our mkdir function to reduce the possibility of a race condition where the folder gets created between the check and the os.mkdir call. This solution pushes the burden to Python, which at least reduces the window for such an issue.

This will append _%d until a new folder could be created.
If the base folder name does not exists, it is used directly, so the
simple, most common case is the cleanest
@boegel boegel added this to the 4.x milestone Feb 3, 2021
@boegel
Copy link
Copy Markdown
Member

boegel commented Feb 3, 2021

@Flamefire What's the use case for this? Always good to know.

Also, I think it makes sense to re-use the existing find_backup_name_candidate function from filetools here, which already has logic to determine a new path (which was added in #2134)?

@Flamefire
Copy link
Copy Markdown
Contributor Author

This is for the build directories in meson, cmake, ... where for each build (iteration, extension, bundle component) a new, empty build dir is (usually) required. We had that on Slack where someone reported a failure and I wanted to get this function into framework instead of hacking it into meson only

As for find_backup_name_candidate: I think it makes sense for that function to use the proposed function here (in a follow-up I'd say) as that has 2 issues: First it might be slow, as a sleep is used instead of simply adding a number and second it is unsafe: A file/folder with that name might be created in the meantime. The function here is as safe as possible.

And finally: I'd like to have a simple folder name, e.g. build in the regular case, not something timestamped. This is especially important for the use case where ECs/builds rely on previous builds/iterations or for debugability: A bundle of 3 components will have component 1 in build, component 2 in build_1, component 3 in build_2, ...
It might also matter to keep the pathnames short to avoid failures due to long paths

@boegel boegel modified the milestones: 4.x, release after 4.3.3 Feb 14, 2021
Comment thread easybuild/tools/filetools.py Outdated
Comment thread test/framework/filetools.py
Comment thread test/framework/filetools.py
@boegel boegel changed the title Add create_unused_dir to create a directory which does not yet exist add create_unused_dir function to create a directory which does not yet exist Apr 5, 2021
@boegel boegel merged commit 9082054 into easybuilders:develop Apr 5, 2021
@Flamefire Flamefire deleted the create_unused_dir branch May 12, 2021 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants