Skip to content

use separate different progress bars for different aspects of the installations being performed#3844

Merged
branfosj merged 27 commits intoeasybuilders:developfrom
boegel:multi_level_progress
Oct 18, 2021
Merged

use separate different progress bars for different aspects of the installations being performed#3844
branfosj merged 27 commits intoeasybuilders:developfrom
boegel:multi_level_progress

Conversation

@boegel
Copy link
Copy Markdown
Member

@boegel boegel commented Sep 21, 2021 •
edited
Loading

  • refactoring to simplify creating/updating/stopping progress bars
  • separate progress bars for:
    • overall progress
    • progress per easyconfig
    • use larger "weight" for long running steps (configure, build, install, extensions)
    • show progress for downloading file (pop-up)
    • only show progress for long downloads (sufficiently large files, or longer than N secs)
    • show overall progress when downloading files (N/M done)
    • extensions progress (pop-up)
  • only show overall progress bar when there's more than 1 installation
  • move code to determine file size before downloading to a dedicated function, and also use it when downloading via requests
  • tweak progress bar output when using --fetch, --stop, ...
  • don't show progress bars in dry run mode
  • show name of running step in easyconfig progress bar
  • use spinner for running step
  • don't print step progress lines when using progress bars (or change to only show them per installation?)
    • for now, I'd like to keep the standard output, so that progress bars are only shown when the session is still running...

fixes #2596, fixes #3041

@boegel boegel added this to the next release (4.5.0?) milestone Sep 21, 2021
@nordmoen nordmoen mentioned this pull request Sep 22, 2021
Closed
@easybuilders easybuilders deleted a comment from boegelbot Sep 27, 2021
@easybuilders easybuilders deleted a comment from boegelbot Oct 9, 2021
@easybuilders easybuilders deleted a comment from boegelbot Oct 13, 2021
boegel added 10 commits October 13, 2021 10:15
@boegel
show progress bar for extensions
0dead5f
@boegel
show separate progress bar to report progress on fetching of sources/…
b956e8b 
…patches + only show download progress for files >= 10MB
@boegel
don't show progress bar if there's only a single task
8d5bf09
@boegel
take into account --stop, --fetch, --module-only when determining num…
a9a9d23 
…ber of steps that will be run
@boegel
also determine file size when downloading file via requests module
9e2e0b0
@boegel
fix broken test_toy_multi_deps by re-disabling showing of progress bars
5350e59
@boegel
add test for toy build with showing of progress bars enabled
a4215be
@boegel
also test show_progress_bars() function in output tests
c61e848
@boegel
remove unused step_id variable in EasyBlock.run_all_steps
4e09da4
@boegel
fix test_det_file_size for Python 2.7 (result of urllib2.urlopen can'…
56660a2 
…t be used as context manager)
@easybuilders easybuilders deleted a comment from boegelbot Oct 13, 2021
@boegel boegel force-pushed the multi_level_progress branch from 6a8d5ff to ab10363 Compare October 13, 2021 20:55
branfosj
branfosj requested changes Oct 14, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@branfosj branfosj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to actually run EB with this to see how I like the look / feel.

Comment thread easybuild/main.py Outdated
Comment thread easybuild/framework/easyblock.py Outdated
@boegel
Copy link
Copy Markdown
Member Author

boegel commented Oct 14, 2021

@branfosj @nordmoen I consider this pretty complete now (see checklist in PR description).

There's probably room for improvement/further tweaking, but I'd like to keep that for smaller follow-up PRs, unless there's something in here w.r.t. code changes or look & feel you'd like to see changed...

branfosj
branfosj requested changes Oct 15, 2021
View reviewed changes
Comment thread easybuild/tools/output.py Outdated
Comment thread easybuild/tools/output.py
"""Shim for Rich's Progress class."""
PROGRESS_BAR_DOWNLOAD_ALL = 'download_all'
PROGRESS_BAR_DOWNLOAD_ONE = 'download_one'
PROGRESS_BAR_EXTENSIONS = 'extensions'
Copy link
Copy Markdown
Member

@branfosj branfosj Oct 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handles extensions, but not components. We should do something similar for them. I'm happy for that to be a follow on enhancement.

Copy link
Copy Markdown
Member Author

@boegel boegel Oct 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Components are specific to the Bundle easyblock, framework doesn't know about them...

I think we can handle this by providing a "neutral" progress bar which can be used by easyblock to display progress on a specific aspect (like installing a list of components). I'd keep that for a follow-up PR though...

Copy link
Copy Markdown
Member Author

@boegel boegel Oct 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a note to #3827 on that...

@boegel boegel mentioned this pull request Oct 16, 2021
Open
@boegel boegel marked this pull request as ready for review October 16, 2021 12:52
@boegel boegel requested a review from branfosj October 16, 2021 13:25
@branfosj
Copy link
Copy Markdown
Member

branfosj commented Oct 18, 2021

Going in, thanks @boegel!

@branfosj branfosj merged commit 344adf2 into easybuilders:develop Oct 18, 2021
@boegel boegel deleted the multi_level_progress branch October 18, 2021 12:31
@boegel boegel mentioned this pull request Oct 20, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@branfosj branfosj branfosj approved these changes

Assignees

No one assigned

Labels

change code cleanup enhancement

Projects

None yet

Milestone

4.5.0

Development

Successfully merging this pull request may close these issues.

Download progress for large files and/or slow downloads Feature: progress bar for "fetching files"

2 participants

@boegel @branfosj