Skip to content

Circle: Build docs in parallel.#20718

Merged
tacaswell merged 2 commits intomatplotlib:mainfrom
QuLogic:circle-parallel
Aug 9, 2022
Merged

Circle: Build docs in parallel.#20718
tacaswell merged 2 commits intomatplotlib:mainfrom
QuLogic:circle-parallel

Conversation

@QuLogic
Copy link
Copy Markdown
Member

@QuLogic QuLogic commented Jul 22, 2021

PR Summary

We are supposedly on medium Docker VMs, which offer 2 vCPUs, so we should try to build in parallel.

PR Checklist

  • [n/a] Has pytest style unit tests (and pytest passes).
  • Is Flake 8 compliant (run flake8 on changed files to check).
  • [n/a] New features are documented, with examples if plot related.
  • Documentation is sphinx and numpydoc compliant (the docs should build without error).
  • [n/a] Conforms to Matplotlib style conventions (install flake8-docstrings and run flake8 --docstring-convention=all).
  • [n/a] New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
  • [n/a] API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).

@jklymak
Copy link
Copy Markdown
Member

jklymak commented Jul 22, 2021

Looks like the build ran out of memory?

@jklymak jklymak marked this pull request as draft July 27, 2021 13:53
@QuLogic
Copy link
Copy Markdown
Member Author

QuLogic commented Aug 4, 2022 •
edited
Loading

On recent Circle builds, I noticed it said we have access to a higher resource class, so I bumped it up to large, which will hopefully mean it won't run out of RAM this time, and will be a little bit faster with double the vCPU.

QuLogic added 2 commits August 4, 2022 22:53
@QuLogic
Circle: Increase resource class to large
9ac36f3 
This doubles the vCPU and RAM, which will hopefully be enough to run in
parallel.
@QuLogic
Circle: Build docs in parallel.
afef2e6 
We are now on large Docker VMs, which offer 4 vCPUs, so we should try to
build in parallel.
@QuLogic
Copy link
Copy Markdown
Member Author

QuLogic commented Aug 5, 2022

Re-reading the resource class docs again, it looks like the system really has 32 cores, and since I passed -jauto, it probably was running in 32 processes, which is likely to OOM on any amount of RAM. Testing locally, single-core takes 2.4GB, 2-process takes 2.5GB, 3-process takes 2.65GB, and 4-process takes 3.1GB. The times are 19.6min, 17.1min, 15.7min, and 15.3min, respectively. Unfortunately, sphinx-gallery takes half the time and doesn't run in parallel, so we can't save too much.

But we might as well save the 5 minutes by going to -j4.

@QuLogic QuLogic marked this pull request as ready for review August 5, 2022 04:18
@tacaswell tacaswell added this to the v3.6.0 milestone Aug 9, 2022
@tacaswell tacaswell merged commit d8005dd into matplotlib:main Aug 9, 2022
@QuLogic QuLogic deleted the circle-parallel branch August 9, 2022 19:31
@StefRe StefRe mentioned this pull request Sep 30, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timhoffm timhoffm timhoffm approved these changes

Assignees

No one assigned

Labels

Build Documentation

Projects

None yet

Milestone

v3.6.0

Development

Successfully merging this pull request may close these issues.

4 participants

@QuLogic @jklymak @timhoffm @tacaswell