docs(testing): document passing pytest options during local development#3367
Conversation
|
@JohnVillalovos - Would you have some time to review this change? Basically the motivation for this came when I was working on #3366 . I wanted to be able to run only selected tests instead of everything while developing locally. Running entire test suite locally is time consuming. Plus, gitlab in docker in local machine requires is not quite stable dependent on available resource. I'm not too familiar with |
This improves the developer experience by making the testing workflow more discoverable and flexible. - Updates `tox.ini` to use `posargs` with default values for the test environments. This allows developers to run the full test suite by default, or easily specify a single test file to run. - Adds `description` fields to the unit and functional test environments in `tox.ini`. This makes instructions on how to run specific tests visible in the output of `tox list`. - Updates `CONTRIBUTING.rst` with concrete examples for running specific unit, API functional, and CLI functional tests, aligning the documentation with the new `tox` configuration.
The `cover` environment was failing in CI with a "No such file or directory: 'pytest'" error. This occurred because the environment did not have a `deps` section to install the testing requirements. While this might work locally due to `tox` reusing environments, it fails in a clean CI run. This change adds the required dependencies to the `[testenv:cover]` section, ensuring it is self-contained and runs reliably.
The functional test environments (`cli_func_v4`, `api_func_v4`) were failing in CI with a `ValueError: no option named 'keep_containers'`. This occurred because the environments were missing the `requirements-test.txt` dependency, so `pytest` and its plugins (like `pytest-docker`) were not installed. A previous refactoring of `tox.ini` made the environments more isolated, exposing this latent dependency issue. This change adds the required test dependencies to the functional test environments, ensuring they are self-contained and run reliably.
04365b5 to
389f2c9
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates the local testing workflow to make tox usage more discoverable and allow targeting specific test paths via posargs, and aligns contributor documentation with the new invocation style.
Changes:
- Updates unit/functional
toxenvironments to use{posargs:<default path>}so runningtox -e <env> -- <path>can target a specific file/dir. - Adds
descriptionmetadata to keytoxenvironments to improvetox listoutput. - Extends
CONTRIBUTING.rstwith concrete examples for running specific unit and functional test files.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| tox.ini | Adds env descriptions and switches pytest commands to {posargs:...} defaults; adjusts deps for cover and functional envs. |
| CONTRIBUTING.rst | Documents how to run single unit/functional test files via tox ... -- <path>. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
JohnVillalovos
left a comment
There was a problem hiding this comment.
Thanks for the PR
The goal of running individual tests locally is reasonable, but the {posargs:tests/functional/cli} change is what's causing the CI failure.
Root cause of CI failure: In CI, the workflow runs tox -- --override-ini='log_cli=True', which sets posargs. With the new {posargs:tests/functional/cli} syntax, that posargs replaces the default test path, so pytest ends up with no test directory and discovers all tests from the repo root. This triggers ValueError: no option named 'keep_containers' because the functional conftest's pytest_addoption isn't loaded properly when tests are collected this way.
The existing main branch tox config already supports running specific tests via -k:
Run a single test
tox -e cli_func_v4 -- -k test_create_project_issue
Run multiple tests
tox -e cli_func_v4 -- -k "test_create_project_issue or test_delete_project"
This works because -k is an additional pytest flag passed through {posargs}, not a replacement for the hardcoded test path. No tox.ini changes needed.
I'd suggest reverting the {posargs:...} changes to the functional test environments and instead just adding the description fields documenting the -k approach.
|
Thank you for all the feedbacks. I'll take a look when I get a chance. One thing I did notice with my current change is that when I run |
This reverts commit bd6b425.
This reverts commit 389f2c9.
No changes are needed to `tox` config to be able to run individual tests.
Added a section into the contributing docs to show how to pass options to `pytest`. This can be useful during local development for selecting specific tests. Running entire test suite is very time consuming. Removed earlier instructions for the same purpose as they are no longer valid.
Changes
This improves the developer experience by making the testing workflow more discoverable and flexible.
Updates
tox.inito useposargswith default values for the test environments. This allows developers to run the full test suite by default, or easily specify a single test file to run.Adds
descriptionfields to the unit and functional test environments intox.ini. This makes instructions on how to run specific tests visible in the output oftox list.Updates
CONTRIBUTING.rstwith concrete examples for running specific unit, API functional, and CLI functional tests, aligning the documentation with the newtoxconfiguration.Documentation and testing
Please consider whether this PR needs documentation and tests. This is not required, but highly appreciated: