Skip to content

ci: add downstream testing#1049

Merged
henryiii merged 12 commits intopypa:mainfrom
henryiii:henryiii/ci/downstream
Mar 7, 2026
Merged

ci: add downstream testing#1049
henryiii merged 12 commits intopypa:mainfrom
henryiii:henryiii/ci/downstream

Conversation

@henryiii
Copy link
Copy Markdown
Contributor

@henryiii henryiii commented Jan 12, 2026

This adds a downstream testing job. Five packages: packaging_legacy, build, pyproject-metadata, setuptools, and pip.

Setuptools takes 4 minutes, otherwise they are pretty fast.

@henryiii henryiii force-pushed the henryiii/ci/downstream branch from 95cef6b to 1246054 Compare January 15, 2026 15:33
@henryiii
Copy link
Copy Markdown
Contributor Author

Dropping this here, trying to add hatch:

index 692dfa7..2c2c083 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -81,6 +81,7 @@ PROJECTS = {
     "packaging_legacy": "https://github.com/di/packaging_legacy/archive/refs/tags/23.0.post0.tar.gz",
     "build": "https://github.com/pypa/build/archive/refs/tags/1.4.0.tar.gz",
     "pyproject_metadata": "https://github.com/pypa/pyproject-metadata/archive/refs/tags/0.10.0.tar.gz",
+    "hatch": "https://github.com/pypa/hatch/archive/refs/tags/hatch-v1.16.2.tar.gz",
 }


@@ -108,6 +109,9 @@ def downstream(session: nox.Session, project: str) -> None:
         session.install("-e.")
         session.run(*pip_cmd, "list")
         session.run("pytest", env=env)
+    elif project == "hatch":
+        session.install("-e.", "pytest", "filelock", "flit-core", "trustme", "editables")
+        session.run("pytest", env=env)
     elif project in {"build", "pyproject_metadata"}:
         session.install("-e.", "-

There still is a known breakage (@ spacing changed), and there are some breakages I think because it expects to run from the git root.

I tried pip too:

    elif project == "pip":
        session.install("-e.", "--group=test")
        session.run("pip", "wheel", "-w", "tests/data/common_wheels", "--group", "test-common-wheels")
        session.run(*pip_cmd, "list")
        session.run("pytest")

but the test suite is quite slow. I didn't get to setting up vendoring to replace its copy of packaging. Another candidate for downstream testing, pip-tools, uses pip._vendor.packaging. Poetry-core vendors as well.

Comment thread .github/workflows/test.yml Outdated
@brettcannon
Copy link
Copy Markdown
Member

trying to add hatch:

Did you want me to point Ofek at this issue?

@henryiii
Copy link
Copy Markdown
Contributor Author

You can, there's also pypa/hatch#2159, which is related.

@abravalheri
Copy link
Copy Markdown
Contributor

(FYI: I am letting the CI run some tests for setuptools: pypa/setuptools#5155, hopefully it will be fine)

@henryiii
Copy link
Copy Markdown
Contributor Author

Great, I just tried that here too; the only downside is it's pretty slow (several minutes), but I can make it work. Added a print inside our code to verify setuptools is picking it up when I patch it in, it's working locally.

diff --git a/noxfile.py b/noxfile.py
index 692dfa7..ed19c1d 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -80,6 +80,7 @@ def tests(session: nox.Session) -> None:
 PROJECTS = {
     "packaging_legacy": "https://github.com/di/packaging_legacy/archive/refs/tags/23.0.post0.tar.gz",
     "build": "https://github.com/pypa/build/archive/refs/tags/1.4.0.tar.gz",
+    "setuptools": "https://github.com/pypa/setuptools/archive/refs/tags/v80.10.1.tar.gz",
     "pyproject_metadata": "https://github.com/pypa/pyproject-metadata/archive/refs/tags/0.10.0.tar.gz",
 }

@@ -87,6 +88,7 @@ PROJECTS = {
 @nox.parametrize("project", list(PROJECTS))
 @nox.session(default=False)
 def downstream(session: nox.Session, project: str) -> None:
+    pkg_dir = Path.cwd() / "src/packaging"
     env = {"FORCE_COLOR": None}
     session.install("-e.")

@@ -107,12 +109,19 @@ def downstream(session: nox.Session, project: str) -> None:
         session.install("-r", "tests/requirements.txt")
         session.install("-e.")
         session.run(*pip_cmd, "list")
-        session.run("pytest", env=env)
+        session.run("pytest", *session.posargs, env=env)
     elif project in {"build", "pyproject_metadata"}:
         session.install("-e.", "--group=test")
         if project != "build":
             session.run(*pip_cmd, "list")
-        session.run("pytest", env=env)
+        session.run("pytest", *session.posargs, env=env)
+    elif project == "setuptools":
+        session.install("-e.[test,cover]")
+        session.run(*pip_cmd, "list")
+        repl_dir = "setuptools/_vendor/packaging"
+        shutil.rmtree(repl_dir)
+        shutil.copytree(pkg_dir, repl_dir)
+        session.run("pytest", *session.posargs, env=env)
     else:
         session.error("Unknown package")

@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Jan 21, 2026

I can run pip using the same idea too, but the test suite takes 53 minutes on my Intel machine. I see the spacing changes, so it's using the right version of packaging.

diff --git a/noxfile.py b/noxfile.py
index ed19c1d..658b806 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -82,6 +82,7 @@ PROJECTS = {
     "build": "https://github.com/pypa/build/archive/refs/tags/1.4.0.tar.gz",
     "setuptools": "https://github.com/pypa/setuptools/archive/refs/tags/v80.10.1.tar.gz",
     "pyproject_metadata": "https://github.com/pypa/pyproject-metadata/archive/refs/tags/0.10.0.tar.gz",
+    "pip": "https://github.com/pypa/pip/archive/refs/tags/25.3.tar.gz",
 }


@@ -122,6 +123,14 @@ def downstream(session: nox.Session, project: str) -> None:
         shutil.rmtree(repl_dir)
         shutil.copytree(pkg_dir, repl_dir)
         session.run("pytest", *session.posargs, env=env)
+    elif project == "pip":
+        session.install("-e.", "--group=test")
+        session.run("pip", "wheel", "-w", "tests/data/common_wheels", "--group", "test-common-wheels")
+        session.run(*pip_cmd, "list")
+        repl_dir = "src/pip/_vendor/packaging"
+        shutil.rmtree(repl_dir)
+        shutil.copytree(pkg_dir, repl_dir)
+        session.run("pytest")
     else:
         session.error("Unknown package")

@notatallshaw
Copy link
Copy Markdown
Member

I would be happy to work on some useful subset of pip unit and feature tests most likely impacted by packaging changes, but I'd be surprised if we got it to less than 10% the full runtime.

@abravalheri
Copy link
Copy Markdown
Contributor

abravalheri commented Jan 21, 2026

the only downside is it's pretty slow (several minutes)

Yeah, our tests are pretty slow 😅.
If you prefer to not have it coded in the packaging test suite, just give me a heads up and we can always run the setuptools CI with the beta release of packaging.

I tested simply creating a PR on setuptools with the newer dependency listed in pyproject.toml and it seems to suffice... (unless I am missing something)

@henryiii henryiii force-pushed the henryiii/ci/downstream branch 2 times, most recently from 8087924 to b124c4f Compare January 23, 2026 17:21
@henryiii henryiii force-pushed the henryiii/ci/downstream branch from b124c4f to 996b8a9 Compare February 12, 2026 14:53
@henryiii henryiii marked this pull request as ready for review February 12, 2026 15:14
@henryiii henryiii force-pushed the henryiii/ci/downstream branch 2 times, most recently from e38e36f to 241c0ec Compare February 12, 2026 21:14
@henryiii henryiii force-pushed the henryiii/ci/downstream branch from 241c0ec to b67d3d9 Compare February 24, 2026 17:17
@henryiii henryiii force-pushed the henryiii/ci/downstream branch from b67d3d9 to fae31ec Compare March 6, 2026 14:33
@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Mar 6, 2026

@brettcannon or @notatallshaw, okay for this to go in? It's basically just more testing.

@notatallshaw
Copy link
Copy Markdown
Member

This would be tested as part of PRs? Do we have a way to protect ourselves from specific flaky tests from downstream projects?

@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Mar 6, 2026

Yes. It's supposed to be pinned to exact versions of the downstreams, so we have to update manually, and that should protect our CI. (Though some test suites might be a bit flaky - setuptools test_editable_with_pyproject just flaked, and it looks like it might be known.)

Copy link
Copy Markdown
Member

@notatallshaw notatallshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More testing = Good

henryiii and others added 6 commits March 7, 2026 01:46
Signed-off-by: Henry Schreiner <[email protected]>
Signed-off-by: Henry Schreiner <[email protected]>
Signed-off-by: Henry Schreiner <[email protected]>
Signed-off-by: Henry Schreiner <[email protected]>
Signed-off-by: Henry Schreiner <[email protected]>
@henryiii henryiii force-pushed the henryiii/ci/downstream branch from 0122a45 to 2f4e5b6 Compare March 7, 2026 06:46
@henryiii henryiii merged commit b9fb15a into pypa:main Mar 7, 2026
55 checks passed
@henryiii henryiii deleted the henryiii/ci/downstream branch March 7, 2026 06:55
ngoldbaum pushed a commit to ngoldbaum/packaging that referenced this pull request Apr 1, 2026
This was referenced Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants