[ty] Remove the mypy_primer CI workflow by sharkdp · Pull Request #24016 · astral-sh/ruff

sharkdp · 2026-03-17T13:02:58Z

Summary

Remove the mypy_primer CI workflow and several files/scripts associated with it.

Test Plan

Successful CI run on this PR

astral-sh-bot · 2026-03-17T13:05:05Z

Typing conformance results

No changes detected ✅

Current numbers

The percentage of diagnostics emitted that were expected errors held steady at 85.29%. The percentage of expected errors that received a diagnostic held steady at 78.13%. The number of fully passing files held steady at 64/132.

astral-sh-bot · 2026-03-17T13:07:06Z

Memory usage report

Memory usage unchanged ✅

astral-sh-bot · 2026-03-17T13:09:27Z

`ecosystem-analyzer` results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

astral-sh-bot · 2026-03-17T13:14:30Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

sharkdp · 2026-03-17T14:01:42Z

.github/workflows/ty-ecosystem-analyzer.yaml

-      - "scripts/mypy_primer.sh"
-      - "scripts/mypy_primer_selector.py"
+      - ".github/ty-ecosystem.toml"
+      - "crates/ty_python_semantic/resources/primer/**"


this is new. seems reasonable to run the workflow if good/bad/flaky.txt change

should we rename the directory to crates/ty_python_semantic/resources/ecosystem/?

that's what I did first, but then I reverted to keep the diff smaller. I like the plan of getting rid of those files, so I'm not going to spend time on renaming the folder now.

sharkdp · 2026-03-17T14:04:01Z

.github/workflows/mypy_primer.yaml

-  # Runs mypy twice against the same ty version to catch any non-deterministic behavior (ideally).
-  # The job is disabled for now because there are some non-deterministic diagnostics.
-  mypy_primer_same_revision:
-    name: Run mypy_primer on same revision
-    runs-on: ${{ github.repository == 'astral-sh/ruff' && 'depot-ubuntu-22.04-32' || 'ubuntu-latest' }}
-    timeout-minutes: 20
-    # TODO: Enable once we fixed the non-deterministic diagnostics
-    if: false && contains(github.event.pull_request.labels.*.name, 'mypy_primer')
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-        with:
-          path: ruff
-          fetch-depth: 0
-          persist-credentials: false
-
-      - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
-        with:
-          version: "0.10.10"
-
-      - uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2.8.2
-        with:
-          workspaces: "ruff"
-          shared-key: "mypy-primer"
-
-      - name: Install Rust toolchain
-        run: rustup show
-
-      - name: Run determinism check
-        env:
-          BASE_REVISION: ${{ github.event.pull_request.head.sha }}
-          PRIMER_SELECTOR: crates/ty_python_semantic/resources/primer/good.txt
-          CLICOLOR_FORCE: "1"
-          DIFF_FILE: mypy_primer_determinism.diff
-        run: |
-          cd ruff
-          scripts/mypy_primer.sh
-
-      - name: Check for non-determinism
-        run: |
-          # Remove ANSI color codes for checking
-          sed -e 's/\x1b\[[0-9;]*m//g' mypy_primer_determinism.diff > mypy_primer_determinism_clean.diff
-
-          # Check if there are any differences (non-determinism)
-          if [ -s mypy_primer_determinism_clean.diff ]; then
-            echo "ERROR: Non-deterministic output detected!"
-            echo "The following differences were found when running ty twice on the same commit:"
-            cat mypy_primer_determinism_clean.diff
-            exit 1
-          else
-            echo "✓ Output is deterministic"
-          fi


@MichaReiser We were never able to activate this check. We now have flaky-detection in ecosystem-primer, which is a more robust way to check the same thing. It's sort-of a "soft" check compared to what you have here, but it should still allow us to catch regressions, I think.

sharkdp · 2026-03-17T14:05:10Z

crates/ty/docs/mypy_primer.md

@@ -1,69 +0,0 @@
-# Running `mypy_primer`


This file was already slightly out of date, and I don't think there are a lot of use cases for running mypy_primer locally anymore, so it seems okay to remove this without replacement.

Let me know if you think otherwise, and I can write a new document that would show how to do those things with ecosystem-analyzer.

sharkdp · 2026-03-17T14:05:54Z

crates/ty/CONTRIBUTING.md

+## Ecosystem CI (`ecosystem-analyzer`)

-GitHub Actions will run your changes against a number of real-world projects from GitHub and
-report on any linter or formatter differences. See [`crates/ty/docs/mypy_primer.md`](./docs/mypy_primer.md)


linter or formatter differences

🙃

sharkdp · 2026-03-17T14:08:08Z

scripts/check_mypy_primer_sha.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python3
-"""Check that the mypy-primer SHA in setup_primer_project.py matches mypy_primer.sh."""


This script made sure that the mypy_primer workflow and setup_primer_project.py use the same Git sha for the mypy_primer dependency. Now that the workflow is gone, there is nothing to check against. Unfortunately, the mypy_primer pin that we use in ecosystem-analyzer is currently not fixed (astral-sh/ecosystem-analyzer#10), but once we have that, maybe there is a way to bring this script back. For now, I changed setup_primer_project.py to always use the latest version of mypy_primer, which seems fine in almost all cases.

sharkdp · 2026-03-17T14:08:43Z

scripts/setup_primer_project.py

 # dependencies = ["mypy-primer"]
 #
 # [tool.uv.sources]
-# mypy-primer = { git = "https://github.com/hauntsaninja/mypy_primer", rev = "850d65d9da947ef9e02498b6f07598e7c8401641" }


See my other comment above

While I agree we'd ideally just make sure we're always using the same pin as ecosystem-analyzer, it still feels not-great from a supply-chain security perspective not to use a pinned dependency here...

we don't even run this script anywhere, and we're contributors + subscribed to updates on that repo, so I think this is a negligible risk.

My agents run it regularly to (attempt to) reproduce ecosystem changes, though

AlexWaygood · 2026-03-17T14:40:36Z

.github/workflows/ty-ecosystem-analyzer.yaml

-      - "scripts/mypy_primer.sh"
-      - "scripts/mypy_primer_selector.py"
+      - ".github/ty-ecosystem.toml"
+      - "crates/ty_python_semantic/resources/primer/**"


should we rename the directory to crates/ty_python_semantic/resources/ecosystem/?

AlexWaygood · 2026-03-17T14:43:49Z

scripts/setup_primer_project.py

 # dependencies = ["mypy-primer"]
 #
 # [tool.uv.sources]
-# mypy-primer = { git = "https://github.com/hauntsaninja/mypy_primer", rev = "850d65d9da947ef9e02498b6f07598e7c8401641" }


While I agree we'd ideally just make sure we're always using the same pin as ecosystem-analyzer, it still feels not-great from a supply-chain security perspective not to use a pinned dependency here...

## Summary Pin mypy_primer (reference: #24016 (comment))

* main: [ty] Filter out unsatisfiable inference attempts during generic call narrowing (#24025) [ty] Avoid inferring intersection types for call arguments (#23933) [ty] Pin mypy_primer in `setup_primer_project.py` (#24020) [`pycodestyle`] Recognize `pyrefly:` as a pragma comment (`E501`) (#24019) Add company AI policy to contributing guide (#24021) [ty] Remove the mypy_primer CI workflow (#24016) Update prek dependencies (#23980) [ty] Smarter semantic token classification for attribute access on union type (#23841) [ty] ecosystem-analyzer: Inline diffs and panic messages (#24015) [ty] Improve `.toml` support in the ty playground (#23476) PEP 639 license information (#19661)

sharkdp added ci Related to internal CI tooling ty Multi-file analysis & type inference labels Mar 17, 2026

[ty] Remove the mypy_primer CI workflow

e8d7c47

sharkdp commented Mar 17, 2026

View reviewed changes

sharkdp force-pushed the david/remove-mypy_primer branch from 440a0c5 to e8d7c47 Compare March 17, 2026 14:08

sharkdp marked this pull request as ready for review March 17, 2026 14:14

sharkdp requested review from AlexWaygood, MichaReiser, carljm, dcreager and ibraheemdev as code owners March 17, 2026 14:14

astral-sh-bot bot assigned BurntSushi Mar 17, 2026

astral-sh-bot bot requested a review from BurntSushi March 17, 2026 14:14

sharkdp assigned AlexWaygood and unassigned BurntSushi Mar 17, 2026

AlexWaygood approved these changes Mar 17, 2026

View reviewed changes

sharkdp merged commit dc59ffa into main Mar 17, 2026
49 checks passed

sharkdp deleted the david/remove-mypy_primer branch March 17, 2026 14:49

sharkdp mentioned this pull request Mar 17, 2026

[ty] Pin mypy_primer in setup_primer_project.py #24020

Merged

sharkdp added a commit that referenced this pull request Mar 17, 2026

[ty] Pin mypy_primer in setup_primer_project.py (#24020)

7f275f4

## Summary Pin mypy_primer (reference: #24016 (comment))

		@@ -1,40 +0,0 @@
		#!/usr/bin/env python3
		"""Check that the mypy-primer SHA in setup_primer_project.py matches mypy_primer.sh."""

Conversation

sharkdp commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

astral-sh-bot bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

No changes detected ✅

Uh oh!

astral-sh-bot bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Memory usage report

Uh oh!

astral-sh-bot bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ecosystem-analyzer results

Uh oh!

astral-sh-bot bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sharkdp Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sharkdp commented Mar 17, 2026 •

edited

Loading

astral-sh-bot bot commented Mar 17, 2026 •

edited

Loading

astral-sh-bot bot commented Mar 17, 2026 •

edited

Loading

astral-sh-bot bot commented Mar 17, 2026 •

edited

Loading

`ecosystem-analyzer` results

astral-sh-bot bot commented Mar 17, 2026 •

edited

Loading

`ruff-ecosystem` results

sharkdp Mar 17, 2026 •

edited

Loading