Extract shared `rustpython-unicode` crate and route core Unicode semantics through it by Copilot · Pull Request #7561 · RustPython/RustPython

Copilot · 2026-04-05T07:11:30Z

This change introduces a shared rustpython-unicode crate to own CPython-aligned Unicode semantics and data access, instead of keeping that logic spread across str, regex, literal escaping, and unicodedata. The goal is to make Unicode behavior reusable across RustPython components while preserving higher-level string behavior in existing runtime crates.

New shared Unicode crate
- Adds crates/unicode with low-level, reusable APIs for:
  - character classification
  - identifier predicates
  - regex Unicode predicates
  - case mapping / casefold helpers
  - normalization
  - unicodedata-style database lookups
- Keeps the API codepoint-oriented where needed and compatible with no_std consumers.
Refactor core runtime call sites
- rustpython-vm now uses rustpython-unicode for:
  - str.isdecimal()
  - str.isspace()
  - str.isprintable()
  - str.isidentifier()
  - WTF-8 lowercase / uppercase / casefold helpers
- rustpython-literal now uses the shared printable classification for repr/escape behavior.
- rustpython-sre_engine now routes its Unicode character-class and case helpers through the shared crate.
Thin unicodedata integration
- crates/stdlib/src/unicodedata.rs now delegates reusable Unicode database and normalization operations to rustpython-unicode.
- This keeps module behavior in place while moving the underlying data/semantics into the shared layer.
Focused coverage
- Adds unit coverage in rustpython-unicode for classification, identifiers, regex predicates, case mapping, normalization, and Unicode database access.
- Adds a RustPython snippet covering integration across str, re, and unicodedata.

Example of the new shared API shape:

use rustpython_unicode::{classify, identifier, normalize::NormalizeForm, normalize};

assert!(classify::is_decimal('५' as u32));
assert!(identifier::is_python_identifier("유니코드"));

let text = rustpython_wtf8::Wtf8Buf::from("e\u{301}");
let normalized = normalize::normalize(NormalizeForm::Nfc, &text);
assert_eq!(normalized, rustpython_wtf8::Wtf8Buf::from("é"));

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/e4115dbc-b1a5-4a77-90a7-38cf9ac1cdf2 Co-authored-by: youknowone <[email protected]>

youknowone

@copilot check CI failures, comments and also missing refactoring about rustpython_common::encodings and rustpython_codegen::string_parser

youknowone · 2026-04-05T08:18:06Z

crates/stdlib/src/unicodedata.rs

    Nfkd,
 }

+impl From<NormalizeForm> for rustpython_unicode::NormalizeForm {


local NormalizeForm is expected to be replaced by rustpython_unicode::NormalizeForm

Replaced the remaining local wrapper path in unicodedata and now parse directly into rustpython_unicode::NormalizeForm in 2934897.

youknowone · 2026-04-05T08:20:20Z

crates/unicode/Cargo.toml

+[features]
+default = ["std", "casefold"]
+std = []
+casefold = ["std", "dep:caseless"]


do we need this feature? when will the feature be disabled?

Kept the std feature because rustpython-unicode is consumed by no_std crates, while casefold depends on caseless for std-enabled consumers. The feature wiring was clarified in 2934897.

youknowone · 2026-04-05T08:25:01Z

crates/stdlib/src/unicodedata.rs

What I expect is the entire Ucd type is placed in the new unicode crate, without Python runtime.

Moved the non-runtime Ucd implementation into rustpython-unicode and left crates/stdlib/src/unicodedata.rs as a thin Python wrapper in 2934897.

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/1d30ae08-d8f0-431c-9299-8aea5c21f7d4 Co-authored-by: youknowone <[email protected]>

Copilot · 2026-04-05T08:55:16Z

@copilot check CI failures, comments and also missing refactoring about rustpython_common::encodings and rustpython_codegen::string_parser

Checked the PR CI/check runs and the earlier branch run showed a stale lint failure while the newer run was otherwise succeeding/in progress. I addressed the review items and the missing rustpython_common::encodings / rustpython_codegen::string_parser refactors in 2934897.

youknowone

@copilot check CI failures and reviews

youknowone · 2026-04-05T11:53:08Z

crates/literal/src/char.rs

@@ -10,17 +8,5 @@ use icu_properties::props::{EnumeratedProperty, GeneralCategory};
 /// * Zp Separator, Paragraph ('\u2029', PARAGRAPH SEPARATOR)
 /// * Zs (Separator, Space) other than ASCII space('\x20').
 pub fn is_printable(c: char) -> bool {


still need this function? why not directly calling is_repr_printable?

Removed the redundant wrapper and now call rustpython_unicode::classify::is_repr_printable directly from escape.rs in 4efa5da.

youknowone · 2026-04-05T11:54:01Z

crates/sre_engine/src/string.rs

-}
-
 #[inline]
 pub(crate) fn is_word(ch: u32) -> bool {


importing rustpython_unicode::regex as unicode_regex will remove needs for all this helpers

Switched the SRE engine over to a direct rustpython_unicode::regex import and removed the pass-through helpers in 4efa5da.

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/b894057a-9bed-4f35-8400-a5731c63602d Co-authored-by: youknowone <[email protected]>

Copilot · 2026-04-06T02:08:25Z

@copilot check CI failures and reviews

Checked the current CI and reviews. I addressed the redundant literal/sre_engine wrappers and the cargo shear dependency failure in 4efa5da. After that cleanup, cargo shear, cargo check, clippy, and parallel_validation pass; the focused CPython failures still reproduce as test_re and test_pkgutil.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

docs.rs
- Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Initial plan

d3af1c5

Copilot AI assigned Copilot and youknowone Apr 5, 2026

Copilot started work on behalf of youknowone April 5, 2026 07:11 View session

Copilot AI linked an issue Apr 5, 2026 that may be closed by this pull request

rustpython-unicode #7560

Open

Copilot AI and others added 3 commits April 5, 2026 07:39

Extract shared unicode crate

67485b5

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/e4115dbc-b1a5-4a77-90a7-38cf9ac1cdf2 Co-authored-by: youknowone <[email protected]>

Tidy shared unicode review feedback

e968d83

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/e4115dbc-b1a5-4a77-90a7-38cf9ac1cdf2 Co-authored-by: youknowone <[email protected]>

Polish unicode regex helpers

5cf1bd6

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/e4115dbc-b1a5-4a77-90a7-38cf9ac1cdf2 Co-authored-by: youknowone <[email protected]>

Copilot AI changed the title ~~[WIP] Add rustpython-unicode shared crate for CPython compatibility~~ Extract shared rustpython-unicode crate and route core Unicode semantics through it Apr 5, 2026

Copilot finished work on behalf of youknowone April 5, 2026 07:50

Copilot AI requested a review from youknowone April 5, 2026 07:50

youknowone requested changes Apr 5, 2026

View reviewed changes

Copilot started work on behalf of youknowone April 5, 2026 08:29 View session

Copilot AI and others added 2 commits April 5, 2026 08:53

Finish unicode crate follow-up refactors

0a340de

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/1d30ae08-d8f0-431c-9299-8aea5c21f7d4 Co-authored-by: youknowone <[email protected]>

Polish unicode follow-up review fixes

2934897

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/1d30ae08-d8f0-431c-9299-8aea5c21f7d4 Co-authored-by: youknowone <[email protected]>

Copilot AI requested a review from youknowone April 5, 2026 08:55

Copilot finished work on behalf of youknowone April 5, 2026 08:55

youknowone requested changes Apr 6, 2026

View reviewed changes

Copilot started work on behalf of youknowone April 6, 2026 01:49 View session

Simplify unicode regex call sites

4efa5da

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/b894057a-9bed-4f35-8400-a5731c63602d Co-authored-by: youknowone <[email protected]>

Copilot finished work on behalf of youknowone April 6, 2026 02:09

Copilot AI requested a review from youknowone April 6, 2026 02:09

Conversation

Copilot AI commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

youknowone Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

youknowone Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

youknowone Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

youknowone Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

youknowone Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Apr 5, 2026 •

edited

Loading

Copilot AI commented Apr 6, 2026 •

edited

Loading