Skip to content

Subarray dtypes#1

Open
vitusbenson wants to merge 81 commits intosehoffmann:subarray_dtypesfrom
vitusbenson:subarray_dtypes
Open

Subarray dtypes#1
vitusbenson wants to merge 81 commits intosehoffmann:subarray_dtypesfrom
vitusbenson:subarray_dtypes

Conversation

@vitusbenson
Copy link
Copy Markdown

  • Rebased to latest main
  • Supports only the subarrays inside structure dtypes

--> Hope this is good for zarr developers to merge then.

hmaarrfk and others added 30 commits December 28, 2025 21:44
* docs: Correct grammar issues in a/an usage

* Update src/zarr/storage/_fsspec.py

---------

Co-authored-by: Tom Augspurger <[email protected]>
…pers#3630)

Bumps the actions group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `7.1.4` | `7.1.6` |
| [actions/cache](https://github.com/actions/cache) | `4` | `5` |
| [scientific-python/upload-nightly-action](https://github.com/scientific-python/upload-nightly-action) | `0.6.2` | `0.6.3` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `5` | `6` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `6` | `7` |



Updates `astral-sh/setup-uv` from 7.1.4 to 7.1.6
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@1e862df...681c641)

Updates `actions/cache` from 4 to 5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v4...v5)

Updates `scientific-python/upload-nightly-action` from 0.6.2 to 0.6.3
- [Release notes](https://github.com/scientific-python/upload-nightly-action/releases)
- [Commits](scientific-python/upload-nightly-action@b36e8c0...5748273)

Updates `actions/upload-artifact` from 5 to 6
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v5...v6)

Updates `actions/download-artifact` from 6 to 7
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v6...v7)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 7.1.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: scientific-python/upload-nightly-action
  dependency-version: 0.6.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: update pre-commit hooks

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.14.3 → v0.14.10](astral-sh/ruff-pre-commit@v0.14.3...v0.14.10)
- [github.com/pre-commit/mirrors-mypy: v1.18.2 → v1.19.1](pre-commit/mirrors-mypy@v1.18.2...v1.19.1)
- [github.com/scientific-python/cookie: 2025.10.20 → 2025.11.21](scientific-python/cookie@2025.10.20...2025.11.21)
- [github.com/numpy/numpydoc: v1.9.0 → v1.10.0](numpy/numpydoc@v1.9.0...v1.10.0)

* update JSON union to include bool

* apply fixes to resolve pre-commit violations

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>
…r-developers#3646)

Bumps the actions group with 1 update: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv).


Updates `astral-sh/setup-uv` from 7.1.6 to 7.2.0
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@681c641...61cb8a9)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 7.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>
* set copy=False in reshape operation

* add compat reshape with conditional

* fix docstring

* fix mypy

* remove tuple unpacking which makes it more readable
* Bump mimimum supported version of numpy

* Update changelog entry filename

---------

Co-authored-by: Davis Bennett <[email protected]>
)

* add benchmarks

* remove failing zipstore

* don't do benchmarking in default pytest runs

* changelog

* codspeed workflow

* lint

* remove pedantic mode

* only run benchmarks in one environment

* use better string id for test params, make test data 1MB, and simplify params

* move layout to an external file

* get workloads to resemble recent sharding perf tests

* test ids

* tweak tests

* tweak tests

* fix typo

* add slice indexing benchmarks

* remove readme

* add docs documentation

* simplify pytest benchmark options

* use --codspeed flag in benchmark ci

* measure walltime in ci

* Update .github/workflows/codspeed.yml

Co-authored-by: Max Jones <[email protected]>

* Apply suggestion from @maxrjones

Co-authored-by: Max Jones <[email protected]>

* add --ignore option to main test and gpu test invocations

* add comment

* ignore codspeed warnings

* update workflow

---------

Co-authored-by: Max Jones <[email protected]>
* Clean up test extra dependencies

* Add changelog

* Put back uv in test deps

Co-authored-by: Davis Bennett <[email protected]>

---------

Co-authored-by: Davis Bennett <[email protected]>
* docs: an python object -> a python object

* docs: a XML -> an XML

* docs: a S3 -> an S3
…rs#3659)

Bumps the actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 5 to 6
- [Release notes](https://github.com/actions/checkout/releases)
- [Commits](actions/checkout@v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>
* bring back pre-commit ci

* restore needs release notes

* restore readme

* add prek workflow

* changelog

* changelog content
* Add benchmarks for sharded + local store indexing

Parameterize with shards and no shards.
Parameterize with local + memory store to have
an example of a store which has some modest latency.

* use latencystore with 10ms get latency instead of localstore

* use .01s of latency

* set get latency to .0001s

---------

Co-authored-by: Davis Bennett <[email protected]>
* add store routines for getting bytes and json

* check for FileNotFoundError when a key is missing

* remove storepath methods

* changelog

* rename methods

* continue renaming / test refactoring

* refactor new test functions

* make new methods private

* remove changelog entry for private API
…pers#3669)

Bumps the actions group with 1 update: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv).


Updates `astral-sh/setup-uv` from 5 to 7
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@v5...v7)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>
…pers#3673)

* only run codspeed benchmarks on PRs labelled 'benchmark'

* changelog

* rename changelog
* - Add .config property to Array
- Make .config attribute of AsyncArray public
- Add .with_config method to Array and AsyncArray
- Use .config attribute in docs

* bring back _config as an alias for config; update docstring

* release note

* add docs for with_config
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.14.10 → v0.14.14](astral-sh/ruff-pre-commit@v0.14.10...v0.14.14)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…pers#3695)

* fix: raise error when encountering nullable string dtype

* add change entry

* fix typing

* move exception on na_object-bearing-string-dtype to inside from_native_dtype

---------

Co-authored-by: Davis Vann Bennett <[email protected]>
* lift array methods to separate functions

* don't use deprecated attribute

* don't use deprecated attribute
* Support with_read_only in LoggingStore and LatencyStore.

Fixes zarr-developers#3699

* Support CacheStore.with_read_only

* Add entry to changes/

* use a dataclass for mutable cache state

---------

Co-authored-by: Davis Bennett <[email protected]>
abishop1990 and others added 29 commits March 11, 2026 08:11
…) (zarr-developers#3752)

Add a clear, simple example in the Compression section of the user guide
showing how to create an array without compression by setting compressors=None.

This addresses user confusion about the default compression behavior and
provides an easy-to-copy example for the common use case of disabling
compression entirely.

Closes zarr-developers#3464

Co-authored-by: Cipher <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>
…lopers#3586) (zarr-developers#3762)

* fix: BaseFloat._check_scalar rejects invalid string values (zarr-developers#3586)

BaseFloat._check_scalar returned True for all strings because the
FloatLike type union includes str. This allowed invalid strings like
'not valid' to pass the check and then raise a confusing ValueError
in _cast_scalar_unchecked instead of the expected TypeError.

Fix _check_scalar to validate string inputs by attempting conversion:
- Valid float strings (e.g. 'NaN', 'inf', '-inf', '1.5') return True
- Invalid strings (e.g. 'not valid') return False, causing cast_scalar
  to raise TypeError as expected

Add test cases for invalid string inputs to invalid_scalar_params.

Fixes zarr-developers#3586

* style: fix ruff formatting and TRY300 in _check_scalar

---------

Co-authored-by: Cipher <[email protected]>
…r-developers#3691) (zarr-developers#3763)

* fix: apply drop_axes squeeze in partial decode path for sharding

When reading sharded arrays with mixed integer/list indexing (e.g.
arr[0:10, 0, [0, 1]]), the outer OrthogonalIndexer produces chunk
selections that have been ix_()-transformed for orthogonal advanced
indexing. Integer indices become single-element ranges (size-1 dims)
via ix_() to enable NumPy orthogonal indexing.

In CodecPipeline.read_batch(), the non-partial path correctly applies
drop_axes.squeeze() to remove those size-1 integer dimensions before
writing to the output buffer. However, the partial decode path (used
by ShardingCodec) was missing this squeeze step.

Fixes zarr-developers#3691

Also: Fix line length violation in test error message to comply with
100 character linting limit.

* fix(mypy): add type ignore comments for dynamic array indexing in sharding test

The test uses complex indexing patterns (mixed integer/list indices) that
mypy's zarr.Array stubs don't recognize as valid. Add specific type ignore
comments for [index] and [union-attr] errors to suppress false positives.

* fix(mypy): correct type-ignore codes for union attribute access in sharding test

- Line 542: Fix assert accessing .shape by changing from [index] to [union-attr]
- Line 544: Add missing type-ignore[union-attr] for f-string .shape access
- Lines 554-555: Remove unused type-ignore[index] comments on assignments

The mypy errors were caused by indexing operations returning union types that
include scalar types (int, float, etc.), which don't have a .shape attribute.
The proper fix uses type-ignore[union-attr] for attribute access, not [index].

---------

Co-authored-by: Cipher <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>
…pers#3779)

* fix: validate that dask-style chunks have regular shapes

* Apply suggestion from @dcherian

Co-authored-by: Deepak Cherian <[email protected]>

* Fix shortcircuit option

---------

Co-authored-by: Deepak Cherian <[email protected]>
* docs: add glossary

* Add glossary

* Update docs/user-guide/glossary.md

Co-authored-by: Davis Bennett <[email protected]>

* Add caveat

---------

Co-authored-by: Davis Bennett <[email protected]>
* add sync methods to codecs

* add CodecChain dataclass and sync codec tests

Introduces CodecChain, a frozen dataclass that chains array-array,
array-bytes, and bytes-bytes codecs with synchronous encode/decode
methods. Pure compute only -- no IO, no threading, no batching.

Also adds sync roundtrip tests for individual codecs (blosc, gzip,
zstd, crc32c, bytes, transpose, vlen) and CodecChain integration tests.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor codecchain

* separate codecs and specs

* add synchronous methods to stores

* fix merge error

* guard storepath methods that rely on underlying sync impl

* scrub out set-range logic

* remove is-zstd-fixed-size test

* remove codecchain

* revert delete comment

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Deepak Cherian <[email protected]>
* remove rich dependency for tree

* lint + pr number

* keyword only

* Apply suggestion from @maxrjones

Co-authored-by: Max Jones <[email protected]>

---------

Co-authored-by: Max Jones <[email protected]>
* doc: add ai use policy to contributing guide

* add changelog entry

* Update docs/contributing.md

Co-authored-by: Davis Bennett <[email protected]>

---------

Co-authored-by: Davis Bennett <[email protected]>
* have towncrier use h2 header for each release note

* remove h1 headers
* Use OIDC for publishing

* Update requires

* Attest
* have towncrier use h2 header for each release note

* remove h1 headers

* release notes

* rebuild release notes docs with zarr-developers#3769

* add more release notes

* style

* Update release-notes.md

Co-authored-by: Max Jones <[email protected]>

---------

Co-authored-by: Max Jones <[email protected]>
…opers#3800)

The reason for this change is the fact that the `DimensionNames` type is does not model the
actual type of the `dimension_names` attribute on the Array V3 Metadata class, but rather a
wider input type that is ultimately narrowed to that actual type. For this reason, it should
use the same `XLike` name convention as the other wide input types that get narrowed to a more
restricted type.
…rs#3797)

* fix: handle special uint32 arrays on Windows correctly

* add changelog entry

---------

Co-authored-by: Davis Bennett <[email protected]>
This change to `release.yml` should allow us to use `workflow-dispatch`, i.e., manual triggering,
to invoke the release workflow, with an explicit tag as input. This supports publishing a tagged
release outside of the commit that creates that tag. This is useful for situations where we issued a
release, but the github actions workflow was broken in some way that prevented us from uploading the
release to pypi.
…ase workflow. (zarr-developers#3822)

* Removes testpypi as a target for the workflow-dispatch-triggered release workflow.

Zarr Python is not configured for OIDC on TestPyPI, and there was a bug in the previous
workflow that prevented command line declaration of the PyPI target (real or test) from
being evaluated correctly. To simplify this, TestPyPI is removed, and only RealPyPI can
be the target for publishing.

* update

---------

Co-authored-by: Max Jones <[email protected]>
* ensure no get on full shard writes

* clean up test

---------

Co-authored-by: Davis Bennett <[email protected]>
…3822 (zarr-developers#3825)

Reverts our `releases.yml` workflow to the state as of 93dd0e4, i.e. no option to publish a tag declared in workflow dispatch.
Remove a leading "-" character that violated github-actions-flavored yaml syntax.
This should prevent email spam from failing workflows, and also prevent the workflow itself
from failing.
…ers#3828)

* Add a new type `GetResult` to `src/zarr/abc/codec.py`.

This type is a typeddict that stores information about a completed
attempt to fetch a stored resource, e.g. a chunk. Currently, `GetResult`
only stores whether the resource was present or missing. We can add more capacity
to this type in the future.

* add tests for codec pipeline

* Remove whitespace

* changelog
* Fix the docstring for `CodecPipeline.write`

The annotation for the `batch_info` parameter in `CodecPipeline.write` omits the final bool from the tuple. This
PR corrects the type annotation in the docstring.

* docs: changelog

* docs: rename changelog file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.