Skip to content

Commit 14082cf

Browse files
authored
Merge pull request hed-standard#1284 from VisLab/fix_extras
Updated documentation on models/ and schemas/
2 parents d0725e2 + f970486 commit 14082cf

29 files changed

Lines changed: 644 additions & 38 deletions

CHANGELOG.md

Lines changed: 133 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,141 @@
1-
# Unreleased
1+
# Release 1.0.0 March 26, 2026
22

3-
## Enhancements
3+
This is a major release with breaking changes. It removes several subsystems that are no longer part of the core `hedtools` package, completes the schema library-extras support across all schema formats, and cleans up the public API.
44

5-
### Prerelease schemas are now always included in version lookups
5+
## Breaking changes
6+
7+
### Removed remodeling tools
8+
9+
The entire `hed/tools/remodeling/` subsystem (dispatcher, operations, backup manager, and CLI entry points `run_remodel` / `run_remodel_backup` / `run_remodel_restore`) has been removed (issue #1180). Remodeling functionality is now maintained as a separate repository: [table-remodeler](https://github.com/hed-standard/table-remodeler):
10+
11+
### Removed visualization code
12+
13+
`hed/tools/visualization/` (`TagWordCloud`, `word_cloud_util`) and the `wordcloud` dependency have been removed. Visualization is maintained in the separate `hedvis` package in repository [hed-vis](https://github.com/hed-standard/hed-vis)
14+
15+
### Removed ontology utilities
16+
17+
`ontology_util.py` and the associated `create_ontology.py` script have been removed (issue #1177) and are now in the [hed-ontology](https://github.com/hed-standard/hed-ontology) HED ID management is now handled by the new `hed/schema/schema_io/hed_id_util.py` module.
18+
19+
### Removed `sequence_map.py`
20+
21+
`hed/tools/analysis/sequence_map.py` has been moved to the `hedvis` package in the [hed-vis](https://github.com/hed-standard/hed-vis) and is no longer part of `hedtools`.
22+
23+
### Removed `HedLogger`
24+
25+
The `HedLogger` logging class has been removed from `hed/errors/`. Use Python's standard `logging` module instead.
26+
27+
### Removed `check_prerelease` parameter from schema loading API
28+
29+
The `check_prerelease` flag has been removed from all public schema-loading functions (`load_schema_version()`, `load_schema()`, `from_string()`, `from_dataframes()`) and all `SchemaLoader` subclasses. Prerelease schemas are now found automatically whenever they are present in the local cache — no flag is needed.
30+
31+
### `check_warnings` renamed to `check_for_warnings`
32+
33+
`ErrorReporter.check_warnings()` has been renamed to `check_for_warnings()` for API consistency.
34+
35+
### Removed `versionner` dependency
36+
37+
Version management has migrated fully to `setuptools-scm`. The `versionner` tool and its configuration have been removed (issue #1181). `hed/_version.py` continues to be auto-generated — do not edit it manually.
38+
39+
## Schema enhancements
40+
41+
### Library schema extras (`inLibrary`) fully supported across all formats
42+
43+
Extra (non-standard) sections in library schemas, marked with `inLibrary`, now round-trip correctly through all three schema file formats:
44+
45+
- **XML**: `inLibrary` attribute written and read for all extra node types.
46+
- **JSON/DataFrame**: `in_library` column propagated through all extra section tables.
47+
- **MediaWiki**: `inLibrary` tags serialized and parsed in extra sections.
48+
49+
Comprehensive roundtrip test suites have been added for each format.
50+
51+
### Consistent `inLibrary` / `in_library` naming
52+
53+
A naming inconsistency between the XML attribute name (`inLibrary`) and the internal DataFrame column name (`in_library`) has been resolved. All internal representations now use `in_library`; serialization to XML uses `inLibrary` as required by the schema specification.
54+
55+
### Prerelease schemas found automatically
56+
57+
When a requested version is present only in the prerelease cache, it is now loaded without any additional flags. The previous `check_prerelease=True` requirement has been eliminated throughout the stack.
58+
59+
### Prerelease library schemas can use their prerelease standard partner
60+
61+
A prerelease library schema that has `withStandard` set may now load against its matching prerelease standard partner instead of requiring a released standard schema.
62+
63+
### Auto-download missing schemas from GitHub
64+
65+
`get_hed_version_path()` now automatically fetches a schema from GitHub when the requested version is absent from the default local cache directory.
66+
67+
### Default schema version resolved dynamically
68+
69+
The default schema version is derived at runtime from the highest released version in the cache rather than being hardcoded in the source. New schema releases no longer require a code change.
70+
71+
### Consistent line-ending handling
72+
73+
CRLF/LF differences in schema files are now normalized on read and write, preventing spurious diffs and test failures on Windows.
74+
75+
### Fixed empty schema DataFrame sections
76+
77+
Writing a schema to DataFrame format no longer fails when a section (e.g., unit modifiers) is empty.
78+
79+
### Fixed loading of early TSV-format schemas
80+
81+
Schemas stored in older TSV formats that predate the current DataFrame layout now load correctly.
82+
83+
### Removed unverified `omn:` columns from TSV format
84+
85+
Columns prefixed with `omn:` that were written but never populated have been removed from the TSV/DataFrame schema format.
86+
87+
### Schema error vs warning strategy revised
88+
89+
The `ErrorReporter` now distinguishes schema-level errors from warnings more precisely. `check_for_warnings()` (formerly `check_warnings()`) returns only genuine warning-level issues; schema structural errors are always reported as errors regardless of warning-filter settings.
90+
91+
### Schema compliance tests check only latest versions
92+
93+
Spec-test compliance checking has been refactored to validate only the latest released version of each schema, reducing test time and avoiding failures from already-superseded versions.
94+
95+
## Bug fixes
96+
97+
- Fixed a race condition in schema-version auto-detection when multiple processes access the cache simultaneously.
98+
- Fixed duplicate HED ID detection — `check_duplicate_hed_ids()` now correctly tracks already-seen IDs across all schema sections.
99+
- Verified all built-in error messages are unique (no two distinct error codes share the same message text).
100+
- Normalized error message punctuation: sentence-ending periods are followed by one space (was inconsistently one or two).
101+
- Fixed extension capitalization check in schema attribute validation.
102+
- Removed stray debug `print` statements from the schema loading path.
103+
104+
## CI/CD and developer tooling
105+
106+
### Migrated GitHub Actions to `uv`
107+
108+
All CI workflows now use the `astral-sh/setup-uv` action and `uv` for environment creation and package installation. `pip`-based setup has been removed. The `black` formatter has been eliminated; `ruff format` handles all code formatting.
109+
110+
### Updated GitHub Actions versions
111+
112+
- `actions/checkout`: v4 → **v6**
113+
- `astral-sh/setup-uv`: v5 → **v7**
114+
- Added `cache-dependency-glob` to all `setup-uv` steps for correct cache invalidation.
115+
116+
### Replaced `codespell` with `typos`
117+
118+
Spell checking in CI now uses [`typos`](https://github.com/crate-ci/typos). Configuration has been consolidated into `pyproject.toml`; the separate `.codespellrc` file has been removed. The `tests/` directory and generated/binary file types are excluded from spell checking.
119+
120+
### Added Claude Code Review and PR Assistant workflows
121+
122+
Two new GitHub Actions workflows provide automated AI code review on pull requests:
123+
124+
- `claude_code_review.yaml` — posts inline review comments via the `gh` CLI.
125+
- `claude_pr_assistant.yaml` — responds to `/review` commands in PR comments.
126+
127+
Bot-authored PRs are excluded from automated review to prevent feedback loops.
128+
129+
### Added project context files for AI assistants
6130

7-
Previously, loading a schema whose version existed only in the prerelease cache required passing `check_prerelease=True` through every layer of the loading API. This flag has been **removed** from all public functions (`load_schema_version`, `load_schema`, `from_string`, `from_dataframes`), all schema loader classes, and all internal helpers. Prerelease schemas are now found automatically whenever they are present in the cache.
131+
- `.github/copilot-instructions.md` — VS Code Copilot context (now tracked in the repository).
132+
- `CLAUDE.md` and `.rules/` — Claude Code project context and coding rules.
8133

9-
**Changes:**
134+
### Consolidated tool configuration
10135

11-
- Removed the `check_prerelease` parameter from `load_schema_version()`, `load_schema()`, `from_string()`, and `from_dataframes()` in `hed_schema_io.py`.
12-
- Removed the parameter from `SchemaLoader` (base class) and all subclasses (`SchemaLoaderXML`, `SchemaLoaderWiki`, `SchemaLoaderJSON`, `SchemaLoaderDF`).
13-
- `get_hed_version_path()` in `hed_cache.py` now always searches both regular and prerelease directories (regular first).
14-
- `get_hed_versions()` in `hed_cache.py` retains the default `check_prerelease=False` (no public API change). Internal callers that need prerelease inclusion (`get_hed_version_path`, error messages) now pass `check_prerelease=True` explicitly.
15-
- Default schema version is now resolved dynamically from the cache (highest released version) instead of being hardcoded, so new schema releases no longer require a code change.
16-
- `get_hed_version_path()` now automatically downloads schemas from GitHub when a requested version is not found in the local cache (default cache directory only).
17-
- `_load_schema_version_sub()` now raises `BAD_PARAMETERS` (was `FILE_NOT_FOUND`) when no version is specified and the cache is empty, since the problem is a missing argument rather than a missing file.
18-
- `check_schema_loading.py` simplified — removed `_is_prerelease_partner()` helper.
19-
- `run_loading_check()` now raises `ValueError` immediately for mutually exclusive flag combinations (`prerelease_only` + `exclude_prereleases`, or `library_filter` + `standard_only`), consistent with the existing CLI-level validation.
136+
- Ruff options updated; submodule directories excluded from linting.
137+
- `lychee.toml` updated to use plain URL format (replaced `{meta}` tags removed in newer lychee releases).
138+
- `pyproject.toml` is now the single source of truth for spell-check word lists.
20139

21140
# Release 0.9.0 January 22, 2026
22141

README.md

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ pip install -e ".[dev,docs,test,examples]"
8080
### Basic usage
8181

8282
```python
83-
from hed import HedString, load_schema_version
83+
from hed import HedString, get_printable_issue_string, load_schema_version
8484

8585

8686
# Load the latest HED schema
@@ -105,17 +105,19 @@ HEDTools provides a unified command-line interface with git-like subcommands:
105105
hedpy --help
106106

107107
# Validate a BIDS dataset
108-
hedpy validate-bids /path/to/bids/dataset
108+
hedpy validate bids-dataset /path/to/bids/dataset
109109

110-
# Extract sidecar template from BIDS dataset
111-
hedpy extract-sidecar /path/to/dataset --suffix events
110+
# Validate a HED string
111+
hedpy validate string "Sensory-event, (Red, Square)" -sv 8.4.0
112+
113+
# Extract sidecar template from a BIDS dataset
114+
hedpy extract bids-sidecar /path/to/dataset
112115

113116
# Validate HED schemas
114117
hedpy schema validate /path/to/schema.xml
115118

116119
# Convert schema between formats (XML, MEDIAWIKI, TSV, JSON)
117120
hedpy schema convert /path/to/schema.xml
118-
119121
```
120122

121123
**Legacy commands** (deprecated, use `hedpy` instead):
@@ -219,6 +221,7 @@ The HED ecosystem consists of several interconnected repositories:
219221
| [hed-specification](https://github.com/hed-standard/hed-specification) | Official HED specification documents |
220222
| [hed-schemas](https://github.com/hed-standard/hed-schemas) | Official HED schema repository |
221223
| [table-remodeler](https://github.com/hed-standard/table-remodeler) | Table transformation and remodeling tools |
224+
| [hed-vis](https://github.com/hed-standard/hed-vis) | HED visualization tools (word clouds, etc.) |
222225
| [ndx-hed](https://github.com/hed-standard/ndx-hed) | HED support for NWB |
223226
| [hed-javascript](https://github.com/hed-standard/hed-javascript) | JavaScript HED validation tools |
224227

@@ -278,7 +281,7 @@ If you use HEDTools in your research, please cite:
278281
@software{hedtools,
279282
author = {Ian Callanan, Robbins, Kay and others},
280283
title = {HEDTools: Python tools for HED},
281-
year = {2024},
284+
year = {2026},
282285
publisher = {GitHub},
283286
url = {https://github.com/hed-standard/hed-python},
284287
doi = {10.5281/zenodo.8056010}

docs/api/models.rst

Lines changed: 141 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Models
2-
=============================
2+
======
33

44
Core data models for working with HED data structures.
55

@@ -40,6 +40,43 @@ DefinitionDict
4040
:undoc-members:
4141
:show-inheritance:
4242

43+
DefinitionEntry
44+
~~~~~~~~~~~~~~~
45+
46+
.. autoclass:: hed.models.definition_entry.DefinitionEntry
47+
:members:
48+
:undoc-members:
49+
:show-inheritance:
50+
51+
DefExpandGatherer
52+
~~~~~~~~~~~~~~~~~
53+
54+
.. autoclass:: hed.models.def_expand_gather.DefExpandGatherer
55+
:members:
56+
:undoc-members:
57+
:show-inheritance:
58+
59+
Constants
60+
---------
61+
62+
Enumerations and named constants used across the models layer.
63+
64+
DefTagNames
65+
~~~~~~~~~~~
66+
67+
.. autoclass:: hed.models.model_constants.DefTagNames
68+
:members:
69+
:undoc-members:
70+
:show-inheritance:
71+
72+
TopTagReturnType
73+
~~~~~~~~~~~~~~~~
74+
75+
.. autoclass:: hed.models.model_constants.TopTagReturnType
76+
:members:
77+
:undoc-members:
78+
:show-inheritance:
79+
4380
Input models
4481
------------
4582

@@ -76,3 +113,106 @@ SpreadsheetInput
76113
:members:
77114
:undoc-members:
78115
:show-inheritance:
116+
117+
TimeseriesInput
118+
~~~~~~~~~~~~~~~
119+
120+
.. autoclass:: hed.models.timeseries_input.TimeseriesInput
121+
:members:
122+
:undoc-members:
123+
:show-inheritance:
124+
125+
ColumnMapper
126+
~~~~~~~~~~~~
127+
128+
.. autoclass:: hed.models.column_mapper.ColumnMapper
129+
:members:
130+
:undoc-members:
131+
:show-inheritance:
132+
133+
ColumnMetadata
134+
~~~~~~~~~~~~~~
135+
136+
.. autoclass:: hed.models.column_metadata.ColumnMetadata
137+
:members:
138+
:undoc-members:
139+
:show-inheritance:
140+
141+
ColumnType
142+
~~~~~~~~~~
143+
144+
.. autoclass:: hed.models.column_metadata.ColumnType
145+
:members:
146+
:undoc-members:
147+
:show-inheritance:
148+
149+
Query models
150+
------------
151+
152+
Classes and functions for searching and querying HED annotations.
153+
154+
QueryHandler
155+
~~~~~~~~~~~~
156+
157+
.. autoclass:: hed.models.query_handler.QueryHandler
158+
:members:
159+
:undoc-members:
160+
:show-inheritance:
161+
162+
SearchResult
163+
~~~~~~~~~~~~
164+
165+
.. autoclass:: hed.models.query_util.SearchResult
166+
:members:
167+
:undoc-members:
168+
:show-inheritance:
169+
170+
get_query_handlers
171+
~~~~~~~~~~~~~~~~~~
172+
173+
.. autofunction:: hed.models.query_service.get_query_handlers
174+
175+
search_hed_objs
176+
~~~~~~~~~~~~~~~
177+
178+
.. autofunction:: hed.models.query_service.search_hed_objs
179+
180+
DataFrame utilities
181+
-------------------
182+
183+
Functions for transforming HED strings within pandas DataFrames.
184+
185+
convert_to_form
186+
~~~~~~~~~~~~~~~
187+
188+
.. autofunction:: hed.models.df_util.convert_to_form
189+
190+
expand_defs
191+
~~~~~~~~~~~
192+
193+
.. autofunction:: hed.models.df_util.expand_defs
194+
195+
shrink_defs
196+
~~~~~~~~~~~
197+
198+
.. autofunction:: hed.models.df_util.shrink_defs
199+
200+
process_def_expands
201+
~~~~~~~~~~~~~~~~~~~
202+
203+
.. autofunction:: hed.models.df_util.process_def_expands
204+
205+
sort_dataframe_by_onsets
206+
~~~~~~~~~~~~~~~~~~~~~~~~
207+
208+
.. autofunction:: hed.models.df_util.sort_dataframe_by_onsets
209+
210+
filter_series_by_onset
211+
~~~~~~~~~~~~~~~~~~~~~~
212+
213+
.. autofunction:: hed.models.df_util.filter_series_by_onset
214+
215+
split_delay_tags
216+
~~~~~~~~~~~~~~~~
217+
218+
.. autofunction:: hed.models.df_util.split_delay_tags

0 commit comments

Comments
 (0)