feat: Add complex type support (Map, JSON, Struct) with schema validation by ntkathole · Pull Request #5974 · feast-dev/feast

ntkathole · 2026-02-16T12:23:43Z

What this PR does / why we need it:

This PR adds comprehensive support for complex data types in Feast and implements schema validation across all compute engines:

Commit 1 (970650b): Fix Map/Dict support and implement schema validation

Fixed Map/Dict type support across offline stores, online stores, and type mappings
Added enable_validation parameter to FeatureView, BatchFeatureView, and StreamFeatureView
Implemented schema validation nodes in Local, Ray, and Spark compute engines (missing columns, type mismatch warnings)
Added Spark and Milvus map type mappings
Updated documentation for Map type backend mappings

Commit 2 (ac28d4d): Add JSON and Struct complex data types

Added JSON, JSON_LIST, STRUCT, STRUCT_LIST to the protobuf ValueType enum and Value message
Introduced Json primitive type and Struct class (schema-aware structured type with named, typed fields)
Implemented full type mapping pipeline: Feast ↔ Proto ↔ PyArrow ↔ backend-native types (PostgreSQL, BigQuery, Snowflake, Redshift, Spark, MSSQL, Athena, DynamoDB, Milvus)
Added JSON well-formedness validation at both proto conversion level (always active) and validation node level (when enable_validation=True)
Implemented Spark-native type validation using from_feast_to_spark_type() and _spark_types_compatible() — eliminates the PyArrow-to-Spark type mismatch gap
Persisted Struct field schemas through the registry via Field tags (feast:struct_schema)
Added Go SDK type conversion support for JSON and Struct types
Added automatic serialization/deserialization for JSON type (Python dict/list ↔ JSON string)
Single _validate_schema() method per engine (missing columns + type checks + JSON content validation)

Commit 3 (a131b73): Modified default template with different types

Added Json, Map and Struct in driver template

Signed-off-by: ntkathole <[email protected]>

franciscojavierarceo

lgtm

huge pr tho

…tion (feast-dev#5974) * feat: Fix Map/Dict support and implement schema validation Signed-off-by: ntkathole <[email protected]> * feat: Modified default example with different data types Signed-off-by: ntkathole <[email protected]> --------- Signed-off-by: ntkathole <[email protected]> Co-authored-by: Francisco Javier Arceo <[email protected]> Signed-off-by: Aniket Paluskar <[email protected]>

# [0.61.0](v0.60.0...v0.61.0) (2026-03-10) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([#6079](#6079)) ([30a3a43](30a3a43)) * Added MLflow metric charts across feature selection ([#6080](#6080)) ([a403361](a403361)) * Check duplicate names for feature view across types ([#5999](#5999)) ([95b9af8](95b9af8)) * Fix integration tests ([#6046](#6046)) ([02d5548](02d5548)) * Fix non-specific label selector on metrics service ([a1a160d](a1a160d)) * Fixed IntegrityError on SqlRegistry ([#6047](#6047)) ([325e148](325e148)) * Fixed pre-commit check ([114b7db](114b7db)) * Fixed uv cache permission error for docker build on mac ([ad807be](ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([#5991](#5991)) ([abfd18a](abfd18a)) * Integration test failures ([#6040](#6040)) ([9165870](9165870)) * Ray offline store tests are duplicated across 3 workflows ([54f705a](54f705a)) * Reenable tests ([#6036](#6036)) ([82ee7f8](82ee7f8)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](35a81e7)) ### Features * Add complex type support (Map, JSON, Struct) with schema validation ([#5974](#5974)) ([1200dbf](1200dbf)) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](2c6be18)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](4d08ddc)), closes [#5835](#5835) * Add OnlineStore for MongoDB ([#6025](#6025)) ([bf4e3fa](bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](547b516)) * Adding optional name to Aggregation (feast-dev[#5994](#5994)) ([#6083](#6083)) ([56469f7](56469f7)) * Feature Server High-Availability on Kubernetes ([#6028](#6028)) ([9c07b4c](9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([#5925](#5925)) ([2b4ec9a](2b4ec9a)) * Horizontal scaling support to the Feast operator ([#6000](#6000)) ([3ec13e6](3ec13e6)) * Making feature view source optional (feast-dev[#6074](#6074)) ([#6075](#6075)) ([76917b7](76917b7)) * Support arm docker build ([#6061](#6061)) ([1e1f5d9](1e1f5d9)) * Use orjson for faster JSON serialization in feature server ([6f5203a](6f5203a)) ### Performance Improvements * Optimize protobuf parsing in Redis online store ([#6023](#6023)) ([59dfdb8](59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([#6024](#6024)) ([9699944](9699944)) * Remove redundant entity key serialization in online_read ([d87283f](d87283f))

…tion (feast-dev#5974) * feat: Fix Map/Dict support and implement schema validation Signed-off-by: ntkathole <[email protected]> * feat: Modified default example with different data types Signed-off-by: ntkathole <[email protected]> --------- Signed-off-by: ntkathole <[email protected]> Co-authored-by: Francisco Javier Arceo <[email protected]>

# [0.61.0](feast-dev/feast@v0.60.0...v0.61.0) (2026-03-10) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](feast-dev@2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](feast-dev@ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([feast-dev#6079](feast-dev#6079)) ([30a3a43](feast-dev@30a3a43)) * Added MLflow metric charts across feature selection ([feast-dev#6080](feast-dev#6080)) ([a403361](feast-dev@a403361)) * Check duplicate names for feature view across types ([feast-dev#5999](feast-dev#5999)) ([95b9af8](feast-dev@95b9af8)) * Fix integration tests ([feast-dev#6046](feast-dev#6046)) ([02d5548](feast-dev@02d5548)) * Fix non-specific label selector on metrics service ([a1a160d](feast-dev@a1a160d)) * Fixed IntegrityError on SqlRegistry ([feast-dev#6047](feast-dev#6047)) ([325e148](feast-dev@325e148)) * Fixed pre-commit check ([114b7db](feast-dev@114b7db)) * Fixed uv cache permission error for docker build on mac ([ad807be](feast-dev@ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([feast-dev#5991](feast-dev#5991)) ([abfd18a](feast-dev@abfd18a)) * Integration test failures ([feast-dev#6040](feast-dev#6040)) ([9165870](feast-dev@9165870)) * Ray offline store tests are duplicated across 3 workflows ([54f705a](feast-dev@54f705a)) * Reenable tests ([feast-dev#6036](feast-dev#6036)) ([82ee7f8](feast-dev@82ee7f8)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](feast-dev@35a81e7)) ### Features * Add complex type support (Map, JSON, Struct) with schema validation ([feast-dev#5974](feast-dev#5974)) ([1200dbf](feast-dev@1200dbf)) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](feast-dev@2c6be18)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](feast-dev@4d08ddc)), closes [feast-dev#5835](feast-dev#5835) * Add OnlineStore for MongoDB ([feast-dev#6025](feast-dev#6025)) ([bf4e3fa](feast-dev@bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](feast-dev@547b516)) * Adding optional name to Aggregation (feast-dev[feast-dev#5994](feast-dev#5994)) ([feast-dev#6083](feast-dev#6083)) ([56469f7](feast-dev@56469f7)) * Feature Server High-Availability on Kubernetes ([feast-dev#6028](feast-dev#6028)) ([9c07b4c](feast-dev@9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([feast-dev#5925](feast-dev#5925)) ([2b4ec9a](feast-dev@2b4ec9a)) * Horizontal scaling support to the Feast operator ([feast-dev#6000](feast-dev#6000)) ([3ec13e6](feast-dev@3ec13e6)) * Making feature view source optional (feast-dev[feast-dev#6074](feast-dev#6074)) ([feast-dev#6075](feast-dev#6075)) ([76917b7](feast-dev@76917b7)) * Support arm docker build ([feast-dev#6061](feast-dev#6061)) ([1e1f5d9](feast-dev@1e1f5d9)) * Use orjson for faster JSON serialization in feature server ([6f5203a](feast-dev@6f5203a)) ### Performance Improvements * Optimize protobuf parsing in Redis online store ([feast-dev#6023](feast-dev#6023)) ([59dfdb8](feast-dev@59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](feast-dev@33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([feast-dev#6024](feast-dev#6024)) ([9699944](feast-dev@9699944)) * Remove redundant entity key serialization in online_read ([d87283f](feast-dev@d87283f))

…tion (feast-dev#5974) * feat: Fix Map/Dict support and implement schema validation Signed-off-by: ntkathole <[email protected]> * feat: Modified default example with different data types Signed-off-by: ntkathole <[email protected]> --------- Signed-off-by: ntkathole <[email protected]> Co-authored-by: Francisco Javier Arceo <[email protected]>

# [0.61.0](feast-dev/feast@v0.60.0...v0.61.0) (2026-03-10) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](feast-dev@2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](feast-dev@ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([feast-dev#6079](feast-dev#6079)) ([30a3a43](feast-dev@30a3a43)) * Added MLflow metric charts across feature selection ([feast-dev#6080](feast-dev#6080)) ([a403361](feast-dev@a403361)) * Check duplicate names for feature view across types ([feast-dev#5999](feast-dev#5999)) ([95b9af8](feast-dev@95b9af8)) * Fix integration tests ([feast-dev#6046](feast-dev#6046)) ([02d5548](feast-dev@02d5548)) * Fix non-specific label selector on metrics service ([a1a160d](feast-dev@a1a160d)) * Fixed IntegrityError on SqlRegistry ([feast-dev#6047](feast-dev#6047)) ([325e148](feast-dev@325e148)) * Fixed pre-commit check ([114b7db](feast-dev@114b7db)) * Fixed uv cache permission error for docker build on mac ([ad807be](feast-dev@ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([feast-dev#5991](feast-dev#5991)) ([abfd18a](feast-dev@abfd18a)) * Integration test failures ([feast-dev#6040](feast-dev#6040)) ([9165870](feast-dev@9165870)) * Ray offline store tests are duplicated across 3 workflows ([54f705a](feast-dev@54f705a)) * Reenable tests ([feast-dev#6036](feast-dev#6036)) ([82ee7f8](feast-dev@82ee7f8)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](feast-dev@35a81e7)) ### Features * Add complex type support (Map, JSON, Struct) with schema validation ([feast-dev#5974](feast-dev#5974)) ([1200dbf](feast-dev@1200dbf)) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](feast-dev@2c6be18)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](feast-dev@4d08ddc)), closes [feast-dev#5835](feast-dev#5835) * Add OnlineStore for MongoDB ([feast-dev#6025](feast-dev#6025)) ([bf4e3fa](feast-dev@bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](feast-dev@547b516)) * Adding optional name to Aggregation (feast-dev[feast-dev#5994](feast-dev#5994)) ([feast-dev#6083](feast-dev#6083)) ([56469f7](feast-dev@56469f7)) * Feature Server High-Availability on Kubernetes ([feast-dev#6028](feast-dev#6028)) ([9c07b4c](feast-dev@9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([feast-dev#5925](feast-dev#5925)) ([2b4ec9a](feast-dev@2b4ec9a)) * Horizontal scaling support to the Feast operator ([feast-dev#6000](feast-dev#6000)) ([3ec13e6](feast-dev@3ec13e6)) * Making feature view source optional (feast-dev[feast-dev#6074](feast-dev#6074)) ([feast-dev#6075](feast-dev#6075)) ([76917b7](feast-dev@76917b7)) * Support arm docker build ([feast-dev#6061](feast-dev#6061)) ([1e1f5d9](feast-dev@1e1f5d9)) * Use orjson for faster JSON serialization in feature server ([6f5203a](feast-dev@6f5203a)) ### Performance Improvements * Optimize protobuf parsing in Redis online store ([feast-dev#6023](feast-dev#6023)) ([59dfdb8](feast-dev@59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](feast-dev@33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([feast-dev#6024](feast-dev#6024)) ([9699944](feast-dev@9699944)) * Remove redundant entity key serialization in online_read ([d87283f](feast-dev@d87283f))

ntkathole self-assigned this Feb 16, 2026

ntkathole added the ok-to-test label Feb 16, 2026

ntkathole force-pushed the fix_complex_type branch from 05522ce to 970650b Compare February 16, 2026 12:59

ntkathole changed the title ~~feat: Fix Map/Dict support and implement schema validation~~ feat: Add complex type support (Map, JSON, Struct) with schema validation Feb 16, 2026

ntkathole marked this pull request as ready for review February 16, 2026 15:49

ntkathole requested a review from a team as a code owner February 16, 2026 15:49