feat: Created DocEmbedder class by patelchaitany · Pull Request #5973 · feast-dev/feast

patelchaitany · 2026-02-16T11:28:33Z

What this PR does / why we need it:

This PR adds a Document Embedder capability to Feast, allowing users to go from raw documents to embeddings stored in the online vector store in a single step. It handles chunking, embedding generation, and writing the results to the online store — providing an end-to-end ingestion pipeline for RAG workflows within Feast.

What changed:

sdk/python/feast/chunker.py

Defines the document chunking layer. Provides:

A BaseChunker abstract class that users can extend with custom chunking strategies
A built-in TextChunker that splits plain text by word count with configurable size, overlap, and thresholds
A chunk_dataframe() convenience method to chunk all documents in a DataFrame

Currently only basic text chunking is implemented. There is room for improvement — future iterations can support more advanced strategies like semantic chunking, sentence-aware splitting, or format-specific chunkers (PDF, HTML, etc.).

sdk/python/feast/embedder.py

Defines the embedding generation layer. Provides:

A BaseEmbedder abstract class with a modality-routing registry for extensibility
A MultiModalEmbedder with built-in support for text (via sentence-transformers) and image (via CLIP) embeddings
Lazy model loading and configurable batch processing

sdk/python/feast/doc_embedder.py

The high-level orchestrator that coordinates chunking, embedding, and storage. Provides:

A DocEmbedder class that runs the full pipeline: chunk -> embed -> logical layer -> write to online store
A user-defined logical layer function to map output to the FeatureView schema
Auto-generation of a default FeatureView if the user does not define their own (controlled by create_feature_view). This is a basic implementation intended as a starting point

sdk/python/feast/init.py

Updated to export DocEmbedder, LogicalLayerFn, BaseChunker, TextChunker, ChunkingConfig, BaseEmbedder, MultiModalEmbedder, and EmbeddingConfig as part of Feast's public API.

Which issue(s) this PR fixes:

Create DocEmbedder class along with RAGRetriever #5426

Misc

patelchaitany · 2026-02-16T11:53:20Z

@ntkathole @jyejare can You pls review this PR and let me know if any changes is needed.

jyejare

Great Addition @patelchaitany , this is a milestone for Feast in RAG. Glad to see multiple types of data are being supported by Embedder.

Few comments and we should be good to go.

sdk/python/tests/unit/test_doc_embedder.py

ntkathole · 2026-03-02T10:58:02Z

@patelchaitany filename typo - examples/rag-retriever/rag_feast_docebedder.ipynb should be rag_feast_docembedder.ipynb

sdk/python/feast/doc_embedder.py

sdk/python/feast/embedder.py

sdk/python/feast/chunker.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

pyproject.toml

sdk/python/feast/chunker.py

…ng them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]>

into fet-DocEmbedder

Signed-off-by: Chaitany patel <[email protected]>

devin-ai-integration

Devin Review found 1 new potential issue.

View 23 additional findings in Devin Review.

sdk/python/feast/doc_embedder.py

ntkathole

lgtm

franciscojavierarceo · 2026-03-16T13:16:03Z

sdk/python/feast/doc_embedder.py

+
+
+@runtime_checkable
+class LogicalLayerFn(Protocol):


Probably we should rename this as it's not particularly intuitive IMO

So i have two name in the mind - SchemaMapperFn or the FeatureViewMapperFn i think this two name are more intuitive if you have particular name in mind then let me know.

* - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * resolving the merge conflict Signed-off-by: Chaitany patel <[email protected]> --------- Signed-off-by: Chaitany patel <[email protected]> Signed-off-by: aaronzuo <[email protected]>

* - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * resolving the merge conflict Signed-off-by: Chaitany patel <[email protected]> --------- Signed-off-by: Chaitany patel <[email protected]> Signed-off-by: Shizoqua <[email protected]>

* - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * resolving the merge conflict Signed-off-by: Chaitany patel <[email protected]> --------- Signed-off-by: Chaitany patel <[email protected]> Signed-off-by: Aniket Paluskar <[email protected]>

* - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * resolving the merge conflict Signed-off-by: Chaitany patel <[email protected]> --------- Signed-off-by: Chaitany patel <[email protected]> Signed-off-by: yuanjun220 <[email protected]>

# [0.61.0](v0.60.0...v0.61.0) (2026-04-07) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([#6079](#6079)) ([30a3a43](30a3a43)) * Added missing jackc/pgx/v5 entries ([94ad0e7](94ad0e7)) * Added MLflow metric charts across feature selection ([#6080](#6080)) ([a403361](a403361)) * Check duplicate names for feature view across types ([#5999](#5999)) ([95b9af8](95b9af8)) * Fix integration tests ([#6046](#6046)) ([02d5548](02d5548)) * Fix missing error handling for resource_counts endpoint ([d9706ce](d9706ce)) * Fix non-specific label selector on metrics service ([a1a160d](a1a160d)) * fix path feature_definitions.py ([7d7df68](7d7df68)) * Fix regstry Rest API tests intermittent failure ([d53a339](d53a339)) * Fixed IntegrityError on SqlRegistry ([#6047](#6047)) ([325e148](325e148)) * Fixed intermittent failures in get_historical_features ([c335ec7](c335ec7)) * Fixed pre-commit check ([114b7db](114b7db)) * Fixed the intermittent FeatureViewNotFoundException ([661ecc7](661ecc7)) * Fixed uv cache permission error for docker build on mac ([ad807be](ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([#5991](#5991)) ([abfd18a](abfd18a)) * Handle existing RBAC role gracefully in namespace registry ([b46a62b](b46a62b)) * Ignore ipynb files during apply ([#6151](#6151)) ([4ea123d](4ea123d)) * Integration test failures ([#6040](#6040)) ([9165870](9165870)) * Mount TLS volumes for init container ([080a9b5](080a9b5)) * **postgres:** Use end_date in synthetic entity_df for non-entity retrieval ([#6110](#6110)) ([088a802](088a802)), closes [#6066](#6066) * Ray offline store tests are duplicated across 3 workflows ([54f705a](54f705a)) * Reenable tests ([#6036](#6036)) ([82ee7f8](82ee7f8)) * SSL/TLS mode by default for postgres connection ([4844488](4844488)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](35a81e7)) ### Features * Add Claude Code agent skills for Feast ([#6081](#6081)) ([1e5b60f](1e5b60f)), closes [#5976](#5976) [#6007](#6007) * Add complex type support (Map, JSON, Struct) with schema validation ([#5974](#5974)) ([1200dbf](1200dbf)) * Add decimal to supported feature types ([#6029](#6029)) ([#6226](#6226)) ([cff6fbf](cff6fbf)) * Add feast apply init container to automate registry population on pod start ([#6106](#6106)) ([6b31a43](6b31a43)) * Add feature view versioning support to PostgreSQL and MySQL online stores ([#6193](#6193)) ([940e0f0](940e0f0)), closes [#6168](#6168) [#6169](#6169) [#2728](#2728) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](2c6be18)) * Add metadata statistics to registry api ([ef1d4fc](ef1d4fc)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](4d08ddc)), closes [#5835](#5835) * Add OnlineStore for MongoDB ([#6025](#6025)) ([bf4e3fa](bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Add Oracle DB as Offline store in python sdk & operator ([#6017](#6017)) ([9d35368](9d35368)) * Add RBAC aggregation labels to FeatureStore ClusterRoles ([daf77c6](daf77c6)) * Add ServiceMonitor auto-generation for Prometheus discovery ([#6126](#6126)) ([56e6d21](56e6d21)) * Add typed_features field to grpc write request (([#6117](#6117)) ([#6118](#6118)) ([eeaa6db](eeaa6db)), closes [#6116](#6116) * Add UUID and TIME_UUID as feature types ([#5885](#5885)) ([#5951](#5951)) ([5d6e311](5d6e311)) * Add version indicators to lineage graph nodes ([#6187](#6187)) ([73805d3](73805d3)) * Add version tracking to FeatureView ([#6101](#6101)) ([ed4a4f2](ed4a4f2)) * Added Agent skills for AI Agents ([#6007](#6007)) ([99008c8](99008c8)) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](547b516)) * Added odfv transformations metrics ([8b5a526](8b5a526)) * Adding optional name to Aggregation (feast-dev[#5994](#5994)) ([#6083](#6083)) ([56469f7](56469f7)) * Created DocEmbedder class ([#5973](#5973)) ([0719c06](0719c06)) * Extended OIDC support to extract groups & namespaces and token injection with multiple methods ([#6089](#6089)) ([7c04026](7c04026)) * Feature Server High-Availability on Kubernetes ([#6028](#6028)) ([9c07b4c](9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([#5925](#5925)) ([2b4ec9a](2b4ec9a)) * Horizontal scaling support to the Feast operator ([#6000](#6000)) ([3ec13e6](3ec13e6)) * Making feature view source optional (feast-dev[#6074](#6074)) ([#6075](#6075)) ([76917b7](76917b7)) * Replace ORJSONResponse with Pydantic response models for faster JSON serialization ([65cf03c](65cf03c)) * Support arm docker build ([#6061](#6061)) ([1e1f5d9](1e1f5d9)) * Support distinct count aggregation [[#6116](#6116)] ([3639570](3639570)) * Support HTTP in MCP ([#6109](#6109)) ([e72b983](e72b983)) * Support nested collection types (Array/Set of Array/Set) ([#5947](#5947)) ([#6132](#6132)) ([ab61642](ab61642)) * Support podAnnotations on Deployment pod template ([1b3cdc1](1b3cdc1)) * Use orjson for faster JSON serialization in feature server ([6f5203a](6f5203a)) * Utilize date partition column in BigQuery ([#6076](#6076)) ([4ea9b32](4ea9b32)) ### Performance Improvements * Online feature response construction in a single pass over read rows ([113fb04](113fb04)) * Optimize protobuf parsing in Redis online store ([#6023](#6023)) ([59dfdb8](59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([#6024](#6024)) ([9699944](9699944)) * Remove redundant entity key serialization in online_read ([d87283f](d87283f))

# [0.62.0](v0.61.0...v0.62.0) (2026-04-08) ### Bug Fixes * Added missing jackc/pgx/v5 entries ([94ad0e7](94ad0e7)) * Fix missing error handling for resource_counts endpoint ([d9706ce](d9706ce)) * fix path feature_definitions.py ([7d7df68](7d7df68)) * Fix regstry Rest API tests intermittent failure ([d53a339](d53a339)) * Fixed intermittent failures in get_historical_features ([c335ec7](c335ec7)) * Fixed the intermittent FeatureViewNotFoundException ([661ecc7](661ecc7)) * Handle existing RBAC role gracefully in namespace registry ([b46a62b](b46a62b)) * Ignore ipynb files during apply ([#6151](#6151)) ([4ea123d](4ea123d)) * Mount TLS volumes for init container ([080a9b5](080a9b5)) * **postgres:** Use end_date in synthetic entity_df for non-entity retrieval ([#6110](#6110)) ([088a802](088a802)), closes [#6066](#6066) * SSL/TLS mode by default for postgres connection ([4844488](4844488)) * Sync v0.61-branch so v0.61.0 tag is reachable from master ([af66878](af66878)) ### Features * Add Claude Code agent skills for Feast ([#6081](#6081)) ([1e5b60f](1e5b60f)), closes [#5976](#5976) [#6007](#6007) * Add decimal to supported feature types ([#6029](#6029)) ([#6226](#6226)) ([cff6fbf](cff6fbf)) * Add feast apply init container to automate registry population on pod start ([#6106](#6106)) ([6b31a43](6b31a43)) * Add feature view versioning support to PostgreSQL and MySQL online stores ([#6193](#6193)) ([940e0f0](940e0f0)), closes [#6168](#6168) [#6169](#6169) [#2728](#2728) * Add metadata statistics to registry api ([ef1d4fc](ef1d4fc)) * Add Oracle DB as Offline store in python sdk & operator ([#6017](#6017)) ([9d35368](9d35368)) * Add RBAC aggregation labels to FeatureStore ClusterRoles ([daf77c6](daf77c6)) * Add ServiceMonitor auto-generation for Prometheus discovery ([#6126](#6126)) ([56e6d21](56e6d21)) * Add typed_features field to grpc write request (([#6117](#6117)) ([#6118](#6118)) ([eeaa6db](eeaa6db)), closes [#6116](#6116) * Add UUID and TIME_UUID as feature types ([#5885](#5885)) ([#5951](#5951)) ([5d6e311](5d6e311)) * Add version indicators to lineage graph nodes ([#6187](#6187)) ([73805d3](73805d3)) * Add version tracking to FeatureView ([#6101](#6101)) ([ed4a4f2](ed4a4f2)) * Added Agent skills for AI Agents ([#6007](#6007)) ([99008c8](99008c8)) * Added odfv transformations metrics ([8b5a526](8b5a526)) * Created DocEmbedder class ([#5973](#5973)) ([0719c06](0719c06)) * Extended OIDC support to extract groups & namespaces and token injection with multiple methods ([#6089](#6089)) ([7c04026](7c04026)) * Replace ORJSONResponse with Pydantic response models for faster JSON serialization ([65cf03c](65cf03c)) * Support distinct count aggregation [[#6116](#6116)] ([3639570](3639570)) * Support HTTP in MCP ([#6109](#6109)) ([e72b983](e72b983)) * Support nested collection types (Array/Set of Array/Set) ([#5947](#5947)) ([#6132](#6132)) ([ab61642](ab61642)) * Support podAnnotations on Deployment pod template ([1b3cdc1](1b3cdc1)) * Utilize date partition column in BigQuery ([#6076](#6076)) ([4ea9b32](4ea9b32)) ### Performance Improvements * Online feature response construction in a single pass over read rows ([113fb04](113fb04))

patelchaitany requested a review from a team as a code owner February 16, 2026 11:28

This comment was marked as resolved.

Sign in to view

patelchaitany changed the title ~~feat: Created DocEmbedder class #5972~~ feat: Created DocEmbedder class Feb 16, 2026

This comment was marked as resolved.

Sign in to view

jyejare reviewed Feb 23, 2026

View reviewed changes

patelchaitany requested a review from jyejare February 24, 2026 10:01

This comment was marked as resolved.

Sign in to view

ntkathole force-pushed the fet-DocEmbedder branch from 1ef9d8e to 083eadb Compare March 2, 2026 05:15

ntkathole added the ok-to-test label Mar 2, 2026

ntkathole force-pushed the fet-DocEmbedder branch from 083eadb to bb74079 Compare March 2, 2026 07:52

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/chunker.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Show resolved Hide resolved

ntkathole removed the ok-to-test label Mar 2, 2026

jyejare approved these changes Mar 3, 2026

View reviewed changes

patelchaitany force-pushed the fet-DocEmbedder branch from daf292f to fcc85cd Compare March 4, 2026 05:38

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch 2 times, most recently from e00ee22 to ee663f5 Compare March 5, 2026 06:31

ntkathole force-pushed the fet-DocEmbedder branch from ee663f5 to 282243b Compare March 6, 2026 15:06

ntkathole reviewed Mar 6, 2026

View reviewed changes

sdk/python/feast/chunker.py Outdated Show resolved Hide resolved

patelchaitany force-pushed the fet-DocEmbedder branch from de74a1f to 23cc11e Compare March 8, 2026 18:48

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch 2 times, most recently from 05ee154 to 510c54a Compare March 9, 2026 12:22

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch from 510c54a to 85ee6e2 Compare March 9, 2026 12:35

ntkathole force-pushed the fet-DocEmbedder branch from 85ee6e2 to 2c66417 Compare March 11, 2026 04:48

Merge branch 'fet-DocEmbedder' of https://github.com/patelchaitany/feast

7562dd0

into fet-DocEmbedder

patelchaitany force-pushed the fet-DocEmbedder branch from 720f0ce to 7562dd0 Compare March 11, 2026 05:28

This comment was marked as resolved.

Sign in to view

Merge branch 'master' into fet-DocEmbedder

f4439cc

This comment was marked as resolved.

Sign in to view

patelchaitany added 2 commits March 11, 2026 11:48

Merge branch 'fet-DocEmbedder' of https://github.com/patelchaitany/feast

d455968

into fet-DocEmbedder

Merge branch 'fet-DocEmbedder' of https://github.com/patelchaitany/feast

b4545aa

into fet-DocEmbedder

ntkathole added the ok-to-test label Mar 14, 2026

patelchaitany added 2 commits March 16, 2026 15:30

Merge remote-tracking branch 'origin/master' into fet-DocEmbedder

9fc01a8

resolving the merge conflict

30dddd7

Signed-off-by: Chaitany patel <[email protected]>

patelchaitany force-pushed the fet-DocEmbedder branch from 523bdfb to 30dddd7 Compare March 16, 2026 11:18

devin-ai-integration bot reviewed Mar 16, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Show resolved Hide resolved

ntkathole approved these changes Mar 16, 2026

View reviewed changes

ntkathole merged commit 0719c06 into feast-dev:master Mar 16, 2026
26 checks passed

franciscojavierarceo reviewed Mar 16, 2026

View reviewed changes

patelchaitany mentioned this pull request Mar 31, 2026

docs: Add DocEmbedder documentation for PR #5973 #6201

Open

3 tasks

Conversation

patelchaitany commented Feb 16, 2026 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

What changed:

sdk/python/feast/chunker.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

sdk/python/feast/init.py

Which issue(s) this PR fixes:

Misc

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

patelchaitany commented Feb 16, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

jyejare left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

ntkathole commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ntkathole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

franciscojavierarceo Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

patelchaitany Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

patelchaitany commented Feb 16, 2026 •

edited by devin-ai-integration bot

Loading