feat: Add OnlineStore for MongoDB#6025
Merged
ntkathole merged 47 commits intofeast-dev:masterfrom Mar 10, 2026
Merged
Conversation
![devin-ai-integration[bot]](proxy.php?url=https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
caseyclements
force-pushed
the
INTPYTHON-297-MongoDB-Feast-Integration
branch
from
5a815be to
6581159
Compare
Collaborator
|
@caseyclements please check the errors in the tests :) |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
caseyclements
force-pushed
the
INTPYTHON-297-MongoDB-Feast-Integration
branch
2 times, most recently
from
6aa5578 to
91dc75c
Compare
Contributor
Author
|
@shuchu Would you please help me investigate the CI timeout? I can run the universal tests for mongodb locally, like so: |
Member
|
@caseyclements Please rebase with master, it would solve the CI issue |
caseyclements
force-pushed
the
INTPYTHON-297-MongoDB-Feast-Integration
branch
from
91dc75c to
bcaa548
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Contributor
Author
|
Thanks @ntkathole , @shuchu . Would you please approve further workflows if applicable. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Contributor
Author
|
Hi @shuchu, @ntkathole, @oavdeev, @ankurs. All checks are now passing on approved workflows. Would you please kick off the others? Of course, your personal feedback would be great as well. In the meantime, I am working on an offline store. |
Contributor
Author
|
Hi @shuchu, @ntkathole, @oavdeev, @ankurs. I made a small change in |
Member
Need to add mongodb in operator as well |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Contributor
Author
@ntkathole Thanks. The go files weren't on my radar. It looks like we're almost there. I updated a few small pieces based on feedback from our product team. Would you please kick off the workflows? |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Member
|
@caseyclements seems linting issue with operator code, you can run make lint in feast-operator dir to fix |
Contributor
Author
I believe this might be fixed by updating the GOLANGCI_LINT version. This fixes the issue which I could reproduce locally. No actual changes were needed though. It was a tooling issue. |
Contributor
Author
|
@ntkathole Can I get a rerun of the workflows please? Is there someone on the west coast that can do another run? |
ntkathole
reviewed
Mar 9, 2026
| We remove any feature views named in tables_to_delete. | ||
| The Entities are serialized in the _id. No schema needs be adjusted. | ||
| """ | ||
| if config.online_store.type != "mongodb": |
Member
There was a problem hiding this comment.
Suggested change
| if config.online_store.type != "mongodb": | |
| if not isinstance(config.online_store, MongoDBOnlineStoreConfig): |
Good to be consistent across
| _client_async: Optional[AsyncMongoClient] = None | ||
| _collection_async: Optional[AsyncCollection] = None | ||
|
|
||
| def online_write_batch( |
Member
There was a problem hiding this comment.
online_write_batch and online_write_batch_async has duplicated logic written in two different styles, we can extract the shared logic into a static helper that both paths call
Contributor
Author
|
HI @ntkathole I'll make the suggested changes. Is the one failing test a red-herring? It appears to be an unrelated timeout. |
Member
yes, unrelated |
Member
|
Please rebase with master, seems there are some conflicts |
Contributor
Author
Can I merge, or is only rebase permitted? |
Comment on lines
+278
to
+283
| async def close(self) -> None: | ||
| """Close the async MongoDB client and release its resources.""" | ||
| if self._client_async is not None: | ||
| await self._client_async.close() | ||
| self._client_async = None | ||
| self._collection_async = None |
Contributor
There was a problem hiding this comment.
🟡 Resource leak: teardown() never closes async client, close() never closes sync client
The MongoDBOnlineStore maintains two independent clients: _client (sync MongoClient) and _client_async (async AsyncMongoClient). The teardown() method (line 273-276) only closes the sync _client and leaves _client_async open. Conversely, the close() method (line 280-283) only closes _client_async and leaves the sync _client open. If both clients are created during the lifetime of a store instance (e.g., sync writes via online_write_batch followed by async reads via online_read_async, or vice versa), the other client's connection pool will leak when cleanup runs through either path.
Suggested change
| async def close(self) -> None: | |
| """Close the async MongoDB client and release its resources.""" | |
| if self._client_async is not None: | |
| await self._client_async.close() | |
| self._client_async = None | |
| self._collection_async = None | |
| async def close(self) -> None: | |
| """Close both async and sync MongoDB clients and release their resources.""" | |
| if self._client_async is not None: | |
| await self._client_async.close() | |
| self._client_async = None | |
| self._collection_async = None | |
| if self._client is not None: | |
| self._client.close() | |
| self._client = None | |
| self._collection = None |
Was this helpful? React with 👍 or 👎 to provide feedback.
Signed-off-by: Casey Clements <[email protected]>
Signed-off-by: Casey Clements <[email protected]>
Signed-off-by: Casey Clements <[email protected]>
Signed-off-by: Casey Clements <[email protected]>
…odb.py Set _client and _collection to None upon synchronous teardown. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Casey Clements <[email protected]>
…ilable In the unit-test-python CI job, pymongo is not installed (it is an optional extra). This caused ModuleNotFoundError during pytest collection. Two changes: 1. mongodb.py: wrap pymongo imports in try/except -> FeastExtrasDependencyImportError (consistent with redis.py, dynamodb.py, datastore.py pattern) 2. test_mongodb_online_retrieval.py: add pytest.importorskip(\pymongo\) so the entire test module is skipped gracefully when pymongo is absent Signed-off-by: Casey Clements <[email protected]>
…never written
MongoDB stores all feature views for an entity in a single document.
If FV 'driver_stats' is written, an entity doc exists for driver_1.
A subsequent read for FV 'pricing' (never written) was previously
returning (None, {all_features: ValueProto()}) - a truthy feature dict
with all-empty protos - instead of the correct (None, None) sentinel.
Feast and downstream callers use (None, None) to signal entity absence.
A non-None feature dict means 'entity found, values are null', which
causes different behaviour in the feature retrieval pipeline.
Fix: after confirming the entity doc exists, also check that the
feature view key is present in doc['features']. If absent, return
(None, None) rather than a dict of empty protos.
Adds test_convert_raw_docs_entity_exists_but_fv_not_written to cover
the multi-feature-view colocation scenario identified in code review.
Signed-off-by: Casey Clements <[email protected]>
…t.toml Any change to pyproject.toml invalidates the pixi.lock manifest hash, causing 'pixi install --locked' to fail in CI even when the changed section (mongodb optional extras) is not used by any pixi environment. Regenerated with: pixi install (v0.63.2) Signed-off-by: Casey Clements <[email protected]>
Stores like mongodb, redis, and dynamodb raise FeastExtrasDependencyImportError at import time when their optional Python extras are not installed. test_all.py only caught ModuleNotFoundError, so any such import caused the entire test_docstrings() function to abort rather than gracefully skipping the module. Extend the except clause to also catch FeastExtrasDependencyImportError so the doctest run completes for all other modules when an optional extra is absent in the test environment. Signed-off-by: Casey Clements <[email protected]>
Directory was renamed from tests/integration/feature_repos to tests/universal/feature_repos in upstream/master. Update the PYTEST_PLUGINS module path in test-python-universal-mongodb-online to match the new location. Signed-off-by: Casey Clements <[email protected]>
…eError and other third-party import errors qdrant_client raises TypeError at import time when a gRPC EnumTypeWrapper is used with the | union operator on Python < 3.12. The previous fix only caught ModuleNotFoundError and FeastExtrasDependencyImportError, leaving the test runner vulnerable to any other exception raised by third-party libraries during pkgutil.walk_packages. Changes: - Catch bare Exception in the import try/except block so any import-time error from an optional dependency causes a graceful skip rather than an abort of the entire test run. - Initialize temp_module = None before the try block and add a continue guard so that a failed import never leaves a stale module reference to be used in the subsequent doctest execution block. Signed-off-by: Casey Clements <[email protected]>
…r package import failures FeastExtrasDependencyImportError inherits from FeastError -> Exception, not from ImportError. pkgutil.walk_packages calls __import__ internally when recursing into sub-packages, and only silently swallows ImportError subclasses; any other exception is re-raised, crashing the entire test run. Passing onerror=lambda _: None tells walk_packages to skip any package that fails to import during the walk phase, regardless of the exception type. The inner importlib.import_module try/except already handles the same errors for the explicit import step used to collect doctests. Signed-off-by: Casey Clements <[email protected]>
…iversal.feature_repos The upstream directory tests/integration/feature_repos was renamed to tests/universal/feature_repos. Three files still referenced the old path: - mongodb_repo_configuration.py: IntegrationTestRepoConfig and MongoDBOnlineStoreCreator imports - tests/.../online_store/mongodb.py: OnlineStoreCreator import - tests/unit/online_store/test_mongodb_online_retrieval.py: TAGS import Updating all three unblocks test collection and allows make test-python-universal-mongodb-online to run locally. Signed-off-by: Casey Clements <[email protected]>
…-operator The Python ONLINE_STORE_CLASS_FOR_TYPE now includes 'mongodb', so the operator's parity check (test-datasources) fails unless the Go API also lists it. Add 'mongodb' to ValidOnlineStoreDBStorePersistenceTypes in both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]>
Add DriverInfo with the Feast name and version to both the sync MongoClient and async AsyncMongoClient so that MongoDB can identify traffic originating from a Feast AI integration. Signed-off-by: Casey Clements <[email protected]>
Signed-off-by: Casey Clements <[email protected]>
…rePersistence The +kubebuilder:validation:Enum annotation on the Type field of OnlineStoreDBStorePersistence was not updated when mongodb was added to ValidOnlineStoreDBStorePersistenceTypes. This annotation drives CRD OpenAPI schema validation at Kubernetes admission time, so any FeatureStore CR specifying type: mongodb would be rejected by the API server. Updated both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]>
Signed-off-by: Casey Clements <[email protected]>
…onsistent with sync counterpart Signed-off-by: Casey Clements <[email protected]>
Signed-off-by: ntkathole <[email protected]>
Signed-off-by: Casey Clements <[email protected]>
…ially identical. Signed-off-by: Casey Clements <[email protected]>
caseyclements
force-pushed
the
INTPYTHON-297-MongoDB-Feast-Integration
branch
from
cb9e1d6 to
e761e10
Compare
Contributor
Author
@ntkathole I've rebased all my commits from the latest master, and run tests locally. Please approve workflows and have a look. |
franciscojavierarceo
pushed a commit
that referenced
this pull request
Mar 10, 2026
# [0.61.0](v0.60.0...v0.61.0) (2026-03-10) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([#6079](#6079)) ([30a3a43](30a3a43)) * Added MLflow metric charts across feature selection ([#6080](#6080)) ([a403361](a403361)) * Check duplicate names for feature view across types ([#5999](#5999)) ([95b9af8](95b9af8)) * Fix integration tests ([#6046](#6046)) ([02d5548](02d5548)) * Fix non-specific label selector on metrics service ([a1a160d](a1a160d)) * Fixed IntegrityError on SqlRegistry ([#6047](#6047)) ([325e148](325e148)) * Fixed pre-commit check ([114b7db](114b7db)) * Fixed uv cache permission error for docker build on mac ([ad807be](ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([#5991](#5991)) ([abfd18a](abfd18a)) * Integration test failures ([#6040](#6040)) ([9165870](9165870)) * Ray offline store tests are duplicated across 3 workflows ([54f705a](54f705a)) * Reenable tests ([#6036](#6036)) ([82ee7f8](82ee7f8)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](35a81e7)) ### Features * Add complex type support (Map, JSON, Struct) with schema validation ([#5974](#5974)) ([1200dbf](1200dbf)) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](2c6be18)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](4d08ddc)), closes [#5835](#5835) * Add OnlineStore for MongoDB ([#6025](#6025)) ([bf4e3fa](bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](547b516)) * Adding optional name to Aggregation (feast-dev[#5994](#5994)) ([#6083](#6083)) ([56469f7](56469f7)) * Feature Server High-Availability on Kubernetes ([#6028](#6028)) ([9c07b4c](9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([#5925](#5925)) ([2b4ec9a](2b4ec9a)) * Horizontal scaling support to the Feast operator ([#6000](#6000)) ([3ec13e6](3ec13e6)) * Making feature view source optional (feast-dev[#6074](#6074)) ([#6075](#6075)) ([76917b7](76917b7)) * Support arm docker build ([#6061](#6061)) ([1e1f5d9](1e1f5d9)) * Use orjson for faster JSON serialization in feature server ([6f5203a](6f5203a)) ### Performance Improvements * Optimize protobuf parsing in Redis online store ([#6023](#6023)) ([59dfdb8](59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([#6024](#6024)) ([9699944](9699944)) * Remove redundant entity key serialization in online_read ([d87283f](d87283f))
ntkathole
added a commit
to red-hat-data-services/feast
that referenced
this pull request
Mar 16, 2026
* Initial commit on INTPYTHON-297-MongoDB-Feast-Integration Signed-off-by: Casey Clements <[email protected]> * Added mongodb to project.optional-dependencies in pyproject.toml. Now pymongo is found as extra Signed-off-by: Casey Clements <[email protected]> * Checkpoint. Passing tests.unit.online_store.test_online_writes.TestOnlineWrites with default args of MongoDBOnlineStoreConfig. Lots to do. Signed-off-by: Casey Clements <[email protected]> * Handle Nan in dfs for test_online_writes.py. Now all tests in the module pass Signed-off-by: Casey Clements <[email protected]> * Removed suffix of implementation: mongodb_openai -> mongodb Signed-off-by: Casey Clements <[email protected]> * Moved MongoDBOnlineStore to feast.infra.online_store.contrib Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Refactor online_read that converts bson to proto. Left two methods for now. simply and transformming Signed-off-by: Casey Clements <[email protected]> * Remove file added early during discovery Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Added version of test that uses testcontainers mongodb instead of assuming one is running Signed-off-by: Casey Clements <[email protected]> * Create Make target for universal tests Signed-off-by: Casey Clements <[email protected]> * Cleanup Signed-off-by: Casey Clements <[email protected]> * Removed temporary integration tests requiring one to spin up own mongodb server. Signed-off-by: Casey Clements <[email protected]> * Format Signed-off-by: Casey Clements <[email protected]> * Typing Signed-off-by: Casey Clements <[email protected]> * Implemented ASync API and Tests Signed-off-by: Casey Clements <[email protected]> * Removed offline store stubs. The first PR will only contain the OnlineStore Signed-off-by: Casey Clements <[email protected]> * Moved mongodb_online_store out of cobtrib package. Signed-off-by: Casey Clements <[email protected]> * Add documentation. Signed-off-by: Casey Clements <[email protected]> * Cleanups and docstrings Signed-off-by: Casey Clements <[email protected]> * Fixed another reference to contrib dir Signed-off-by: Casey Clements <[email protected]> * Typos Signed-off-by: Casey Clements <[email protected]> * Made _convert_raw_docs_to_proto staticmethods private Signed-off-by: Casey Clements <[email protected]> * After benchmarking two alogithm for conevrting read results from bson to proto, removed the naive one. It was outcompeted 3X across dimensions Signed-off-by: Casey Clements <[email protected]> * Add extra unit tests Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Fixes in pyproject.toml Signed-off-by: Casey Clements <[email protected]> * Fixed Detect secrets false positives. Signed-off-by: Casey Clements <[email protected]> * Update sdk/python/feast/infra/online_stores/mongodb_online_store/mongodb.py Set _client and _collection to None upon synchronous teardown. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Casey Clements <[email protected]> * Fix CI: guard pymongo imports and skip test module when pymongo unavailable In the unit-test-python CI job, pymongo is not installed (it is an optional extra). This caused ModuleNotFoundError during pytest collection. Two changes: 1. mongodb.py: wrap pymongo imports in try/except -> FeastExtrasDependencyImportError (consistent with redis.py, dynamodb.py, datastore.py pattern) 2. test_mongodb_online_retrieval.py: add pytest.importorskip(\pymongo\) so the entire test module is skipped gracefully when pymongo is absent Signed-off-by: Casey Clements <[email protected]> * Fix: return (None, None) when entity doc exists but feature view was never written MongoDB stores all feature views for an entity in a single document. If FV 'driver_stats' is written, an entity doc exists for driver_1. A subsequent read for FV 'pricing' (never written) was previously returning (None, {all_features: ValueProto()}) - a truthy feature dict with all-empty protos - instead of the correct (None, None) sentinel. Feast and downstream callers use (None, None) to signal entity absence. A non-None feature dict means 'entity found, values are null', which causes different behaviour in the feature retrieval pipeline. Fix: after confirming the entity doc exists, also check that the feature view key is present in doc['features']. If absent, return (None, None) rather than a dict of empty protos. Adds test_convert_raw_docs_entity_exists_but_fv_not_written to cover the multi-feature-view colocation scenario identified in code review. Signed-off-by: Casey Clements <[email protected]> * Update pixi.lock after adding mongodb optional dependency to pyproject.toml Any change to pyproject.toml invalidates the pixi.lock manifest hash, causing 'pixi install --locked' to fail in CI even when the changed section (mongodb optional extras) is not used by any pixi environment. Regenerated with: pixi install (v0.63.2) Signed-off-by: Casey Clements <[email protected]> * fix: catch FeastExtrasDependencyImportError in doctest runner Stores like mongodb, redis, and dynamodb raise FeastExtrasDependencyImportError at import time when their optional Python extras are not installed. test_all.py only caught ModuleNotFoundError, so any such import caused the entire test_docstrings() function to abort rather than gracefully skipping the module. Extend the except clause to also catch FeastExtrasDependencyImportError so the doctest run completes for all other modules when an optional extra is absent in the test environment. Signed-off-by: Casey Clements <[email protected]> * fix: update PYTEST_PLUGINS path for mongodb online store tests Directory was renamed from tests/integration/feature_repos to tests/universal/feature_repos in upstream/master. Update the PYTEST_PLUGINS module path in test-python-universal-mongodb-online to match the new location. Signed-off-by: Casey Clements <[email protected]> * fix: broaden import exception handling in doctest runner to catch TypeError and other third-party import errors qdrant_client raises TypeError at import time when a gRPC EnumTypeWrapper is used with the | union operator on Python < 3.12. The previous fix only caught ModuleNotFoundError and FeastExtrasDependencyImportError, leaving the test runner vulnerable to any other exception raised by third-party libraries during pkgutil.walk_packages. Changes: - Catch bare Exception in the import try/except block so any import-time error from an optional dependency causes a graceful skip rather than an abort of the entire test run. - Initialize temp_module = None before the try block and add a continue guard so that a failed import never leaves a stale module reference to be used in the subsequent doctest execution block. Signed-off-by: Casey Clements <[email protected]> * fix: pass onerror to pkgutil.walk_packages to suppress non-ImportError package import failures FeastExtrasDependencyImportError inherits from FeastError -> Exception, not from ImportError. pkgutil.walk_packages calls __import__ internally when recursing into sub-packages, and only silently swallows ImportError subclasses; any other exception is re-raised, crashing the entire test run. Passing onerror=lambda _: None tells walk_packages to skip any package that fails to import during the walk phase, regardless of the exception type. The inner importlib.import_module try/except already handles the same errors for the explicit import step used to collect doctests. Signed-off-by: Casey Clements <[email protected]> * fix: update stale tests.integration.feature_repos imports to tests.universal.feature_repos The upstream directory tests/integration/feature_repos was renamed to tests/universal/feature_repos. Three files still referenced the old path: - mongodb_repo_configuration.py: IntegrationTestRepoConfig and MongoDBOnlineStoreCreator imports - tests/.../online_store/mongodb.py: OnlineStoreCreator import - tests/unit/online_store/test_mongodb_online_retrieval.py: TAGS import Updating all three unblocks test collection and allows make test-python-universal-mongodb-online to run locally. Signed-off-by: Casey Clements <[email protected]> * feat: add mongodb to ValidOnlineStoreDBStorePersistenceTypes in feast-operator The Python ONLINE_STORE_CLASS_FOR_TYPE now includes 'mongodb', so the operator's parity check (test-datasources) fails unless the Go API also lists it. Add 'mongodb' to ValidOnlineStoreDBStorePersistenceTypes in both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]> * feat: add Feast driver metadata to MongoDB client instantiations Add DriverInfo with the Feast name and version to both the sync MongoClient and async AsyncMongoClient so that MongoDB can identify traffic originating from a Feast AI integration. Signed-off-by: Casey Clements <[email protected]> * docs: update MongoDB online store status from alpha to preview Signed-off-by: Casey Clements <[email protected]> * fix: add mongodb to kubebuilder Enum annotations for OnlineStoreDBStorePersistence The +kubebuilder:validation:Enum annotation on the Type field of OnlineStoreDBStorePersistence was not updated when mongodb was added to ValidOnlineStoreDBStorePersistenceTypes. This annotation drives CRD OpenAPI schema validation at Kubernetes admission time, so any FeatureStore CR specifying type: mongodb would be rejected by the API server. Updated both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]> * Update +GOLANGCI_LINT_VERSION to fix upstream issue golang/go#74462 Signed-off-by: Casey Clements <[email protected]> * fix: raise ValueError in _get_client_async for invalid config type, consistent with sync counterpart Signed-off-by: Casey Clements <[email protected]> * fix: Added mongodb to operator yamls Signed-off-by: ntkathole <[email protected]> * Small change suggested by ntkathole Signed-off-by: Casey Clements <[email protected]> * Factor out write logic into utility function making sync/async essentially identical. Signed-off-by: Casey Clements <[email protected]> --------- Signed-off-by: Casey Clements <[email protected]> Signed-off-by: ntkathole <[email protected]> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ntkathole <[email protected]>
ntkathole
pushed a commit
to red-hat-data-services/feast
that referenced
this pull request
Mar 16, 2026
# [0.61.0](feast-dev/feast@v0.60.0...v0.61.0) (2026-03-10) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](feast-dev@2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](feast-dev@ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([feast-dev#6079](feast-dev#6079)) ([30a3a43](feast-dev@30a3a43)) * Added MLflow metric charts across feature selection ([feast-dev#6080](feast-dev#6080)) ([a403361](feast-dev@a403361)) * Check duplicate names for feature view across types ([feast-dev#5999](feast-dev#5999)) ([95b9af8](feast-dev@95b9af8)) * Fix integration tests ([feast-dev#6046](feast-dev#6046)) ([02d5548](feast-dev@02d5548)) * Fix non-specific label selector on metrics service ([a1a160d](feast-dev@a1a160d)) * Fixed IntegrityError on SqlRegistry ([feast-dev#6047](feast-dev#6047)) ([325e148](feast-dev@325e148)) * Fixed pre-commit check ([114b7db](feast-dev@114b7db)) * Fixed uv cache permission error for docker build on mac ([ad807be](feast-dev@ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([feast-dev#5991](feast-dev#5991)) ([abfd18a](feast-dev@abfd18a)) * Integration test failures ([feast-dev#6040](feast-dev#6040)) ([9165870](feast-dev@9165870)) * Ray offline store tests are duplicated across 3 workflows ([54f705a](feast-dev@54f705a)) * Reenable tests ([feast-dev#6036](feast-dev#6036)) ([82ee7f8](feast-dev@82ee7f8)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](feast-dev@35a81e7)) ### Features * Add complex type support (Map, JSON, Struct) with schema validation ([feast-dev#5974](feast-dev#5974)) ([1200dbf](feast-dev@1200dbf)) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](feast-dev@2c6be18)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](feast-dev@4d08ddc)), closes [feast-dev#5835](feast-dev#5835) * Add OnlineStore for MongoDB ([feast-dev#6025](feast-dev#6025)) ([bf4e3fa](feast-dev@bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](feast-dev@547b516)) * Adding optional name to Aggregation (feast-dev[feast-dev#5994](feast-dev#5994)) ([feast-dev#6083](feast-dev#6083)) ([56469f7](feast-dev@56469f7)) * Feature Server High-Availability on Kubernetes ([feast-dev#6028](feast-dev#6028)) ([9c07b4c](feast-dev@9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([feast-dev#5925](feast-dev#5925)) ([2b4ec9a](feast-dev@2b4ec9a)) * Horizontal scaling support to the Feast operator ([feast-dev#6000](feast-dev#6000)) ([3ec13e6](feast-dev@3ec13e6)) * Making feature view source optional (feast-dev[feast-dev#6074](feast-dev#6074)) ([feast-dev#6075](feast-dev#6075)) ([76917b7](feast-dev@76917b7)) * Support arm docker build ([feast-dev#6061](feast-dev#6061)) ([1e1f5d9](feast-dev@1e1f5d9)) * Use orjson for faster JSON serialization in feature server ([6f5203a](feast-dev@6f5203a)) ### Performance Improvements * Optimize protobuf parsing in Redis online store ([feast-dev#6023](feast-dev#6023)) ([59dfdb8](feast-dev@59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](feast-dev@33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([feast-dev#6024](feast-dev#6024)) ([9699944](feast-dev@9699944)) * Remove redundant entity key serialization in online_read ([d87283f](feast-dev@d87283f))
ntkathole
added a commit
to red-hat-data-services/feast
that referenced
this pull request
Mar 16, 2026
* Initial commit on INTPYTHON-297-MongoDB-Feast-Integration Signed-off-by: Casey Clements <[email protected]> * Added mongodb to project.optional-dependencies in pyproject.toml. Now pymongo is found as extra Signed-off-by: Casey Clements <[email protected]> * Checkpoint. Passing tests.unit.online_store.test_online_writes.TestOnlineWrites with default args of MongoDBOnlineStoreConfig. Lots to do. Signed-off-by: Casey Clements <[email protected]> * Handle Nan in dfs for test_online_writes.py. Now all tests in the module pass Signed-off-by: Casey Clements <[email protected]> * Removed suffix of implementation: mongodb_openai -> mongodb Signed-off-by: Casey Clements <[email protected]> * Moved MongoDBOnlineStore to feast.infra.online_store.contrib Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Refactor online_read that converts bson to proto. Left two methods for now. simply and transformming Signed-off-by: Casey Clements <[email protected]> * Remove file added early during discovery Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Added version of test that uses testcontainers mongodb instead of assuming one is running Signed-off-by: Casey Clements <[email protected]> * Create Make target for universal tests Signed-off-by: Casey Clements <[email protected]> * Cleanup Signed-off-by: Casey Clements <[email protected]> * Removed temporary integration tests requiring one to spin up own mongodb server. Signed-off-by: Casey Clements <[email protected]> * Format Signed-off-by: Casey Clements <[email protected]> * Typing Signed-off-by: Casey Clements <[email protected]> * Implemented ASync API and Tests Signed-off-by: Casey Clements <[email protected]> * Removed offline store stubs. The first PR will only contain the OnlineStore Signed-off-by: Casey Clements <[email protected]> * Moved mongodb_online_store out of cobtrib package. Signed-off-by: Casey Clements <[email protected]> * Add documentation. Signed-off-by: Casey Clements <[email protected]> * Cleanups and docstrings Signed-off-by: Casey Clements <[email protected]> * Fixed another reference to contrib dir Signed-off-by: Casey Clements <[email protected]> * Typos Signed-off-by: Casey Clements <[email protected]> * Made _convert_raw_docs_to_proto staticmethods private Signed-off-by: Casey Clements <[email protected]> * After benchmarking two alogithm for conevrting read results from bson to proto, removed the naive one. It was outcompeted 3X across dimensions Signed-off-by: Casey Clements <[email protected]> * Add extra unit tests Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Fixes in pyproject.toml Signed-off-by: Casey Clements <[email protected]> * Fixed Detect secrets false positives. Signed-off-by: Casey Clements <[email protected]> * Update sdk/python/feast/infra/online_stores/mongodb_online_store/mongodb.py Set _client and _collection to None upon synchronous teardown. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Casey Clements <[email protected]> * Fix CI: guard pymongo imports and skip test module when pymongo unavailable In the unit-test-python CI job, pymongo is not installed (it is an optional extra). This caused ModuleNotFoundError during pytest collection. Two changes: 1. mongodb.py: wrap pymongo imports in try/except -> FeastExtrasDependencyImportError (consistent with redis.py, dynamodb.py, datastore.py pattern) 2. test_mongodb_online_retrieval.py: add pytest.importorskip(\pymongo\) so the entire test module is skipped gracefully when pymongo is absent Signed-off-by: Casey Clements <[email protected]> * Fix: return (None, None) when entity doc exists but feature view was never written MongoDB stores all feature views for an entity in a single document. If FV 'driver_stats' is written, an entity doc exists for driver_1. A subsequent read for FV 'pricing' (never written) was previously returning (None, {all_features: ValueProto()}) - a truthy feature dict with all-empty protos - instead of the correct (None, None) sentinel. Feast and downstream callers use (None, None) to signal entity absence. A non-None feature dict means 'entity found, values are null', which causes different behaviour in the feature retrieval pipeline. Fix: after confirming the entity doc exists, also check that the feature view key is present in doc['features']. If absent, return (None, None) rather than a dict of empty protos. Adds test_convert_raw_docs_entity_exists_but_fv_not_written to cover the multi-feature-view colocation scenario identified in code review. Signed-off-by: Casey Clements <[email protected]> * Update pixi.lock after adding mongodb optional dependency to pyproject.toml Any change to pyproject.toml invalidates the pixi.lock manifest hash, causing 'pixi install --locked' to fail in CI even when the changed section (mongodb optional extras) is not used by any pixi environment. Regenerated with: pixi install (v0.63.2) Signed-off-by: Casey Clements <[email protected]> * fix: catch FeastExtrasDependencyImportError in doctest runner Stores like mongodb, redis, and dynamodb raise FeastExtrasDependencyImportError at import time when their optional Python extras are not installed. test_all.py only caught ModuleNotFoundError, so any such import caused the entire test_docstrings() function to abort rather than gracefully skipping the module. Extend the except clause to also catch FeastExtrasDependencyImportError so the doctest run completes for all other modules when an optional extra is absent in the test environment. Signed-off-by: Casey Clements <[email protected]> * fix: update PYTEST_PLUGINS path for mongodb online store tests Directory was renamed from tests/integration/feature_repos to tests/universal/feature_repos in upstream/master. Update the PYTEST_PLUGINS module path in test-python-universal-mongodb-online to match the new location. Signed-off-by: Casey Clements <[email protected]> * fix: broaden import exception handling in doctest runner to catch TypeError and other third-party import errors qdrant_client raises TypeError at import time when a gRPC EnumTypeWrapper is used with the | union operator on Python < 3.12. The previous fix only caught ModuleNotFoundError and FeastExtrasDependencyImportError, leaving the test runner vulnerable to any other exception raised by third-party libraries during pkgutil.walk_packages. Changes: - Catch bare Exception in the import try/except block so any import-time error from an optional dependency causes a graceful skip rather than an abort of the entire test run. - Initialize temp_module = None before the try block and add a continue guard so that a failed import never leaves a stale module reference to be used in the subsequent doctest execution block. Signed-off-by: Casey Clements <[email protected]> * fix: pass onerror to pkgutil.walk_packages to suppress non-ImportError package import failures FeastExtrasDependencyImportError inherits from FeastError -> Exception, not from ImportError. pkgutil.walk_packages calls __import__ internally when recursing into sub-packages, and only silently swallows ImportError subclasses; any other exception is re-raised, crashing the entire test run. Passing onerror=lambda _: None tells walk_packages to skip any package that fails to import during the walk phase, regardless of the exception type. The inner importlib.import_module try/except already handles the same errors for the explicit import step used to collect doctests. Signed-off-by: Casey Clements <[email protected]> * fix: update stale tests.integration.feature_repos imports to tests.universal.feature_repos The upstream directory tests/integration/feature_repos was renamed to tests/universal/feature_repos. Three files still referenced the old path: - mongodb_repo_configuration.py: IntegrationTestRepoConfig and MongoDBOnlineStoreCreator imports - tests/.../online_store/mongodb.py: OnlineStoreCreator import - tests/unit/online_store/test_mongodb_online_retrieval.py: TAGS import Updating all three unblocks test collection and allows make test-python-universal-mongodb-online to run locally. Signed-off-by: Casey Clements <[email protected]> * feat: add mongodb to ValidOnlineStoreDBStorePersistenceTypes in feast-operator The Python ONLINE_STORE_CLASS_FOR_TYPE now includes 'mongodb', so the operator's parity check (test-datasources) fails unless the Go API also lists it. Add 'mongodb' to ValidOnlineStoreDBStorePersistenceTypes in both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]> * feat: add Feast driver metadata to MongoDB client instantiations Add DriverInfo with the Feast name and version to both the sync MongoClient and async AsyncMongoClient so that MongoDB can identify traffic originating from a Feast AI integration. Signed-off-by: Casey Clements <[email protected]> * docs: update MongoDB online store status from alpha to preview Signed-off-by: Casey Clements <[email protected]> * fix: add mongodb to kubebuilder Enum annotations for OnlineStoreDBStorePersistence The +kubebuilder:validation:Enum annotation on the Type field of OnlineStoreDBStorePersistence was not updated when mongodb was added to ValidOnlineStoreDBStorePersistenceTypes. This annotation drives CRD OpenAPI schema validation at Kubernetes admission time, so any FeatureStore CR specifying type: mongodb would be rejected by the API server. Updated both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]> * Update +GOLANGCI_LINT_VERSION to fix upstream issue golang/go#74462 Signed-off-by: Casey Clements <[email protected]> * fix: raise ValueError in _get_client_async for invalid config type, consistent with sync counterpart Signed-off-by: Casey Clements <[email protected]> * fix: Added mongodb to operator yamls Signed-off-by: ntkathole <[email protected]> * Small change suggested by ntkathole Signed-off-by: Casey Clements <[email protected]> * Factor out write logic into utility function making sync/async essentially identical. Signed-off-by: Casey Clements <[email protected]> --------- Signed-off-by: Casey Clements <[email protected]> Signed-off-by: ntkathole <[email protected]> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ntkathole <[email protected]>
ntkathole
pushed a commit
to red-hat-data-services/feast
that referenced
this pull request
Mar 16, 2026
# [0.61.0](feast-dev/feast@v0.60.0...v0.61.0) (2026-03-10) ### Bug Fixes * Add grpcio dependency group to transformation server Dockerfile ([2c2150a](feast-dev@2c2150a)) * Add https readiness check for rest-registry tests ([ea85e63](feast-dev@ea85e63)) * Add website build check for PRs and fix blog frontmatter YAML error ([feast-dev#6079](feast-dev#6079)) ([30a3a43](feast-dev@30a3a43)) * Added MLflow metric charts across feature selection ([feast-dev#6080](feast-dev#6080)) ([a403361](feast-dev@a403361)) * Check duplicate names for feature view across types ([feast-dev#5999](feast-dev#5999)) ([95b9af8](feast-dev@95b9af8)) * Fix integration tests ([feast-dev#6046](feast-dev#6046)) ([02d5548](feast-dev@02d5548)) * Fix non-specific label selector on metrics service ([a1a160d](feast-dev@a1a160d)) * Fixed IntegrityError on SqlRegistry ([feast-dev#6047](feast-dev#6047)) ([325e148](feast-dev@325e148)) * Fixed pre-commit check ([114b7db](feast-dev@114b7db)) * Fixed uv cache permission error for docker build on mac ([ad807be](feast-dev@ad807be)) * Fixes a `PydanticDeprecatedSince20` warning for trino_offline_store ([feast-dev#5991](feast-dev#5991)) ([abfd18a](feast-dev@abfd18a)) * Integration test failures ([feast-dev#6040](feast-dev#6040)) ([9165870](feast-dev@9165870)) * Ray offline store tests are duplicated across 3 workflows ([54f705a](feast-dev@54f705a)) * Reenable tests ([feast-dev#6036](feast-dev#6036)) ([82ee7f8](feast-dev@82ee7f8)) * Use commitlint pre-commit hook instead of a separate action ([35a81e7](feast-dev@35a81e7)) ### Features * Add complex type support (Map, JSON, Struct) with schema validation ([feast-dev#5974](feast-dev#5974)) ([1200dbf](feast-dev@1200dbf)) * Add materialization, feature freshness, request latency, and push metrics to feature server ([2c6be18](feast-dev@2c6be18)) * Add non-entity retrieval support for ClickHouse offline store ([4d08ddc](feast-dev@4d08ddc)), closes [feast-dev#5835](feast-dev#5835) * Add OnlineStore for MongoDB ([feast-dev#6025](feast-dev#6025)) ([bf4e3fa](feast-dev@bf4e3fa)), closes [golang/go#74462](golang/go#74462) * Added CodeQL SAST scanning and detect-secrets pre-commit hook ([547b516](feast-dev@547b516)) * Adding optional name to Aggregation (feast-dev[feast-dev#5994](feast-dev#5994)) ([feast-dev#6083](feast-dev#6083)) ([56469f7](feast-dev@56469f7)) * Feature Server High-Availability on Kubernetes ([feast-dev#6028](feast-dev#6028)) ([9c07b4c](feast-dev@9c07b4c)), closes [Hi#Availability](https://github.com/Hi/issues/Availability) [Hi#Availability](https://github.com/Hi/issues/Availability) * **go:** Implement metrics and tracing for http and grpc servers ([feast-dev#5925](feast-dev#5925)) ([2b4ec9a](feast-dev@2b4ec9a)) * Horizontal scaling support to the Feast operator ([feast-dev#6000](feast-dev#6000)) ([3ec13e6](feast-dev@3ec13e6)) * Making feature view source optional (feast-dev[feast-dev#6074](feast-dev#6074)) ([feast-dev#6075](feast-dev#6075)) ([76917b7](feast-dev@76917b7)) * Support arm docker build ([feast-dev#6061](feast-dev#6061)) ([1e1f5d9](feast-dev@1e1f5d9)) * Use orjson for faster JSON serialization in feature server ([6f5203a](feast-dev@6f5203a)) ### Performance Improvements * Optimize protobuf parsing in Redis online store ([feast-dev#6023](feast-dev#6023)) ([59dfdb8](feast-dev@59dfdb8)) * Optimize timestamp conversion in _convert_rows_to_protobuf ([33a2e95](feast-dev@33a2e95)) * Parallelize DynamoDB batch reads in sync online_read ([feast-dev#6024](feast-dev#6024)) ([9699944](feast-dev@9699944)) * Remove redundant entity key serialization in online_read ([d87283f](feast-dev@d87283f))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds a first-class MongoDB online store integration at
feast.infra.online_stores.mongodb_online_store.Schema
Each entity is stored as a single MongoDB document keyed by its serialized entity key:
{ "_id": "<serialized_entity_key>", "features": { "<feature_view>": { "<feature>": <value> } }, "event_timestamps": { "<feature_view>": "<datetime>" }, "created_timestamp": "<datetime>" }Because MongoDB has a loose schema and supports upsert semantics natively,
update()requires nopre-creation — it only removes feature views named in
tables_to_deletevia$unset. Multiplefeature views for the same entity share one document.
Implementation highlights
online_write_batch,online_read,online_write_batch_async,online_read_async, andasync def close().async_supportedreturnsread=True, write=True.requested_featuresprojection — both read paths build a MongoDB field projection so onlythe requested feature columns are returned from the server.
_convert_raw_docs_to_prototransforms column-wise tominimise calls to
python_values_to_proto_values(one call per feature across all entities,rather than one call per entity × feature). Benchmarking confirmed this is ~3–4× faster than a
naïve row-wise approach across all scaling dimensions (entities, features, feature views).
serialize_entity_key, so compositekeys (e.g.
customer_id + driver_id) are handled without schema changes.Tests
test_mongodb_online_features) — spins up a real MongoDB containervia
testcontainers, writes to three feature views (single int key, single string key, compositekey), and asserts correct retrieval including type coercion and missing-entity handling.
test_convert_raw_docs_missing_entity— entity absent from query results →(None, None)test_convert_raw_docs_partial_doc— entity present but a feature key missing → emptyValueProto(schema-migration safety)test_convert_raw_docs_ordering— result order follows the requestedidslist regardlessof MongoDB cursor order
All tests pass. Code is clean under
mypy,ruff check, andruff format.