NEW BACKEND! MongoDB Atlas by caseyclements · Pull Request #1883 · docarray/docarray

caseyclements · 2024-04-23T19:21:31Z

Description

This pull request introduces MongoDB integration as a document index backend, enhancing search capabilities within the application. Below are the details of the implementation and its supported functionalities:

Simple Usage

from docarray.index import MongoAtlasDocumentIndex
import numpy as np

class MyDoc(BaseDoc):
    text: str
    embedding: NdArray[10]

docs = [MyDoc(text=f'text {i}', embedding=np.random.rand(10)) for i in range(10)]
query = np.random.rand(10)
db = MongoAtlasDocumentIndex[MyDoc](host='localhost')
db.index(docs)
results = index.find(query, search_field='embedding', limit=10)

Supported Functionality

Find (vector search): Enables vector-based search.
Filter: Allows filtering on textual and numeric data using MongoDB syntax.
Text Search: Supports text search using regex match.
Get/Del: Retrieve and delete operations.
Subindex: Ability to create subindexes for better organization.

Integration Tests and documentation

tests/index/mongo_atlas
docs/API_reference/doc_index/backends/mongodb.md

Coming soon

Implementation of QueryBuilder and Hybrid Search.

JoanFM · 2024-04-24T06:14:20Z

Hello @caseyclements ,

Thanks a lot for this amazing contribution. May I ask if you could please sign off the commits so that we can pass the DCO and merge the PR if accepted?

docarray/index/backends/mongodb_atlas.py

Signed-off-by: Casey Clements <[email protected]>

…equired_args Signed-off-by: Casey Clements <[email protected]>

Signed-off-by: Casey Clements <[email protected]>

prakul · 2024-04-24T23:09:12Z

All the comments have been addressed @JoanFM

JoanFM

Can you please check how to pass the black check? You can find in our CONTRIBUTING guidelines the steps to find

docarray/index/backends/mongodb_atlas.py

Signed-off-by: Casey Clements <[email protected]>

JoanFM · 2024-04-26T07:09:40Z

We are gettting these errors in the tests caused by the type annotations , maybe my requests were not ggood enough.

docarray/index/backends/mongodb_atlas.py:197: in MongoDBAtlasDocumentIndex
    ) -> Tuple[list[dict], list[float]]:
E   TypeError: 'type' object is not subscriptable

Signed-off-by: Casey Clements <[email protected]>

caseyclements · 2024-04-26T12:58:51Z

Hi @JoanFM. Thank you for your working with me on this. I created a python 3.8 poetry environment for better coverage. black appears to have changed its mind between versions with regards to the ellipsis .... Running from 3.8, it moved it to the following line, where the day before, my 3.11 black moved it to the same one! :) I changed it to pass. ¯_(ツ)_/¯

-    class RuntimeConfig(BaseDocIndex.RuntimeConfig): ...
+    class RuntimeConfig(BaseDocIndex.RuntimeConfig):
+        ...

And now

-    class RuntimeConfig(BaseDocIndex.RuntimeConfig): ...
+    class RuntimeConfig(BaseDocIndex.RuntimeConfig):
+        pass

I'll turn to the typing issues, mypy with py3.8.

Signed-off-by: Casey Clements <[email protected]>

caseyclements · 2024-04-26T13:15:47Z

I'm sorry that I missed list to List. I recognized the required change while updating tuple, but it was a busy morning. I pushed a new commit.

codecov · 2024-04-26T13:20:50Z

Codecov Report

Attention: Patch coverage is 41.11675% with 116 lines in your changes are missing coverage. Please review.

Project coverage is 44.75%. Comparing base (febbdc4) to head (5c01811).

❗ Current head 5c01811 differs from pull request most recent head 3b03f06. Consider uploading reports for the commit 3b03f06 to get more accurate results

Files	Patch %	Lines
docarray/index/backends/mongodb_atlas.py	40.10%	115 Missing ⚠️
docarray/index/__init__.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1883       +/-   ##
===========================================
- Coverage   84.69%   44.75%   -39.95%     
===========================================
  Files         136      137        +1     
  Lines        9263     9459      +196     
===========================================
- Hits         7845     4233     -3612     
- Misses       1418     5226     +3808

Flag	Coverage Δ
docarray	`44.75% <41.11%> (-39.95%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

caseyclements · 2024-04-26T16:04:20Z

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

JoanFM · 2024-04-26T17:21:38Z

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

no need to worry about code coverage

caseyclements · 2024-04-26T17:36:25Z

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

no need to worry about code coverage

Cool. What remains then?

In the next two weeks (I'm in London this coming one) when we add the QueryBuilder, we'll also set up the testing on your end. We are already running on our CI against Atlas on our end. Maybe we could set up a face-to-face meeting in a couple weeks. We can optimize to get the most of MongoDB's API once we know the scope of the use cases for the Indexes, and the data types.

JoanFM · 2024-04-26T17:44:32Z

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

no need to worry about code coverage

Cool. What remains then?

In the next two weeks (I'm in London this coming one) when we add the QueryBuilder, we'll also set up the testing on your end. We are already running on our CI against Atlas on our end. Maybe we could set up a face-to-face meeting in a couple weeks. We can optimize to get the most of MongoDB's API once we know the scope of the use cases for the Indexes, and the data types.

there seems to be a test timing out but not sure if it comes from your changes.

So what is the plan for the upcoming changes feom your side?

JoanFM requested changes Apr 24, 2024

View reviewed changes

docarray/index/backends/mongodb_atlas.py Outdated Show resolved Hide resolved

docarray/index/backends/mongodb_atlas.py Outdated Show resolved Hide resolved

docarray/index/backends/mongodb_atlas.py Show resolved Hide resolved

WaVEV and others added 28 commits April 24, 2024 12:36

Mongo backend index initial.

98dda1c

Signed-off-by: Casey Clements <[email protected]>

find unit test.

3150375

Signed-off-by: Casey Clements <[email protected]>

unit test of configuration, find, del, get and persist data.

386b5ab

Signed-off-by: Casey Clements <[email protected]>

query builder refactor.

08cbc90

Signed-off-by: Casey Clements <[email protected]>

Rename function _collect_query_args_required_args => _collect_query_r…

6d45d1f

…equired_args Signed-off-by: Casey Clements <[email protected]>

filter by parent id and query builder refactor.

12c7216

Signed-off-by: Casey Clements <[email protected]>

Add nested schema fixture.

df0e461

Signed-off-by: Casey Clements <[email protected]>

refactor test find, moving text and filter test.

f0ff913

Signed-off-by: Casey Clements <[email protected]>

Add test filter.

028f1be

Signed-off-by: Casey Clements <[email protected]>

Add test text search.

d2f1f03

Signed-off-by: Casey Clements <[email protected]>

Add query builder test.

2cafd61

Signed-off-by: Casey Clements <[email protected]>

Set collection name by schema name.

61b9943

Signed-off-by: Casey Clements <[email protected]>

Fix unit test.

b29290e

Signed-off-by: Casey Clements <[email protected]>

Add index name property.

b3b35cf

Signed-off-by: Casey Clements <[email protected]>

set scope for mongo_fixture_env.

fdbb334

Signed-off-by: Casey Clements <[email protected]>

add subindex test.

4cb814d

Signed-off-by: Casey Clements <[email protected]>

subindex find.

00de14f

Signed-off-by: Casey Clements <[email protected]>

Update readme and manage exception when an Index is missing.

2c083f9

Signed-off-by: Casey Clements <[email protected]>

test find without index.

b52cd14

Signed-off-by: Casey Clements <[email protected]>

refactor fixtures import.

63ede54

Signed-off-by: Casey Clements <[email protected]>

Importing all fixtures to avoid importing dependencies.

1b1ab1c

Signed-off-by: Casey Clements <[email protected]>

fix poetry lock.

04cd2a6

Signed-off-by: Casey Clements <[email protected]>

QueryBuilder: hybrid search implementation.

de1d74f

Signed-off-by: Casey Clements <[email protected]>

Hybrid search: fix score.

df57c22

Signed-off-by: Casey Clements <[email protected]>

Refactor: project now takes extra fields to project scores.

53e93f4

Signed-off-by: Casey Clements <[email protected]>

Moved query builder to query-builder-implementation branch.

addac74

Signed-off-by: Casey Clements <[email protected]>

Refactor and clean up.

95bf4f3

Signed-off-by: Casey Clements <[email protected]>

Rename MongoAtlasDocumentIndex to MongoDBAtlasDocumentIndex

78a670d

Signed-off-by: Casey Clements <[email protected]>

caseyclements added 2 commits April 24, 2024 12:36

Update env variable name

09c6e70

Signed-off-by: Casey Clements <[email protected]>

Add MongoDB Atlas setup instructions.

b5b73d1

Signed-off-by: Casey Clements <[email protected]>

caseyclements force-pushed the mongodb-atlas-backend branch from d74b174 to b5b73d1 Compare April 24, 2024 16:36

caseyclements added 2 commits April 24, 2024 15:38

Updates in response to maintainer comments

62be05c

Signed-off-by: Casey Clements <[email protected]>

Added detailed README in tests/ for setup of indexes

aed03a2

Signed-off-by: Casey Clements <[email protected]>

JoanFM requested changes Apr 25, 2024

View reviewed changes

docarray/index/backends/mongodb_atlas.py Outdated Show resolved Hide resolved

docarray/index/backends/mongodb_atlas.py Outdated Show resolved Hide resolved

caseyclements added 2 commits April 25, 2024 13:17

black formatted

9df653b

Signed-off-by: Casey Clements <[email protected]>

Changed typing tuple > Tuple for python<=3.8

c2178de

Signed-off-by: Casey Clements <[email protected]>

black formatting ellipsis to pass

30d2b40

Signed-off-by: Casey Clements <[email protected]>

Updated typing of Lists for backward compatibility

5c01811

Signed-off-by: Casey Clements <[email protected]>

Merge branch 'main' into mongodb-atlas-backend

3b03f06

JoanFM approved these changes Apr 26, 2024

View reviewed changes

JoanFM merged commit f5c9ab0 into docarray:main Apr 29, 2024

blink1073 mentioned this pull request Jul 30, 2024

Request for support for MongoDB Atlas Vector Search as DocIndex #1768

Closed

7 tasks

Conversation

caseyclements commented Apr 23, 2024

Description

Simple Usage

Supported Functionality

Integration Tests and documentation

Coming soon

Uh oh!

JoanFM commented Apr 24, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prakul commented Apr 24, 2024

Uh oh!

JoanFM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JoanFM commented Apr 26, 2024

Uh oh!

caseyclements commented Apr 26, 2024

Uh oh!

caseyclements commented Apr 26, 2024

Uh oh!

codecov bot commented Apr 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

caseyclements commented Apr 26, 2024

Uh oh!

JoanFM commented Apr 26, 2024

Uh oh!

caseyclements commented Apr 26, 2024

Uh oh!

JoanFM commented Apr 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Apr 26, 2024 •

edited

Loading