Skip to content

fix: native milvus len #885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JohannesMessner merged 3 commits into from
Dec 5, 2022
Merged

fix: native milvus len #885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d2bfe24
feat: native len for milvus
JohannesMessner Dec 2, 2022
e8422c8
fix: make implementing len non-optional
JohannesMessner Dec 2, 2022
4f32ec6
fix: only use milvus len as fallback
JohannesMessner Dec 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docarray/array/storage/base/seqlike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ def __eq__(self, other):
...

def __len__(self):
return len(self._offset2ids)
...

def __iter__(self) -> Iterator['Document']:
for _id in self._offset2ids:
Expand Down
15 changes: 14 additions & 1 deletion docarray/array/storage/milvus/seqlike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import Iterable, Iterator, Union, TYPE_CHECKING
from docarray.array.storage.base.seqlike import BaseSequenceLikeMixin
from docarray.array.storage.milvus.backend import _batch_list
from docarray.array.storage.milvus.backend import _batch_list, _always_true_expr
from docarray import Document


Expand Down Expand Up @@ -56,3 +56,16 @@ def _extend(self, values: Iterable['Document'], **kwargs):
payload = self._docs_to_milvus_payload(docs_batch)
self._collection.insert(payload, **kwargs)
self._offset2ids.extend([doc.id for doc in docs_batch])

def __len__(self):
if self._list_like:
return len(self._offset2ids)
else:
# Milvus has no native way to get num of entities
# so only use it as fallback option
with self.loaded_collection():
res = self._collection.query(
expr=_always_true_expr('document_id'),
output_fields=['document_id'],
)
return len(res)
7 changes: 3 additions & 4 deletions docs/advanced/document-store/extend.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,9 @@ class SequenceLikeMixin(BaseSequenceLikeMixin):
def __add__(self, other: Union['Document', Iterable['Document']]):
...

def __len__(self):
...

def insert(self, index: int, value: 'Document'):
# Optional. By default, this will add a new item and update offset2id
# if you want to customize this, make sure to handle offset2id
Expand All @@ -158,10 +161,6 @@ class SequenceLikeMixin(BaseSequenceLikeMixin):
# Optional. Override this if you have better implementation than appending one by one
...

def __len__(self):
# Optional. By default, this will rely on offset2id to get the length
...

def __iter__(self) -> Iterator['Document']:
# Optional. By default, this will rely on offset2id to iterate
...
Expand Down