Skip to content

fix: milvus _get_docs_by_ids#859

Merged
JohannesMessner merged 4 commits intomainfrom
fix-milvus-getbyids
Nov 29, 2022
Merged

fix: milvus _get_docs_by_ids#859
JohannesMessner merged 4 commits intomainfrom
fix-milvus-getbyids

Conversation

@AnneYang720
Copy link
Copy Markdown
Contributor

@AnneYang720 AnneYang720 commented Nov 28, 2022

Goals:

This PR is related to issue #857

  • codes
  • add related test

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Nov 28, 2022

Codecov Report

Base: 84.78% // Head: 86.36% // Increases project coverage by +1.58% 🎉

Coverage data is based on head (57f0a4c) compared to base (86c4cd4).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #859      +/-   ##
==========================================
+ Coverage   84.78%   86.36%   +1.58%     
==========================================
  Files         138      138              
  Lines        7117     7116       -1     
==========================================
+ Hits         6034     6146     +112     
+ Misses       1083      970     -113     
Flag Coverage Δ
docarray 86.36% <100.00%> (+1.58%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/array/storage/milvus/getsetdel.py 97.05% <100.00%> (-0.05%) ⬇️
docarray/array/storage/qdrant/getsetdel.py 80.51% <0.00%> (+1.29%) ⬆️
docarray/array/storage/redis/getsetdel.py 97.14% <0.00%> (+1.42%) ⬆️
docarray/array/storage/base/getsetdel.py 91.39% <0.00%> (+1.98%) ⬆️
docarray/array/storage/sqlite/getsetdel.py 97.77% <0.00%> (+2.22%) ⬆️
docarray/document/mixins/porting.py 94.36% <0.00%> (+2.81%) ⬆️
docarray/array/storage/base/helper.py 90.56% <0.00%> (+3.77%) ⬆️
docarray/array/storage/elastic/getsetdel.py 100.00% <0.00%> (+4.76%) ⬆️
docarray/array/storage/annlite/getsetdel.py 100.00% <0.00%> (+4.87%) ⬆️
docarray/array/storage/weaviate/getsetdel.py 100.00% <0.00%> (+7.50%) ⬆️
... and 8 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Copy Markdown
Member

@JohannesMessner JohannesMessner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - just one small performance concern.

# sort output docs according to input id sorting
id_to_index = {id_: i for i, id_ in enumerate(ids)}
return DocumentArray(sorted(docs, key=lambda d: id_to_index[d.id]))
return DocumentArray([docs[d] for d in ids])
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep the dict-based approach, but have every id_ point to a list of positions, and leverage that in key=... somehow?
The reason it was done this way is performance, we want to gather the id-to-position mapping only once, and then delegate everything else to sorted(), which leverages a fist implementation in C.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't think of a python built-in function. sort doesn't make the list longer than the original one.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I see, then let's keep it.

@AnneYang720 AnneYang720 marked this pull request as ready for review November 29, 2022 06:56
@JohannesMessner JohannesMessner merged commit 67209b8 into main Nov 29, 2022
@JohannesMessner JohannesMessner deleted the fix-milvus-getbyids branch November 29, 2022 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants