* fix(plot): be robust against non-existing subindices

* chore: add file structure

* feat: first draft of backend implementation

* feat: docarray can now connect to running milvus database

* feat: implement basics of getsetdel and seqlike

* fix: type hint

* fix: saving and loading offset2ids

* feat: first implementation of vector search

* refactor: declare static methods

* feat: add consistency level as a configuration parameter

* feat: change default consistency to session

* refactor: in clear_storage, drop and re-create collection

* feat: implement filter and hybrid search

* feat: implement columns feature

* feat: allow consistency level to be passed to extend, append, insert

* fix: fix columns feature in milvus

* test: add test for milvus columns feature

* fix: import types for type hints

* fix: add pymilvus requirement

* test: fix docker compose for milvus

* test: add milvus to test construct

* fix: backend naming

* fix: sort returned docs when accessing by id

* test: add milvus to sequence tests

* fix: raise keyerror on non existing id

* refactor: use context managar to laod and release collections

* fix: map embeddings to np before inserting

* test: add milvus to advanced indexing tests

* test: add milvus to test pull out

* fix: deleting to size zero

* fix: when accessing with no ids, return empty da instead of rasing

* test: add milvus to content tests

* test: add milvus to del tests

* test: add milvus to embed tests

* test: add milvus to test empty

* test: add milvus to test eval class

* test: add milvus to test find

* test: add milvus to getset tests

* test: add milvus to all remaining tests

* feat: load and release collection in context manager

* test: use context manager to speed up milvus tests

* test: add milvus to test plot

* test: fix plot tests for milvus

* test: fix multimodal find test

* test: fix test embed

* test: use context manager to speed up milvus

* fix: implement state methods to enable pickling

* test: fix more tests

* feat: add overloaded milvus init

* test: fix even more tests

* refactor: remove some comments

* test: use context manager even more

* test: moar context manager usage

* test: fix test fixture input

* test: remove milvus from test that it can't handle

* test: remove milvus from one more test

* refactor: better mechanism for automatic collection loading

* docs: add docs section for milvus

* chore: remove comment

* fix: enforce limit for filter only queries

* docs: fix typo

* docs: add milvus to comparison and add docstring

* feat: bulk extend

* docs: document advanced milvus options

* feat: allo passing of kwargs to insert

* chore: remove comment

* chore: update accepted array types

* test: add milvus specific tests

* docs: apply changes from code review

Co-authored-by: Nicholas Dunham <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* docs: update docs/advanced/document-store/milvus.md

Co-authored-by: Nicholas Dunham <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* refactor: update type hint

Co-authored-by: samsja <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* refactor: apply suggestions from code review

* docs: clarify docstring

Signed-off-by: Johannes Messner <[email protected]>

* docs: fix fautly docstring

Signed-off-by: Johannes Messner <[email protected]>

* refactor: another round of review changes

* fix: set consistency level for offset id loading

* fix: set stricter defaut consistency level

* perf: optimize sorting of retrieved documents

* docs: document loading context manager

* refactor: find unboxing done by base class

* feat: add batching

* test: add test for batch size

* test: restart milvus if it breaks

* test: dont remove-orphans where it is not needed

* feat: add ability to disble list like behaviour

Signed-off-by: Johannes Messner <[email protected]>

* ci: increase timeout to see how long it really takes

* docs: apply suggestions from code review

Co-authored-by: AlaeddineAbdessalem <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* ci: change timeouts

Signed-off-by: Johannes Messner <[email protected]>

* chore: pin pymilvus version, only for testing if it fixes it

Signed-off-by: Johannes Messner <[email protected]>

* test: pin pymilvus test dependency

Signed-off-by: Johannes Messner <[email protected]>

* chore: loosen pymilvus version restriction

Signed-off-by: Johannes Messner <[email protected]>

* ci: add prints for debugging

Signed-off-by: Johannes Messner <[email protected]>

* ci: add more debugging prints

Signed-off-by: Johannes Messner <[email protected]>

* ci: remove prints

Signed-off-by: Johannes Messner <[email protected]>

Signed-off-by: Johannes Messner <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>
Co-authored-by: Nicholas Dunham <[email protected]>
Co-authored-by: samsja <[email protected]>
Co-authored-by: AlaeddineAbdessalem <[email protected]>

* fix(plot): be robust against non-existing subindices

* chore: add file structure

* feat: first draft of backend implementation

* feat: docarray can now connect to running milvus database

* feat: implement basics of getsetdel and seqlike

* fix: type hint

* fix: saving and loading offset2ids

* feat: first implementation of vector search

* refactor: declare static methods

* feat: add consistency level as a configuration parameter

* feat: change default consistency to session

* refactor: in clear_storage, drop and re-create collection

* feat: implement filter and hybrid search

* feat: implement columns feature

* feat: allow consistency level to be passed to extend, append, insert

* fix: fix columns feature in milvus

* test: add test for milvus columns feature

* fix: import types for type hints

* fix: add pymilvus requirement

* test: fix docker compose for milvus

* test: add milvus to test construct

* fix: backend naming

* fix: sort returned docs when accessing by id

* test: add milvus to sequence tests

* fix: raise keyerror on non existing id

* refactor: use context managar to laod and release collections

* fix: map embeddings to np before inserting

* test: add milvus to advanced indexing tests

* test: add milvus to test pull out

* fix: deleting to size zero

* fix: when accessing with no ids, return empty da instead of rasing

* test: add milvus to content tests

* test: add milvus to del tests

* test: add milvus to embed tests

* test: add milvus to test empty

* test: add milvus to test eval class

* test: add milvus to test find

* test: add milvus to getset tests

* test: add milvus to all remaining tests

* feat: load and release collection in context manager

* test: use context manager to speed up milvus tests

* test: add milvus to test plot

* test: fix plot tests for milvus

* test: fix multimodal find test

* test: fix test embed

* test: use context manager to speed up milvus

* fix: implement state methods to enable pickling

* test: fix more tests

* feat: add overloaded milvus init

* test: fix even more tests

* refactor: remove some comments

* test: use context manager even more

* test: moar context manager usage

* test: fix test fixture input

* test: remove milvus from test that it can't handle

* test: remove milvus from one more test

* refactor: better mechanism for automatic collection loading

* docs: add docs section for milvus

* chore: remove comment

* fix: enforce limit for filter only queries

* docs: fix typo

* docs: add milvus to comparison and add docstring

* feat: bulk extend

* docs: document advanced milvus options

* feat: allo passing of kwargs to insert

* chore: remove comment

* chore: update accepted array types

* test: add milvus specific tests

* docs: apply changes from code review

Co-authored-by: Nicholas Dunham <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* docs: update docs/advanced/document-store/milvus.md

Co-authored-by: Nicholas Dunham <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* refactor: update type hint

Co-authored-by: samsja <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* refactor: apply suggestions from code review

* docs: clarify docstring

Signed-off-by: Johannes Messner <[email protected]>

* docs: fix fautly docstring

Signed-off-by: Johannes Messner <[email protected]>

* refactor: another round of review changes

* fix: set consistency level for offset id loading

* fix: set stricter defaut consistency level

* perf: optimize sorting of retrieved documents

* docs: document loading context manager

* refactor: find unboxing done by base class

* feat: add batching

* test: add test for batch size

* test: restart milvus if it breaks

* test: dont remove-orphans where it is not needed

* feat: add ability to disble list like behaviour

Signed-off-by: Johannes Messner <[email protected]>

* ci: increase timeout to see how long it really takes

* docs: apply suggestions from code review

Co-authored-by: AlaeddineAbdessalem <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>

* ci: change timeouts

Signed-off-by: Johannes Messner <[email protected]>

* chore: pin pymilvus version, only for testing if it fixes it

Signed-off-by: Johannes Messner <[email protected]>

* test: pin pymilvus test dependency

Signed-off-by: Johannes Messner <[email protected]>

* chore: loosen pymilvus version restriction

Signed-off-by: Johannes Messner <[email protected]>

* ci: add prints for debugging

Signed-off-by: Johannes Messner <[email protected]>

* ci: add more debugging prints

Signed-off-by: Johannes Messner <[email protected]>

* ci: remove prints

Signed-off-by: Johannes Messner <[email protected]>

Signed-off-by: Johannes Messner <[email protected]>
Signed-off-by: Johannes Messner <[email protected]>
Co-authored-by: Nicholas Dunham <[email protected]>
Co-authored-by: samsja <[email protected]>
Co-authored-by: AlaeddineAbdessalem <[email protected]>