refactor: doc index structure by jupyterjazz · Pull Request #1266 · docarray/docarray

jupyterjazz · 2023-03-21T16:05:46Z

Two suggestions:

shorter imports

Since developers will most likely only interact with specific backend implementations, i.e. will only import those classes, let's make the import shorter.

Before:

from docarray.doc_index.backends.hnswlib_doc_index import HnswDocumentIndex

Now:

from docarray.index import HnswDocumentIndex

less verbose file names

Being inside doc_index module, it's already clear that abstract_doc_index.py (same with hnswlib_doc_index.py) is about doc_index, so let's make it less verbose.

Before:

doc_index/abstract_doc_index.py
doc_index/backend/hnswlib_doc_index.py

Now:

index/abstract.py
index/backend/hnswlib.py

Signed-off-by: jupyterjazz <[email protected]>

docarray/doc_index/__init__.py

samsja

we need to add __all__ to init .py

JohannesMessner · 2023-03-21T16:32:20Z

from docarray.doc_index import HnswDocumentIndex

I think we should also rename the doc_index part, i find it a bit verbose and i don't like having an underscore. Just index might be not so clear though, any other suggestions?

samsja · 2023-03-21T16:34:15Z

from docarray.doc_index import HnswDocumentIndex
I think we should also rename the doc_index part, i find it a bit verbose and i don't like having an underscore. Just index might be not so clear though, any other suggestions?

fully agree, what about :

from docarray.vectordb import HnswDocumentIndex

from docarray.vectorDB import HnswDocumentIndex

from docarray.index import HnswDocumentIndex

from docarray.search import HnswDocumentIndex

from docarray.db import HnswDocumentIndex

jupyterjazz · 2023-03-21T16:40:53Z

I'm also not a big fan of doc_index.

I like docarray.index, it's simple, but maybe a bit less informative though.

Regarding other options Sami listed, I think if we use something like docarray.vectordb we should also adjust class names similarly for it to be more intuitive, e.g. HnswVectorDB instead of HnswDocumentIndex

Signed-off-by: jupyterjazz <[email protected]>

jupyterjazz · 2023-03-22T09:32:41Z

@samsja @JohannesMessner
Let's go with

from docarray.index import HnswDocumentIndex

for now and if it doesn't suit docarray we can refine it later. wdyt?

samsja · 2023-03-22T09:36:55Z

makes sense

Signed-off-by: jupyterjazz <[email protected]>

github-actions · 2023-03-22T11:02:55Z

📝 Docs are deployed on https://ft-refactor-doc-index-imports--jina-docs.netlify.app 🎉

* refactor: doc index structure Signed-off-by: jupyterjazz <[email protected]> * fix: import unused error Signed-off-by: jupyterjazz <[email protected]> * fix: run black Signed-off-by: jupyterjazz <[email protected]> * refactor: remove noqa Signed-off-by: jupyterjazz <[email protected]> * refactor: doc_index to index Signed-off-by: jupyterjazz <[email protected]> * refactor: small changes Signed-off-by: jupyterjazz <[email protected]> * refactor: readme Signed-off-by: jupyterjazz <[email protected]> * refactor: add init file Signed-off-by: jupyterjazz <[email protected]> * refactor: move docindex outside integration tests Signed-off-by: jupyterjazz <[email protected]> * refactor: adjust config paths Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]>

* refactor: rename predefined documents (#1208) * refactor: rename Image to ImageDoc Signed-off-by: samsja <[email protected]> * refactor: rename Text to TextDoc Signed-off-by: samsja <[email protected]> * refactor: rename Audio to AudioDoc Signed-off-by: samsja <[email protected]> * refactor: rename Video to VideoDOc Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: support other text formats (#1207) * feat: add validation function and filetypes feat: add validation function and filetypes fix: formatting Signed-off-by: [email protected] <[email protected]> * feat: added extension validation tests for TextURL Signed-off-by: [email protected] <[email protected]> * fix: fix for tests and PR codereview Signed-off-by: [email protected] <[email protected]> * fix: added internet mark to json dump test for texturl Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: refactor query builder (#1213) * refactor: query builder Signed-off-by: Johannes Messner <[email protected]> * docs: add guidance for query builder Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat(index): automatically convert a dict to Document (#1215) * feat: add _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * fix: str cut Signed-off-by: AnneY <[email protected]> * feat: _get_items add return type Signed-off-by: AnneY <[email protected]> * fix: mypy Signed-off-by: AnneY <[email protected]> * test: add test for _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * refactor: simplify _convert_to_doc_list Signed-off-by: AnneY <[email protected]> * docs: add docstring for _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * feat: filter find text add return type Signed-off-by: AnneY <[email protected]> * fix: type judge Signed-off-by: AnneY <[email protected]> * refactor: change search_field default value Signed-off-by: AnneY <[email protected]> * fix: always check length Signed-off-by: AnneY <[email protected]> * refactor: create private result class Signed-off-by: AnneY <[email protected]> * refactor: seperate helper method Signed-off-by: AnneY <[email protected]> * refactor: create private result class Signed-off-by: AnneY <[email protected]> --------- Signed-off-by: AnneY <[email protected]> Signed-off-by: Anne Yang <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <[email protected]> * fix: remove impl Signed-off-by: samsja <[email protected]> * refactor: move some tests Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add document view Signed-off-by: samsja <[email protected]> * feat: add tests Signed-off-by: samsja <[email protected]> * refactor: rename storage to column storage Signed-off-by: samsja <[email protected]> * refactor: order of function in da stacked Signed-off-by: samsja <[email protected]> * refactor: add inner doc in test Signed-off-by: samsja <[email protected]> * refactor: rename storage to column Signed-off-by: samsja <[email protected]> * feat: add from storage Signed-off-by: samsja <[email protected]> * refactor: rename file Signed-off-by: samsja <[email protected]> * refactor: storage take columns as init Signed-off-by: samsja <[email protected]> * feat: add back getitiem Signed-off-by: samsja <[email protected]> * fix: fix getatr view problem Signed-off-by: samsja <[email protected]> * feat: add get array attribute Signed-off-by: samsja <[email protected]> * feat: add setitem back Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * feat: add to Signed-off-by: samsja <[email protected]> * feat: add ttests Signed-off-by: samsja <[email protected]> * feat: add unstack Signed-off-by: samsja <[email protected]> * feat: add validation da set atr Signed-off-by: samsja <[email protected]> * feat: add da and any Signed-off-by: samsja <[email protected]> * feat: remove deleitem overload Signed-off-by: samsja <[email protected]> * fix: remove context manager Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: move tests Signed-off-by: samsja <[email protected]> * fix: add tidi Signed-off-by: samsja <[email protected]> * fix: fix copy in ndarray mypy Signed-off-by: samsja <[email protected]> * feat: add indexing sequence mixin Signed-off-by: samsja <[email protected]> * feat: add list index class Signed-off-by: samsja <[email protected]> * fix: add cast from tuple to list Signed-off-by: samsja <[email protected]> * fix: fix indexing test Signed-off-by: samsja <[email protected]> * feat: add back traversle flat Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: remove unstack mode Signed-off-by: samsja <[email protected]> * feat: add proto Signed-off-by: samsja <[email protected]> * feat: add init mixin Signed-off-by: samsja <[email protected]> * fix: add back staticmethod Signed-off-by: samsja <[email protected]> * fix: add len to mixin Signed-off-by: samsja <[email protected]> * fix: add del to index seq Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <[email protected]> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <[email protected]> * refactor: move docs to column storage to da stack Signed-off-by: samsja <[email protected]> * refactor: remove useless arg Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix nested array in da stack Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: remove type hint from setitiem Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix getitem Signed-off-by: samsja <[email protected]> * merge: merge featrewrite v2 Signed-off-by: samsja <[email protected]> * fix: black does nto format pb file Signed-off-by: samsja <[email protected]> * fix: remove comment Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix proto3 Signed-off-by: samsja <[email protected]> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <[email protected]> * fix: fix del Signed-off-by: samsja <[email protected]> * fix: fix scalar value Signed-off-by: samsja <[email protected]> * fix: remove useless test Signed-off-by: samsja <[email protected]> * feat: raise error when docs are empty Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix type hint Signed-off-by: samsja <[email protected]> * fix: fix settatr Signed-off-by: samsja <[email protected]> * fix: apply suggestion Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Signed-off-by: samsja <[email protected]> * fix: docstring Signed-off-by: samsja <[email protected]> * fix: last fix hopefully Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: add comments Signed-off-by: samsja <[email protected]> * fix: remove classvar Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: remvoe proprety Signed-off-by: samsja <[email protected]> * fix: bring back proprety Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> Signed-off-by: samsja <[email protected]> Co-authored-by: Jackmin801 <[email protected]> Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Signed-off-by: RStar2022 <[email protected]> * add equal function Signed-off-by: RStar2022 <[email protected]> * Add equality function Signed-off-by: RStar2022 <[email protected]> * Add equal to array and document Signed-off-by: RStar2022 <[email protected]> * changes a bit Signed-off-by: RStar2022 <[email protected]> * add __eq__ function in abstract_comp_backend and add test_array Signed-off-by: RStar2022 <[email protected]> * feat: ellipsis in tensor shape definition (#1228) * feat: ellipsis in tensor shape definition Signed-off-by: jupyterjazz <[email protected]> * fix: type hint Signed-off-by: jupyterjazz <[email protected]> * refactor: add tests for other tensors Signed-off-by: jupyterjazz <[email protected]> * test: fix tensorflow test Signed-off-by: jupyterjazz <[email protected]> * docs: update docstrings with examples Signed-off-by: jupyterjazz <[email protected]> * docs: clarify needed dimensions Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * chore: update contributing guideline (#1240) * chore: update contrubing guideline Signed-off-by: samsja <[email protected]> * fix: mark dl test slow Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix equality funtion Signed-off-by: RStar2022 <[email protected]> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <[email protected]> * fix: remove impl Signed-off-by: samsja <[email protected]> * refactor: move some tests Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add document view Signed-off-by: samsja <[email protected]> * feat: add tests Signed-off-by: samsja <[email protected]> * refactor: rename storage to column storage Signed-off-by: samsja <[email protected]> * refactor: order of function in da stacked Signed-off-by: samsja <[email protected]> * refactor: add inner doc in test Signed-off-by: samsja <[email protected]> * refactor: rename storage to column Signed-off-by: samsja <[email protected]> * feat: add from storage Signed-off-by: samsja <[email protected]> * refactor: rename file Signed-off-by: samsja <[email protected]> * refactor: storage take columns as init Signed-off-by: samsja <[email protected]> * feat: add back getitiem Signed-off-by: samsja <[email protected]> * fix: fix getatr view problem Signed-off-by: samsja <[email protected]> * feat: add get array attribute Signed-off-by: samsja <[email protected]> * feat: add setitem back Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * feat: add to Signed-off-by: samsja <[email protected]> * feat: add ttests Signed-off-by: samsja <[email protected]> * feat: add unstack Signed-off-by: samsja <[email protected]> * feat: add validation da set atr Signed-off-by: samsja <[email protected]> * feat: add da and any Signed-off-by: samsja <[email protected]> * feat: remove deleitem overload Signed-off-by: samsja <[email protected]> * fix: remove context manager Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: move tests Signed-off-by: samsja <[email protected]> * fix: add tidi Signed-off-by: samsja <[email protected]> * fix: fix copy in ndarray mypy Signed-off-by: samsja <[email protected]> * feat: add indexing sequence mixin Signed-off-by: samsja <[email protected]> * feat: add list index class Signed-off-by: samsja <[email protected]> * fix: add cast from tuple to list Signed-off-by: samsja <[email protected]> * fix: fix indexing test Signed-off-by: samsja <[email protected]> * feat: add back traversle flat Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: remove unstack mode Signed-off-by: samsja <[email protected]> * feat: add proto Signed-off-by: samsja <[email protected]> * feat: add init mixin Signed-off-by: samsja <[email protected]> * fix: add back staticmethod Signed-off-by: samsja <[email protected]> * fix: add len to mixin Signed-off-by: samsja <[email protected]> * fix: add del to index seq Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <[email protected]> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <[email protected]> * refactor: move docs to column storage to da stack Signed-off-by: samsja <[email protected]> * refactor: remove useless arg Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix nested array in da stack Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: remove type hint from setitiem Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix getitem Signed-off-by: samsja <[email protected]> * merge: merge featrewrite v2 Signed-off-by: samsja <[email protected]> * fix: black does nto format pb file Signed-off-by: samsja <[email protected]> * fix: remove comment Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix proto3 Signed-off-by: samsja <[email protected]> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <[email protected]> * fix: fix del Signed-off-by: samsja <[email protected]> * fix: fix scalar value Signed-off-by: samsja <[email protected]> * fix: remove useless test Signed-off-by: samsja <[email protected]> * feat: raise error when docs are empty Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix type hint Signed-off-by: samsja <[email protected]> * fix: fix settatr Signed-off-by: samsja <[email protected]> * fix: apply suggestion Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Signed-off-by: samsja <[email protected]> * fix: docstring Signed-off-by: samsja <[email protected]> * fix: last fix hopefully Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: add comments Signed-off-by: samsja <[email protected]> * fix: remove classvar Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: remvoe proprety Signed-off-by: samsja <[email protected]> * fix: bring back proprety Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> Signed-off-by: samsja <[email protected]> Co-authored-by: Jackmin801 <[email protected]> Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> * chore: update contribution guideline (#1247) * chore: update contribution guideline Signed-off-by: samsja <[email protected]> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: samsja <[email protected]> * feat: add poetry add Signed-off-by: samsja <[email protected]> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: RStar2022 <[email protected]> * make default search field empty string (#1249) Signed-off-by: azayz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * delete import numpy Signed-off-by: RStar2022 <[email protected]> * fix key name Signed-off-by: RStar2022 <[email protected]> * feat:changes-equality-operation Signed-off-by: RStar2022 <[email protected]> * feat:add if Signed-off-by: RStar2022 <[email protected]> * docs: add explanation about id field (#1242) * docs: add explanation about id field Signed-off-by: Johannes Messner <[email protected]> * docs: update docs/tutorials/add_doc_index.md Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: Johannes Messner <[email protected]> Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: shift to mkdocs (#1244) * chore: remove sphinx and mkdocs Signed-off-by: samsja <[email protected]> * fix: add DocumentArrayStacked to init py Signed-off-by: samsja <[email protected]> * feat: add mkdocs Signed-off-by: samsja <[email protected]> * feat: apply alex setting Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * fix: fix import Signed-off-by: samsja <[email protected]> * fix: fix import Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * feat: uses sphinx style Signed-off-by: samsja <[email protected]> * feat: udpate docs Signed-off-by: samsja <[email protected]> * feat: add inherited menbers Signed-off-by: samsja <[email protected]> * feat: add more Signed-off-by: samsja <[email protected]> * fix: code block syntax highlight in docstring Signed-off-by: Alex C-G <[email protected]> * feat: add typing Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: Alex C-G <[email protected]> Co-authored-by: Alex C-G <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix:equal func Signed-off-by: RStar2022 <[email protected]> * refactor: rename filter to filter_docs to avoid shadowing of filtern (#1257) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: ad user defined mapping for python type to db type (#1252) * feat: user defined mapping for python type to db type Signed-off-by: anna-charlotte <[email protected]> * feat: check if col_type available Signed-off-by: anna-charlotte <[email protected]> * test: add test for base classes Signed-off-by: anna-charlotte <[email protected]> * fix: clean up Signed-off-by: anna-charlotte <[email protected]> * fix: test Signed-off-by: anna-charlotte <[email protected]> * docs: add documentation for db type and python type Signed-off-by: anna-charlotte <[email protected]> * docs: add doumentation for runtime config Signed-off-by: anna-charlotte <[email protected]> * fix: add and test illegal col types Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix:tensor type func Signed-off-by: RStar2022 <[email protected]> * fix: comp_backend code delete Signed-off-by: RStar2022 <[email protected]> * feat(index): index data with union types (#1220) * refactor: split flattening into separate method Signed-off-by: Johannes Messner <[email protected]> * refactor: don't build column info during schema check Signed-off-by: Johannes Messner <[email protected]> * feat: allos unions and optional in indexed data Signed-off-by: Johannes Messner <[email protected]> * fix: mypy Signed-off-by: Johannes Messner <[email protected]> * fix: mypy Signed-off-by: Johannes Messner <[email protected]> * fix: import from typing inspect instead of typing Signed-off-by: Johannes Messner <[email protected]> * fix: equality and hash for parametrized tensors Signed-off-by: Johannes Messner <[email protected]> * test: add test for flatten docs Signed-off-by: Johannes Messner <[email protected]> * refactor: apply suggestions Signed-off-by: Johannes Messner <[email protected]> * docs: better docstrings Signed-off-by: Johannes Messner <[email protected]> * refactor: use construct to create docarray Signed-off-by: Johannes Messner <[email protected]> * fix: check for nonetype Signed-off-by: Johannes Messner <[email protected]> * fix: none in equals check Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix eq Signed-off-by: RStar2022 <[email protected]> * fix: move test to integration test (#1260) * fix: move test to integration test Signed-off-by: samsja <[email protected]> * fix: use a different runner for doc index Signed-off-by: samsja <[email protected]> * fix: use a different runner for doc index Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: disable pycharm da property detection (#1262) * fix: disable unresolved attr detection for da in pycharm Signed-off-by: anna-charlotte <[email protected]> * fix: add docstring Signed-off-by: anna-charlotte <[email protected]> * fix: call super instead of pass Signed-off-by: anna-charlotte <[email protected]> * fix: getattr Signed-off-by: anna-charlotte <[email protected]> * fix: getattribute Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: add minimal logger (#1254) * feat: add minimal logger Signed-off-by: jupyterjazz <[email protected]> * docs: add an example in contributing Signed-off-by: jupyterjazz <[email protected]> * docs: try different format Signed-off-by: jupyterjazz <[email protected]> * refactor: set one logger name Signed-off-by: jupyterjazz <[email protected]> * docs: add quotes Signed-off-by: jupyterjazz <[email protected]> * refactor: put logger as a class attr Signed-off-by: jupyterjazz <[email protected]> * docs: small change Signed-off-by: jupyterjazz <[email protected]> * fix: typo Signed-off-by: jupyterjazz <[email protected]> * docs: rephrase text Signed-off-by: jupyterjazz <[email protected]> * refactor: requested changes Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: Saba Sturua <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: bytes type in `TextDoc` and `VideoDoc` (#1270) * fix: bytes type is None in predefined documents Signed-off-by: anna-charlotte <[email protected]> * fix: remove defaults to none Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: doc index structure (#1266) * refactor: doc index structure Signed-off-by: jupyterjazz <[email protected]> * fix: import unused error Signed-off-by: jupyterjazz <[email protected]> * fix: run black Signed-off-by: jupyterjazz <[email protected]> * refactor: remove noqa Signed-off-by: jupyterjazz <[email protected]> * refactor: doc_index to index Signed-off-by: jupyterjazz <[email protected]> * refactor: small changes Signed-off-by: jupyterjazz <[email protected]> * refactor: readme Signed-off-by: jupyterjazz <[email protected]> * refactor: add init file Signed-off-by: jupyterjazz <[email protected]> * refactor: move docindex outside integration tests Signed-off-by: jupyterjazz <[email protected]> * refactor: adjust config paths Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor(da): remove tensor type from `DocumentArray` init (#1268) * fix: remove tensor type from DocumentArray Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix tensorflow test Signed-off-by: samsja <[email protected]> * fix: docstrng Signed-off-by: samsja <[email protected]> * feat: apply charllote suggestion Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply saba suggestion Co-authored-by: Saba Sturua <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: samsja <[email protected]> Co-authored-by: Charlotte Gerhaher <[email protected]> Co-authored-by: Saba Sturua <[email protected]> * refactor: bytes to bytes_ in predefined documents (#1273) * refactor: bytes to bytes_ Signed-off-by: anna-charlotte <[email protected]> * refactor: missed fields Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: doc summary for dict and set attributes (#1279) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: add `get_paths()` instead of v1 `from_files()` (#1267) * feat: add from_files() Signed-off-by: anna-charlotte <[email protected]> * feat: add da classmethod from_files() Signed-off-by: anna-charlotte <[email protected]> * docs: update docstring Signed-off-by: anna-charlotte <[email protected]> * docs: add example usage Signed-off-by: anna-charlotte <[email protected]> * fix: add get_paths, rm from_files Signed-off-by: anna-charlotte <[email protected]> * fix: add print to debug ci Signed-off-by: anna-charlotte <[email protected]> * fix: test Signed-off-by: anna-charlotte <[email protected]> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: proto ser and deser for nested tuple/dict/list (#1278) * feat: add failing test Signed-off-by: samsja <[email protected]> * refactor: shorten if else statememt Signed-off-by: samsja <[email protected]> * refactor: shorten if else statememt Signed-off-by: samsja <[email protected]> * fix: fix proto and list Signed-off-by: samsja <[email protected]> * fix: fix proto and dict Signed-off-by: samsja <[email protected]> * feat: add very complex test Signed-off-by: samsja <[email protected]> * feat: fix pure tensor stuff Signed-off-by: samsja <[email protected]> * feat: fix pure tensor stuff Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix from protobuf Signed-off-by: samsja <[email protected]> * fix: fix from protobuf tensorflow Signed-off-by: samsja <[email protected]> * fix: add more test Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: add more test Signed-off-by: samsja <[email protected]> * fix: import ndarray Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: create documents from dict (#1283) * feat: create documents from dict Signed-off-by: jupyterjazz <[email protected]> * fix: ignore type Signed-off-by: jupyterjazz <[email protected]> * refactor: change fn names Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: fix up english (#1285) Signed-off-by: Alex C-G <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: add int, float and others to doc summary (#1287) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: hnswlib doc index (#1277) * fix: support for torch and tf Signed-off-by: Johannes Messner <[email protected]> * fix: allow arbitrary payloads, including tensors Signed-off-by: Johannes Messner <[email protected]> * test: mark tf tests Signed-off-by: Johannes Messner <[email protected]> * test: another attempt at fixing tf tests Signed-off-by: Johannes Messner <[email protected]> * test: remove parametrization of test Signed-off-by: Johannes Messner <[email protected]> * test: fix test Signed-off-by: Johannes Messner <[email protected]> * fix: add suggestion Signed-off-by: Johannes Messner <[email protected]> * ci: exlude tf tests from index tests Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix Signed-off-by: RStar2022 <[email protected]> * fix: tensorflow Signed-off-by: RStar2022 <[email protected]> * feat(test): DocumentArray method tests similar to list methods like reverse, sort, remove, pop (#1291) * feat: isort format fix Signed-off-by: agaraman0 <[email protected]> * refactor: comment fixes Signed-off-by: agaraman0 <[email protected]> * refactor: comment fixes Signed-off-by: agaraman0 <[email protected]> --------- Signed-off-by: agaraman0 <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix:doc Signed-off-by: RStar2022 <[email protected]> * feat: implement push/pull interface from JAC, file and s3 (#1182) * refactor: move streaming serialization into separate method Signed-off-by: Jackmin801 <[email protected]> * refactor: add binary io like protocol definition Signed-off-by: Jackmin801 <[email protected]> * feat: ported push pull to JAC Signed-off-by: Jackmin801 <[email protected]> * fix: protocol is not in 3.7 typing Signed-off-by: Jackmin801 <[email protected]> * fix: make mypy happy Signed-off-by: Jackmin801 <[email protected]> * fix: patch missing waterfall Signed-off-by: Jackmin801 <[email protected]> * refactor: jit import backends Signed-off-by: Jackmin801 <[email protected]> * feat: implement cache in jinaai pull Signed-off-by: Jackmin801 <[email protected]> * fix: add hubble dependency to jina group Signed-off-by: Jackmin801 <[email protected]> * refactor: better division of concerns Signed-off-by: Jackmin801 <[email protected]> * feat: add concept of namespace Signed-off-by: Jackmin801 <[email protected]> * fix: ignore missing hubble stubs Signed-off-by: Jackmin801 <[email protected]> * feat: streaming protocol stubs Signed-off-by: Jackmin801 <[email protected]> * refactor: make more general buffered caching reader Signed-off-by: Jackmin801 <[email protected]> * test: add tests for hubble pushpull Signed-off-by: Jackmin801 <[email protected]> * test: add tests for file backend Signed-off-by: Jackmin801 <[email protected]> * fix: remove hubble dependency from jina group This reverts commit b3044213d58517becb9d71194af34f3833560ebc. Signed-off-by: Jackmin801 <[email protected]> * feat: implement push pull for local filesystem Signed-off-by: Jackmin801 <[email protected]> * test: test concurrent pushes and pulls in file protocol Signed-off-by: Jackmin801 <[email protected]> * fix: resolve concurrent pushes and pulls correctly Signed-off-by: Jackmin801 <[email protected]> * fix: rename text to textdoc Signed-off-by: Jackmin801 <[email protected]> * feat: added some logging Signed-off-by: Jackmin801 <[email protected]> * test: s3 tests Signed-off-by: Jackmin801 <[email protected]> * feat: s3 pushpull Signed-off-by: Jackmin801 <[email protected]> * fix: add smart open dependency Signed-off-by: Jackmin801 <[email protected]> * fix: add smart opens silly python bound Signed-off-by: Jackmin801 <[email protected]> * test: update hubble tests (failing) Signed-off-by: Jackmin801 <[email protected]> * fix: fix delete return in hubble pushpull Signed-off-by: Jackmin801 <[email protected]> * Revert "fix: add smart open dependency" This reverts commit cf78c6cc6d2b367501d2358c18773a456426a448. This reverts commit eb0e52b4c521f2b638bf5de850701546a4996bc3. Signed-off-by: Jackmin801 <[email protected]> * fix: add hubble and smart open dependencies Signed-off-by: Jackmin801 <[email protected]> * fix: mypy fixes Signed-off-by: Jackmin801 <[email protected]> * ci: allow tests to see jina auth token Signed-off-by: Jackmin801 <[email protected]> * feat: add progress bars for streaming Signed-off-by: Jackmin801 <[email protected]> * style: blacken Signed-off-by: Jackmin801 <[email protected]> * feat: buffer writes to s3 Signed-off-by: Jackmin801 <[email protected]> * fix: mypy no like sequence Signed-off-by: Jackmin801 <[email protected]> * fix: make progress bar quieter when disabled Signed-off-by: Jackmin801 <[email protected]> * test: skip failing tests Signed-off-by: Jackmin801 <[email protected]> * feat: add tables when listing Signed-off-by: Jackmin801 <[email protected]> * test: add jina auth token to uncaped test Signed-off-by: Jackmin801 <[email protected]> * test: mock s3 tests with minio container Signed-off-by: Jackmin801 <[email protected]> * fix: silly error that cost me 2 hours of life Signed-off-by: Jackmin801 <[email protected]> * test: use tolerance ratio in file tests Signed-off-by: Jackmin801 <[email protected]> * feat: add caching to s3 pull Signed-off-by: Jackmin801 <[email protected]> * feat: add log messages for unused parameters Signed-off-by: Jackmin801 <[email protected]> * refactor: take out unneeded buffering smart open already buffers Signed-off-by: Jackmin801 <[email protected]> * feat: pick fastest protocol compression configuration for s3 Signed-off-by: Jackmin801 <[email protected]> * test: bump tolerance ratio for s3 test Signed-off-by: Jackmin801 <[email protected]> * refactor: reduce code duplication Signed-off-by: Jackmin801 <[email protected]> * refactor: put reader chunk size constant at top of file Signed-off-by: Jackmin801 <[email protected]> * test: reduce reader chunk size for memory tests Signed-off-by: Jackmin801 <[email protected]> * fix: multipart uploads get stuck frequently lets just do big uploads for now... Signed-off-by: Jackmin801 <[email protected]> * docs: add docstrings to mixin and file backend Signed-off-by: Jackmin801 <[email protected]> * docs: add docstring for s3 and hubble backends Signed-off-by: Jackmin801 <[email protected]> * test: remove unused test Signed-off-by: Jackmin801 <[email protected]> * refactor: use literal in protocol Co-authored-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> * refactor: protocols dont need to be inherited Co-authored-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> * fix: add make mypy happy with the literals Signed-off-by: Jackmin801 <[email protected]> * fix: literals not in 3.7 Signed-off-by: Jackmin801 <[email protected]> * refactor: move mixin out of init file Signed-off-by: Jackmin801 <[email protected]> * refactor: move cache path resolution to utils Signed-off-by: Jackmin801 <[email protected]> * feat: cache path is only evaluated once Signed-off-by: Jackmin801 <[email protected]> * refactor: loading backends makes more sense as debug log Signed-off-by: Jackmin801 <[email protected]> * tests: add slow and internet marks Signed-off-by: Jackmin801 <[email protected]> * refactor: pin image tag Signed-off-by: Jackmin801 <[email protected]> * refactor: use abc instead of protocol for typing backends Signed-off-by: Jackmin801 <[email protected]> * fix: revert - add hubble and smart open dependencies This reverts commit 1d1d2eeaf2b51be6ef00e6ab6ee5b9fd1bcf1d92. Signed-off-by: Jackmin801 <[email protected]> * fix: add hubble and aws dependencies Signed-off-by: Jackmin801 <[email protected]> * refactor: change all push pull mixin methods to class methods Signed-off-by: Jackmin801 <[email protected]> * fix: misstyped class method self reference Signed-off-by: Jackmin801 <[email protected]> * refactor: rename pushpull to docstore and use more classmethods Signed-off-by: Jackmin801 <[email protected]> * refactor: separate remote backend implementations from mixin Signed-off-by: Jackmin801 <[email protected]> * fix: missed import refactor Signed-off-by: Jackmin801 <[email protected]> * refactor: change submodule name to store Signed-off-by: Jackmin801 <[email protected]> * refactor: remove list and delete from mixin Signed-off-by: Jackmin801 <[email protected]> * tests: clear all the garbage in ci account Signed-off-by: Jackmin801 <[email protected]> * tests: skip test that is broken on ci Signed-off-by: Jackmin801 <[email protected]> * refactor: standardize naming to jac Signed-off-by: Jackmin801 <[email protected]> --------- Signed-off-by: Jackmin801 <[email protected]> Co-authored-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: rename `Document` to `Doc` (#1293) * refactor: rename document to doc Signed-off-by: samsja <[email protected]> * refactor: rename document to doc in da Signed-off-by: samsja <[email protected]> * refactor: rename base doc in md files Signed-off-by: samsja <[email protected]> * refactor: rename base base document ot base doc Signed-off-by: samsja <[email protected]> * fix(docs): fix docs building Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> * chore(docs): add ci and fix docs ui (#1295) * refactor: rename document to doc Signed-off-by: samsja <[email protected]> * refactor: rename document to doc in da Signed-off-by: samsja <[email protected]> * refactor: rename base doc in md files Signed-off-by: samsja <[email protected]> * refactor: rename base base document ot base doc Signed-off-by: samsja <[email protected]> * fix(docs): fix docs building Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> * docs: add userguide install Signed-off-by: samsja <[email protected]> * docs: add awesome-pages Signed-off-by: samsja <[email protected]> * docs: add install Signed-off-by: samsja <[email protected]> * docs: rename tutorials to how to Signed-off-by: samsja <[email protected]> * chore: add pre commit blacken docs Signed-off-by: samsja <[email protected]> * chore: add blacken docs Signed-off-by: samsja <[email protected]> * docs: arr warning docarray version Signed-off-by: samsja <[email protected]> * docs: repo url Signed-off-by: samsja <[email protected]> * docs: add social Signed-off-by: samsja <[email protected]> * docs: add logo Signed-off-by: samsja <[email protected]> * docs: add first step emtpy page Signed-off-by: samsja <[email protected]> * docs: add document docs Signed-off-by: samsja <[email protected]> * feat: add markdown documentation test Signed-off-by: samsja <[email protected]> * docs: remove content Signed-off-by: samsja <[email protected]> * docs: fix ci Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * chore: add docstring test (#1298) * wip Signed-off-by: samsja <[email protected]> * fix: cleanup namespace utils Signed-off-by: samsja <[email protected]> * feat: add docstring test Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: rename DocArrayProto to DocumentArrayProto (#1297) Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: docstring polish typing (#1299) * wip Signed-off-by: samsja <[email protected]> * fix: cleanup namespace utils Signed-off-by: samsja <[email protected]> * feat: add docstring test Signed-off-by: samsja <[email protected]> * fix: fix video url docstring Signed-off-by: samsja <[email protected]> * fix: fix text url Signed-off-by: samsja <[email protected]> * fix: fix image url Signed-off-by: samsja <[email protected]> * fix: fic audio url Signed-off-by: samsja <[email protected]> * fix: mesh 3d url Signed-off-by: samsja <[email protected]> * fix: mesh 3d url Signed-off-by: samsja <[email protected]> * fix: remove useless data Signed-off-by: samsja <[email protected]> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <[email protected]> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <[email protected]> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <[email protected]> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <[email protected]> * fix: fix video tensor Signed-off-by: samsja <[email protected]> * fix: fix video tensor Signed-off-by: samsja <[email protected]> * fix: fix audio bytes Signed-off-by: samsja <[email protected]> * fix: video and image bytes Signed-off-by: samsja <[email protected]> * docs: move typing section Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: remove files (#1305) Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: flatten schema of abstract index (#1294) * fix: flatten schema of abstract index Signed-off-by: AnneY <[email protected]> * fix: _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * fix: catch exception when flatten schema Signed-off-by: AnneY <[email protected]> * refactor: remove useless assignemnt Signed-off-by: AnneY <[email protected]> * fix: use Abstractensor as tensor doc_type Signed-off-by: AnneY <[email protected]> * fix: add AbstractTensor to hnswlib Signed-off-by: AnneY <[email protected]> * docs: AbstractTensor as doc_type Signed-off-by: AnneY <[email protected]> * docs: complete description about AbstracTensor Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: Anne Yang <[email protected]> --------- Signed-off-by: AnneY <[email protected]> Signed-off-by: Anne Yang <[email protected]> Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: add utils section (#1307) * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <[email protected]> * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <[email protected]> * feat: add utils for find and fix docstring Signed-off-by: samsja <[email protected]> * fix: fix video ndaray docstrng Signed-off-by: samsja <[email protected]> * fix: fix video find docstrng Signed-off-by: samsja <[email protected]> * fix: fix map docstring Signed-off-by: samsja <[email protected]> * fix: fix fileter docstring Signed-off-by: samsja <[email protected]> * fix: fix add reduce Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: fix docstring example of find_batched (#1308) Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: fix map docstring (#1311) * fix: fix utils Signed-off-by: samsja <[email protected]> * fix: fix map Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: elasticsearch document index (#1196) * feat: __init__ of ElasticDocumentIndex Signed-off-by: AnneY <[email protected]> * feat: add index func Signed-off-by: AnneY <[email protected]> * feat: get and del funcs Signed-off-by: AnneY <[email protected]> * fix: init and index creation Signed-off-by: AnneY <[email protected]> * fix: __init__ and _index Signed-off-by: AnneY <[email protected]> * fix: _get_items Signed-off-by: AnneY <[email protected]> * feat: add _find Signed-off-by: AnneY <[email protected]> * feat: add filter text and their batch version Signed-off-by: AnneY <[email protected]> * feat: store id and get nested doc Signed-off-by: AnneY <[email protected]> * fix: vector cannot be all zero Signed-off-by: AnneY <[email protected]> * fix: __getitem__ raise error Signed-off-by: AnneY <[email protected]> * feat: support more python types Signed-off-by: AnneY <[email protected]> * fix: mypy Signed-off-by: AnneY <[email protected]> * test: elastic index tests Signed-off-by: AnneY <[email protected]> * test: comment scripts before ci setup Signed-off-by: AnneY <[email protected]> * chore: add elasticsearch dependency to poetry Signed-off-by: AnneY <[email protected]> * test: elastic index ci setup Signed-off-by: AnneY <[email protected]> * feat: add num_candidates to rumtime config Signed-off-by: AnneY <[email protected]> * fix: let user pass index_settings Signed-off-by: AnneY <[email protected]> * feat: degrade to v7 and add query builder Signed-off-by: AnneY <[email protected]> * fix: remove elastic_transport Signed-off-by: AnneY <[email protected]> * feat: minor features Signed-off-by: AnneY <[email protected]> * refactor: style fix Signed-off-by: AnneY <[email protected]> * fix: fix mypy Signed-off-by: AnneY <[email protected]> * feat: add chunk size to runtime config Signed-off-by: AnneY <[email protected]> * fix: chunk size Signed-off-by: AnneY <[email protected]> * feat: add chunk_size to funcs Signed-off-by: AnneY <[email protected]> * feat: rewrite elastic v7 query builder Signed-off-by: AnneY <[email protected]> * fix: poetry Signed-off-by: AnneY <[email protected]> * fix: db_type should be elastic types Signed-off-by: AnneY <[email protected]> * fix: minor adjustment Signed-off-by: AnneY <[email protected]> * refactor: rename elastic index files Signed-off-by: AnneY <[email protected]> * refactor: remove comments Signed-off-by: AnneY <[email protected]> * feat: rename, batch operations, etc Signed-off-by: AnneY <[email protected]> * test: add test for persistency and col config Signed-off-by: AnneY <[email protected]> * feat: support more field types and subclass Signed-off-by: AnneY <[email protected]> * feat: support more python types Signed-off-by: AnneY <[email protected]> * test: tf, tensor and more elastic field types Signed-off-by: AnneY <[email protected]> * fix: elastic should be optional in toml Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: Anne Yang <[email protected]> * refactor: rename class Signed-off-by: AnneY <[email protected]> * fix: change Dict to Mapping Signed-off-by: AnneY <[email protected]> * fix: add AbstractTensor Signed-off-by: AnneY <[email protected]> * test: rename class and add tests Signed-off-by: AnneY <[email protected]> * fix: poetry Signed-off-by: AnneY <[email protected]> --------- Signed-off-by: AnneY <[email protected]> Signed-off-by: Anne Yang <[email protected]> Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: torch backend basic operation tests (#1306) Signed-off-by: agaraman0 <[email protected]> Signed-off-by: RStar2022 <[email protected]> * chore: add instructions to pip installs and group extras (#1281) * chore: group extras and add instructions for pip installs Signed-off-by: anna-charlotte <[email protected]> * fix: throw runtime error with install instructions for hnswlib Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for video imports Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for audio imports Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for 3d imports Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for image imports Signed-off-by: anna-charlotte <[email protected]> * fix: import only audiosegment from pydub Signed-off-by: anna-charlotte <[email protected]> * fix: generalize audio and image imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for web imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for web imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for protobuf imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for lz4 imports Signed-off-by: anna-charlotte <[email protected]> * fix: fastapi import Signed-off-by: anna-charlotte <[email protected]> * fix: revert changes in protobuf import Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for torch, without raising error Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for torch, with raising error Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for tensorflow Signed-off-by: anna-charlotte <[email protected]> * fix: base doc io imports Sign…

* refactor: dummy change Signed-off-by: jupyterjazz <[email protected]> * feat: implement == for document and document array (#1224) * refactor: rename predefined documents (#1208) * refactor: rename Image to ImageDoc Signed-off-by: samsja <[email protected]> * refactor: rename Text to TextDoc Signed-off-by: samsja <[email protected]> * refactor: rename Audio to AudioDoc Signed-off-by: samsja <[email protected]> * refactor: rename Video to VideoDOc Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: support other text formats (#1207) * feat: add validation function and filetypes feat: add validation function and filetypes fix: formatting Signed-off-by: [email protected] <[email protected]> * feat: added extension validation tests for TextURL Signed-off-by: [email protected] <[email protected]> * fix: fix for tests and PR codereview Signed-off-by: [email protected] <[email protected]> * fix: added internet mark to json dump test for texturl Signed-off-by: [email protected] <[email protected]> --------- Signed-off-by: [email protected] <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: refactor query builder (#1213) * refactor: query builder Signed-off-by: Johannes Messner <[email protected]> * docs: add guidance for query builder Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat(index): automatically convert a dict to Document (#1215) * feat: add _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * fix: str cut Signed-off-by: AnneY <[email protected]> * feat: _get_items add return type Signed-off-by: AnneY <[email protected]> * fix: mypy Signed-off-by: AnneY <[email protected]> * test: add test for _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * refactor: simplify _convert_to_doc_list Signed-off-by: AnneY <[email protected]> * docs: add docstring for _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * feat: filter find text add return type Signed-off-by: AnneY <[email protected]> * fix: type judge Signed-off-by: AnneY <[email protected]> * refactor: change search_field default value Signed-off-by: AnneY <[email protected]> * fix: always check length Signed-off-by: AnneY <[email protected]> * refactor: create private result class Signed-off-by: AnneY <[email protected]> * refactor: seperate helper method Signed-off-by: AnneY <[email protected]> * refactor: create private result class Signed-off-by: AnneY <[email protected]> --------- Signed-off-by: AnneY <[email protected]> Signed-off-by: Anne Yang <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <[email protected]> * fix: remove impl Signed-off-by: samsja <[email protected]> * refactor: move some tests Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add document view Signed-off-by: samsja <[email protected]> * feat: add tests Signed-off-by: samsja <[email protected]> * refactor: rename storage to column storage Signed-off-by: samsja <[email protected]> * refactor: order of function in da stacked Signed-off-by: samsja <[email protected]> * refactor: add inner doc in test Signed-off-by: samsja <[email protected]> * refactor: rename storage to column Signed-off-by: samsja <[email protected]> * feat: add from storage Signed-off-by: samsja <[email protected]> * refactor: rename file Signed-off-by: samsja <[email protected]> * refactor: storage take columns as init Signed-off-by: samsja <[email protected]> * feat: add back getitiem Signed-off-by: samsja <[email protected]> * fix: fix getatr view problem Signed-off-by: samsja <[email protected]> * feat: add get array attribute Signed-off-by: samsja <[email protected]> * feat: add setitem back Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * feat: add to Signed-off-by: samsja <[email protected]> * feat: add ttests Signed-off-by: samsja <[email protected]> * feat: add unstack Signed-off-by: samsja <[email protected]> * feat: add validation da set atr Signed-off-by: samsja <[email protected]> * feat: add da and any Signed-off-by: samsja <[email protected]> * feat: remove deleitem overload Signed-off-by: samsja <[email protected]> * fix: remove context manager Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: move tests Signed-off-by: samsja <[email protected]> * fix: add tidi Signed-off-by: samsja <[email protected]> * fix: fix copy in ndarray mypy Signed-off-by: samsja <[email protected]> * feat: add indexing sequence mixin Signed-off-by: samsja <[email protected]> * feat: add list index class Signed-off-by: samsja <[email protected]> * fix: add cast from tuple to list Signed-off-by: samsja <[email protected]> * fix: fix indexing test Signed-off-by: samsja <[email protected]> * feat: add back traversle flat Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: remove unstack mode Signed-off-by: samsja <[email protected]> * feat: add proto Signed-off-by: samsja <[email protected]> * feat: add init mixin Signed-off-by: samsja <[email protected]> * fix: add back staticmethod Signed-off-by: samsja <[email protected]> * fix: add len to mixin Signed-off-by: samsja <[email protected]> * fix: add del to index seq Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <[email protected]> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <[email protected]> * refactor: move docs to column storage to da stack Signed-off-by: samsja <[email protected]> * refactor: remove useless arg Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix nested array in da stack Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: remove type hint from setitiem Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix getitem Signed-off-by: samsja <[email protected]> * merge: merge featrewrite v2 Signed-off-by: samsja <[email protected]> * fix: black does nto format pb file Signed-off-by: samsja <[email protected]> * fix: remove comment Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix proto3 Signed-off-by: samsja <[email protected]> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <[email protected]> * fix: fix del Signed-off-by: samsja <[email protected]> * fix: fix scalar value Signed-off-by: samsja <[email protected]> * fix: remove useless test Signed-off-by: samsja <[email protected]> * feat: raise error when docs are empty Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix type hint Signed-off-by: samsja <[email protected]> * fix: fix settatr Signed-off-by: samsja <[email protected]> * fix: apply suggestion Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Signed-off-by: samsja <[email protected]> * fix: docstring Signed-off-by: samsja <[email protected]> * fix: last fix hopefully Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: add comments Signed-off-by: samsja <[email protected]> * fix: remove classvar Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: remvoe proprety Signed-off-by: samsja <[email protected]> * fix: bring back proprety Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> Signed-off-by: samsja <[email protected]> Co-authored-by: Jackmin801 <[email protected]> Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Signed-off-by: RStar2022 <[email protected]> * add equal function Signed-off-by: RStar2022 <[email protected]> * Add equality function Signed-off-by: RStar2022 <[email protected]> * Add equal to array and document Signed-off-by: RStar2022 <[email protected]> * changes a bit Signed-off-by: RStar2022 <[email protected]> * add __eq__ function in abstract_comp_backend and add test_array Signed-off-by: RStar2022 <[email protected]> * feat: ellipsis in tensor shape definition (#1228) * feat: ellipsis in tensor shape definition Signed-off-by: jupyterjazz <[email protected]> * fix: type hint Signed-off-by: jupyterjazz <[email protected]> * refactor: add tests for other tensors Signed-off-by: jupyterjazz <[email protected]> * test: fix tensorflow test Signed-off-by: jupyterjazz <[email protected]> * docs: update docstrings with examples Signed-off-by: jupyterjazz <[email protected]> * docs: clarify needed dimensions Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * chore: update contributing guideline (#1240) * chore: update contrubing guideline Signed-off-by: samsja <[email protected]> * fix: mark dl test slow Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix equality funtion Signed-off-by: RStar2022 <[email protected]> * refactor: da stack full column wise (#1183) * refactor: wip add storage class Signed-off-by: samsja <[email protected]> * fix: remove impl Signed-off-by: samsja <[email protected]> * refactor: move some tests Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add storage view Signed-off-by: samsja <[email protected]> * feat: add document view Signed-off-by: samsja <[email protected]> * feat: add tests Signed-off-by: samsja <[email protected]> * refactor: rename storage to column storage Signed-off-by: samsja <[email protected]> * refactor: order of function in da stacked Signed-off-by: samsja <[email protected]> * refactor: add inner doc in test Signed-off-by: samsja <[email protected]> * refactor: rename storage to column Signed-off-by: samsja <[email protected]> * feat: add from storage Signed-off-by: samsja <[email protected]> * refactor: rename file Signed-off-by: samsja <[email protected]> * refactor: storage take columns as init Signed-off-by: samsja <[email protected]> * feat: add back getitiem Signed-off-by: samsja <[email protected]> * fix: fix getatr view problem Signed-off-by: samsja <[email protected]> * feat: add get array attribute Signed-off-by: samsja <[email protected]> * feat: add setitem back Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * fix: fix some tests Signed-off-by: samsja <[email protected]> * feat: add to Signed-off-by: samsja <[email protected]> * feat: add ttests Signed-off-by: samsja <[email protected]> * feat: add unstack Signed-off-by: samsja <[email protected]> * feat: add validation da set atr Signed-off-by: samsja <[email protected]> * feat: add da and any Signed-off-by: samsja <[email protected]> * feat: remove deleitem overload Signed-off-by: samsja <[email protected]> * fix: remove context manager Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: move tests Signed-off-by: samsja <[email protected]> * fix: add tidi Signed-off-by: samsja <[email protected]> * fix: fix copy in ndarray mypy Signed-off-by: samsja <[email protected]> * feat: add indexing sequence mixin Signed-off-by: samsja <[email protected]> * feat: add list index class Signed-off-by: samsja <[email protected]> * fix: add cast from tuple to list Signed-off-by: samsja <[email protected]> * fix: fix indexing test Signed-off-by: samsja <[email protected]> * feat: add back traversle flat Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: add to protobuf Signed-off-by: samsja <[email protected]> * feat: remove unstack mode Signed-off-by: samsja <[email protected]> * feat: add proto Signed-off-by: samsja <[email protected]> * feat: add init mixin Signed-off-by: samsja <[email protected]> * fix: add back staticmethod Signed-off-by: samsja <[email protected]> * fix: add len to mixin Signed-off-by: samsja <[email protected]> * fix: add del to index seq Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * fix: fix mypy del Signed-off-by: samsja <[email protected]> * docs: change DocumentArrayStacked docstring Signed-off-by: samsja <[email protected]> * docs: add docstring dor ColumnStorage Signed-off-by: samsja <[email protected]> * refactor: move docs to column storage to da stack Signed-off-by: samsja <[email protected]> * refactor: remove useless arg Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix mypy 2 Signed-off-by: samsja <[email protected]> * fix: fix nested array in da stack Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: remove type hint from setitiem Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix getitem Signed-off-by: samsja <[email protected]> * merge: merge featrewrite v2 Signed-off-by: samsja <[email protected]> * fix: black does nto format pb file Signed-off-by: samsja <[email protected]> * fix: remove comment Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix proto3 Signed-off-by: samsja <[email protected]> * fix: fix multimodal dataset tests The initialization of all the subclasses is no longer necessary Signed-off-by: Jackmin801 <[email protected]> * fix: fix del Signed-off-by: samsja <[email protected]> * fix: fix scalar value Signed-off-by: samsja <[email protected]> * fix: remove useless test Signed-off-by: samsja <[email protected]> * feat: raise error when docs are empty Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix tests Signed-off-by: samsja <[email protected]> * fix: fix type hint Signed-off-by: samsja <[email protected]> * fix: fix settatr Signed-off-by: samsja <[email protected]> * fix: apply suggestion Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Signed-off-by: samsja <[email protected]> * fix: docstring Signed-off-by: samsja <[email protected]> * fix: last fix hopefully Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: add comments Signed-off-by: samsja <[email protected]> * fix: remove classvar Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: rename Signed-off-by: samsja <[email protected]> * fix: remvoe proprety Signed-off-by: samsja <[email protected]> * fix: bring back proprety Signed-off-by: samsja <[email protected]> * feat: apply johannes suggestion Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> Signed-off-by: samsja <[email protected]> Co-authored-by: Jackmin801 <[email protected]> Co-authored-by: Johannes Messner <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> * chore: update contribution guideline (#1247) * chore: update contribution guideline Signed-off-by: samsja <[email protected]> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: samsja <[email protected]> * feat: add poetry add Signed-off-by: samsja <[email protected]> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply alex suggestion Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Co-authored-by: Alex Cureton-Griffiths <[email protected]> Signed-off-by: RStar2022 <[email protected]> * make default search field empty string (#1249) Signed-off-by: azayz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * delete import numpy Signed-off-by: RStar2022 <[email protected]> * fix key name Signed-off-by: RStar2022 <[email protected]> * feat:changes-equality-operation Signed-off-by: RStar2022 <[email protected]> * feat:add if Signed-off-by: RStar2022 <[email protected]> * docs: add explanation about id field (#1242) * docs: add explanation about id field Signed-off-by: Johannes Messner <[email protected]> * docs: update docs/tutorials/add_doc_index.md Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: Johannes Messner <[email protected]> Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: shift to mkdocs (#1244) * chore: remove sphinx and mkdocs Signed-off-by: samsja <[email protected]> * fix: add DocumentArrayStacked to init py Signed-off-by: samsja <[email protected]> * feat: add mkdocs Signed-off-by: samsja <[email protected]> * feat: apply alex setting Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * fix: fix import Signed-off-by: samsja <[email protected]> * fix: fix import Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * fix: fix ci Signed-off-by: samsja <[email protected]> * feat: uses sphinx style Signed-off-by: samsja <[email protected]> * feat: udpate docs Signed-off-by: samsja <[email protected]> * feat: add inherited menbers Signed-off-by: samsja <[email protected]> * feat: add more Signed-off-by: samsja <[email protected]> * fix: code block syntax highlight in docstring Signed-off-by: Alex C-G <[email protected]> * feat: add typing Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: Alex C-G <[email protected]> Co-authored-by: Alex C-G <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix:equal func Signed-off-by: RStar2022 <[email protected]> * refactor: rename filter to filter_docs to avoid shadowing of filtern (#1257) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: ad user defined mapping for python type to db type (#1252) * feat: user defined mapping for python type to db type Signed-off-by: anna-charlotte <[email protected]> * feat: check if col_type available Signed-off-by: anna-charlotte <[email protected]> * test: add test for base classes Signed-off-by: anna-charlotte <[email protected]> * fix: clean up Signed-off-by: anna-charlotte <[email protected]> * fix: test Signed-off-by: anna-charlotte <[email protected]> * docs: add documentation for db type and python type Signed-off-by: anna-charlotte <[email protected]> * docs: add doumentation for runtime config Signed-off-by: anna-charlotte <[email protected]> * fix: add and test illegal col types Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix:tensor type func Signed-off-by: RStar2022 <[email protected]> * fix: comp_backend code delete Signed-off-by: RStar2022 <[email protected]> * feat(index): index data with union types (#1220) * refactor: split flattening into separate method Signed-off-by: Johannes Messner <[email protected]> * refactor: don't build column info during schema check Signed-off-by: Johannes Messner <[email protected]> * feat: allos unions and optional in indexed data Signed-off-by: Johannes Messner <[email protected]> * fix: mypy Signed-off-by: Johannes Messner <[email protected]> * fix: mypy Signed-off-by: Johannes Messner <[email protected]> * fix: import from typing inspect instead of typing Signed-off-by: Johannes Messner <[email protected]> * fix: equality and hash for parametrized tensors Signed-off-by: Johannes Messner <[email protected]> * test: add test for flatten docs Signed-off-by: Johannes Messner <[email protected]> * refactor: apply suggestions Signed-off-by: Johannes Messner <[email protected]> * docs: better docstrings Signed-off-by: Johannes Messner <[email protected]> * refactor: use construct to create docarray Signed-off-by: Johannes Messner <[email protected]> * fix: check for nonetype Signed-off-by: Johannes Messner <[email protected]> * fix: none in equals check Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix eq Signed-off-by: RStar2022 <[email protected]> * fix: move test to integration test (#1260) * fix: move test to integration test Signed-off-by: samsja <[email protected]> * fix: use a different runner for doc index Signed-off-by: samsja <[email protected]> * fix: use a different runner for doc index Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: disable pycharm da property detection (#1262) * fix: disable unresolved attr detection for da in pycharm Signed-off-by: anna-charlotte <[email protected]> * fix: add docstring Signed-off-by: anna-charlotte <[email protected]> * fix: call super instead of pass Signed-off-by: anna-charlotte <[email protected]> * fix: getattr Signed-off-by: anna-charlotte <[email protected]> * fix: getattribute Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: add minimal logger (#1254) * feat: add minimal logger Signed-off-by: jupyterjazz <[email protected]> * docs: add an example in contributing Signed-off-by: jupyterjazz <[email protected]> * docs: try different format Signed-off-by: jupyterjazz <[email protected]> * refactor: set one logger name Signed-off-by: jupyterjazz <[email protected]> * docs: add quotes Signed-off-by: jupyterjazz <[email protected]> * refactor: put logger as a class attr Signed-off-by: jupyterjazz <[email protected]> * docs: small change Signed-off-by: jupyterjazz <[email protected]> * fix: typo Signed-off-by: jupyterjazz <[email protected]> * docs: rephrase text Signed-off-by: jupyterjazz <[email protected]> * refactor: requested changes Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: Saba Sturua <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: bytes type in `TextDoc` and `VideoDoc` (#1270) * fix: bytes type is None in predefined documents Signed-off-by: anna-charlotte <[email protected]> * fix: remove defaults to none Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: doc index structure (#1266) * refactor: doc index structure Signed-off-by: jupyterjazz <[email protected]> * fix: import unused error Signed-off-by: jupyterjazz <[email protected]> * fix: run black Signed-off-by: jupyterjazz <[email protected]> * refactor: remove noqa Signed-off-by: jupyterjazz <[email protected]> * refactor: doc_index to index Signed-off-by: jupyterjazz <[email protected]> * refactor: small changes Signed-off-by: jupyterjazz <[email protected]> * refactor: readme Signed-off-by: jupyterjazz <[email protected]> * refactor: add init file Signed-off-by: jupyterjazz <[email protected]> * refactor: move docindex outside integration tests Signed-off-by: jupyterjazz <[email protected]> * refactor: adjust config paths Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor(da): remove tensor type from `DocumentArray` init (#1268) * fix: remove tensor type from DocumentArray Signed-off-by: samsja <[email protected]> * fix: fix test Signed-off-by: samsja <[email protected]> * fix: fix tensorflow test Signed-off-by: samsja <[email protected]> * fix: docstrng Signed-off-by: samsja <[email protected]> * feat: apply charllote suggestion Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: samsja <[email protected]> * feat: apply saba suggestion Co-authored-by: Saba Sturua <[email protected]> Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: samsja <[email protected]> Co-authored-by: Charlotte Gerhaher <[email protected]> Co-authored-by: Saba Sturua <[email protected]> * refactor: bytes to bytes_ in predefined documents (#1273) * refactor: bytes to bytes_ Signed-off-by: anna-charlotte <[email protected]> * refactor: missed fields Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: doc summary for dict and set attributes (#1279) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: add `get_paths()` instead of v1 `from_files()` (#1267) * feat: add from_files() Signed-off-by: anna-charlotte <[email protected]> * feat: add da classmethod from_files() Signed-off-by: anna-charlotte <[email protected]> * docs: update docstring Signed-off-by: anna-charlotte <[email protected]> * docs: add example usage Signed-off-by: anna-charlotte <[email protected]> * fix: add get_paths, rm from_files Signed-off-by: anna-charlotte <[email protected]> * fix: add print to debug ci Signed-off-by: anna-charlotte <[email protected]> * fix: test Signed-off-by: anna-charlotte <[email protected]> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <[email protected]> --------- Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: proto ser and deser for nested tuple/dict/list (#1278) * feat: add failing test Signed-off-by: samsja <[email protected]> * refactor: shorten if else statememt Signed-off-by: samsja <[email protected]> * refactor: shorten if else statememt Signed-off-by: samsja <[email protected]> * fix: fix proto and list Signed-off-by: samsja <[email protected]> * fix: fix proto and dict Signed-off-by: samsja <[email protected]> * feat: add very complex test Signed-off-by: samsja <[email protected]> * feat: fix pure tensor stuff Signed-off-by: samsja <[email protected]> * feat: fix pure tensor stuff Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: fix from protobuf Signed-off-by: samsja <[email protected]> * fix: fix from protobuf tensorflow Signed-off-by: samsja <[email protected]> * fix: add more test Signed-off-by: samsja <[email protected]> * fix: fix mypy Signed-off-by: samsja <[email protected]> * fix: add more test Signed-off-by: samsja <[email protected]> * fix: import ndarray Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: create documents from dict (#1283) * feat: create documents from dict Signed-off-by: jupyterjazz <[email protected]> * fix: ignore type Signed-off-by: jupyterjazz <[email protected]> * refactor: change fn names Signed-off-by: jupyterjazz <[email protected]> --------- Signed-off-by: jupyterjazz <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: fix up english (#1285) Signed-off-by: Alex C-G <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: add int, float and others to doc summary (#1287) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: hnswlib doc index (#1277) * fix: support for torch and tf Signed-off-by: Johannes Messner <[email protected]> * fix: allow arbitrary payloads, including tensors Signed-off-by: Johannes Messner <[email protected]> * test: mark tf tests Signed-off-by: Johannes Messner <[email protected]> * test: another attempt at fixing tf tests Signed-off-by: Johannes Messner <[email protected]> * test: remove parametrization of test Signed-off-by: Johannes Messner <[email protected]> * test: fix test Signed-off-by: Johannes Messner <[email protected]> * fix: add suggestion Signed-off-by: Johannes Messner <[email protected]> * ci: exlude tf tests from index tests Signed-off-by: Johannes Messner <[email protected]> --------- Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix Signed-off-by: RStar2022 <[email protected]> * fix: tensorflow Signed-off-by: RStar2022 <[email protected]> * feat(test): DocumentArray method tests similar to list methods like reverse, sort, remove, pop (#1291) * feat: isort format fix Signed-off-by: agaraman0 <[email protected]> * refactor: comment fixes Signed-off-by: agaraman0 <[email protected]> * refactor: comment fixes Signed-off-by: agaraman0 <[email protected]> --------- Signed-off-by: agaraman0 <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix:doc Signed-off-by: RStar2022 <[email protected]> * feat: implement push/pull interface from JAC, file and s3 (#1182) * refactor: move streaming serialization into separate method Signed-off-by: Jackmin801 <[email protected]> * refactor: add binary io like protocol definition Signed-off-by: Jackmin801 <[email protected]> * feat: ported push pull to JAC Signed-off-by: Jackmin801 <[email protected]> * fix: protocol is not in 3.7 typing Signed-off-by: Jackmin801 <[email protected]> * fix: make mypy happy Signed-off-by: Jackmin801 <[email protected]> * fix: patch missing waterfall Signed-off-by: Jackmin801 <[email protected]> * refactor: jit import backends Signed-off-by: Jackmin801 <[email protected]> * feat: implement cache in jinaai pull Signed-off-by: Jackmin801 <[email protected]> * fix: add hubble dependency to jina group Signed-off-by: Jackmin801 <[email protected]> * refactor: better division of concerns Signed-off-by: Jackmin801 <[email protected]> * feat: add concept of namespace Signed-off-by: Jackmin801 <[email protected]> * fix: ignore missing hubble stubs Signed-off-by: Jackmin801 <[email protected]> * feat: streaming protocol stubs Signed-off-by: Jackmin801 <[email protected]> * refactor: make more general buffered caching reader Signed-off-by: Jackmin801 <[email protected]> * test: add tests for hubble pushpull Signed-off-by: Jackmin801 <[email protected]> * test: add tests for file backend Signed-off-by: Jackmin801 <[email protected]> * fix: remove hubble dependency from jina group This reverts commit b3044213d58517becb9d71194af34f3833560ebc. Signed-off-by: Jackmin801 <[email protected]> * feat: implement push pull for local filesystem Signed-off-by: Jackmin801 <[email protected]> * test: test concurrent pushes and pulls in file protocol Signed-off-by: Jackmin801 <[email protected]> * fix: resolve concurrent pushes and pulls correctly Signed-off-by: Jackmin801 <[email protected]> * fix: rename text to textdoc Signed-off-by: Jackmin801 <[email protected]> * feat: added some logging Signed-off-by: Jackmin801 <[email protected]> * test: s3 tests Signed-off-by: Jackmin801 <[email protected]> * feat: s3 pushpull Signed-off-by: Jackmin801 <[email protected]> * fix: add smart open dependency Signed-off-by: Jackmin801 <[email protected]> * fix: add smart opens silly python bound Signed-off-by: Jackmin801 <[email protected]> * test: update hubble tests (failing) Signed-off-by: Jackmin801 <[email protected]> * fix: fix delete return in hubble pushpull Signed-off-by: Jackmin801 <[email protected]> * Revert "fix: add smart open dependency" This reverts commit cf78c6cc6d2b367501d2358c18773a456426a448. This reverts commit eb0e52b4c521f2b638bf5de850701546a4996bc3. Signed-off-by: Jackmin801 <[email protected]> * fix: add hubble and smart open dependencies Signed-off-by: Jackmin801 <[email protected]> * fix: mypy fixes Signed-off-by: Jackmin801 <[email protected]> * ci: allow tests to see jina auth token Signed-off-by: Jackmin801 <[email protected]> * feat: add progress bars for streaming Signed-off-by: Jackmin801 <[email protected]> * style: blacken Signed-off-by: Jackmin801 <[email protected]> * feat: buffer writes to s3 Signed-off-by: Jackmin801 <[email protected]> * fix: mypy no like sequence Signed-off-by: Jackmin801 <[email protected]> * fix: make progress bar quieter when disabled Signed-off-by: Jackmin801 <[email protected]> * test: skip failing tests Signed-off-by: Jackmin801 <[email protected]> * feat: add tables when listing Signed-off-by: Jackmin801 <[email protected]> * test: add jina auth token to uncaped test Signed-off-by: Jackmin801 <[email protected]> * test: mock s3 tests with minio container Signed-off-by: Jackmin801 <[email protected]> * fix: silly error that cost me 2 hours of life Signed-off-by: Jackmin801 <[email protected]> * test: use tolerance ratio in file tests Signed-off-by: Jackmin801 <[email protected]> * feat: add caching to s3 pull Signed-off-by: Jackmin801 <[email protected]> * feat: add log messages for unused parameters Signed-off-by: Jackmin801 <[email protected]> * refactor: take out unneeded buffering smart open already buffers Signed-off-by: Jackmin801 <[email protected]> * feat: pick fastest protocol compression configuration for s3 Signed-off-by: Jackmin801 <[email protected]> * test: bump tolerance ratio for s3 test Signed-off-by: Jackmin801 <[email protected]> * refactor: reduce code duplication Signed-off-by: Jackmin801 <[email protected]> * refactor: put reader chunk size constant at top of file Signed-off-by: Jackmin801 <[email protected]> * test: reduce reader chunk size for memory tests Signed-off-by: Jackmin801 <[email protected]> * fix: multipart uploads get stuck frequently lets just do big uploads for now... Signed-off-by: Jackmin801 <[email protected]> * docs: add docstrings to mixin and file backend Signed-off-by: Jackmin801 <[email protected]> * docs: add docstring for s3 and hubble backends Signed-off-by: Jackmin801 <[email protected]> * test: remove unused test Signed-off-by: Jackmin801 <[email protected]> * refactor: use literal in protocol Co-authored-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> * refactor: protocols dont need to be inherited Co-authored-by: samsja <[email protected]> Signed-off-by: Jackmin801 <[email protected]> * fix: add make mypy happy with the literals Signed-off-by: Jackmin801 <[email protected]> * fix: literals not in 3.7 Signed-off-by: Jackmin801 <[email protected]> * refactor: move mixin out of init file Signed-off-by: Jackmin801 <[email protected]> * refactor: move cache path resolution to utils Signed-off-by: Jackmin801 <[email protected]> * feat: cache path is only evaluated once Signed-off-by: Jackmin801 <[email protected]> * refactor: loading backends makes more sense as debug log Signed-off-by: Jackmin801 <[email protected]> * tests: add slow and internet marks Signed-off-by: Jackmin801 <[email protected]> * refactor: pin image tag Signed-off-by: Jackmin801 <[email protected]> * refactor: use abc instead of protocol for typing backends Signed-off-by: Jackmin801 <[email protected]> * fix: revert - add hubble and smart open dependencies This reverts commit 1d1d2eeaf2b51be6ef00e6ab6ee5b9fd1bcf1d92. Signed-off-by: Jackmin801 <[email protected]> * fix: add hubble and aws dependencies Signed-off-by: Jackmin801 <[email protected]> * refactor: change all push pull mixin methods to class methods Signed-off-by: Jackmin801 <[email protected]> * fix: misstyped class method self reference Signed-off-by: Jackmin801 <[email protected]> * refactor: rename pushpull to docstore and use more classmethods Signed-off-by: Jackmin801 <[email protected]> * refactor: separate remote backend implementations from mixin Signed-off-by: Jackmin801 <[email protected]> * fix: missed import refactor Signed-off-by: Jackmin801 <[email protected]> * refactor: change submodule name to store Signed-off-by: Jackmin801 <[email protected]> * refactor: remove list and delete from mixin Signed-off-by: Jackmin801 <[email protected]> * tests: clear all the garbage in ci account Signed-off-by: Jackmin801 <[email protected]> * tests: skip test that is broken on ci Signed-off-by: Jackmin801 <[email protected]> * refactor: standardize naming to jac Signed-off-by: Jackmin801 <[email protected]> --------- Signed-off-by: Jackmin801 <[email protected]> Co-authored-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: rename `Document` to `Doc` (#1293) * refactor: rename document to doc Signed-off-by: samsja <[email protected]> * refactor: rename document to doc in da Signed-off-by: samsja <[email protected]> * refactor: rename base doc in md files Signed-off-by: samsja <[email protected]> * refactor: rename base base document ot base doc Signed-off-by: samsja <[email protected]> * fix(docs): fix docs building Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> * chore(docs): add ci and fix docs ui (#1295) * refactor: rename document to doc Signed-off-by: samsja <[email protected]> * refactor: rename document to doc in da Signed-off-by: samsja <[email protected]> * refactor: rename base doc in md files Signed-off-by: samsja <[email protected]> * refactor: rename base base document ot base doc Signed-off-by: samsja <[email protected]> * fix(docs): fix docs building Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> * fix: ingore hubble test Signed-off-by: samsja <[email protected]> * docs: add userguide install Signed-off-by: samsja <[email protected]> * docs: add awesome-pages Signed-off-by: samsja <[email protected]> * docs: add install Signed-off-by: samsja <[email protected]> * docs: rename tutorials to how to Signed-off-by: samsja <[email protected]> * chore: add pre commit blacken docs Signed-off-by: samsja <[email protected]> * chore: add blacken docs Signed-off-by: samsja <[email protected]> * docs: arr warning docarray version Signed-off-by: samsja <[email protected]> * docs: repo url Signed-off-by: samsja <[email protected]> * docs: add social Signed-off-by: samsja <[email protected]> * docs: add logo Signed-off-by: samsja <[email protected]> * docs: add first step emtpy page Signed-off-by: samsja <[email protected]> * docs: add document docs Signed-off-by: samsja <[email protected]> * feat: add markdown documentation test Signed-off-by: samsja <[email protected]> * docs: remove content Signed-off-by: samsja <[email protected]> * docs: fix ci Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * chore: add docstring test (#1298) * wip Signed-off-by: samsja <[email protected]> * fix: cleanup namespace utils Signed-off-by: samsja <[email protected]> * feat: add docstring test Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: rename DocArrayProto to DocumentArrayProto (#1297) Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: docstring polish typing (#1299) * wip Signed-off-by: samsja <[email protected]> * fix: cleanup namespace utils Signed-off-by: samsja <[email protected]> * feat: add docstring test Signed-off-by: samsja <[email protected]> * fix: fix video url docstring Signed-off-by: samsja <[email protected]> * fix: fix text url Signed-off-by: samsja <[email protected]> * fix: fix image url Signed-off-by: samsja <[email protected]> * fix: fic audio url Signed-off-by: samsja <[email protected]> * fix: mesh 3d url Signed-off-by: samsja <[email protected]> * fix: mesh 3d url Signed-off-by: samsja <[email protected]> * fix: remove useless data Signed-off-by: samsja <[email protected]> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <[email protected]> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <[email protected]> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <[email protected]> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <[email protected]> * fix: fix video tensor Signed-off-by: samsja <[email protected]> * fix: fix video tensor Signed-off-by: samsja <[email protected]> * fix: fix audio bytes Signed-off-by: samsja <[email protected]> * fix: video and image bytes Signed-off-by: samsja <[email protected]> * docs: move typing section Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: remove files (#1305) Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * fix: flatten schema of abstract index (#1294) * fix: flatten schema of abstract index Signed-off-by: AnneY <[email protected]> * fix: _convert_dict_to_doc Signed-off-by: AnneY <[email protected]> * fix: catch exception when flatten schema Signed-off-by: AnneY <[email protected]> * refactor: remove useless assignemnt Signed-off-by: AnneY <[email protected]> * fix: use Abstractensor as tensor doc_type Signed-off-by: AnneY <[email protected]> * fix: add AbstractTensor to hnswlib Signed-off-by: AnneY <[email protected]> * docs: AbstractTensor as doc_type Signed-off-by: AnneY <[email protected]> * docs: complete description about AbstracTensor Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: Anne Yang <[email protected]> --------- Signed-off-by: AnneY <[email protected]> Signed-off-by: Anne Yang <[email protected]> Co-authored-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: add utils section (#1307) * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <[email protected]> * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <[email protected]> * feat: add utils for find and fix docstring Signed-off-by: samsja <[email protected]> * fix: fix video ndaray docstrng Signed-off-by: samsja <[email protected]> * fix: fix video find docstrng Signed-off-by: samsja <[email protected]> * fix: fix map docstring Signed-off-by: samsja <[email protected]> * fix: fix fileter docstring Signed-off-by: samsja <[email protected]> * fix: fix add reduce Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: fix docstring example of find_batched (#1308) Signed-off-by: Johannes Messner <[email protected]> Signed-off-by: RStar2022 <[email protected]> * docs: fix map docstring (#1311) * fix: fix utils Signed-off-by: samsja <[email protected]> * fix: fix map Signed-off-by: samsja <[email protected]> --------- Signed-off-by: samsja <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: elasticsearch document index (#1196) * feat: __init__ of ElasticDocumentIndex Signed-off-by: AnneY <[email protected]> * feat: add index func Signed-off-by: AnneY <[email protected]> * feat: get and del funcs Signed-off-by: AnneY <[email protected]> * fix: init and index creation Signed-off-by: AnneY <[email protected]> * fix: __init__ and _index Signed-off-by: AnneY <[email protected]> * fix: _get_items Signed-off-by: AnneY <[email protected]> * feat: add _find Signed-off-by: AnneY <[email protected]> * feat: add filter text and their batch version Signed-off-by: AnneY <[email protected]> * feat: store id and get nested doc Signed-off-by: AnneY <[email protected]> * fix: vector cannot be all zero Signed-off-by: AnneY <[email protected]> * fix: __getitem__ raise error Signed-off-by: AnneY <[email protected]> * feat: support more python types Signed-off-by: AnneY <[email protected]> * fix: mypy Signed-off-by: AnneY <[email protected]> * test: elastic index tests Signed-off-by: AnneY <[email protected]> * test: comment scripts before ci setup Signed-off-by: AnneY <[email protected]> * chore: add elasticsearch dependency to poetry Signed-off-by: AnneY <[email protected]> * test: elastic index ci setup Signed-off-by: AnneY <[email protected]> * feat: add num_candidates to rumtime config Signed-off-by: AnneY <[email protected]> * fix: let user pass index_settings Signed-off-by: AnneY <[email protected]> * feat: degrade to v7 and add query builder Signed-off-by: AnneY <[email protected]> * fix: remove elastic_transport Signed-off-by: AnneY <[email protected]> * feat: minor features Signed-off-by: AnneY <[email protected]> * refactor: style fix Signed-off-by: AnneY <[email protected]> * fix: fix mypy Signed-off-by: AnneY <[email protected]> * feat: add chunk size to runtime config Signed-off-by: AnneY <[email protected]> * fix: chunk size Signed-off-by: AnneY <[email protected]> * feat: add chunk_size to funcs Signed-off-by: AnneY <[email protected]> * feat: rewrite elastic v7 query builder Signed-off-by: AnneY <[email protected]> * fix: poetry Signed-off-by: AnneY <[email protected]> * fix: db_type should be elastic types Signed-off-by: AnneY <[email protected]> * fix: minor adjustment Signed-off-by: AnneY <[email protected]> * refactor: rename elastic index files Signed-off-by: AnneY <[email protected]> * refactor: remove comments Signed-off-by: AnneY <[email protected]> * feat: rename, batch operations, etc Signed-off-by: AnneY <[email protected]> * test: add test for persistency and col config Signed-off-by: AnneY <[email protected]> * feat: support more field types and subclass Signed-off-by: AnneY <[email protected]> * feat: support more python types Signed-off-by: AnneY <[email protected]> * test: tf, tensor and more elastic field types Signed-off-by: AnneY <[email protected]> * fix: elastic should be optional in toml Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: Anne Yang <[email protected]> * refactor: rename class Signed-off-by: AnneY <[email protected]> * fix: change Dict to Mapping Signed-off-by: AnneY <[email protected]> * fix: add AbstractTensor Signed-off-by: AnneY <[email protected]> * test: rename class and add tests Signed-off-by: AnneY <[email protected]> * fix: poetry Signed-off-by: AnneY <[email protected]> --------- Signed-off-by: AnneY <[email protected]> Signed-off-by: Anne Yang <[email protected]> Co-authored-by: Charlotte Gerhaher <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <[email protected]> Signed-off-by: RStar2022 <[email protected]> * feat: torch backend basic operation tests (#1306) Signed-off-by: agaraman0 <[email protected]> Signed-off-by: RStar2022 <[email protected]> * chore: add instructions to pip installs and group extras (#1281) * chore: group extras and add instructions for pip installs Signed-off-by: anna-charlotte <[email protected]> * fix: throw runtime error with install instructions for hnswlib Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for video imports Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for audio imports Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for 3d imports Signed-off-by: anna-charlotte <[email protected]> * feat: add instructions for image imports Signed-off-by: anna-charlotte <[email protected]> * fix: import only audiosegment from pydub Signed-off-by: anna-charlotte <[email protected]> * fix: generalize audio and image imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for web imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for web imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for protobuf imports Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for lz4 imports Signed-off-by: anna-charlotte <[email protected]> * fix: fastapi import Signed-off-by: anna-charlotte <[email protected]> * fix: revert changes in protobuf import Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for torch, without raising error Signed-off-by: anna-charlotte <[email protected]> * fix: add instructions for torch, with raising error Signed-off-by: anna-charlotte <[email protected]> * fix: add …

refactor: doc index structure

78795bb

Signed-off-by: jupyterjazz <[email protected]>

github-actions bot added size/s area/core area/testing labels Mar 21, 2023

jupyterjazz linked an issue Mar 21, 2023 that may be closed by this pull request

shorten doc_index imports #1265

Closed

jupyterjazz added 2 commits March 21, 2023 17:09

fix: import unused error

346a154

Signed-off-by: jupyterjazz <[email protected]>

fix: run black

a58e447

Signed-off-by: jupyterjazz <[email protected]>

jupyterjazz requested review from JohannesMessner and samsja March 21, 2023 16:17

jupyterjazz marked this pull request as ready for review March 21, 2023 16:23

samsja reviewed Mar 21, 2023

View reviewed changes

docarray/doc_index/__init__.py Outdated Show resolved Hide resolved

samsja requested changes Mar 21, 2023

View reviewed changes

refactor: remove noqa

ef10e8c

Signed-off-by: jupyterjazz <[email protected]>

jupyterjazz added 2 commits March 22, 2023 10:47

Merge branch 'feat-rewrite-v2' into refactor-doc-index-imports

091c28c

refactor: doc_index to index

7b1cd4d

Signed-off-by: jupyterjazz <[email protected]>

github-actions bot added area/cicd area/housekeeping area/setup labels Mar 22, 2023

jupyterjazz added 5 commits March 22, 2023 11:00

refactor: small changes

8eba744

Signed-off-by: jupyterjazz <[email protected]>

refactor: readme

be93c29

Signed-off-by: jupyterjazz <[email protected]>

refactor: add init file

d17ebd1

Signed-off-by: jupyterjazz <[email protected]>

refactor: move docindex outside integration tests

58e3085

Signed-off-by: jupyterjazz <[email protected]>

refactor: adjust config paths

b171afa

Signed-off-by: jupyterjazz <[email protected]>

jupyterjazz requested a review from samsja March 22, 2023 11:06

samsja approved these changes Mar 22, 2023

View reviewed changes

samsja merged commit 6707f4c into feat-rewrite-v2 Mar 22, 2023

samsja deleted the refactor-doc-index-imports branch March 22, 2023 12:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: doc index structure#1266

refactor: doc index structure#1266
samsja merged 11 commits intofeat-rewrite-v2from
refactor-doc-index-imports

jupyterjazz commented Mar 21, 2023 •

edited

Loading

Uh oh!

Uh oh!

samsja left a comment

Uh oh!

JohannesMessner commented Mar 21, 2023

Uh oh!

samsja commented Mar 21, 2023 •

edited

Loading

Uh oh!

jupyterjazz commented Mar 21, 2023

Uh oh!

jupyterjazz commented Mar 22, 2023

Uh oh!

samsja commented Mar 22, 2023

Uh oh!

github-actions bot commented Mar 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jupyterjazz commented Mar 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

shorter imports

less verbose file names

Uh oh!

Uh oh!

samsja left a comment

Choose a reason for hiding this comment

Uh oh!

JohannesMessner commented Mar 21, 2023

Uh oh!

samsja commented Mar 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jupyterjazz commented Mar 21, 2023

Uh oh!

jupyterjazz commented Mar 22, 2023

Uh oh!

samsja commented Mar 22, 2023

Uh oh!

github-actions bot commented Mar 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jupyterjazz commented Mar 21, 2023 •

edited

Loading

samsja commented Mar 21, 2023 •

edited

Loading