Skip to content

feat: add flag to disable list-like structure in DocArray#730

Merged
JoanFM merged 1 commit intomainfrom
feat-add-flag-to-disable-offset2id
Nov 10, 2022
Merged

feat: add flag to disable list-like structure in DocArray#730
JoanFM merged 1 commit intomainfrom
feat-add-flag-to-disable-offset2id

Conversation

@dongxiang123
Copy link
Copy Markdown
Contributor

@dongxiang123 dongxiang123 commented Nov 8, 2022

Goals:

  • ...
  • ...DocArray allows list-like behavior by adding offset2id structure to storage backends. This means the database stores, along with documents, meta information about document order.
    This information is only updated/synced when a DA is destroyed or quit the context manager. Furthermore, offset2id is updated in memory in all update operations.
    However, offset2id is not useful in indexers where concurrent usage is possible and users do not need information about document location. Besides, updating offset2id is not efficient.
    Therefore, it's important to have an option that disables list-like behavior (and offset2id with it) and use it in DocumentArray indexers.
@dataclass
class QdrantConfig:
    n_dim: int
    distance: str = 'cosine'
    list_like: bool = True
    collection_name: Optional[str] = None
    host: Optional[str] = field(default="localhost")
    port: Optional[int] = field(default=6333)
    serialize_config: Dict = field(default_factory=dict)
    scroll_batch_size: int = 64
    ef_construct: Optional[int] = None
    full_scan_threshold: Optional[int] = None
    m: Optional[int] = None
    columns: Optional[Union[List[Tuple[str, str]], Dict[str, str]]] = None
  • ...
  • ...
  • check and update documentation, if required. See guide

@dongxiang123 dongxiang123 changed the title chore: split tests with older proto version (#725) feat: add flag to disable offsettoid in Offsettoid class Nov 8, 2022
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 3 times, most recently from f5d0536 to 8421596 Compare November 8, 2022 11:09
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Nov 8, 2022

Codecov Report

Base: 83.50% // Head: 36.53% // Decreases project coverage by -46.97% ⚠️

Coverage data is based on head (a09ef37) compared to base (938c061).
Patch coverage: 36.84% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #730       +/-   ##
===========================================
- Coverage   83.50%   36.53%   -46.98%     
===========================================
  Files         133      133               
  Lines        6651     6703       +52     
===========================================
- Hits         5554     2449     -3105     
- Misses       1097     4254     +3157     
Flag Coverage Δ
docarray 36.53% <36.84%> (-46.98%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/array/storage/annlite/backend.py 0.00% <0.00%> (-97.34%) ⬇️
docarray/array/storage/annlite/getsetdel.py 0.00% <0.00%> (-100.00%) ⬇️
docarray/array/storage/base/getsetdel.py 39.72% <0.00%> (-46.58%) ⬇️
docarray/array/storage/redis/backend.py 0.00% <0.00%> (-97.40%) ⬇️
docarray/array/storage/redis/getsetdel.py 0.00% <0.00%> (-96.97%) ⬇️
docarray/array/storage/redis/seqlike.py 0.00% <0.00%> (-86.67%) ⬇️
docarray/array/storage/sqlite/backend.py 0.00% <0.00%> (-94.88%) ⬇️
docarray/array/storage/sqlite/getsetdel.py 0.00% <0.00%> (-100.00%) ⬇️
docarray/array/storage/weaviate/backend.py 0.00% <0.00%> (-90.26%) ⬇️
docarray/array/storage/weaviate/getsetdel.py 0.00% <0.00%> (-100.00%) ⬇️
... and 117 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Copy Markdown
Member

@JoanFM JoanFM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure to Document this behavior

@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 3 times, most recently from a34c453 to 8c7bcb8 Compare November 9, 2022 03:35
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch from 8c7bcb8 to 623b8ae Compare November 9, 2022 03:40
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch from af1f0b4 to 0647cfb Compare November 9, 2022 04:17
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 3 times, most recently from faf1528 to 46297dc Compare November 9, 2022 04:37
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 2 times, most recently from 21d8e69 to 207725f Compare November 9, 2022 06:21
Copy link
Copy Markdown
Member

@alaeddine-13 alaeddine-13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also methods that create the offset2id schema should be disabled if the flag is false

@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 3 times, most recently from 6730cdf to 85db92e Compare November 9, 2022 11:57
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 4 times, most recently from da36af1 to 23ff90f Compare November 9, 2022 13:44
@dongxiang123 dongxiang123 changed the title feat: add flag to disable offsettoid in Offsettoid class feat: add flag to disable list-like structure in DocArray Nov 9, 2022
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 2 times, most recently from 36d1a31 to f300d8b Compare November 9, 2022 14:36
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 3 times, most recently from 1502b68 to a028e01 Compare November 10, 2022 02:07
@JoanFM JoanFM requested a review from alaeddine-13 November 10, 2022 07:16
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 3 times, most recently from 9d6a9d0 to 2a04fa6 Compare November 10, 2022 08:10
@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch 2 times, most recently from 867f5ca to 6d668b7 Compare November 10, 2022 08:39
@github-actions
Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-add-flag-to-disable-offset2id--jina-docs.netlify.app 🎉

@dongxiang123 dongxiang123 force-pushed the feat-add-flag-to-disable-offset2id branch from 6d668b7 to a09ef37 Compare November 10, 2022 10:11
@github-actions
Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-add-flag-to-disable-offset2id--jina-docs.netlify.app 🎉

@JoanFM JoanFM merged commit 706e884 into main Nov 10, 2022
@JoanFM JoanFM deleted the feat-add-flag-to-disable-offset2id branch November 10, 2022 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants