Skip to content

feat: allow empty to get kwargs#91

Merged
hanxiao merged 7 commits intomainfrom
feat-empty-kwargs
Feb 3, 2022
Merged

feat: allow empty to get kwargs#91
hanxiao merged 7 commits intomainfrom
feat-empty-kwargs

Conversation

@davidbp
Copy link
Copy Markdown
Contributor

@davidbp davidbp commented Feb 3, 2022

This PR allows .empty to work with storage backends that need "configuration information".

Consider the following setting

from docarray import Document, DocumentArray
import numpy as np

n_docs = 500
n_features = 128

A user can do

# works in sqlite
storage='sqlite'
da = DocumentArray(storage=storage).empty(n_docs)
da[:, 'embedding'] = [x for x in np.random.random((n_docs,n_features))]
assert da[:,'embedding'].shape == (n_docs, n_features)
da.embeddings = np.random.random((n_docs,n_features))
assert da[:,'embedding'].shape == (n_docs, n_features)

but can't do

storage='pqlite'
da_pqlite = DocumentArray(storage=storage, config={'n_dim': n_features}).empty(n_docs)
da_pqlite[:, 'embedding'] = [x for x in np.random.random((len(da_pqlite),n_features))]
assert da_pqlite[:,'embedding'].shape == (500, 128)
da_pqlite.embeddings = np.random.random((len(da_pqlite),n_features))
assert da_pqlite[:,'embedding'].shape == (500, 128)

Because the information from config is ignored with .empty is called.

With this PR users can do

storage='pqlite'
da_pqlite = DocumentArray(storage=storage).empty(n_docs, config={'n_dim': n_features})
da_pqlite[:, 'embedding'] = [x for x in np.random.random((len(da_pqlite),n_features))]
assert da_pqlite[:,'embedding'].shape == (500, 128)
da_pqlite.embeddings = np.random.random((len(da_pqlite),n_features))
assert da_pqlite[:,'embedding'].shape == (500, 128)

@codecov
Copy link
Copy Markdown

codecov bot commented Feb 3, 2022

Codecov Report

Merging #91 (cb25463) into main (2a80abf) will increase coverage by 0.11%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #91      +/-   ##
==========================================
+ Coverage   85.13%   85.24%   +0.11%     
==========================================
  Files          98       98              
  Lines        4392     4386       -6     
==========================================
  Hits         3739     3739              
+ Misses        653      647       -6     
Flag Coverage Δ
docarray 85.24% <100.00%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/array/mixins/io/binary.py 93.65% <ø> (ø)
docarray/array/mixins/io/from_gen.py 83.63% <ø> (ø)
docarray/__init__.py 100.00% <100.00%> (ø)
docarray/array/mixins/empty.py 87.50% <100.00%> (ø)
docarray/array/mixins/io/pushpull.py 94.93% <100.00%> (ø)
docarray/array/mixins/pydantic.py 82.35% <100.00%> (ø)
docarray/array/storage/weaviate/backend.py 87.73% <0.00%> (+0.46%) ⬆️
docarray/array/storage/sqlite/backend.py 92.10% <0.00%> (+0.85%) ⬆️
docarray/array/mixins/plot.py 64.92% <0.00%> (+2.01%) ⬆️
docarray/array/storage/memory/backend.py 97.82% <0.00%> (+2.17%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74ce861...cb25463. Read the comment docs.

Copy link
Copy Markdown
Member

@hanxiao hanxiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only DocumentArray class method need to be extended. not Documet class method

@github-actions github-actions bot added size/s and removed size/m labels Feb 3, 2022
@davidbp
Copy link
Copy Markdown
Contributor Author

davidbp commented Feb 3, 2022

only DocumentArray class method need to be extended. not Documet class method

Ok, reverted

@hanxiao hanxiao merged commit 569f871 into main Feb 3, 2022
@hanxiao hanxiao deleted the feat-empty-kwargs branch February 3, 2022 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants