Skip to content

perf: only check if field is set#519

Merged
alaeddine-13 merged 3 commits intomainfrom
perf-filter-exists
Sep 5, 2022
Merged

perf: only check if field is set#519
alaeddine-13 merged 3 commits intomainfrom
perf-filter-exists

Conversation

@alaeddine-13
Copy link
Copy Markdown
Member

@alaeddine-13 alaeddine-13 commented Sep 2, 2022 •
edited
Loading

speed up the filter case described in #502 (comment)
code:

import time
from docarray import DocumentArray, Document

num = 10_000
da = DocumentArray(Document(text='lsdkfjsl') for x in range(num)) + DocumentArray(Document(blob=b'lsdkfjsl') for x in range(num))

start = time.time()
print(len(da.find(query={'text': {'$exists': True}})))
print(f'time with find: {time.time() - start}')

main: 0.24
feat: 0.077

closes: https://github.com/jina-ai/internal-tasks/issues/426

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 2, 2022 •
edited
Loading

Codecov Report

Merging #519 (09fa4fe) into main (b268f6f) will increase coverage by 2.16%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #519      +/-   ##
==========================================
+ Coverage   84.05%   86.22%   +2.16%     
==========================================
  Files         142      142              
  Lines        7013     7018       +5     
==========================================
+ Hits         5895     6051     +156     
+ Misses       1118      967     -151
Flag Coverage Δ
docarray 86.22% <100.00%> (+2.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/array/queryset/lookup.py 78.26% <100.00%> (+0.15%) ⬆️
docarray/document/data.py 91.83% <100.00%> (+0.34%) ⬆️
docarray/array/mixins/traverse.py 95.04% <0.00%> (+4.13%) ⬆️
docarray/document/mixins/sugar.py 68.00% <0.00%> (+12.00%) ⬆️
docarray/helper.py 81.90% <0.00%> (+13.12%) ⬆️
docarray/array/mixins/group.py 87.17% <0.00%> (+17.94%) ⬆️
docarray/array/mixins/post.py 93.02% <0.00%> (+74.41%) ⬆️
docarray/array/mixins/embed.py 89.47% <0.00%> (+78.94%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@alaeddine-13 alaeddine-13 merged commit d615233 into main Sep 5, 2022
@alaeddine-13 alaeddine-13 deleted the perf-filter-exists branch September 5, 2022 16:14
@hanxiao hanxiao mentioned this pull request Sep 22, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JoanFM JoanFM JoanFM approved these changes

Assignees

No one assigned

Labels

area/core component/array component/document size/s

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alaeddine-13 @JoanFM