perf: sync sub index only when parent is synced#719
Conversation
Codecov Report
@@ Coverage Diff @@
## main #719 +/- ##
=======================================
Coverage 88.28% 88.28%
=======================================
Files 134 134
Lines 6640 6641 +1
=======================================
+ Hits 5862 5863 +1
Misses 778 778
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
| with subindex_da: | ||
| subindex_da.clear() | ||
| subindex_da.extend(docs) | ||
| subindex_da.clear() |
There was a problem hiding this comment.
I do not understand the problem, maybe a more clear description of the problem would clarify the scope
There was a problem hiding this comment.
The issue is the usage of the context manager for subindices. It is used "all the time", even if the main (parent) da does not use it, so subindex offset2ids are always synced, and there is no control over it.
This change undoes this, and instead ties the syncing of the subindices to the syncing of the parent, by making .sync() operate on all the subindices. I like it.
|
I would like to have a better description of this and understand the problem with a potential test |
Currently, any elemental operation of a DA with subindices will attempt to sync the underlying sub indices even though the parent is not synced.
This PR tries to align the behavior and avoid the performance cost of syncing subindices at each operation.