-
Notifications
You must be signed in to change notification settings - Fork 238
feat: add sqlite storage backend #69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
68 commits
Select commit
Hold shift + click to select a range
2736dde
refactor(array): use extend in add improve docs
hanxiao d9739f6
Merge branch 'main' into feat-persist
hanxiao 590961f
chore: bump to 0.2
hanxiao 06b4b51
Merge branch 'main' into feat-persist
hanxiao 7369901
chore: fix typo
hanxiao 8cb7baa
chore: fix typo
hanxiao 95b4a37
fix(document): serialize tag value in the correct priority
hanxiao b3b378f
fix(document): complete the schema for namedscore
hanxiao c1d0084
feat(array): add storage backend
hanxiao eac7071
Merge branch 'main' into feat-persist
hanxiao dee8457
feat(array): add storage backend
hanxiao 34ad9e1
feat(array): add storage backend
hanxiao 1621b94
feat(array): add storage backend
hanxiao c8800b4
feat(array): add storage backend
hanxiao 6029a14
feat(array): add storage backend
hanxiao 8936d77
feat(array): add storage backend
hanxiao 58da76e
feat(array): add storage backend
hanxiao 80193cd
feat: implement sugar API
alaeddine-13 8ea8935
test: add sqlite tests
alaeddine-13 aa7e4ca
fix: fix sqlite set_doc_by_id
alaeddine-13 c6a6d6f
fix: fix set_doc_by_id
alaeddine-13 c1423b7
fix: DocumentArraySqlit._set_doc_by_id should behave like DocumentArr…
alaeddine-13 6397a46
fix: raise TypeError when assigning non iterable with slice
alaeddine-13 f15fd77
fix: fix del attribute
alaeddine-13 5eb3881
test: fix test_tensor_attribute_selector
alaeddine-13 8deaacd
fix: delete by offset was missing shift of ids
davidbp 3aa538b
fix: _set_doc_by_id should behave like DocumentArrayInMemory
alaeddine-13 6b0b567
fix: raise TypeError when assigning non iterable with slice
alaeddine-13 27c34df
fix: fix del attribute
alaeddine-13 9b9c2d5
test: fix test_tensor_attribute_selector
alaeddine-13 f6f60b9
fix: delete by offset was missing shift of ids
davidbp 160118f
fix: fix linting
alaeddine-13 a912f5b
Merge branch 'feat-persist' of https://github.com/jina-ai/docarray in…
davidbp 51401d4
fix: black
alaeddine-13 8a89396
fix: cover negative offset
davidbp 313e844
Merge branch 'feat-persist' of https://github.com/jina-ai/docarray in…
davidbp 2215d81
test: cover negative indices
davidbp 183c079
fix(storage): fix doc id usage in _set_doc_value_pairs
hanxiao 829c5b4
fix(storage): fix doc id usage in _set_doc_value_pairs
hanxiao 23cd905
fix(storage): fix _set_doc_value_pairs
hanxiao e1f30df
fix: offset set by id
davidbp 0c2a825
Merge branch 'feat-persist' of https://github.com/jina-ai/docarray in…
davidbp 901e0d0
refactor: add boolean slicing in sqlite
davidbp f4ef442
refactor: correct set item
davidbp 906ff8b
test: fix test_advance_selector_mixed
alaeddine-13 f99766b
style: fix black style
hanxiao d9f1c6f
style: fix black style
hanxiao e7b5a89
fix(sqlite): fix iter method in sqlite
hanxiao 1c9cb0c
fix(sqlite): fix iter method in sqlite
hanxiao 3c38084
fix(sqlite): revert shift index right backward
hanxiao 30e05cf
fix: use separate cursor for iterator
alaeddine-13 baa25f3
fix: use separate cursor
alaeddine-13 d2c2ff6
fix: merge conflicts
alaeddine-13 63cb55e
fix: persist when table_name is specified
alaeddine-13 681aa84
test: add base tests
alaeddine-13 41f38ad
fix(sqlite): fix _get_docs_by_offsets
hanxiao cb6ce42
fix: linting
alaeddine-13 47bcade
Merge branch 'feat-persist' of https://github.com/jina-ai/docarray in…
alaeddine-13 be1c7c3
fix(sqlite): fix _get_docs_by_offsets
hanxiao 4ae4413
Merge branch 'feat-persist' of https://github.com/jina-ai/docarray in…
hanxiao fc54273
test: test base
alaeddine-13 0418f2d
Merge branch 'feat-persist' of https://github.com/jina-ai/docarray in…
alaeddine-13 c83a755
fix: linting
alaeddine-13 6a8db70
feat(sqlite): improve type hint
hanxiao 4e678e1
Merge remote-tracking branch 'origin/feat-persist' into feat-persist
hanxiao 62aa9da
feat(sqlite): improve type hint
hanxiao 778aae4
chore: bump version to 0.5
hanxiao f4c1449
Merge branch 'main' into feat-persist
hanxiao File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| __version__ = '0.4.5' | ||
| __version__ = '0.5.0' | ||
|
|
||
| from .document import Document | ||
| from .array import DocumentArray |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| from typing import MutableSequence | ||
|
|
||
| from .. import Document | ||
|
|
||
|
|
||
| class BaseDocumentArray(MutableSequence[Document]): | ||
| def __init__(self, *args, storage: str = 'memory', **kwargs): | ||
| super().__init__() | ||
| self._init_storage(*args, **kwargs) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,12 +7,13 @@ | |
| ) | ||
|
|
||
| from .document import DocumentArray | ||
| from .memory import DocumentArrayInMemory | ||
|
|
||
| if TYPE_CHECKING: | ||
| from ..document import Document | ||
|
|
||
|
|
||
| class ChunkArray(DocumentArray): | ||
| class ChunkArray(DocumentArrayInMemory): | ||
| """ | ||
| :class:`ChunkArray` inherits from :class:`DocumentArray`. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please don't forget to update the doc str accordingly |
||
| It's a subset of Documents. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is
_init_storage()still necessary?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh im actually not so sure now, will wait for @alaeddine-13