Skip to content

feat: add in-memory doc index#1441

Merged
samsja merged 19 commits intomainfrom
feat-in-memory-doc-index
Apr 26, 2023
Merged

feat: add in-memory doc index#1441
samsja merged 19 commits intomainfrom
feat-in-memory-doc-index

Conversation

@anna-charlotte
Copy link
Copy Markdown
Contributor

Add an in-memory DocIndex class

Signed-off-by: anna-charlotte <[email protected]>

def execute_query(self, query: List[str], *args, **kwargs) -> Any:
"""
Execute a query on the HnswDocumentIndex.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Execute a query on the HnswDocumentIndex.
Execute a query on the InMemoryIndex.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WHat is this Query? what QL will it follow?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One can build a query consisting of find() and filter() calls, where the QL for the filtering follows the QL of our filter_docs() function. I will put more information on this in the docstrings

anna-charlotte added 4 commits April 25, 2023 08:09
Signed-off-by: anna-charlotte <[email protected]>
Signed-off-by: anna-charlotte <[email protected]>
Signed-off-by: anna-charlotte <[email protected]>
Signed-off-by: anna-charlotte <[email protected]>
anna-charlotte added 5 commits April 25, 2023 10:18
Signed-off-by: anna-charlotte <[email protected]>
Signed-off-by: anna-charlotte <[email protected]>
Signed-off-by: anna-charlotte <[email protected]>
@github-actions github-actions bot added size/l and removed size/m labels Apr 25, 2023
@anna-charlotte anna-charlotte marked this pull request as ready for review April 25, 2023 11:09
Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: Charlotte Gerhaher <[email protected]>
@anna-charlotte anna-charlotte requested a review from alexcg1 April 25, 2023 12:53
Comment on lines +148 to +149
def execute_query(self, query: List[Tuple[str, Dict]], *args, **kwargs) -> Any:
"""
Copy link
Copy Markdown
Member

@JohannesMessner JohannesMessner Apr 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method has significant overlap with its counterpart in the hnswlib implementation. Do you think we can share that code somehow? either have helper methods somewhere, or through inheritance?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i put it into a helper file in the backends dir

Copy link
Copy Markdown
Member

@JohannesMessner JohannesMessner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! just some small things

Signed-off-by: anna-charlotte <[email protected]>
anna-charlotte and others added 3 commits April 25, 2023 16:09
Signed-off-by: anna-charlotte <[email protected]>
Co-authored-by: Alex Cureton-Griffiths <[email protected]>
Signed-off-by: Charlotte Gerhaher <[email protected]>
Signed-off-by: anna-charlotte <[email protected]>
Copy link
Copy Markdown
Contributor

@alexcg1 alexcg1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please only ask me to review after all feedback has been resolved 😅

@alexcg1
Copy link
Copy Markdown
Contributor

alexcg1 commented Apr 25, 2023

Please only ask me to review after all feedback has been resolved sweat_smile

Oops. My bad. I saw the notification and I assumed a review had been re-requested. Can ignore my prior comment

anna-charlotte added 2 commits April 25, 2023 17:02
Signed-off-by: anna-charlotte <[email protected]>
@samsja
Copy link
Copy Markdown
Member

samsja commented Apr 26, 2023

this pr might fix this bug : #1445

@github-actions
Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-in-memory-doc-index--jina-docs.netlify.app 🎉

Copy link
Copy Markdown
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets wait to merge it until we decide if it go into the next release or not

@samsja samsja merged commit f65e023 into main Apr 26, 2023
@samsja samsja deleted the feat-in-memory-doc-index branch April 26, 2023 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants