Skip to content

feat(redis): add full-text search and io#535

Merged
bwanglzu merged 11 commits intodocarray:mainfrom
AnneYang720:feat-redis-text
Sep 14, 2022
Merged

feat(redis): add full-text search and io#535
bwanglzu merged 11 commits intodocarray:mainfrom
AnneYang720:feat-redis-text

Conversation

@AnneYang720
Copy link
Copy Markdown
Contributor

@AnneYang720 AnneYang720 commented Sep 9, 2022

Goals:
This PR will add support of full-text search in redis according to #500 and add io functions in redis.

  • add code for redis full-text search
  • add test for redis full-text search
  • add redis full-text search documentation
  • add code for redis io
  • add test for redis io

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 9, 2022

Codecov Report

Merging #535 (1c07793) into main (638f362) will decrease coverage by 1.77%.
The diff coverage is 97.43%.

@@            Coverage Diff             @@
##             main     #535      +/-   ##
==========================================
- Coverage   86.25%   84.47%   -1.78%     
==========================================
  Files         142      142              
  Lines        7050     7087      +37     
==========================================
- Hits         6081     5987      -94     
- Misses        969     1100     +131     
Flag Coverage Δ
docarray 84.47% <97.43%> (-1.78%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/array/storage/redis/find.py 94.11% <95.23%> (+0.14%) ⬆️
docarray/array/storage/redis/backend.py 96.61% <100.00%> (+0.41%) ⬆️
docarray/array/storage/redis/getsetdel.py 97.01% <100.00%> (+0.24%) ⬆️
docarray/array/mixins/embed.py 10.52% <0.00%> (-78.95%) ⬇️
docarray/array/mixins/post.py 18.60% <0.00%> (-74.42%) ⬇️
docarray/array/mixins/group.py 69.23% <0.00%> (-17.95%) ⬇️
docarray/helper.py 68.77% <0.00%> (-13.13%) ⬇️
docarray/document/mixins/sugar.py 56.00% <0.00%> (-12.00%) ⬇️
docarray/array/mixins/traverse.py 90.90% <0.00%> (-4.14%) ⬇️
docarray/proto/pb2/docarray_pb2.py 100.00% <0.00%> (+39.62%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Copy Markdown

@bwanglzu bwanglzu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some comments

# (DocumentArrayAnnlite, PqliteConfig(n_dim=3)), # TODO: enable this
# (DocumentArrayQdrant, QdrantConfig(n_dim=3)),
# (DocumentArrayElastic, ElasticConfig(n_dim=3)), # Elastic needs config
# (DocumentArrayRedis, RedisConfig(n_dim=3, flush=True)), # Redis needs config
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why commented?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Annlite, Qdrant and Elastic were already commented. I think Redis is the same as Elastic that they both need a non-empty config, so they cannot use from_base64

@bwanglzu
Copy link
Copy Markdown

make sure the BM25 and TF-IDF is documented

Copy link
Copy Markdown

@bwanglzu bwanglzu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comments

Copy link
Copy Markdown

@bwanglzu bwanglzu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Copy Markdown
Member

@alaeddine-13 alaeddine-13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great if we also test scorer. We probably will not test the scoring itself but we can check that using the parameter doesn't break anything

Copy link
Copy Markdown
Member

@alaeddine-13 alaeddine-13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bwanglzu bwanglzu merged commit ea2a7a8 into docarray:main Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants