Signed-off-by: Michael Guenther [email protected]

Goals:

• Support recall@k and F1 measure@k for labeled datasets.
• check and update documentation, if required. See guide

For labeled datasets it is not trivial to calculate metrics like recall and F1 measure, which require the knowledge of the number of relevant documents in the document collection for each query since neither the whole set of relevant documents nor the number of documents with a specific label is provided to the evaluate function.

To enable the calculation of recall and F1 measure, this PR

• adds a parameter max_rel_per_label: Dict to the evaluate function which provides the number of relevant documents for each label, i.e., the number of documents in the collection with this label.
• calculates those label counts for max_rel_per_label in the embed_and_evaluate_function.

Code Example:

# example DocumentArray with matches and labels for evaluation 
da = DocumentArray([Document(text=str(i), tags={'label': i}) for i in range(3)])
for d in da:
  d.matches = da
# each label occurs one time in the data collection (not provided to the evaluate function)
max_rel_per_label = {i: 1 for i in range(3)}
# evaluate matches
metrics = da.evaluate(['recall_at_k'], max_rel_per_label=max_rel_per_label)
print(metrics)

{'recall_at_k': 1.0}