Context

refactor the proto serialization under the hood.

In DocArray we have different types with different functionality that will be stored using the same raw type (bytes, str, NdArray). So once we serialize this types we loose the information of what is the true DocArray type. Is the NdArray in the proto and AudioTensor or a VideoTensor ?

Therefore we need to have a way to encode this information somewhere in the proto.

Before this PR

Before this PR we were storing this information (which DocArray type does the object belongs to) in the key of the proto itself. docarray.proto looked like this :

message NodeProto {

  oneof content {
    bytes blob = 1;
    // the ndarray of the image/audio/video document
    NdArrayProto ndarray = 2;
    // a text
    string text = 3;
    // a sub Document
    
    NdArrayProto audio_ndarray = 4
    NdArrayProto video_ndarray = 5
  }

  oneof docarray_type {
    string type = 6;
  }

}

But this does not scale well as we need to modify the proto each time we add a new modalities.

What this PR do

This PR change the behavior and instead of storing the information as a key we store it as a extra string. We basically save the schema of the Document in the proto.

Therefore for each of our Type we need to assign a corresponding string that will be used when serializing. In class is store its key so we have mapping from class to key. Nevertheless for deserializing we need to inverse, i.e, a mapping from key to class.

To avoid to store this double mapping in two place and doing copy pasting we introduce the
register_proto class decorator that take a key as parameters and will assign the key to the class and at the same time register the class and the key into a dict to have the mapping from key to class. This dict will then be used in the deserializing part.

The proto looks like this

message NodeProto {

  oneof content {
    bytes blob = 1;
    // the ndarray of the image/audio/video document
    NdArrayProto ndarray = 2;
    // a text
    string text = 3;
    // a sub Document

  }

  oneof docarray_type {
    string type = 6;
  }

}