Of the new tests added, 1 of them is expected to fail.

DocArrays backed by Weaviate seem to have 2 references to them upon creation.
This means they cannot be garbage collected when they go out of scope, possibly creating a memory leak

Hey @Jackmin801 , please solve the commit linting problem to have the tests run

Ah ok. I purposely failed it cause I wasnt sure if I should be constantly running the CI. Seems like they are a bit expensive

Codecov Report

Merging #604 (f9fcba4) into main (262b2d1) will decrease coverage by 9.18%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##             main     #604      +/-   ##
==========================================
- Coverage   84.91%   75.72%   -9.19%     
==========================================
  Files         133      133              
  Lines        6714     6723       +9     
==========================================
- Hits         5701     5091     -610     
- Misses       1013     1632     +619     

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

4. I also think it's clearer to put just _save_offset2id (or any logic that cannot run during program exit) in the exit hook, not all of del.

I like this idea

After some discussion internally among the maintainers, it was decided DocArray will only support explicit sync and sync when exiting context manager.

There are four candidate places we can sync:

1. [self.sync] When the user explicitly calls da.sync()
2. [self.__exit__] When the user exits the DocArray’s context manager
3. [self.__del__] When the variable gets garbage collected
4. [atexit.register] When the program exits

DocArray will only support (1) and (2) -- explicit sync and sync when exiting context manager.

We will not support syncing when the variable gets garbage collected or when the program exits.
Because:

1. Not supporting them would make it such that we do not need to keep track of a _synced flag. Which simplifies the logic.
2. They are too implicit and difficult to predict. Using explicit sync and the pythonic context manager are intentional and have easy to understand, predictable behavior. Garbage collection and trying to sync on program exit can have edge cases that can make them tricky for users to use and would breed bad practice in their code.

We are no longer pursuing adding the behavior of persistence to atexit. The intended persistence behavior will be implemented in #625