This folder contains benchmarks used to evaluate design decisions in audb.

The reported results were calculated on the following machine:

• CPU: 13th Gen Intel Core i7-1355U, 10-core (2-mt/8-st)
• RAM: 15.29 GiB
• Hard drive: KIOXIA KXG8AZNV1T02
• Linux: Ubuntu 22.04
• Python: 3.10

Benchmarks all methods of audb.Dependencies besides audb.Dependencies.load() and audb.Dependencies.save(). This benchmark provides insights how to best represent the dependency table internally.

To run the benchmark execute:

$ uv run --python 3.10 benchmark-dependencies-methods.py

Execution times in seconds of audb.Dependencies methods for a dependency table containing 1,000,000 files stored as a pandas.DataFrame using different dtype representations (storing string as string, storing string as object, using pyarrow dtypes).

Handling of the dependency table with pandas was further compared to handling it with polars, by reimplementing all methods of audb.Dependencies using polars.

This benchmark was executed on:

• CPU: 12th Gen Intel Core i7-1255U
• RAM: 15.66 GB
• Hard Drive: KBG5AZNT1T02 LA KIOXIA
• Linux: Ubuntu 22.04.4 LTS
• Python 3.11.9

To run the benchmark execute:

$ uv run --python 3.11 benchmark-dependencies-methods-polars.py

The data were compared to the results from benchmark-dependencies-methods.py using

$ uv run --python 3.11 compare_dependency_methods_polars.py

Both steps require that benchmark-dependencies-methods.py has been run previously in order to create the test data and results. The comparison in the pandas column is based on the pyarrow column in the tabulation in the previous section.

Benchmarks how fast a dependency table can be written to disk or read from disk using different file formats (csv, pickle, parquet) and different internal representations (pandas.DataFrame with different dtypes, and pyarrow.Table).

To run the benchmark execute:

$ uv run --python 3.10 benchmark-dependencies-save-and-load.py

Writing

Execution times in seconds for a dependency table containing 1,000,000 files.

Reading

Execution times in seconds for a dependency table containing 1,000,000 files. -c--> stands for using the c engine and -pa-> for using the pyarrow engine with pandas.read_csv().

Storing a dependency table with 1,000,000 entries resulted in:

• 102 MB for csv
• 131 MB for pickle
• 20 MB for parquet

When zipped all files can be further reduced by 50%.

Besides the execution time, memory consumption is also measured when running the benchmark script. We use memray v1.11.0, to measure it. It stores its evaluation results in binary files in the folder results. Those files are then manually evaluated using memray tree <file>.

Writing

When writing to files there is no memory overhead when converting a pandas.DataFrame first to pyarrow.Table. Hence, we don't have to compare results.

Reading

Peak memory consumption when reading a dependency table containing 1,000,000 files.