We currently don't have the best observability into the status of the data processing pipeline runs.

Specifically it would be ideal to collect information on DAG runtimes and learn if some runs have failed.

Airflow supports this by producing otel compatible metrics.

As part of this task we should:

• Add otel collector support for our ansible roles (see: https://opentelemetry.io/blog/2024/scaling-collectors/)
• Configure airflow to write to this collector locally (see: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-opentelemetry)
• Expose these metrics so that they can be scraped by prometheus using the otel prometheusexporter (see: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md)