Proposal

Hi,

Using prometheus at scale (we run prom under two level of thanos federation, sometimes three), we are regularly faced with a UX pain point where people try to query prometheus directly / one of the intermediate thanos, and mess up using the proper labels. For instance they will specify prometheus="foobar" in their query on the actual prometheus, which will return no data.

We were wondering what you would think about enabling prometheus (as a first step) to handle this natively.

I can see three levels of various implementations of this:

1. most naive: just strip any label that is currently defined as an external label.
   a. i.e. if I have prometheus: foo defined as an external label, I can query up{prometheus="bar"}, which will be rewritten to up{}. This obviously returns false positives.
2. strict matches: check against the value defined in the config, only allow strict matches
   a. i.e. if I have prometheus: foo defined as an external label, I can query up{prometheus="foo"} but not up{prometheus=~"fo."}
   b. I don't like this solution, but I'm mentioning it because I've read other places where only strict matches are allowed. Assuming that it's still true, that there is a technical justification for this, and that we want to match that behavior... here it is
3. actual match: just check against the value, allowing strict and regex match

Either of these solutions would be handle at query time on the fly, without actually writing the external labels to the metric (i.e. not changing the behavior on the actual metrics). I don't know off hand why we would implement 1 and 2 over 3, but it's cheap to list them as options.

Hopefully this makes sense. Is that something we would be open to implementing? Or accept a PR for?

Let me know if you have any question. And happy to read about any concerns / why this wouldn't be possible.
Thank you for your time!