This PR has been split from #10192

Thanks for working on this, @armru and @simonapencea!

I've been thinking about this approach and wanted to share some concerns.
My main hesitation is that checking for the existence of the VolumeSnapshot object alone doesn't give us a complete picture. A PVC can stay in Pending state for several reasons beyond a missing VolumeSnapshot:

• The VolumeSnapshotContent object may not exist
• The VolumeSnapshotContent may reference a snapshot that was deleted at the storage layer
• The CSI driver may be experiencing issues
• The snapshot may have readyToUse: false

Since #10029 already addresses the root cause by preventing the operator from selecting missing snapshots as candidates for new replicas, I'm wondering if this additional cleanup logic is necessary.
The preventive approach should stop this situation from occurring in the first place.

I'm also a bit cautious about the operator automatically deleting PVCs based on this heuristic.
If we can't be certain why a PVC is stuck in Pending, it feels safer to leave it for an administrator to investigate rather than risk deleting something based on an incomplete check.

Would you be open to dropping this part and relying on the preventive fix from #10029?

I agree with @leonardoce's concerns. The heuristic of checking only for the VolumeSnapshot object is indeed incomplete, and there are several other reasons a PVC can remain in Pending state that we wouldn't catch.

As i mentioned in the original PR #10029, there is a small window of time where a volume snapshot that passed the existence check can be deleted before the job starts and the pvc gets created - and as far as i understood, this PR intended to cover for that situation.
However, i believe that this window is relatively small so it is essentially a corner case from where we can recover easily now - just by deleting the pending PVC and job, whereas before it was pretty involved.
So PR #10029 already improves the situation by a lot.
What this PR does is, as far as i understand, provide the basis for a sort of self-healing, which is always complicated to get right. It would be a nice feature to have, but atm it may be a bit biased toward the use-case we discussed and maybe misses some other use cases.
I think this one can wait until we can gain more context.

That being said, another path may be to allow users to essentially opt-out of using volumesnapshots for replicas. This will empower the users to select what best fits their use-case. Just a thought.