Determine the percentage of valid invariants inferred by TrainCheck, w.r.t at least the original input pipeline.
-
Manual investigation of a small set of invariants.
-
Experimentally,
- apply the invariants inferred from one input to itself with different argument/data
- apply the invariants inferred from one input / one class of input to one class of input or different classes of input
- Self-transfer invariants bar chart, each bar group corresponds to a input, and bars within a group will show different settings (e.g. different version, different optim, different size etc.)
- Cross pipeline invariants false positivity x-axis: num inputs y-axis: fp rate
Note for the cross pipeline evaluation it is important to remove the compound effect of transferability.