We should investigate how to track quality metrics for perf comparison:
- number of ambiguous comparison results
- number of false positives
After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.
This issue can be close with:
- a list of metrics to be tracked with their descriptions (i.e. number of ambiguities, false positives etc.)
- methodology for getting these metrics (i.e. do I run a script that stores number of ambg? how do I tell a result is false positive?)
We can defer the following to a different issue:
- how to compare two sets of metrics (i.e. number of amb is lower but false positives is larger, is it better?)
We should investigate how to track quality metrics for perf comparison:
After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.
This issue can be close with:
We can defer the following to a different issue: