Skip to content

Spike on tracking quality metrics for nvbench comparison scripts #415

Description

@gevtushenko

We should investigate how to track quality metrics for perf comparison:

  • number of ambiguous comparison results
  • number of false positives

After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.

This issue can be close with:

  • a list of metrics to be tracked with their descriptions (i.e. number of ambiguities, false positives etc.)
  • methodology for getting these metrics (i.e. do I run a script that stores number of ambg? how do I tell a result is false positive?)

We can defer the following to a different issue:

  • how to compare two sets of metrics (i.e. number of amb is lower but false positives is larger, is it better?)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

  • Status
    Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions