Reproducability & Auditability

[sethtroisi]

Some comments from offline discussion between Andrew & Seth

• Launch things via scripts & flag files
• record CLI & flag files into a directories
• model table
• run table
• eval table
  • v1 was ratings.db in ratings.py
  • v2 is bigtable
  • Backfill script [oneoff] Script to push sgfs to bigtable eval_games #776
  • [] Run and verified backfill script
    ...

[sethtroisi]

Model registry

• Models get a parent (which should also be in the registry)
• save the commands invoked (python dual_net.py train ...)
• save the master repo diff
• any local code diff
  • How to do this?

Results table

• Uses bigtable tag Add bigtable eval tags #590 to name a comparison
• Use launch_eval.py and record command (somewhere)
  • First pass done in Initial work on eval-server #601

---

Each entry should be

• contained in a google cloud directory (probably in minigo-pub)
• contain the information to reproduce / understand where it came from
• force push code to a testing branch and create the url for diff between that branch and master commit

[sethtroisi]

#919 and #893 are both related