Fix completion-hang deadlock so all clients report BENCHMARK_DONE (#699)

[charles-typ]

Summary:

The distributed run never finalized cleanly: warmup completed (warmup_done=N) but benchmark_done stalled below num_clients (e.g. 13/16). Because the admin server only broadcasts ALL_DONE once benchmarkCompleteClients_ >= numExpectedClients_, the missing clients block EVERY client forever at waitForNotification(0), so the run is torn down with no clean client-side results (only server-aggregated metrics survive).

Root cause is a google::log_mutex/stdio deadlock, not a coroutine hang. gstack on a stuck client showed 48 of 189 threads blocked forever (futex abstime=0x0) on google::log_mutex inside LogMessage::Flush(); the lock holder was blocked in __libc_write(fd=1) to stdout, its buffer full of Benchmark GET error (sample 1/1000): mc_res_tko repeated. The stdout/stderr pipe to benchpress/automark was backpressured, so write() blocked while holding the stdio FILE lock and glog's log_mutex, wedging every logging thread including the coro worker event-base threads -> mainScope.joinAsync() never returns -> BENCHMARK_DONE is never sent. Under the mc_res_tko storm at multi-M QPS, even 1-in-1000 error sampling is thousands of synchronous prints/sec, which is enough to back the pipe up.

Fix: the four hot-path sampled error prints now fire only on the FIRST error per worker (errors == 1 instead of errors % 1000 == 1) -> at most one line per worker for the whole run, flood-proof, while keeping the error-type diagnostic (still gated by --verbose). The error counts themselves are unchanged and still reported at the end.

This change also folds in the related straggler-drain robustness the completion path needs: per-request co_await now uses folly::makePromiseContract + SemiFuture::within(10s) (global-timekeeper based, abandons the underlying future so the await always completes even if the McRouter callback never fires), and the warmup/measurement worker scopes use CancellableAsyncScope::cancelAndJoinAsync() to cancel rather than wait on requests still outstanding at window-end.

This entire diff stack contributed to the final result.

Reviewed By: excelle08

Differential Revision: D110091576

[meta-codesync]

@charles-typ has exported this pull request. If you are a Meta employee, you can view the originating Diff in D110091576.