spring-util: SHiP log inspection/repair/split/merge tooling + nodeos state-history-force-write

[heifner]

Gives node operators in-box tooling for state history (SHiP) logs — the capabilities requested in #1664, motivated by AntelopeIO/leap#1593 (corruption at block 112M with 8.1 TB of salvageable history after it, and no recovery short of a multi-month replay).

Problem

state_history_log's constructor auto-mutates: merely opening a log truncates a corrupt tail, regenerates a bad index, and vacuums pruned logs, so a damaged log cannot even be inspected without modifying it — and only via a full nodeos. Anything beyond tail corruption ("is corrupted and cannot be repaired") is fatal, discarding everything after the first bad byte even when terabytes of valid data follow.

spring-util ship-log

All subcommands operate on a --state-history-dir / --log pair, accept retained <stem>-N-M bundles, and the read-only ones really are read-only (the scanner never goes through state_history_log).

Subcommand	Writes	Purpose
info	no	per-bundle endpoint report: version, pruned flag, block range, endpoint ids, index health
block-id	no	the id a log records for any block number, across the whole directory or one bundle
smoke-test [--deep]	no	validate every entry (headers, position trailers, block continuity, index agreement); --deep also decompresses every payload (adler32 catches bit rot); prints a map of valid/damaged regions
make-index	index	standalone index rebuild; sequential forward walk, byte-identical to the library's backward regeneration
trim --first/--last	yes	trim to a block range; end trim is an in-place truncate, front trim rewrites position trailers via temp files + atomic rename
extract-blocks	new bundle	copy a block range into a fresh bundle, source untouched
repair [--keep-tail --dry-run --deep --output-dir]	yes	default truncates at first damage + rebuilds the index (offline, previewable equivalent of nodeos's auto-recovery); --keep-tail salvages the last valid range instead — the leap#1593 case
vacuum	yes	pruned → non-pruned
split --stride	new bundles	rotation-compatible retained bundles (boundaries on stride multiples) plus head remainder; loads directly into a log_catalog
merge	new bundle	merge contiguous retained bundles back into one log

nodeos --state-history-force-write

Never let damaged or inconsistent ship logs stop the node. An index that disagrees with its log is regenerated instead of fatal; a head log that fails its startup checks or cannot accept the next block (a gap after a snapshot restore, divergent fork history) is renamed aside to <stem>-corrupt-<n> — kept on disk, never deleted — and writing continues into a fresh log, escalating to setting aside the retained bundles only when a write predates the whole catalog. Holes in the retained set are tolerated with a warning, leaving just those blocks unserved. Orphaned bundles remain valid logs that ship-log can inspect, repair, and trim (and merge back once renamed off the -corrupt- suffix).

Better fork-change diagnostics

The three fork-change asserts in state_history_log::pack_and_write_entry now report the block being written, the previous id it carries, and the id actually recorded for the prior block; the index-backed path additionally decodes the recorded id's own block number and states that a block number mismatch means a corrupt index rather than a fork, pointing at spring-util ship-log block-id and make-index. On the incident that motivated this it would have printed "the index resolves block 8028595 to id 000000021a92..., an id for block 2", turning a day of forensics into one log line.

Implementation notes

• Primitives live in libraries/state_history/log_utils.{hpp,cpp} next to the format they understand; the CLI is a thin binding in programs/spring-util/actions/shiplog.{hpp,cpp} following the block-log action pattern.
• Entry position trailers store absolute file offsets, so every operation that moves entries (front trim, extract, split, merge, keep-tail) rewrites them — and must walk every entry including fork-superseded ones, not just the index's slots.
• After damage, the scanner resynchronizes by searching for the 8-byte ship magic and accepting a candidate only if its position trailer points back at it exactly, making a false resync on payload bytes practically impossible.
• A bundle salvaged from mid-log must start at its "canonical" entry (the earliest one not superseded by a later fork switch); starting at a stale fork entry would produce a log whose first block is not a floor for the rest, which state_history_log cannot open safely.
• A write below the whole catalog (chain wiped and resynced) is the one case force-write resolves by setting aside the retained bundles too; plain head-gap and fork-mismatch cases only rotate the head log out.
• ~state_history_log no longer lets a vacuum-on-close failure throw out of the destructor (which would std::terminate).

Drive-by fixes

• spring-util block-log print-log --print-from is now an option rather than a flag: a CLI11 flag only accepts its value in --print-from=<value> form, so the conventional space-separated spelling failed validation.
• log_config.hpp's boost_test_print_type is now inline — it is defined in a header that this change makes reach more than one translation unit per binary, which without inline is a multiple-definition link error.

Validation

• tests/ship_log_utils.cpp — 33 cases: torn tails, mid-file damage, payload bit flips, fork-overwritten entries, pruned logs, index corruption variants, trim/extract/split/merge round-trips all verified by reopening the results through real state_history_log/log_catalog instances, and every force-write tier.
• tests/ship_log_util_test.py — end-to-end against a real nodeos: corrupt, repair with spring-util, byte-compare against pristine files, relaunch, plus both force-write scenarios (lying index regenerated; gapped log orphaned while the head advances).
• Full plugin_test and ship_restart_test pass unchanged.
• This is a port of sys-util: SHiP log inspection/repair/split/merge tooling + nodeop state-history-force-write Wire-Network/wire-sysio#392, where the same code was additionally validated against a production 61 GB / 8.05M-block chain_state log: full structural scan + slot-by-slot index verification in ~3 minutes; deep payload validation of a 3.5 GB trace log in 38 s; 100-block extract from the 61 GB log in 37 ms; split→merge of the 3.5 GB log round-trips to a byte-exact prefix of the original; corrupt-then-repair cycles left both the 61 GB log and its rebuilt index byte-identical to a pristine backup.