poc(kernel): pure-Go (CGO_ENABLED=0) dynamic loader — control + data plane, no latency regression

[msrathore-db]

Draft / proof-of-concept. Not for merge as-is. Opened to validate the dynamic-loading approach in-repo. See internal/backend/kernel/DYNAMIC_LOADER_POC.md.

What this proves

The shipped kernel backend static-links libdatabricks_sql_kernel.a through cgo, forcing CGO_ENABLED=1, a C toolchain on every builder, and no cross-compilation — the blockers the SEA/kernel release design flags against making SEA the default. This PoC drives the closed-source kernel from a pure-Go (CGO_ENABLED=0) binary by dlopen-ing the kernel shared library at run time via purego — the model gosnowflake uses for its own closed-source core.

Now complete: both control plane AND data plane, verified live on pecotesting.

Control plane

dlopen → config → session open → execute → query-id / affected-rows → teardown. No cgo.

Data plane (the previously-blocked part — now solved)

arrow-go v12's cdata importer is itself cgo, so it can't run under CGO_ENABLED=0. cdata_pure.go is a pure-Go port of that importer's C-Data import path: reads the flat ArrowSchema/ArrowArray structs via unsafe, invokes each struct's release callback via purego.SyscallN, builds arrow.Records zero-copy with array.NewData. dynamic_rows.go pulls batches through the dlopen'd kernel_result_stream_* and scans with the same arrowscan scanner the cgo path uses — so values are identical by construction.

Verified live (TestDynamicLoaderDataPlane, CGO_ENABLED=0): scalars, null, decimal-as-exact-string, high-precision DECIMAL(38,2) byte-exact (9999999999999999999999999999.99 — proves buffer import correctness), date/timestamp/binary/float, nested array/map/struct as JSON, empty result set, and 100k rows across batches.

Latency: no regression (head-to-head on pecotesting)

Identical 500k-row × 3-col query, same drain loop, same scanner; both kernel artifacts built from the same pinned rev with tls-rustls. BenchmarkCgoLargeResult vs BenchmarkDynLargeResult:

Path	ns/op (500k rows)
cgo static	7.91 s
purego dyn	6.17 / 6.61 / 8.59 s (repeated)

End-to-end dominated by warehouse + network → statistically indistinguishable; purego is never meaningfully slower and crosses no per-cell cgo boundary. No regression.

Test matrix (all green)

• Default pure-Go suite (CGO_ENABLED=0, no tags): go test ./... — pass.
• Dynamic-tagged (CGO_ENABLED=0 -tags databricks_kernel_dynamic): unit pass; e2e/data-plane pass live on pecotesting.
• cgo-tagged (CGO_ENABLED=1 -tags databricks_kernel): unit pass; full TestKernelE2E* passes live on pecotesting, 0 failures — confirms the existing static path is not regressed.

No user CUJ change

WithUseKernel and the existing build tags are untouched. The loader lives behind a new, separate tag databricks_kernel_dynamic; the default pure-Go build is unchanged and does not pull in purego. No kernel source change (its Cargo.toml already declares cdylib).

Files

• dynamic_loader.go — purego dlopen + C ABI binding (control plane + result stream)
• cdata_pure.go, cdata_pure_finalizer.go — pure-Go Arrow C-Data importer
• dynamic_rows.go — driver.Rows over the pure-Go importer
• dynamic_bench_test.go, cgo_bench.go, cgo_bench_test.go — head-to-head benchmarks
• dynamic_loader_test.go — control + data-plane e2e
• DYNAMIC_LOADER_POC.md — full writeup

Still PoC (must change before merge)

• Local replace for purego so the branch builds offline (real PR pins v0.10.2 via proxy).
• dylib path via env var (real code resolves via rpath / documented env var).
• Hand-mirrored cKernelError/C-Data struct layouts (real PR adds unsafe.Sizeof/Offsetof asserts).
• cdata_pure.go ports the type cases the kernel emits; union/run-end-encoded and the ArrowArrayStream reader are intentionally omitted (kernel uses next_batch pull).

This pull request and its description were written by Isaac.