fix: robustly strip psql meta commands

[ignatremizov]

Summary

This PR makes PostgreSQL schema preprocessing semantically safer for pg_dump / psql-flavored input without turning sqlc into a partial psql interpreter.

It preserves real PostgreSQL SQL, strips or warns on client-side commands, rejects script semantics that cannot be supported safely, and shares the apply-time schema loading path used by command setup and PostgreSQL sqltest seeding.

Changes

PostgreSQL schema preprocessing

• Replace the old line-based PostgreSQL filter with a single-pass state machine that only strips top-level psql meta-commands outside SQL lexical context.
• Keep explicit coverage of documented psql backslash commands as a reference fixture. See PostgreSQL psql docs
• Preserve backslashes that appear in valid PostgreSQL SQL, including:
  • string literals, including E'...'
  • quoted identifiers
  • dollar-quoted bodies
  • comments and nested block comments
• Support the valid \meta ... \\ SQL... separator form and keep the trailing SQL intact.
• Handle Unicode dollar-quote tags, identifier-boundary checks, CR/LF normalization, and \copy ... from stdin blocks.

Semantic command handling

• Strip semantic psql commands such as \connect, \copy, \gexec, \i, and \ir in parse/codegen paths with warnings so users get told when sqlc is ignoring client-side behavior.
• Reject those same semantic commands in apply/live-DB paths where silently flattening them would be unsafe.
• Reject psql conditionals (\if, \elif, \else, \endif) instead of pretending to support them.
• Keep standard_conforming_strings handling as a best-effort parse aid and warn when preprocessing has to approximate session/script semantics rather than emulate psql.

Rollout across schema-loading paths

• Apply the shared PostgreSQL preprocessing behavior in:
  • compiler parsing / generate
  • createdb
  • verify
  • managed vet
  • PostgreSQL sqltest seeding
• Surface preprocessing warnings through the relevant callers instead of keeping them internal to the migration helper.
• Re-enable the pg_dump end-to-end fixture now that its schema parses and seeds successfully again.

Follow-up refactor

• Extract the repeated apply-time glob/read/preprocess/warn loop into schemautil.LoadSchemasForApply().
• Reuse that loader across command setup and PostgreSQL sqltest helpers so apply-time behavior stays aligned as preprocessing evolves.
• Preserve read-only PostgreSQL cache hashing on preprocessed DDL.
• Handle plain pg_dump schema replays in PostgreSQL sqltest helpers by dropping the default public schema before applying dumps that recreate it.

Behavioral impact

• sqlc is now much more tolerant of pg_dump / psql-flavored PostgreSQL schema files without corrupting valid SQL that happens to contain backslashes.
• Parse/codegen paths are intentionally permissive with warnings when users are in footgun territory.
• Apply/live-DB paths stay stricter and reject client-side script behavior that sqlc cannot reproduce safely.
• This does not try to become a full psql interpreter; full arbitrary client-side script semantics remain out of scope here.

Testing

• go test ./internal/migrations ./internal/compiler ./internal/cmd/...
• go test ./internal/sqltest/... -run '^$'
• go test ./internal/endtoend -run 'TestReplay/base/pg_dump|TestValidSchema/endtoend-testdata/pg_dump/sqlc.json-0'
• internal/migrations/migrations.go is covered at 100%

Related

Co-authored-by: Andrew Benton andrew@sqlc.dev

Addresses gbarr's comment in #4082 which closes #4065

[ignatremizov]

@andrewmbenton please review

[gbarr]

Thanks @ignat980 this looks a much more complete solution than I was expecting

[kyleconroy]

@ignat980 if you open this against main I can get this merged. Just make sure to includes Andrew's commits.

[ignatremizov]

@kyleconroy Thanks! I rebased to latest sqlc/main and changed this PR's merge-into branch as sqlc/main. Just waiting on the test CI to finish

[ignatremizov]

@kyleconroy bump for visibility - not sure why the CI tests haven't run, maybe you need to push something?

[ignatremizov]

@kyleconroy bump again ⭐

[atombender]

@kyleconroy Can you please merge this? It's been in approved state for 5 months now.

[kyleconroy]

I've merged the simple fix in #4390

If you rebase in that we can get this merged

[ignatremizov]

@kyleconroy Rebased on top of #4390 (current main), but I didn’t just replay the old patch unchanged.

After checking the merged simple fix more closely, I found several correctness gaps if we wanted this to be safe against real-world pg_dump / psql-flavored schema input, not just the narrow \restrict / \unrestrict case.

So the branch now does more than just “robust stripping”, it does "correct stripping":

• keeps the single-pass state machine so we only strip top-level psql meta-commands outside real SQL literal/comment/body context
• preserves valid PostgreSQL SQL containing backslashes in:
  • string literals, including E'...'
  • quoted identifiers
  • dollar-quoted bodies
  • comments / nested block comments
• handles the valid \meta ... \\ SQL... separator form
• strips semantic psql commands in parse/codegen paths with warnings, so if users rely on psql-side behavior they get told they are now in footgun territory
• rejects those same semantic commands in apply/live-DB paths, where silently flattening them would be unsafe
• rejects psql conditionals (\if, \elif, \else, \endif) instead of pretending to support them

I did start going further down the path of "full correctness", but pretty quickly that becomes "reimplement psql", which is not the scope of this PR.

So I stopped at:

• lexically correct stripping
• semantically safer behavior for sqlc users
• warnings/errors where preprocessing becomes approximate

If we ever want to go further, the honest solution is probably not "keep accreting special cases here", but something closer to "run the SQL through a real PostgreSQL environment and reason about the resulting state after psql-side processing".

If users want full arbitrary-psql-script semantics in sqlc today, that is outside the scope of this PR and they can suck a lemon.

Also split out the repeated mechanical caller refactor as a separate follow-up commit so the actual behavior change stays readable.