feat(data): add PostgresSource for live config reads#900
Conversation
…ray reads json_agg returns json, so coalescing it with a '[]'::jsonb fallback errored (SQLSTATE 42846) on every non-empty config table. Use a '[]'::json fallback so array kinds read back; output text is unchanged.
zulkhair
force-pushed
the
daim/adr057-postgres-source
branch
from
734786c to
324ac5a
Compare
zulkhair
requested review from
AnthonyWadham,
rmrt1n,
ryanditjia and
winton-library
rmrt1n
left a comment
rmrt1n
left a comment
There was a problem hiding this comment.
mostly code style nits, though I don't mind merging now and refactoring later if it already works.
| } | ||
|
|
||
| // tableFor returns the table registered for file, falling back to the name derived from the file. | ||
| func (p *PostgresSource) tableFor(file string) string { |
There was a problem hiding this comment.
nit: I'd inline this, and tableFromFile, into Fetch so u don't have to jump between 3 different functions just to understand what Fetch does.
| type SingletonRegistrar interface { | ||
| RegisterSingleton(file string) | ||
| } | ||
|
|
||
| // KindRegistrar is implemented by sources that accept an explicit table name for a kind's JSON file. | ||
| type KindRegistrar interface { | ||
| RegisterKind(file, table string) | ||
| } |
There was a problem hiding this comment.
nit: Remove these. Either add these to Source or use a tagged union pattern in Register.
There was a problem hiding this comment.
nit: We don't need these tests. The function that's most complicated is readTableJSON, which is better tested w an integration test against a real DB, but that takes a lot more work and is probably overkill ATM. Testing with fakeReader doesn't do much here as the logic being tested is Go's stdlib functions (e.g. sha256), and the other functions are pretty straightforward, so I think we should just get rid of this file.
2 participants
What
The
dataplugin can read each config kind live from Postgres instead of build-time embedded JSON — so a deployment changes config without a rebuild.PickSourcechooses the source at runtime byDB_DSN.Source selection (
PickSource)flowchart LR K["config kind read"] --> PICK{"DB_DSN set?"} PICK -->|yes| PG["PostgresSource<br/>(fail-loud, no embed fallback)"] PICK -->|no| EM["EmbedSource<br/>(embedded JSON, unchanged)"] PG --> DB[("Postgres<br/>(id, doc) tables")]DB_DSNset but unusable → panic (never silently serve stale embedded config).Table contract
One table per kind; one row = one record, JSON in a
doc jsonbcolumn. Table name = the kind'sName(), registered when the kind registers — shardmain.gostays unchanged.SELECT coalesce(json_agg(doc ORDER BY id), '[]'::json)::text FROM <t>Singletonmarker)SELECT doc::text FROM <t> LIMIT 1Notes
EmbedSource): ignores the requested content hash, returns current rows.postgres_internal_test.go— fakeconfigReader, no DB (table resolution + registrar override, read shapes, rows+sha256, error propagation).DB_DSNis the same env the monorepo docker backend injects (paired change: Argus-Labs/monorepo#711).