Add elasticgraph-proto_ingestion schema artifacts#1080
Open
jwils wants to merge 1 commit into
Open
Conversation
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
2e0e4a1 to
53ea492
Compare
jwils
force-pushed
the
joshauw/pluggable-ingestion-serializers
branch
from
a3cfa5c to
9e6f39a
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
3 times, most recently
from
4325d51 to
fd8dd47
Compare
jwils
force-pushed
the
joshauw/pluggable-ingestion-serializers
branch
from
9e6f39a to
c408a50
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
fd8dd47 to
bdce7fb
Compare
jwils
force-pushed
the
joshauw/pluggable-ingestion-serializers
branch
from
c408a50 to
5717334
Compare
jwils
force-pushed
the
joshauw/pluggable-ingestion-serializers
branch
12 times, most recently
from
0607029 to
40815e1
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
6 times, most recently
from
6591c29 to
751db75
Compare
jwils
force-pushed
the
joshauw/pluggable-ingestion-serializers
branch
4 times, most recently
from
b2e8d34 to
6969f84
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
5a5bb53 to
c5cd7b8
Compare
jwils
force-pushed
the
joshuaw/indexer-ingestion-adapters
branch
5 times, most recently
from
908b4c9 to
247ac28
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
c5cd7b8 to
2f7471a
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
3 times, most recently
from
070849e to
ce625bc
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
2 times, most recently
from
e9902f1 to
f069179
Compare
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
2 times, most recently
from
5efeb79 to
7d428d5
Compare
myronmarston
reviewed
Jul 2, 2026
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
7d428d5 to
9acfed5
Compare
jwils
changed the title
Adds `elasticgraph-proto_ingestion`, a self-contained schema-definition extension that generates Protocol Buffers schema artifacts from an ElasticGraph schema. Running `schema_artifacts:dump` emits: - `schema.proto` — the generated Protobuf schema for the indexed types - `proto_field_numbers.yaml` — a sidecar that reserves field numbers so they stay stable as the schema evolves (including across field renames) Capabilities: - Maps built-in ElasticGraph scalars to proto types, with `proto_field` to configure custom scalars. - Generates messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping nested lists so the output stays valid. - `proto_enum_mappings` reuses enum values already maintained elsewhere. The generator emits `proto3` by default and can emit `proto2` via `syntax: :proto2` (which labels every field `optional`/`repeated`). Arbitrary file-level headers (e.g. `option` declarations) can be injected verbatim via `headers:`, so language-specific options can be set without baking any particular convention into the gem.
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
9acfed5 to
4fe8938
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
What
elasticgraph-proto_ingestionextension gem that generatesschema.protoandproto_field_numbers.yamlschema.protoon public GraphQL field names while storing privatename_in_indexoverrides in the sidecar mapping fileschema.protoartifact preamble with protobuf//comments so it remains valid proto syntaxfield.renamed_fromand update repo wiring/docs/testsReferencing external proto enum types (
proto_external_types) is split out into the stacked follow-up PR #1286.Notes for review
elasticgraph-proto_ingestionis fully independent ofelasticgraph-json_ingestion— no runtime or development dependency. Its specs activate only the proto extension; the rename-preservation specs use a minimalFieldRenameSupportspec extension that recordsrenamed_frommetadata on the schema-definition state, since proto generation duck-types against that state rather than depending on a specific ingestion extension. When no extension provides rename metadata, rename migration is skipped (covered by a dedicated spec).1..n), so removing or inserting an enum value renumbers subsequent values. Message field numbers are pinned viaproto_field_numbers.yaml; enum value numbers are not yet. If we want wire-format stability for enums before first release, the sidecar mechanism could be extended to them.Verification
script/quick_build(the one spec failure on this machine —cli_new_specunder bundler 4 — reproduces identically onmainand is environmental)script/type_check,script/lint,script/spellcheckscript/run_gem_specs elasticgraph-proto_ingestion(100% line + branch coverage)run_misc_checkssteps that CI previously never reached: config artifacts, dependency diagrams, CI yaml, licenses, README snippets,rake site:validateReferences