perf: Optimize /names-and-versions API with lightweight DAO query#1011
Draft
akhilpathivada wants to merge 2 commits intoconductor-oss:mainfrom
Draft
perf: Optimize /names-and-versions API with lightweight DAO query#1011akhilpathivada wants to merge 2 commits intoconductor-oss:mainfrom
akhilpathivada wants to merge 2 commits intoconductor-oss:mainfrom
Conversation
20ec5cf to
5e611fe
Compare
Signed-off-by: akhilpathivada <akhil.pathivada.21@gmail.com>
5e611fe to
c7b67a2
Compare
Signed-off-by: akhilpathivada <akhil.pathivada.21@gmail.com>
2e7b5e2 to
74da20e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /metadata/workflow/names-and-versionsendpoint backend to avoid loading fullWorkflowDefobjects. Previously, this endpoint calledgetAllWorkflowDefs()which runsSELECT json_data— reading and deserializing every workflow definition just to extract name, version, and timestamps.getWorkflowNamesAndVersions()DAO override that queries onlySELECT name, version, created_on, modified_on— nojson_dataTOAST column read, no JSON deserialization.HashMap + TreeSetwithLinkedHashMap + ArrayListin the service layer, leveraging the pre-sorted data from the DAO to avoid TreeSet rebalancing overhead.updateTimefield toWorkflowDefSummarywith gRPC/protobuf compatibility.Changes
Backend
MetadataDAO: Addeddefault getWorkflowNamesAndVersions()method — backward-compatible fallback that iteratesgetAllWorkflowDefs()and constructsWorkflowDefSummaryobjectsPostgresMetadataDAO: Optimized override usingSELECT name, version, created_on, modified_on FROM meta_workflow_def ORDER BY name, version— nojson_datacolumn, no deserializationMetadataServiceImpl: RewrittengetWorkflowNamesAndVersions()to call the new DAO method directly and group results usingLinkedHashMap + ArrayListinstead ofHashMap + TreeSetModel
WorkflowDefSummary: AddedupdateTimefield with@ProtoField(id = 4)workflowdefsummary.proto: Addedint64 update_time = 4AbstractProtoMapper: UpdatedtoProto()andfromProto()for the new fieldTests
MetadataServiceTest: Updated mock to usegetWorkflowNamesAndVersions()and verify ascending version order with the newArrayList-based groupingPostgresMetadataDAOTest: Integration test for the optimized DAO queryBackward Compatibility
MetadataDAO.getWorkflowNamesAndVersions()is adefaultmethod — existing DAO implementations (Redis, MySQL, etc.) continue to work without changesupdateTimefield inWorkflowDefSummaryuses@ProtoField(id = 4)— additive and backward compatible for gRPC/protobuf clientsTest Plan
./gradlew spotlessApplypasses./gradlew testpasses (core, postgres-persistence)/metadata/workflow/names-and-versionsreturns correct data withupdateTime