Skip to content

perf: Optimize /names-and-versions API with lightweight DAO query#1011

Draft
akhilpathivada wants to merge 2 commits intoconductor-oss:mainfrom
akhilpathivada:feature/optimize-workflow-names-loading
Draft

perf: Optimize /names-and-versions API with lightweight DAO query#1011
akhilpathivada wants to merge 2 commits intoconductor-oss:mainfrom
akhilpathivada:feature/optimize-workflow-names-loading

Conversation

@akhilpathivada
Copy link
Copy Markdown
Contributor

@akhilpathivada akhilpathivada commented Apr 14, 2026

Summary

  • Optimize the GET /metadata/workflow/names-and-versions endpoint backend to avoid loading full WorkflowDef objects. Previously, this endpoint called getAllWorkflowDefs() which runs SELECT json_data — reading and deserializing every workflow definition just to extract name, version, and timestamps.
  • Add a Postgres-optimized getWorkflowNamesAndVersions() DAO override that queries only SELECT name, version, created_on, modified_on — no json_data TOAST column read, no JSON deserialization.
  • Replace HashMap + TreeSet with LinkedHashMap + ArrayList in the service layer, leveraging the pre-sorted data from the DAO to avoid TreeSet rebalancing overhead.
  • Add updateTime field to WorkflowDefSummary with gRPC/protobuf compatibility.

Changes

Backend

  • MetadataDAO: Added default getWorkflowNamesAndVersions() method — backward-compatible fallback that iterates getAllWorkflowDefs() and constructs WorkflowDefSummary objects
  • PostgresMetadataDAO: Optimized override using SELECT name, version, created_on, modified_on FROM meta_workflow_def ORDER BY name, version — no json_data column, no deserialization
  • MetadataServiceImpl: Rewritten getWorkflowNamesAndVersions() to call the new DAO method directly and group results using LinkedHashMap + ArrayList instead of HashMap + TreeSet

Model

  • WorkflowDefSummary: Added updateTime field with @ProtoField(id = 4)
  • workflowdefsummary.proto: Added int64 update_time = 4
  • AbstractProtoMapper: Updated toProto() and fromProto() for the new field

Tests

  • MetadataServiceTest: Updated mock to use getWorkflowNamesAndVersions() and verify ascending version order with the new ArrayList-based grouping
  • PostgresMetadataDAOTest: Integration test for the optimized DAO query

Backward Compatibility

  • No database schema changes or migrations
  • No existing endpoints modified or removed
  • MetadataDAO.getWorkflowNamesAndVersions() is a default method — existing DAO implementations (Redis, MySQL, etc.) continue to work without changes
  • New updateTime field in WorkflowDefSummary uses @ProtoField(id = 4) — additive and backward compatible for gRPC/protobuf clients

Test Plan

  • ./gradlew spotlessApply passes
  • ./gradlew test passes (core, postgres-persistence)
  • Verify /metadata/workflow/names-and-versions returns correct data with updateTime
  • Verify version dropdown still works correctly on Workbench page
  • Verify no regression on Workflow Definition editor pages

@akhilpathivada akhilpathivada force-pushed the feature/optimize-workflow-names-loading branch from 20ec5cf to 5e611fe Compare April 14, 2026 11:28
Signed-off-by: akhilpathivada <akhil.pathivada.21@gmail.com>
@akhilpathivada akhilpathivada force-pushed the feature/optimize-workflow-names-loading branch from 5e611fe to c7b67a2 Compare April 14, 2026 12:17
Signed-off-by: akhilpathivada <akhil.pathivada.21@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant