Skip to content

Fix(Fabric): Skip Catalog Jump#5780

Open
fresioAS wants to merge 6 commits intoSQLMesh:mainfrom
fresioAS:fabric_catalog_jump
Open

Fix(Fabric): Skip Catalog Jump#5780
fresioAS wants to merge 6 commits intoSQLMesh:mainfrom
fresioAS:fabric_catalog_jump

Conversation

@fresioAS
Copy link
Copy Markdown
Contributor

Description

Reduces excessive catalog-switching churn in the Fabric engine adapter.

Root cause:
get_current_catalog() was inherited from GetCurrentCatalogFromFunctionMixin, executing SELECT db_name() on every call — including the two calls made by the set_catalog decorator (switch-in + restore) wrapping every catalog-scoped operation. For Fabric, each SQL statement runs in an independent session, so this also triggered a connection close() + reopen on every single restore, even when restoring to the same catalog.

Changes:

  1. Override get_current_catalog() to read thread-local state (_target_catalog) instead of querying the database. Eliminates all SELECT db_name() round-trips.

  2. Treat the default catalog as neutral (None) via _normalize_catalog(). The configured database is the implicit connection default — there is no meaningful distinction between "connected to default" and "no catalog set", so both map to None. This prevents the decorator from seeing a mismatch and unnecessarily switching back to the default after every operation.

  3. Lazy connection management via _connected_catalog tracking. set_current_catalog() now distinguishes between the logical catalog target (_target_catalog) and the physical catalog the open connection is using (_connected_catalog). The connection is only closed and reopened when the physical connection actually needs to change:

    • Restoring to neutral after a catalog-scoped call: _target_catalog is updated, but the open connection is kept alive for reuse on the next call to the same catalog.
    • Switching to the same non-default catalog a second time: no close, the existing connection is reused directly.
    • Switching to a different catalog: one close + reopen as before.
  4. Fix _drop_catalog to check both _connected_catalog and get_current_catalog() (with default-db fallback) before closing all connections, so dropping the default warehouse is still handled correctly.

Test Plan

  • Unit tests added covering: neutral state behaviour, no DB queries from get_current_catalog, default catalog clearing, restore-to-neutral without connection close, same-catalog reuse (1 close for N consecutive ops), and cross-catalog switching (1 close per genuine switch).
  • Tested and verified on live data locally. Catalog-switching log lines reduced from 175 → 51 per plan run.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant