Parallelize test execution using xdist

[koletzilla]

Summary of changes

Test users (the DBT_TEST_USER problem)

• Before: the three test users were created once, at the start of the whole test session, and their names were passed around through environment variables. This broke with parallel runs.
• Now: each test class that actually needs users creates its own three users, with random unique names, right before the class runs, and drops them when the class is done. The environment variables are still set (upstream dbt tests require them), but only for the duration of that class, and they are restored afterwards.
• Only the grants and dbt_clone test folders need users, so each of those folders got a tiny new conftest.py that turns this on. Everyone else never pays for it.

Docker handling

• Sequential runs (the old default) still work exactly as before: set DBT_CH_TEST_USE_DOCKER and the test session starts and stops the ClickHouse cluster for you.
• Parallel runs must start the cluster themselves first (docker compose up -d). If you try to combine DBT_CH_TEST_USE_DOCKER with -n, you get a clear error explaining what to do instead of a confusing failure.
• CONTRIBUTING.md now documents both ways of running the tests.

Making tests safe to run at the same time

• Each test gets a schema name that includes the worker id plus a random suffix, so two workers can never step on each other's tables.
• SYSTEM FLUSH LOGS runs once per worker at startup, so system tables exist before tests query them.
• pytest.ini: makes pytest's doctest collector to skip conftest.py files (having several files with the same name broke imports).

CI (GitHub Actions)

• The workflows now start the ClickHouse cluster as their own step, then run pytest with -n 5 --dist loadscope --timeout=300 for both the HTTP and native test jobs. pytest-xdist was added to dev_requirements.txt.

Adapter fixes for flaky tests found by running in parallel

• New shared helper retry_on_database_error in util.py.
• dbclient.py: command() and get_ch_setting() can now retry transient failures internally (the drivers implement the raw _command / _get_ch_setting versions). A failed check of the allow_nondeterministic_mutations setting is no longer cached for the whole process. Before, one bad read silently disabled lightweight deletes for every later connection in that process. Warning messages now say clearly what got disabled and show the real error.
• nativeclient.py: reading a setting now raises a proper error instead of returning None, which used to crash with a confusing message.

ClickHouse test cluster config (test_config.xml)

• Keeper election timings widened (heartbeat 1s, election window 4–8s): all three keeper nodes share one machine with the test workers, and short CPU stalls were triggering pointless leader elections that killed in-flight ON CLUSTER statements (the "Code 999" flake).
• The distributed DDL queue now cleans itself up (finished tasks kept 5 minutes instead of 7 days, cleanup every 15s instead of 60s): before, the queue filled up to its 1000-task cap and stayed there forever, causing sporadic timeouts.

[koletzilla]

This query was ok if run sequentially. If other run at the same time, it caused OOMs in the replica, making the rest of the tests to fail

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0b64e6a. Configure here.}

[koletzilla]

One tests if failing in a flaky way. Fixing would require a few changes so I have created an issue to track it #724