[AURON #2323] Refactor and rename cast test suites for ANSI mode on and off

[cxzl25]

Which issue does this PR close?

Closes #2323

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

How was this patch tested?

GHA

[weiqingy]

Thanks for taking this on — the rename correctly tracks Spark 4's base-class refactor: extending CastWithAnsiOnSuite/CastWithAnsiOffSuite and dropping the AuronAnsiCast*WithAnsiMode* wrappers (Spark 4 folded the standalone AnsiCast path into Cast, so those base classes no longer exist) is the right mapping, and CI confirms both modules now compile past #2323. One question on the overall check state.

1. The spark-4.0 and spark-4.1 checks are still red on the latest run, but they now fail in the scalatest phase rather than at compile — so the rename did its job: both modules reach scala-test-compile and run the suites. The remaining failures are in other suites in the auron-spark-tests-spark40/41 modules that only surface now that the module compiles — AuronTryCastSuite and the parquet/JSON read suites hit TASK_WRITE_FAILED and RemoteClassLoaderError: org.apache.spark.sql.catalyst.expressions.Object, alongside native panics like map_from_arrays does not support null map keys. The two renamed cast suites themselves run with every case reported !!! IGNORED !!!, so they aren't the cause. Is the intent here to unblock compilation while those Spark 4 runtime failures are tracked separately? If so, would it be worth a line in the description noting spark-4.0/4.1 will stay red pending those follow-ups, so it's clear the check isn't expected to go green with this merge?

[Copilot]

Pull request overview

Refactors Spark 4.x cast-related test suites to align with Spark’s ANSI on/off suite naming and removes Spark 4.x test wrappers that referenced non-existent upstream suites (fixing the CI compilation failures in #2323). It also hardens the native execution runtime shutdown path to avoid error-reporting races during finalization, and adjusts Spark 4.x test settings to disable some suites that can crash and cause cascading failures in CI.

Changes:

• Rename/retarget Spark 4.0/4.1 cast suite wrappers to CastWithAnsiOnSuite / CastWithAnsiOffSuite, and remove AnsiCastSuiteWithAnsiMode{On,Off} wrappers for Spark 4.x.
• Prevent concurrent set_error calls during native runtime finalization by gating error reporting with an is_finalizing flag.
• Disable certain Spark 4.x suites in AuronSparkTestSettings to avoid crash-driven cascade failures; add Spark 4.1-specific excludes for datetime rebasing metadata tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
native-engine/auron/src/rt.rs	Adds finalization-aware guarding to avoid set_error after shutdown (and related concurrency hazards).
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronCastWithAnsiOnSuite.scala	Updates Spark 4.1 cast wrapper to extend CastWithAnsiOnSuite.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronCastWithAnsiOffSuite.scala	Updates Spark 4.1 cast wrapper to extend CastWithAnsiOffSuite.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronAnsiCastSuiteWithAnsiModeOn.scala	Removes Spark 4.1 wrapper referencing missing upstream suite.
auron-spark-tests/spark41/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronAnsiCastSuiteWithAnsiModeOff.scala	Removes Spark 4.1 wrapper referencing missing upstream suite.
auron-spark-tests/spark41/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Disables additional crash-prone suites; adds Spark 4.1 datetime rebase excludes.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronCastWithAnsiOnSuite.scala	Updates Spark 4.0 cast wrapper to extend CastWithAnsiOnSuite.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronCastWithAnsiOffSuite.scala	Updates Spark 4.0 cast wrapper to extend CastWithAnsiOffSuite.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronAnsiCastSuiteWithAnsiModeOn.scala	Removes Spark 4.0 wrapper referencing missing upstream suite.
auron-spark-tests/spark40/src/test/scala/org/apache/spark/sql/catalyst/expressions/AuronAnsiCastSuiteWithAnsiModeOff.scala	Removes Spark 4.0 wrapper referencing missing upstream suite.
auron-spark-tests/spark40/src/test/scala/org/apache/auron/utils/AuronSparkTestSettings.scala	Disables additional crash-prone suites to prevent cascade failures.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Tartarus0zm]

@richox @merrily01 PTAL

[SteNicholas]

LGTM. The cast-suite rename is correct (CastWithAnsiOn/OffSuite and TryCastSuite exist in both Spark 4.0 and 4.1, and this fixes the compile mismatch from #2284), the new ParquetRebaseDatetimeSuite excludes match the upstream test names exactly, and the rt.rs is_finalizing change is safe (next_batch/finalize are same-thread serialized; GlobalRef is Arc-refcounted). The remaining items (whole-suite .disable coverage trade-off, the 'date/partition' comment wording, and the freed-GlobalRef comment) are non-blocking.