You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native NUMERIC_OUT_OF_SUPPORTED_RANGE error surfaces as SparkException instead of SparkArithmeticException #7216
Under ANSI mode, when the native string to decimal cast rejects a value as out of range with NUMERIC_OUT_OF_SUPPORTED_RANGE, such as '0e38', Comet throws a plain org.apache.spark.SparkException. Spark throws org.apache.spark.SparkArithmeticException, from outOfDecimalTypeRangeError (DataTypeErrors on 4.x, QueryExecutionErrors on 3.4 and 3.5). The error class, SQLSTATE and message are the same. Only the exception type differs, so code that catches ArithmeticException behaves differently with Comet enabled.
The native side raises SparkError::NumericOutOfRange, which exception_class() in native/common/src/error.rs groups with the SparkArithmeticException errors. But none of the three ShimSparkErrorConverters (spark-3.4, spark-3.5, spark-4.x) has a "NumericOutOfRange" case. So SparkErrorConverter.convertToSparkException falls through to its generic new SparkException(errorClass = ..., ...). In the 4.x shim, the only other native error type without a case is Arrow.
Steps to reproduce
SETspark.sql.ansi.enabled=true;
CREATETABLEt (s STRING) USING parquet;
INSERT INTO t VALUES ('0e38');
SELECT CAST(s ASDECIMAL(10, 2)) FROM t;
On main at 330b29c with Spark 4.1.3, both engines fail with [NUMERIC_OUT_OF_SUPPORTED_RANGE] The value 0e38 cannot be interpreted as a numeric since it has more than 38 digits. SQLSTATE: 22003. Spark's exception is a SparkArithmeticException, and Comet's is a SparkException.
A SQL file test with query expect_error_class(NUMERIC_OUT_OF_SUPPORTED_RANGE) on that query fails with class org.apache.spark.SparkException did not equal class org.apache.spark.SparkArithmeticException.
Expected behavior
Comet throws the same SparkArithmeticException as Spark. Adding a "NumericOutOfRange" case to each shim that calls that version's outOfDecimalTypeRangeError would do it, with the expect_error_class query above as the test.
The cast tests compare messages, not exception classes. castTest in CometNativeCastSuite compares the start of the message on Spark 4, which is why this hasn't shown up.
Activity