Skip to content

Native NUMERIC_OUT_OF_SUPPORTED_RANGE error surfaces as SparkException instead of SparkArithmeticException #7216

Open

Description

@andygrove

Describe the bug

Under ANSI mode, when the native string to decimal cast rejects a value as out of range with NUMERIC_OUT_OF_SUPPORTED_RANGE, such as '0e38', Comet throws a plain org.apache.spark.SparkException. Spark throws org.apache.spark.SparkArithmeticException, from outOfDecimalTypeRangeError (DataTypeErrors on 4.x, QueryExecutionErrors on 3.4 and 3.5). The error class, SQLSTATE and message are the same. Only the exception type differs, so code that catches ArithmeticException behaves differently with Comet enabled.

The native side raises SparkError::NumericOutOfRange, which exception_class() in native/common/src/error.rs groups with the SparkArithmeticException errors. But none of the three ShimSparkErrorConverters (spark-3.4, spark-3.5, spark-4.x) has a "NumericOutOfRange" case. So SparkErrorConverter.convertToSparkException falls through to its generic new SparkException(errorClass = ..., ...). In the 4.x shim, the only other native error type without a case is Arrow.

Steps to reproduce

SET spark.sql.ansi.enabled=true;
CREATE TABLE t (s STRING) USING parquet;
INSERT INTO t VALUES ('0e38');
SELECT CAST(s AS DECIMAL(10, 2)) FROM t;

On main at 330b29c with Spark 4.1.3, both engines fail with [NUMERIC_OUT_OF_SUPPORTED_RANGE] The value 0e38 cannot be interpreted as a numeric since it has more than 38 digits. SQLSTATE: 22003. Spark's exception is a SparkArithmeticException, and Comet's is a SparkException.

A SQL file test with query expect_error_class(NUMERIC_OUT_OF_SUPPORTED_RANGE) on that query fails with class org.apache.spark.SparkException did not equal class org.apache.spark.SparkArithmeticException.

Expected behavior

Comet throws the same SparkArithmeticException as Spark. Adding a "NumericOutOfRange" case to each shim that calls that version's outOfDecimalTypeRangeError would do it, with the expect_error_class query above as the test.

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmetic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions