[VL] Fallback happens to vanilla spark if there is a cast from varchar to timestamp

[ayushi-agarwal]

Description

Currently there is only a support of conversion from date to timestamp -

incubator-gluten/cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc

Line 273 in ff89539

if (toType->kind() == TypeKind::TIMESTAMP && !input->type()->isDate()) {

An example expression for which fallback happens: cast(from_utc_timestamp(from_unixtime((cast(col as double) / 1000.0), 'yyyy-MM-dd HH:mm:ss'), 'PST') as date)

Is there a plan to support other data types also? @dcoliversun

[ayushi-agarwal]

I modified the code so that varchar to timestamp cast is also gets pushed to velox. For PST it fails with below error but works fine if I give America/Los_Angeles
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: Unknown time zone: 'PST'
Retriable: False
Expression: toTimeZone != nullptr
Context: Top-level Expression: from_utc_timestamp(try_cast((from_unixtime(try_cast((divide(try_cast((n0_0) as DOUBLE), 1000:DOUBLE)) as BIGINT), yyyy-MM-dd HH:mm:ss.SSS :VARCHAR)) as TIMESTAMP), PST:VARCHAR)
Function: call
File: /home/ayushi/workspace/incubator-gluten/ep/build-velox/build/velox_ep/./velox/functions/sparksql/DateTimeFunctions.h
Line: 374
This error might be coming because PST is not present in https://github.com/facebookincubator/velox/blob/main/velox/type/tz/TimeZoneDatabase.cpp
@PHILO-HE Is this the reason we have this validation check in gluten to not offload it to velox?

Also at present timestamp to date cast is also not offloaded but when I tried doing that it gave correct result, shall we enable it?

[FelixYBW]

can #8357 fix it?

[ayushi-agarwal]

@FelixYBW I did the same change in my local and it resolves the cast from varchar to timestamp offload issue but then I got the above error for from_utc_timestamp expression when I gave PST as parameter. Any suggestions for this as this would need change in velox?

[FelixYBW]

@FelixYBW I did the same change in my local and it resolves the cast from varchar to timestamp offload issue but then I got the above error for from_utc_timestamp expression when I gave PST as parameter. Any suggestions for this as this would need change in velox?

I see. I don't think we can fix this in Gluten itself.

[PHILO-HE]

@ayushi-agarwal, I think abbreviated timezone name like "PST" is not supported in Velox, because it's ambiguous. And it is not recommended by Spark (see link).

[ayushi-agarwal]

@ayushi-agarwal, I think abbreviated timezone name like "PST" is not supported in Velox, because it's ambiguous. And it is not recommended by Spark (see link).

That makes sense now. Thank you for directing me to this code.