When the to_timestamp_millis function is used and the output format is csv, it will panic

[ZuoTiJia]

To Reproduce

datafusion-cli --format csv
DataFusion CLI v16.0.0
❯ SELECT to_timestamp_millis(1926632005177685347);
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', arrow-csv-29.0.0/src/writer.rs:253:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The bug of csv writer occurs when unwrap() is called during conversion, which seems to be a bug of arrow-csv.
Expected behavior
Throw an error, no panic

[comphead]

@ZuoTiJia
It looks like None returns because the real error is supressed, if I run it without CSV format

DataFusion CLI v16.0.0
❯ select to_timestamp_millis(1926632005177685347);
+-----------------------------------------------+
| totimestampmillis(Int64(1926632005177685347)) |
+-----------------------------------------------+
| ERROR CONVERTING DATE                         |
+-----------------------------------------------+

[ZuoTiJia]

@ZuoTiJia It looks like None returns because the real error is supressed, if I run it without CSV format

DataFusion CLI v16.0.0
❯ select to_timestamp_millis(1926632005177685347);
+-----------------------------------------------+
| totimestampmillis(Int64(1926632005177685347)) |
+-----------------------------------------------+
| ERROR CONVERTING DATE                         |
+-----------------------------------------------+

Mixing correct and incorrect results is not good, it would be nice to be able to throw an error.

DataFusion CLI v16.0.0
❯ CREATE TABLE test(time BIGINT) AS VALUES (1), (1926632005177685347);
0 rows in set. Query took 0.006 seconds.
❯ select * from test;
+---------------------+
| time                |
+---------------------+
| 1                   |
| 1926632005177685347 |
+---------------------+
2 rows in set. Query took 0.003 seconds.
❯ select to_timestamp_millis(time) from test;
+------------------------------+
| totimestampmillis(test.time) |
+------------------------------+
| 1970-01-01T00:00:00.001      |
| ERROR CONVERTING DATE        |
+------------------------------+
2 rows in set. Query took 0.003 seconds.
❯

[comphead]

Agree, in old spark versions, if timestamp cast cannot be done then null returned that was really confusing, we should consider a runtime error if such invalid cast happened. I'll fix it in apache/arrow-rs#3547 soon

[comphead]

Waiting apache/arrow-rs#3584 and apache/arrow-rs#3514

[comphead]

@ZuoTiJia The initial problem is solved.

./target/debug/datafusion-cli --format csv

DataFusion CLI v19.0.0
❯ select to_timestamp_millis(1926632005177685347);
Arrow error: Csv error: Error formatting row 1 and column 1: Cast error: Failed to convert 1926632005177685347 to datetime for Timestamp(Millisecond, None)
❯ 

Another problem still in place is to remove mixed values, lets do that in separate issue if needed

[alamb]

(venv-310) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs$ datafusion-cli --format csv
DataFusion CLI v42.1.0
> SELECT to_timestamp_millis(1926632005177685347);
to_timestamp_millis(Int64(1926632005177685347))
Arrow error: Csv error: Error processing row 1, col 1: Cast error: Failed to convert 1926632005177685347 to datetime for Timestamp(Millisecond, None)
>

Seems like the panic is fixed and the code is working as expected

[alamb]

Thanks (again) to @drauschenbach for pointing this out