Erros messages weired rendering

[ptabandzelic]

Hi, this is how i get error messages in both terminal and UI. Any idea?

RuntimeError: Command execution failed. Traceback from cluster: 
[0;31m---------------------------------------------------------------------------[0m
[0;31mKeyError[0m                                  Traceback (most recent call last)
[0;32m<command--1>[0m in [0;36m<cell line: 3>[0;34m()[0m
[1;32m      1[0m [0;32mimport[0m [0mpkg_resources[0m[0;34m[0m[0;34m[0m[0m
[1;32m      2[0m [0m_func[0m [0;34m=[0m [0mpkg_resources[0m[0;34m.[0m[0mload_entry_point[0m[0;34m([0m[0;34m"rca_nbo_rbal"[0m[0;34m,[0m [0;34m"console_scripts"[0m[0;34m,[0m
[0;34m"preprocessing"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m----> 3[0;31m [0m_func[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m      4[0m [0;34m[0m[0m

[0;32m/local_disk0/.ephemeral_nfs/envs/pythonEnv-b8ac7c01-e8f2-4cfa-80ca-669026119966/lib/python3.9/site-packages/rca_nbo_rbal/tasks/preprocessing/entrypoint.py[0m in 
[0;36msubmit_task[0;34m()[0m
[1;32m     33[0m [0;32mdef[0m [0msubmit_task[0m[0;34m([0m[0;34m)[0m [0;34m->[0m [0;32mNone[0m[0;34m:[0m[0;34m[0m[0;34m[0m[0m
[1;32m     34[0m     [0mtask[0m [0;34m=[0m [0mPreprocessingTask[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 35[0;31m     [0mtask[0m[0;34m.[0m[0mlaunch[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m
[0;32m/local_disk0/.ephemeral_nfs/envs/pythonEnv-b8ac7c01-e8f2-4cfa-80ca-669026119966/lib/python3.9/site-packages/rca_nbo_rbal/tasks/preprocessing/entrypoint.py[0m in 
[0;36mlaunch[0;34m(self)[0m
[1;32m     27[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0minfo[0m[0;34m([0m[0;34m"Starting execution of preprocessing task"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m     28[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0mdebug[0m[0;34m([0m[0;34mf"Logging execution of mlflow run under mlflow experiment: 
{self.conf.experiment_id}"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 29[0;31m         [0mself[0m[0;34m.[0m[0m_process_data[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m     30[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0minfo[0m[0;34m([0m[0;34m"Successfully finished preprocessing task"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m     31[0m [0;34m[0m[0m

[0;32m/local_disk0/.ephemeral_nfs/envs/pythonEnv-b8ac7c01-e8f2-4cfa-80ca-669026119966/lib/python3.9/site-packages/rca_nbo_rbal/tasks/preprocessing/entrypoint.py[0m in 
[0;36m_process_data[0;34m(self)[0m
[1;32m     21[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0minfo[0m[0;34m([0m[0;34m"Successfully read raw parquet data into PySpark 
DataFrame"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m     22[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0mdebug[0m[0;34m([0m[0;34mf"Shape of raw PySpark DataFrame: {rawDF.count(), 
len(rawDF.columns)}"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;32m---> 23[0;31m         [0mfilter_expr[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_provide_filter[0m[0;34m([0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0m[1;32m     24[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0mdebug[0m[0;34m([0m[0;34mf"FILTER:{filter_expr}"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m     25[0m [0;34m[0m[0m

[0;32m/local_disk0/.ephemeral_nfs/envs/pythonEnv-b8ac7c01-e8f2-4cfa-80ca-669026119966/lib/python3.9/site-packages/rca_nbo_rbal/common/abstract_task.py[0m in 
[0;36m_provide_filter[0;34m(self)[0m
[1;32m    142[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0minfo[0m[0;34m([0m[0;34m"Reading filters based on --segment and --product from task 
option"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    143[0m         [0msegment[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0m_args_parser[0m[0;34m([0m[0;34m)[0m[0;34m[[0m[0;34m"segment"[0m[0;34m][0m[0;34m[0m[0;34m[0m[0m
[0;32m--> 144[0;31m         [0mproduct[0m [0;34m=[0m `[0mself[0m[0;34m.[0m[0m_args_parser[0m[0;34m([0m[0;34m)[0m[0;34m[[0m[0;34m"product"[0m[0;34m][0m[0;34m[0m[0;34m[0m[0m`
[0m[1;32m    145[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0mdebug[0m[0;34m([0m[0;34mf"Provided segment:{segment} 
product:{product}"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[1;32m    146[0m         [0mself[0m[0;34m.[0m[0mlogger[0m[0;34m.[0m[0mdebug[0m[0;34m([0m[0;34mf"Reading filters logic from 
{self.conf.filters}"[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m

[copdips]

is this from a Windows console ?

[ptabandzelic]

Nope, ubuntu, but its like this in Workflows UI as well

[Squaess]

I'm having the same issue. The error messages contain parts of the ANSI escape characters, both in my terminal and on the Databricks instance.

[renardeinside]

hi @Squaess ,
could you please create a GH issue? I've seen this error once, but cannot repro it with dbx 0.8.2. Could you please check it with the latest version, and if it's still there - please open a GH issue, unfortunately I'm dot get notifications from discussions section.

[Squaess]

@ptabandzelic Can you check what Databricks runtime the cluster is using? I just check this and I'm having the same issue when using the newest 11.3 LTS, but when I switch the cluster to 10.4 LTS it's working fine. Both deployments are done with the dbx version 0.8.7.

Created an issue for this: #590

[copdips]

does this error happen during the running of the jobs, or during the dbx deployment ?

[Squaess]

To be specific it's not about the error happening, it's about how it's being display.
It's happening when I run the job.