[Bug] [dinky-client]Submitting a JAR task to the Kubernetes (k8s) Flink application (version 1.19.1) results in a ClassNotFoundException, as it cannot find the Protobuf classes contained within the JAR task.

[jiangwwwei]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

In higher versions of Flink, the construction method for the class loader has changed (for example, org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.getOrResolveClassLoader()).

If the pipeline.jars of the JAR task are not correctly passed during task execution, the class loaders constructed by the JobMaster and TaskManager will not load the classes within the JAR. This results in a ClassNotFoundException when the JobMaster tries to deserialize job data during initialization because it cannot find the necessary classes.

Dinky calls addJar after downloading the JAR, but this only adds the configuration for the tableEnv. However, the configuration for executing the JAR pipeline comes from streamEnv, leading to a situation where the pipeline.jars in the submitted task's configuration only contains [file:/dinky/dinky-app-1.19-1.2.0-jar-with-dependencies.jar] instead of including the task's JAR as well.

After modifying the addJar() method in org.dinky.executor.CustomTableEnvironment, the classes within the JAR can now be loaded correctly.

Additionally, there is a similar issue with the addJar() method in org.dinky.executor.Executor. It appears that this method also suffers from the same problem regarding the usage of addJar syntax. However, since I have not utilized this feature yet, I have not made any modifications to it.

What you expected to happen

Successfully submit the JAR task to Kubernetes (k8s) and load the classes contained within it.

How to reproduce

1.Using Dinky version 1.2 with Flink version 1.19.1
2.Package the JAR task, ensuring that any Protobuf classes required during task execution are included within the JAR.
3.Submit the JAR task to a Kubernetes (k8s) Flink application cluster via Dinky.

Anything else

There is no issue with Flink 1.16.

Additionally, there is no problem when submitting JAR tasks to a Flink 1.19 Kubernetes session cluster via the web interface (Kubernetes Session Web).

Version

1.2.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Hello @jiangwwwei, this issue is about K8S, so I assign it to @gaoyan1998 and @zackyoungh. If you have any questions, you can comment and reply.

你好 @jiangwwwei, 这个 issue 是关于 K8S 的，所以我把它分配给了 @gaoyan1998 和 @zackyoungh。如有任何问题，可以评论回复。

[github-actions]

Hello @jiangwwwei, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @jiangwwwei, 这个 issue 是关于 CDC/CDCSOURCE 的，所以我把它分配给了 @aiwenmo。如有任何问题，可以评论回复。

[github-actions]

Hello @jiangwwwei, this issue is about web, so I assign it to @Zzm0809. If you have any questions, you can comment and reply.

你好 @jiangwwwei, 这个 issue 是关于 web 的，所以我把它分配给了 @Zzm0809。如有任何问题，可以评论回复。

[aiwenmo]

Hello, your discovery is wonderful.
Regarding the submission of the JAR task, are you willing to submit a pull request?