Using sparkmonitor for remote kernels in Jupyter Enterprise Gateway setup.

[IMAM9AIS]

Hello,

We have a setup of jupyter where we are spawing ipython kernels remotely on a spark cluster via jupyter enterprise gateway setup.
This means that the kernel extension has to be installed on the remote machines before startup.
Although we have managed to have sparkmonitor installed on the remote machines, configuring the ipython_kernel_config.py does not seem possible for us on these remote machines.
Since these kernels are being launched using YARN as a resource manager providing a command line option to load this extension also does not seem to work.

I tried doing %load_extension post which i can load the extension and the conf variable shows all the relevant details. I have also installed sparkmonitor on my local machine for notebook ui extension and notebook server extension.

But although the spark jobs are executing properly, the extension does not seem to show the UI or display.

Do you have any ideas how can we fix this issue ?
Let me know if you need additional info. (we have tried installing this on local setup and things were working fine)