Fix inconsistent spark driver and executor image

[chi-yelp]

Description

The Spark driver and executor image version can be inconsistent in a Tron action run when a job is already scheduled before the image is updated, which causes an InvalidClassException if the image's Spark version was changed.

Behavior:

• The action was run with old spark.kubernetes.container.image
• The image is different with the version in the latest Config command
• In the Tron web's META section, the PAASTA_DOCKER_IMAGE used is the latest version in the Config command

Solution

Set spark.kubernetes.container.image to $PAASTA_DOCKER_IMAGE when generating the spark-submit command, to read the environment variable set by Tron, which is the image that Spark driver uses.

Test

Checking the environment variable we are referencing for Tron in this PR

$ paasta spark-run --aws-profile=dev --cmd '/bin/env | grep PAASTA_DOCKER_IMAGE'
...
PAASTA_DOCKER_IMAGE=services-spark:paasta-<iamge_tag>

and spark.kubernetes.container.image should be something like: docker-paasta.yelpcorp.com:443/services-spark:<iamge_tag>

Release Plan

Pin the paasta version everywhere except pnw-devc, to test in devc first.