Executor pods not starting while submitting spark application from operator

[sp-matrix]

Hello,

The operator was installed in our openshift cluster (organization). When the example spark application (spark-examples_2.11-2.4.5.jar) was submitted with the help of operator, submitter pod and driver pod was getting created but the executor pod is not getting created and failing with the below error.

INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes.
ERROR Utils: Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc/api/v1/namespaces/nm/pods. Message: Pod "my-spark-app-1653416287945-exec-4" is invalid: spec.containers[0].resources.requests: Invalid value: "1": must be less than or equal to cpu limit. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.containers[0].resources.requests, message=Invalid value: "1": must be less than or equal to cpu limit, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, name=my-spark-app-1653416287945-exec-4, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "my-spark-app-1653416287945-exec-4" is invalid: spec.containers[0].resources.requests: Invalid value: "1": must be less than or equal to cpu limit, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure

After going through the documentation, we tried limiting the CPU and core but the issue is not getting resolved.

SparkApplication yaml:
...
...
spec:
driver:
coreLimit: 500m
cores: 0.2
executor:
coreLimit: 1000m
coreRequest: 0.5
cores: 1 # we can't give below 1 (float values), when we skip this parameter the default value 1 is assigned.
cpuLimit: 1000m
instances: 1
...

In configmap of the driver:

spark.executor.memory=512m
spark.driver.blockManager.port=7079
spark.ui.reverseProxy=true
spark.executorEnv.APPLICATION_NAME=my-spark-app
spark.kubernetes.container.image=quay.io/radanalyticsio/openshift-spark:2.4-latest
spark.jars=/opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar
spark.ui.reverseProxyUrl=/
spark.kubernetes.driver.limit.cores=500m
spark.kubernetes.submitInDriver=true
spark.driver.memory=512m
spark.submit.deployMode=cluster
spark.kubernetes.driverEnv.APPLICATION_NAME=my-spark-app
spark.kubernetes.executor.label.radanalytics.io/SparkApplication=my-spark-app
spark.kubernetes.driver.label.radanalytics.io/SparkApplication=my-spark-app
spark.executor.cores=1
spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator
spark.jars.ivy=/tmp/.ivy2
spark.kubernetes.driver.pod.name=my-spark-app-1653416287945-driver
spark.executor.instances=1
spark.kubernetes.namespace=nm-np
spark.app.id=spark-d7cd179d47a047dd9d11811a99e1060c
spark.app.name=my-spark-app
spark.kubernetes.driver.label.version=2.3.0
spark.driver.cores=0.2
spark.driver.port=7078

Note: we tried starting the service 'cluster-limreq' with 4 cpu limit for executors as mentioned in READme but the issue was not resolved.

Resource Quota of my namespace allocated by my cluster manager

Name: core-resource-limits-hermes
Namespace: nm-np
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio

---

Container cpu 25m 4 150m 400m -
Container memory 25Mi 4Gi 256Mi 512Mi -
Pod cpu 25m 4 - - -
Pod memory 25Mi 4Gi - - -

Can anyone please help to find the reason why the executor Pod is not staring even though we request only 1 cpu from the allocated quota of 4 cpus.

Thanks!

[elmiko]

this looks like kubernetes is complaining about your Pod spec for the application. do you have both requests and limits defined in the pod.spec.containers[0].resoures ?

could you share the Pod or Deployment spec you are using?

[sp-matrix]

Thank you @elmiko for the reply.

I am getting into the operator in OC and then submitting the application in UI. While submitting, below is my complete .yaml configuration.

apiVersion: radanalytics.io/v1
kind: SparkApplication
metadata:
name: my-spark-app
namespace: xxx-np
spec:
mainApplicationFile: 'local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar'
mainClass: org.apache.spark.examples.SparkPi
driver:
cores: 0.2
coreLimit: 500m
executor:
coreLimit: 1000m
coreRequest: 0.5
cores: 1
cpuLimit: 1000m
instances: 1

Not using the requests and limits options in the .yaml.

[elmiko]

it's possible we have a bug in that logic for the requests/limits

[sp-matrix]

@elmiko Is there any workaround ?

[elmiko]

i need to see more details about the Pod and Deployments that are being created, knowing the SparkApplication yaml is not giving enough detail as it looks like the kubernetes API does not like the Pod definition.

would it be possible to share those records? (and please keep the formatting)

[sp-matrix]

@elmiko : Update on the issue.
I tried to run a spark application in the same namespace via spark-submit, when the 'executor.cores' is <= 0.4, I was able to run it, If i use value beyond 0.4 cpu for executors, it is failing.
When I use operator, even though I set cpuRequest as 0.4 or less than 1, the default value 1 is set to executor cores. whereas I can pass any value greater than 1, but the available cpu is 0.4, it is failing.

Operator Deployment yaml :
kind: Deployment
apiVersion: apps/v1
metadata:
annotations:
deployment.kubernetes.io/revision: '1'
resourceVersion: '1876495850'
name: spark-operator
uid: 503bc9cd-ebdd-45c1-b575-e02fc04b8163
creationTimestamp: '2022-05-17T22:33:56Z'
generation: 2
managedFields:
- manager: olm
operation: Update
apiVersion: apps/v1
time: '2022-05-17T22:33:57Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:labels':
.: {}
'f:olm.deployment-spec-hash': {}
'f:olm.owner': {}
'f:olm.owner.kind': {}
'f:olm.owner.namespace': {}
'f:operators.coreos.com/radanalytics-spark.agl-np': {}
'f:ownerReferences':
.: {}
'k:{"uid":"0dbf1c06-ac44-4337-836a-68edb418f7d4"}':
.: {}
'f:apiVersion': {}
'f:blockOwnerDeletion': {}
'f:controller': {}
'f:kind': {}
'f:name': {}
'f:uid': {}
'f:spec':
'f:progressDeadlineSeconds': {}
'f:replicas': {}
'f:revisionHistoryLimit': {}
'f:selector': {}
'f:strategy':
'f:rollingUpdate':
.: {}
'f:maxSurge': {}
'f:maxUnavailable': {}
'f:type': {}
'f:template':
'f:metadata':
'f:annotations':
'f:olm.operatorNamespace': {}
'f:olm.properties': {}
'f:createdAt': {}
'f:alm-examples': {}
'f:description': {}
'f:olm.operatorGroup': {}
'f:capabilities': {}
.: {}
'f:containerImage': {}
'f:categories': {}
'f:certified': {}
'f:operatorframework.io/properties': {}
'f:support': {}
'f:olm.targetNamespaces': {}
'f:repository': {}
'f:labels':
.: {}
'f:app.kubernetes.io/name': {}
'f:spec':
'f:containers':
'k:{"name":"spark-operator"}':
.: {}
'f:env':
.: {}
'k:{"name":"HTTPS_PROXY"}':
.: {}
'f:name': {}
'f:value': {}
'k:{"name":"HTTP_PROXY"}':
.: {}
'f:name': {}
'f:value': {}
'k:{"name":"NO_PROXY"}':
.: {}
'f:name': {}
'f:value': {}
'k:{"name":"OPERATOR_CONDITION_NAME"}':
.: {}
'f:name': {}
'f:value': {}
'k:{"name":"WATCH_NAMESPACE"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:fieldRef':
.: {}
'f:apiVersion': {}
'f:fieldPath': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:name': {}
'f:resources': {}
'f:terminationMessagePath': {}
'f:terminationMessagePolicy': {}
'f:dnsPolicy': {}
'f:restartPolicy': {}
'f:schedulerName': {}
'f:securityContext': {}
'f:serviceAccount': {}
'f:serviceAccountName': {}
'f:terminationGracePeriodSeconds': {}
- manager: kube-controller-manager
operation: Update
apiVersion: apps/v1
time: '2022-05-17T22:34:26Z'
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
.: {}
'f:deployment.kubernetes.io/revision': {}
'f:status':
'f:availableReplicas': {}
'f:conditions':
.: {}
'k:{"type":"Available"}':
.: {}
'f:lastTransitionTime': {}
'f:lastUpdateTime': {}
'f:message': {}
'f:reason': {}
'f:status': {}
'f:type': {}
'k:{"type":"Progressing"}':
.: {}
'f:lastTransitionTime': {}
'f:lastUpdateTime': {}
'f:message': {}
'f:reason': {}
'f:status': {}
'f:type': {}
'f:observedGeneration': {}
'f:readyReplicas': {}
'f:replicas': {}
'f:updatedReplicas': {}
namespace: agl-np
ownerReferences:
- apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
name: sparkoperator.v1.1.0
uid: 0dbf1c06-ac44-4337-836a-68edb418f7d4
controller: false
blockOwnerDeletion: false
labels:
olm.deployment-spec-hash: d48ccd4b6
olm.owner: sparkoperator.v1.1.0
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: xxx-np
operators.coreos.com/radanalytics-spark.agl-np: ''
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: radanalytics-spark-operator
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/name: radanalytics-spark-operator
annotations:
certified: 'false'
olm.targetNamespaces: xxx-np
operatorframework.io/properties: >-
{"properties":[{"type":"olm.gvk","value":{"group":"radanalytics.io","kind":"SparkApplication","version":"v1"}},{"type":"olm.gvk","value":{"group":"radanalytics.io","kind":"SparkCluster","version":"v1"}},{"type":"olm.gvk","value":{"group":"radanalytics.io","kind":"SparkHistoryServer","version":"v1"}},{"type":"olm.maxOpenShiftVersion","value":"4.8"},{"type":"olm.package","value":{"packageName":"radanalytics-spark","version":"1.1.0"}}]}
repository: 'https://github.com/radanalyticsio/spark-operator'
support: [email protected]
olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8"}]'
alm-examples: |-
[
{
"apiVersion": "radanalytics.io/v1",
"kind": "SparkCluster",
"metadata": {
"name": "my-spark-cluster"
},
"spec": {
"worker": {
"instances": "2"
},
"master": {
"instances": "1"
}
}
},
{
"apiVersion": "radanalytics.io/v1",
"kind": "SparkApplication",
"metadata": {
"name": "my-spark-app"
},
"spec": {
"mainApplicationFile": "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar",
"mainClass": "org.apache.spark.examples.SparkPi",
"driver": {
"cores": 0.2,
"coreLimit": "500m"
},
"executor": {
"instances": 2,
"cores": 1,
"coreLimit": "1000m"
}
}
},
{
"apiVersion": "radanalytics.io/v1",
"kind": "SparkHistoryServer",
"metadata": {
"name": "my-history-server"
},
"spec": {
"type": "remoteStorage",
"expose": true,
"logDirectory": "s3a://my-history-server/",
"updateInterval": 10,
"retainedApplications": 50,
"customImage": "quay.io/jkremser/openshift-spark:2.4.0-aws",
"sparkConfiguration": [
{
"name": "spark.hadoop.fs.s3a.impl",
"value": "org.apache.hadoop.fs.s3a.S3AFileSystem"
},
{
"name": "spark.hadoop.fs.s3a.access.key",
"value": "foo"
},
{
"name": "spark.hadoop.fs.s3a.secret.key",
"value": "bar"
},
{
"name": "spark.hadoop.fs.s3a.endpoint",
"value": "http://ceph-nano-0:8000"
}
]
}
}
]
capabilities: Deep Insights
olm.operatorNamespace: agl-np
containerImage: 'quay.io/radanalyticsio/spark-operator:1.1.0'
createdAt: '2019-01-17 12:00:00'
categories: Big Data
description: >-
An operator for managing the Apache Spark clusters and intelligent
applications that spawn those clusters.
olm.operatorGroup: radanalytics-spark
spec:
containers:
- name: spark-operator
image: 'quay.io/radanalyticsio/spark-operator:1.1.0'
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: 'metadata.annotations[''olm.targetNamespaces'']
- name: OPERATOR_CONDITION_NAME
value: sparkoperator.v1.1.0
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 5
dnsPolicy: ClusterFirst
serviceAccountName: spark-crd-operator
serviceAccount: spark-crd-operator
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 1
progressDeadlineSeconds: 600
status:
observedGeneration: 2
replicas: 1
updatedReplicas: 1
readyReplicas: 1
availableReplicas: 1
conditions:
- type: Available
status: 'True'
lastUpdateTime: '2022-05-17T22:34:26Z'
lastTransitionTime: '2022-05-17T22:34:26Z'
reason: MinimumReplicasAvailable
message: Deployment has minimum availability.
- type: Progressing
status: 'True'
lastUpdateTime: '2022-05-17T22:34:26Z'
lastTransitionTime: '2022-05-17T22:33:56Z'
reason: NewReplicaSetAvailable
message: ReplicaSet "spark-operator-97bdcc5fc" has successfully progressed.

Submitter POD's Log after submitting the application

cmd:
$SPARK_HOME/bin/spark-submit
--class org.apache.spark.examples.SparkPi
--master k8s://https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT
--conf spark.kubernetes.namespace=xxx-np
--deploy-mode cluster
--conf spark.app.name=my-spark-app
--conf spark.kubernetes.container.image=quay.io/radanalyticsio/openshift-spark:2.4-latest
--conf spark.kubernetes.submission.waitAppCompletion=false
--conf spark.driver.cores=0.5
--conf spark.kubernetes.driver.limit.cores=500m
--conf spark.driver.memory=512m
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator
--conf spark.kubernetes.driver.label.version=2.3.0
--conf spark.kubernetes.driver.label.radanalytics.io/SparkApplication=my-spark-app
--conf spark.kubernetes.executor.label.radanalytics.io/SparkApplication=my-spark-app
--conf spark.kubernetes.driverEnv.APPLICATION_NAME=my-spark-app
--conf spark.executorEnv.APPLICATION_NAME=my-spark-app
--conf spark.executor.instances=1
--conf spark.executor.cores=1
--conf spark.executor.memory=512m
--conf spark.jars.ivy=/tmp/.ivy2 local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar && echo -e

Thanks !

[elmiko]

thanks for the report back @sp-matrix . it sounds like we might have a bug in the operator around converting the cpu cores value properly. i haven't looked at this code in awhile so i can't predict any sort of fix, but it certainly sounds like we are not converting the value properly.