Get started (free)

Spark applications

Spark applications are submitted to the Spark Operator as SparkApplication resources. These resources are used to define the configuration of the Spark job, including the image to use, the main application file, and the number of executors to start.

Upon creation, the application’s status set to Unknown. As the operator creates the necessary resources, the status of the application transitions through different phases that reflect the phase of the driver Pod. A successful application eventually reaches the Succeeded phase.

The operator never reconciles an application once it has been created. To resubmit an application, a new SparkApplication resource must be created.

Clean-up of finished applications

Once the driver Pod finished, the operator keeps it for a while, so that its logs can still be read, and deletes it afterwards. The time to live is set with spec.cleanup, separately for a driver Pod that succeeded and one that failed:

  • afterSucceeded: the time to live if the driver Pod succeeded. The default is 10m.

  • afterFailed: the time to live if the driver Pod failed. The default is 24h.

0s deletes the driver Pod as soon as it finished.

spec:
  cleanup:
    afterSucceeded: 0s
    afterFailed: 3d

The time to live counts from when the last container of the driver Pod terminated. If Kubernetes lost track of a container (ContainerStatusUnknown, for example after a node restart), it counts from when the driver Pod stopped being ready. Whether an application counts as failed depends on the phase of the driver Pod. cleanup can also come from a SparkApplicationTemplate, field by field. The times to live are fixed when the operator submits the application and are shown in status.effectiveCleanup of the SparkApplication.

The executors are owned by the driver Pod and are deleted with it, if Spark has not already deleted them. Executors that are still Pending when the driver finished are deleted right away, because they can never connect to the driver. The phase of the application is kept in the status of the SparkApplication after the driver Pod was deleted.

Spark makes the driver Pod the owner of PersistentVolumeClaims it creates on demand (claimName: OnDemand), so they can live as long as the driver Pod. If the driver exits normally, Spark deletes them itself, except the driver’s own PVC, which is only released once the driver Pod is deleted. If the driver crashes, the executor PVCs are kept as well. Kept PVCs count against storage ResourceQuotas, so consider a shorter afterFailed when using on-demand PVCs.

The spark-submit Job is kept for 10 minutes after it finished, independent of spec.cleanup. The ServiceAccount, RoleBinding and ConfigMaps the operator created are kept until the SparkApplication is deleted.

If spark.kubernetes.driver.pod.name is set, the operator deletes the driver pod as soon as it finished, so that the next submission or retry can create a driver pod with the same name. spec.cleanup must then be unset or set both afterSucceeded and afterFailed to 0s, otherwise the operator rejects the application. This also applies to a cleanup that comes from a SparkApplicationTemplate.
A driver Pod whose SparkApplication was deleted with kubectl delete --cascade=orphan is left alone and not deleted by the operator, also if a SparkApplication with the same name is created again.

Monitoring

Starting with version 25.7, the built-in Prometheus servlet is enabled. The JMX exporter is available but not used for applications. It has never been used automatically for applications and now it is deprecated.

Application driver pods expose Prometheus metrics at the following endpoints:

  • /metrics/prometheus for driver instances

  • /metrics/executors/prometheus for executor instances.

These endpoints are available on the same port as the Spark UI, which is 4040 by default.