Assigning Kubernetes Queues and User Ids for Grouping and Chargeback Reporting
In Kubernetes clusters, you can associate a queue group with an application by configuring a
queue label for the executor Pod, and then using the Pepperdata dashboard to associate the label (an abstraction for an attribute) with specific queues.
Likewise, you can associate the
user label with an application, and then use the Pepperdata dashboard to associate that label with specific users.
Add the applicable Spark properties for labels—
queuefor queue groups;
userfor users—to all Pepperdata-monitored Spark applications.
Add the properties to the same
<spark-job>.yamlfiles that you configured for Pepperdata instrumentation (see Activate Pepperdata for Spark Applications).
Be sure to replace
the-*placeholder names with the actual names.
# For queue groups: "spark.kubernetes.executor.label.queue": "the-queue-name" # # For users: "spark.kubernetes.executor.label.user": "the-user-name"
What to do next
- Associate the
userlabel attributes with specific queues and/or users, respectively; see Configure Labels.