Assigning Kubernetes Queues and User Ids for Grouping and Chargeback Reporting

In Kubernetes clusters, you can associate a queue group with an application by configuring a queue label for the executor Pod, and then using the Pepperdata dashboard to associate the label (an abstraction for an attribute) with specific queues. Likewise, you can associate the user label with an application, and then use the Pepperdata dashboard to associate that label with specific users.

Procedure

• Add the applicable Spark properties for labels—queue for queue groups; user for users—to all Pepperdata-monitored Spark applications.

  • Add the properties to the same <spark-job>.yaml files that you configured for Pepperdata instrumentation (see Activate Pepperdata for Spark Applications).

  • Be sure to replace the-* placeholder names with the actual names.

  # For queue groups:
  "spark.kubernetes.executor.label.queue": "the-queue-name"
  #
  # For users:
  "spark.kubernetes.executor.label.user": "the-user-name"
  

What to do next

• Associate the queue and/or user label attributes with specific queues and/or users, respectively; see Configure Labels.