If you’re using Application Profiler to fetch history data for Spark apps, you can customize the connection timeout value and/or add a second Spark History Server for monitoring.

Configure Connection Timeout for Spark History Server

In environments with extreme network latency or frequent connectivity issues, it can be helpful to increase the connection timeout setting for REST requests that fetch Spark app data from the Spark History Server. By default, Pepperdata waits five (5) seconds before timing out, but you can change this value to suit your environment.

Related Topics

• Configure History Fetcher Retries. To ensure that application history is successfully fetched from the applicable component (MapReduce Job History Server for MapReduce apps, or Spark History Server for Spark apps), the Pepperdata Supervisor uses a two-phase approach. Phase 1 makes the initial attempt to fetch the history, and if it fails, makes up to three retries. Phase 2 adds an additional try and by default up to five retries, with the interval between retries increased by a factor of five every time. You can customize the number of retries for each phase, which might be required for environments with extreme network latency or frequent connectivity issues.

Procedure

1. Add the environment variable for the connection timeout, PD_JOBHISTORY_SPARK_CONNECTION_TIMEOUT_SEC.

   Use Cloudera Manager to add the environment variable for the connection timeout to the Pepperdata > Configuration > Pepperdata Service Environment Advanced Configuration Snippet (Safety Valve) template.

   Add the environment variable in the following format. Be sure to replace the default connection timeout (5 seconds) with your custom value.

   PD_JOBHISTORY_SPARK_CONNECTION_TIMEOUT_SEC=5
   

2. Restart the Pepperdata services.

   In Cloudera Manager, select the Restart action for the Pepperdata service.

Read from Two Spark History Servers

If you have two Spark History Servers (typically because you’re in the midst of upgrading to a newer version, and you’re temporarily running both versions during the migration), you can configure Pepperdata to read from both of them. Perform the configuration for the second Spark History Server on any host other than the MapReduce Job History Server host (which is the primary history fetcher host for Pepperdata). On the chosen host, add environment variables for the second Spark History Server to the Pepperdata configuration.

Prerequisites

1. Complete the regular Pepperdata installation steps; see Installing Pepperdata.

2. Complete the regular Pepperdata configuration steps, including the configuration of the first Spark History Server; see Configuring Pepperdata.

3. Start up Pepperdata; see Starting Up Pepperdata.

Procedure

1. Choose any host other than the MapReduce Job History Server host, which is the primary history fetcher host for Pepperdata.

2. Add the configuration for the second Spark History Server.

   Use Cloudera Manager to add the following snippet to the Pepperdata > Configuration > PepAgent Environment Advanced Configuration Snippet (Safety Valve) template.

   PD_SPARK_HISTORY_SERVER_ADDRESS=your-server-url
   

   Where your-server=url is the fully-qualified URL of the second Spark History Server.

3. Disambiguate the two Spark History Servers.

   1. In Cloudera Manager, navigate to Pepperdata > Instances.

   2. Click an instance of PepAgent that is not a job history host, and click Configuration.

      An alert appears that recommends that you make configuration changes on the service configuration page instead.

      DO NOT click Cancel or Service Configuration.

   3. Dismiss the alert by clicking the × in the upper-right corner of the alert.

   4. Add the following snippet to the PepAgent Environment Advanced Configuration Snippet (Safety Valve) template.

      PD_BYPASS_JOBHISTORY_IS_LOCAL_CHECK=1
      PD_DO_JOBHISTORY_STARTUP_CHECK=0
      PD_JOBHISTORY_FETCHERS="spark"
      

   5. Click Save Changes.

4. In Cloudera Manager, select the Restart action for the PepAgent service.