Apache Hadoop with YARN: Configuring HBase Monitoring (RPM/DEB)

Supported versions: See the Apache Hadoop entries for Pepperdata 8.0.x in the table of Supported Platforms by Pepperdata Version

To configure HBase, activate Pepperdata in the HBase RegionServers, and add the required properties to the Pepperdata site configuration to enable collecting the HBase metrics. If there are different authentication types for the HBase Web UI and the other cluster services, override the Pepperdata-default setting (pepperdata.agent.hbase.http.authentication.type) for HBase. If the HTTP service policy for HBase is different than the HTTP service policy for YARN, override the Pepperdata-default setting (pepperdata.agent.hbase.http.policy) for HBase.

Prerequisites

Before you perform any of this procedure’s tasks (Task 1–Task 4), ensure that you know the HBase URL endpoint’s HTTP service policy (HTTPS_ONLY or HTTP_ONLY). You’ll need to know the HTTP service policy when you:

Task 1: Configure Pepperdata to Collect HBase Filesystem Metrics

To activate Pepperdata—inject the necessary instrumentation—for HBase in a manually configured cluster, add the activation script to each Hadoop service’s environment shell executable file.

Procedure

  1. Beginning with any host, add the following snippet to the HBase RegionServer configuration file, hbase-env.sh, typically located in /etc/hbase/conf.

    PEPPERDATA_ACTIVATE_SCRIPT_PATH="/opt/pepperdata/supervisor/lib/pepperdata-activate.sh"
    if [ -e $PEPPERDATA_ACTIVATE_SCRIPT_PATH ]; then
     . $PEPPERDATA_ACTIVATE_SCRIPT_PATH
    fi
    
  2. Repeat step 1 on every host in the cluster.

  3. On every host in the cluster, restart the HBase RegionServer daemon.

Task 2: Turn On HBase Monitoring and Configure Ports and Polling

To configure Pepperdata to collect HBase metrics, set the pepperdata.agent.genericJsonFetch.hbase.enabled property in the Pepperdata site configuration to true, and optionally add properties to override the default ports and polling intervals for the RegionServer and Master Server.

Procedure

  1. Beginning with any host, enable HBase metrics collection.

    1. Open the host’s Pepperdata site file, pepperdata-site.xml, for editing.

      By default, the Pepperdata site file, pepperdata-site.xml, is located in /etc/pepperdata. If you customized the location, the file is specified by the PD_CONF_DIR environment variable. See Change the Location of pepperdata-site.xml for details.

    2. Add the following snippet.

      <!-- For HBase Fielsystem Metrics Collection -->
      <property>
          <name>pepperdata.agent.genericJsonFetch.hbase.enabled</name>
          <value>true</value>
      </property>
      
      Malformed XML files can cause operational errors that can be difficult to debug. To prevent such errors, we recommend that you use a linter, such as xmllint, after you edit any .xml configuration file.
  2. (Optional) To use ports and settings other than the defaults, configure any/all of the override properties.

    The table shows the default values.

    Setting Default Value
    RegionServer port 16030
    RegionServer polling interval 60000 milliseconds (1 minute)
    HBase Master Server port 16010
    HBase Master Server polling interval 60000 milliseconds (1 minute)
    1. To change the RegionServer port, add the pepperdata.agent.hbase.jmx.regionserver.port property.

      <!-- For an HBase Region Server port other than the default Hadoop port (16030) -->
      <property>
        <name>pepperdata.agent.hbase.jmx.regionserver.port</name>
        <value>your-region-server-portnum</value>
      </property>
      
    2. To change the RegionServer polling interval, add the pepperdata.agent.hbase.web.regionservermetrics.monitor.interval property.

      <!-- For an HBase Region Server polling interval other than the default (60000 milliseconds) -->
      <property>
        <name>pepperdata.agent.hbase.web.regionservermetrics.monitor.interval</name>
        <value>your-region-server-polling-interval-in-milliseconds</value>
      </property>
      
    3. To change the Master Server port, add the pepperdata.agent.hbase.jmx.master.port property.

      <!-- For an HBase Master Server port other than the default Hadoop port (16010) -->
      <property>
        <name>pepperdata.agent.hbase.jmx.master.port</name>
        <value>your-master-server-portnum</value>
      </property>
      
    4. To change the Master Server polling interval, add the pepperdata.agent.hbase.web.mastermetrics.monitor.interval property.

      <!-- For an HBase Master Server polling interval other than the default (60000 milliseconds) -->
      <property>
        <name>pepperdata.agent.hbase.web.mastermetrics.monitor.interval</name>
        <value>your-master-server-polling-interval-in-milliseconds</value>
      </property>
      
    Malformed XML files can cause operational errors that can be difficult to debug. To prevent such errors, we recommend that you use a linter, such as xmllint, after you edit any .xml configuration file.
  3. Save your changes and close the file.

  4. Repeat steps 1–3 on every host in the cluster.

  5. Restart the HBase RegionServer daemons on every host in the cluster.

Task 3: (Different Kerberization for HBase Web UI and Other Cluster Services) Override the Pepperdata-Default Authentication Type for HBase

Although the HBase Web UI and the remaining cluster services typically use the same authentication mechanism—and therefore by default, Pepperdata assumes that HBase Web UI is Kerberized or not according to whether the YARN services are Kerberized—they can be different. That is, a cluster can be Kerberized for other services, but not for the HBase Web UI, or vice versa (Kerberized HBase Web UI but nothing else Kerberized in the cluster).

Procedure

  1. Determine/verify the authentication type for the HBase endpoint by running the following cURL command, where {your-protocol} is http or https:

    curl --tlsv1.2 -kIs {your-protocol}://YOUR_HBASE_HOST:YOUR_PORT/jmx | grep WWW-Authenticate

    • If the returned response is WWW-Authenticate: Negotiate, the authentication type (your-hbase-auth-type) is kerberos.
    • Otherwise nothing is returned, and the authentication type (your-hbase-auth-type) is simple.
  2. Beginning with any host, override its authentication settings for the HBase Web UI by assigning the authentication type you found in step 1 (your-base-auth-type) to the pepperdata.agent.hbase.http.authentication.type property.

    1. Open the host’s Pepperdata site file, pepperdata-site.xml, for editing.

      By default, the Pepperdata site file, pepperdata-site.xml, is located in /etc/pepperdata. If you customized the location, the file is specified by the PD_CONF_DIR environment variable. See Change the Location of pepperdata-site.xml for details.

    2. Add the following snippet.

      Be sure to substitute the authentication type for the HBase Web UI (simple or kerberos, as indicated by the cURL command’s output in step 1) for the your-hbase-auth-type placeholder.

      <!-- Override for PepAgent authentication for HBase Web UI HTTP endpoints -->
      <property>
          <name>pepperdata.agent.hbase.http.authentication.type</name>
          <value>your-hbase-auth-type</value>
      </property>
      
      Malformed XML files can cause operational errors that can be difficult to debug. To prevent such errors, we recommend that you use a linter, such as xmllint, after you edit any .xml configuration file.
    3. Save your changes and close the file.

  3. Repeat step 2 on every host in the cluster.

  4. On every host in the cluster, restart the PepAgent.

    You can use either the service (if provided by your OS) or systemctl command:

    • sudo service pepagentd restart
    • sudo systemctl restart pepagentd

Task 4: (Different HTTP Service Policy for HBase YARN) Override the Pepperdata-Default HTTP Service Policy for HBase

Although HBase and YARN typically use the same HTTP service policy—and therefore by default, Pepperdata assumes that the service policy for HBase is HTTPS_ONLY or HTTP_ONLY according to the policy for YARN—they can be different. That is, HBase might be configured for HTTPS_ONLY when the HTTP service policy for YARN is HTTP_ONLY, or vice versa (HTTP_ONLY for HBase, but HTTPS_ONLY for YARN).

Procedure

  1. Beginning with any host, override its HTTP service policy for HBase.

    1. Open the host’s Pepperdata site file, pepperdata-site.xml, for editing.

      By default, the Pepperdata site file, pepperdata-site.xml, is located in /etc/pepperdata. If you customized the location, the file is specified by the PD_CONF_DIR environment variable. See Change the Location of pepperdata-site.xml for details.

    2. Add the following snippet.

      Be sure to substitute the HTTP service policy for HBase (HTTPS_ONLY or HTTP_ONLY) for the your-hbase-http-service-policy placeholder.

      <!-- Override the default HTTP service policy for HBase -->
      <property>
        <name>pepperdata.agent.hbase.http.policy</name>
        <value>your-hbase-http-service-policy</value>
      </property>
      
      Malformed XML files can cause operational errors that can be difficult to debug. To prevent such errors, we recommend that you use a linter, such as xmllint, after you edit any .xml configuration file.
    3. Save your changes and close the file.

  2. Repeat step 1 on every host in the cluster.

  3. On every host in the cluster, restart the PepAgent.

    You can use either the service (if provided by your OS) or systemctl command:

    • sudo service pepagentd restart
    • sudo systemctl restart pepagentd