Ambari 2.1.2 Upgrade Guide

Pivotal HD

Pivotal HD

Ambari Upgrade Guide


Chapter 1. Upgrading Ambari

Ambari and the PHD cluster being managed by Ambari can be upgraded independently. This section describes the process to upgrade Ambari. You are strongly encouraged to read completely through this entire document before starting the upgrade process, to that you understand the interdependencies and order of the steps. It is highly recommended you validate these steps in a test environment to adjust + account for any special configurations for your cluster.

Preparing to Upgrade

  • Be sure to review the Release Notes for Known Issues and Behavioral Changes.

  • You must have root, administrative, or root-equivalent authorization on the Ambari server host and all servers in the cluster.

  • You must backup the Ambari Server database.

  • You must make a safe copy of the Ambari Server configuration file found at /etc/ambari-server/conf/ambari.properties.

  • If you are managing Ganglia and Nagios from Ambari:

    • Support for managing Ganglia and Nagios from Ambari has been removed.

    • Record the location of the Nagios server before you begin the upgrade process.

    • Record the location of the Ganglia server before you begin the upgrade process.

    • You must stop the Nagios and Ganglia services from Ambari Web.

    • After upgrading Ambari, you must remove Nagios and Ganglia from your cluster and replace with Ambari Alerts and Metrics. For more information, see Migrate to Ambari Alerts and Metrics in Ambari.

  • If you are on Ambari 1.7 or earlier:

    • If your cluster has Kerberos enabled, you must disable Kerberos prior to performing the upgrade. Be sure to review Adjusting for Kerberos for more information on the additional steps required pre- and post-upgrade.

  • Proceed to Upgrade Ambari.

[Note]Note

During Ambari upgrade, the existing /var/lib/ambari-server/ambari-env.sh file is overwritten and a backup copy of ambari-env.sh (with extension .rpmsave) is created. If you have manually modified ambari-env.sh (for example, to change Ambari Server heap), you will need to re-apply your changes to the new file.

Upgrade Ambari

  1. If you are running Ambari Metrics service in your cluster, stop the service. From Ambari Web, browse to Services > Ambari Metrics and select Stop from the Service Actions menu.

  2. Stop the Ambari Server. On the host running Ambari Server:

    ambari-server stop

  3. Stop all Ambari Agents. On each host in your cluster running an Ambari Agent:

    ambari-agent stop

  4. Use these helper commands to clean up the old Ambari repo files from all of the PHD cluster nodes:

    • For RHEL/CentOS:

      ssh root@{hostname} mkdir ~/old_repo_files
      ssh root@{hostname} \
      "mv /etc/yum.repos.d/ambari.repo  ~/old_repo_files”
    • For SLES:

      Remove the repository on all of the cluster nodes. For example:

      ssh root@hostname zypper rr AMBARI-2.1.2.2
  5. Download the correct Ambari RPM single repository tarball for your operarting system from Pivotal Network.

  6. Ambari-2.1.2 is required for installing the PHD-3.0.1 stack. Set up the local YUM repository using the instructions provided in [2.1 Setup YUM Repository Server] in the Pivotal HD Automated Install with Ambari 2.1.2 guide.

  7. Untar and run setup-repo.sh on the Ambari tarball:

    tar -zxvf  AMBARI-2.1.2.2-163-centos6.tar.gz
    ./AMBARI-2.1.2.2/setup_repo.sh
  8. Copy ambari.repo to ALL of the cluster nodes:

    • For RHEL/CentOS:

      scp /etc/yum.repos.d/ambari.repo \
         $cluster_host:/etc/yum.repos.d/ambari.repo
      
    • For SLES:

      scp /etc/zypp/repos.d/ambari.repo \
         $cluster_host:/etc/zypp/repos.d/ambari.repo
  9. Upgrade Ambari Server. On the host running Ambari Server:

    • For RHEL/CentOS:

      yum clean all
      yum info ambari-server

      In the info output, visually validate that there is an available version containing "2.1.2"

      yum upgrade ambari-server
    • For SLES:

      zypper clean
      zypper info ambari-server

      In the info output, visually validate that there is an available version containing "2.1.2"

      zypper up ambari-server
    [Important]Important

    When performing upgrade on SLES, you will see a message "There is an update candidate for 'ambari-server', but it is from different vendor. Use 'zypper install ambari-server-2.1.2-101.noarch' to install this candidate". You will need to to use yast to update the package, as follows:

    1. From the command line run: > yast.

      > yast

      You will see command line UI for YaST program.

    2. Choose Software > Software Management, then click the Enter button.

    3. In the Search Phrase field, enter ambari-server, then click the Enter button.

    4. On the right side you will see the search result ambari-server 2.1.2. Click Actions, choose Update, then click the Enter button.

    5. Go to Accept, and click enter.

  10. Check for upgrade success by noting progress during the Ambari Server installation process you started in Step 5.

    • As the process runs, the console displays output similar, although not identical, to the following:

      Setting up Upgrade Process
      Resolving Dependencies
      --> Running transaction check
    • If the upgrade fails, the console displays output similar to the following:

      Setting up Upgrade Process
      No Packages marked for Update
    • A successful upgrade displays output similar to the following:

      Updated:
         ambari-server.noarch 0:2.1.2-111
      Complete!
    [Note]Note

    Confirm there is only one ambari-server*.jar file in /usr/lib/ambari-server. If there is more than one JAR file with name ambari-server*.jar, move all JARs except ambari-server-2.1.2.*.jar to /tmp before proceeding with upgrade.

  11. Upgrade all Ambari Agents. On each host in your cluster running an Ambari Agent:

    • For RHEL/CentOS:

      yum upgrade ambari-agent
    • For SLES:

      zypper up ambari-agent
      [Note]Note

      Ignore the warning that begins with "There are some running programs that use files deleted by recent upgrade".

      [Important]Important

      When performing upgrade on SLES, you will see a message "There is an update candidate for 'ambari-agent', but it is from different vendor. Use 'zypper install ambari-agent-2.1.2-101.noarch' to install this candidate". You will need to to use yast to update the package, as follows:

      1. From the command line run: > yast

        > yast

        You will see command line UI for YaST program.

      2. Choose Software > Software Management, then click the Enter button.

      3. In the Search Phrase field, enter ambari-agent, then click the Enter button.

      4. On the right side you will see the search result ambari-agent 2.1.2. Click Actions, choose Update, then click the Enter button.

      5. Go to Accept, and click enter.

  12. After the upgrade process completes, check each host to make sure the new files have been installed:

    rpm -qa | grep ambari-agent
  13. Upgrade Ambari Server database schema. On the host running Ambari Server:

    ambari-server upgrade
  14. Start the Ambari Server. On the host running Ambari Server:

    ambari-server start
  15. Start all Ambari Agents. On each host in your cluster running an Ambari Agent:

    ambari-agent start
  16. Open Ambari Web.

    Point your browser to http://<your.ambari.server>:8080

    where <your.ambari.server> is the name of your ambari server host. For example, c6401.ambari.apache.org.

    [Important]Important

    Refresh your browser so that it loads the new version of the Ambari Web code. If you have problems, clear your browser cache manually, then restart Ambari Server.

  17. Log in, using the Ambari administrator credentials that you have set up.

    For example, the default name/password is admin/admin.

  18. You will see a Restart indicator next to each service after upgrading. Ambari upgrade has added to/adjusted the configuration properties of your cluster based on new configuration types and properties being made available for each service with this release of Ambari. Review these changes by comparing the previous configuration with the latest version created by "ambari-upgrade".

  19. Review the PHD-UTILS repository Base URL setting in Ambari:

    • Browse to Ambari Web > Admin > Stack and Versions.

    • Click on the Versions tab.

    • You will see the current installed PHD Stack version displayed.

    • Click the Edit repositories icon in the upper-right of the version display and confirm the value of the PHD-UTILS repository Base URL is correct for your environment.

    • If you are using a local repository for PHD-UTILS, be sure to confirm the Base URL is correct for your locally hosted PHD-UTILS repository.

  20. If you have configured Ambari to authenticate against an external LDAP or Active Directory, you must re-run "ambari-server setup-ldap”. For more information, see Set Up LDAP or Active Directory Authentication.

  21. If you have configured your cluster for Hive or Oozie with an external database (Oracle, MySQL or PostgreSQL), you must re-run “ambari-server setup --jdbc-db and --jdbc-driver” to get the JDBC driver JAR file in place. For more information, see Using Non-Default Databases - Hive and Using Non-Default Databases - Oozie.

  22. If your cluster includes Ganglia and Nagios:

    You must remove Nagios and Ganglia from your cluster and replace with Ambari Alerts and Metrics. For more information, see Migrate to Ambari Alerts and Metrics.

  23. If you have upgraded from Ambari 1.7:

    • If your cluster was configured for Kerberos, you must adjust your cluster for Kerberos. For more information, see Adjusting for Kerberos.

    • Y must get the cluster hosts to advertise the "current version". This is done by restarting a master or slave component (such as a DataNode ) on each host to have the host. For example, in Ambari Web, navigate to the Hosts page, select any Host that has the DataNode component and restart that DataNode component on that single host.

Post-Upgrade Tasks

Depending on the configuration of your cluster and the Ambari version you are starting with, review the following post-upgrade tasks.

Task

Description

Migrate to Ambari Alerts and Metrics

If your cluster includes legacy Nagios and Ganglia services, they must be removed and replaced with Ambari Alerts and Metrics.

Adjusting for Kerberos

If you have upgraded from Ambari 1.7 or earlier and your cluster was configured for Kerberos, you must adjust Ambari.

Migrate to Ambari Alerts and Metrics

Starting with Ambari 2.0, Ambari includes built-in systems for alerting and metrics collection. Therefore, when upgrading to Ambari 2.1, the legacy Nagios and Ganglia services must be removed and replaced with the new systems. You only need to perform these steps and migrate to Ambari Alerts and Metrics if you are running Nagios and Ganglia.

[Important]Important

We highly recommended that you perform and validate this procedure in a test environment prior to attempting the Ambari upgrade on production systems.

Moving from Nagios to Ambari Alerts

After upgrading to Ambari 2.1, the Nagios service will be removed from the cluster. The Nagios server and packages will remain on the existing installed host but Nagios itself is removed from Ambari management.

[Important]Important

Nagios used the operating system sendmail utility to dispatch email alerts on changes. With Ambari Alerts, the email dispatch is handled from the Ambari Server via Javamail. Therefore, you must provide SMTP information to Ambari for sending email alerts. Have this information ready. You will use it after the Ambari 2.1 upgrade to get Ambari Alert email notifications configured in the new Ambari Alerts system.

The Ambari Alerts system is configured automatically to replace Nagios but you must:

  1. Configure email notifications in Ambari to handle dispatch of alerts. Browse to Ambari Web > Alerts.

  2. In the Actions menu, select Manage Notifications.

  3. Click to Create a new Notification. Enter information about the SMTP host, port to and from email addresses and select the Alerts to receive notifications.

  4. Click Save.

[Note]Note

(Optional) Remove the Nagios packages (nagios, nagios-www) from the Nagios host.

For more information Ambari Alerts, see Managing Alerts in the Ambari User’s Guide.

Moving from Ganglia to Ambari Metrics

After upgrading to Ambari 2.1, the Ganglia service stays intact in the cluster. You must perform the following steps to remove Ganglia from the cluster and to move to the new Ambari Metrics system.

  1. Stop Ganglia service via Ambari Web.

  2. Using the Ambari REST API, remove the Ganglia service by executing the following:

    curl -u admin.user:admin.password -H 'X-Requested-By:ambari' -X DELETE 'http://ambari.server:8080/api/v1/clusters/cluster.name/services/GANGLIA'

    where admin.user and admin.password are credentials for an Ambari Administrator, ambari.server is the Ambari Server host and cluster.name is the name of your cluster.

  3. Refresh Ambari Web and make sure that Ganglia service is no longer visible.

  4. In the Actions menu on the left beneath the list of Services, use the "Add Service" wizard to add Ambari Metrics to the cluster.

  5. This will install an Ambari Metrics Collector into the cluster, and an Ambari Metrics Monitor on each host.

  6. Pay careful attention to following service configuration for Ambari Metrics.

    By default, Ambari Metrics uses the local filesystem as the default storage backend. This is embedded mode and the rootdir for HBase is set to a local filesystem path by default when using Ambari Metrics in embedded mode. If you want to run Ambari Metrics in distributed mode (i.e. use HDFS instead of local filesystem for storage), set this rootdir value to an HDFS dir. For example: hdfs://namenode.example.org:8020/apps/ams/metrics. See Tuning Ambari Metrics for more information.

    Section

    Property

    Default Value

    Advanced ams-hbase-site

    hbase.rootdir

    file:///var/lib/ambari-metrics-collector/hbase

  7. For the cluster services to start sending metrics to Ambari Metrics, restart all services. For example, restart HDFS, YARN, and HBase.

[Note]Note

(Optional) Remove the Ganglia packages (ganglia-gmetad and ganglia-gmond) from the hosts.

Adjusting for Kerberos

[Important]Important

You only need to perform these Kerberos steps if you are running Ambari 1.7 or earlier and your cluster is Kerberos enabled. You do not need need to perform these steps if you are running Ambari 2.0 or later.

If you are upgrading to Ambari 2.1 from an Ambari-managed cluster that is already Kerberos-enabled, because of new Kerberos features introduced in Ambari, you need perform the following steps:

  1. Review the procedure for Configuring Ambari and Hadoop for Kerberos in the Ambari Security Guide.

  2. Have your Kerberos environment information readily available, including your KDC Admin account credentials.

  3. Take note of current Kerberos security settings for your cluster.

    1. Browse to Services > HDFS > Configs.

    2. Record the core-site auth-to-local property value.

  4. Disable Kerberos before performing the Ambari upgrade. In Ambari Web, browse to Admin > Kerberos, and click Disable Kerberos. You can re-enable Kerberos after performing the Ambari upgrade.

  5. Upgrade Ambari according to the steps in Upgrading to Ambari 2.1.

  6. Ensure your cluster and the Services are healthy.

  7. Browse to Admin > Kerberos and you’ll notice Ambari thinks that Kerberos is not enabled. Run the Enable Kerberos Wizard, following the instructions in the Ambari Security Guide. Be sure to pay close attention to the principal names in the Configure Identities step in the wizard to confirm principals names match what you expect for your environment.

  8. Ensure your cluster and the Services are healthy.

  9. Verify the Kerberos security settings for your cluster are correct.

    • Browse to Services > HDFS > Configs.

    • Check the core-site auth-to-local property value.

    • Adjust as necessary, based on the pre-upgrade value recorded in Step 3.