Promoting secondary ElastiCenter and backing up configurations

The following sections explain how to set up a secondary ElastiCenter (secondary setup) to ensure backup in the case your primary ElastiCenter (primary setup) fails. If a failure occurs, you can promote your secondary setup.

As shown in diagram 1, the primary ElastiCenter manages and monitors multiple Storage Nodes spread across datacenters. All configurations on primary ElastiCenter are replicated on the secondary. In case the primary ElastiCenter fails, the secondary can be promoted as shown in diagram 2.

Diagram 1:

ElastiCenter-01

Diagram 2:

ElastiCenter_1-02

Important:The primary and secondary setups cannot be used at the same time. Ensure that only the database service is operational on the secondary setup. Promote the secondary setup only when the primary setup is down.

Prerequisites

  • The primary and secondary setups are on same subnet
  • ElastiCenter and Node are installed separately

Setting up the secondary ElastiCenter

The procedures involve action at both the primary and secondary setups.

Note: If you copy and paste the commands in the section, ensure that they appear correctly on the CLI.

    1. (On both the setups) Log in as root and access (over SSH or directly) the command prompt on the machine where the ElastiCenters are installed.
    2. (On the secondary setup) Run the following command:
      service rabbitmq onestop
    3. (On the secondary setup) Run the following command to ensure that the RabbitMQ service is stopped:
      service rabbitmq onestatus
    4. (On secondary setup) Run the following command to ensure that no Java process is running:
	ps –auxwww | grep java
    1. If any of the Java processes or RabbitMQ service is running, kill it using the following command:
      kill -9 PID_of_process
    2. Else, ensure the following:
      • Size of
        /tpool
        folder is same on both primary ElastiCenter and secondary ElastiCenter using the following command:
        zpool list
      • /
        folder in secondary ElastiCenter database has enough space for backup. This means the available space in / directory on the secondary ElastiCenter must be greater than the used space in /tpool directory on the primary ElastiCenter.
    3. (Optional) On the secondary setup, start the mysql service using the command:
      service mysql -server onestart
      .
    4. (On the secondary setup) Run the script
      /usr/local/cb/bin/cb_config_slave.sh
      .
    5. Perform the steps as shown in the following command line :
       
Select your choice:
1) Configure secondary ElastiCenter.
2) Promote secondary to primary ElastiCenter.
3) Exit 
1
Here, you have to select 1 as highlighted.
You have selected to configure secondary ElastiCenter.
Do you want to continue? (Yes/No) Yes
Here, you have to select Yes as highlighted.
			
Specify primary ElastiCenter's IP address:
20.10.97.1

Wait for the script to execute.
Stopping Tomcat and RabbitMQ services...
Tomcat and RabbitMQ services stopped.
Restarting MySql service...
MySql service started.
Starting backup of primary ElastiCenter...
Primary Elasticenter backup completed. 
Starting the setting up of secondary ElastiCenter...
Setting up the secondary ElastiCenter completed. 
Syncing reports...

Promoting the secondary setup

Before you proceed

If the primary setup is down, try the following before promoting the secondary setup:

  • Verify the error logs in
    java/Rabbitmq/Mysql
    or the service status. For example,
    service mysql-server onestatus
    .
  • Restart the service that is not running.

If downtime prevails despite the service restart, promote the secondary setup.

Procedures

Note: If you copy and paste the commands in the section, ensure that they appear correctly on the CLI.

    1. Power off the primary ElastiCenter.
    2. (On the secondary ElastiCenter console) Run the script
       /usr/local/cb/bin/cb_config_slave.sh 
      .
    3. Perform the steps as shown in the following command line :
Select your choice:
1) Configure secondary ElastiCenter.
2) Promote secondary to primary Elasticenter.
3) Exit 
2
Here, you have to select 2 as highlighted.
You have selected to promote secondary Elasticenter to primary Elasticenter.
Do you want to continue ? (Yes/No)Yes
Here, you have to select Yes as highlighted.
            
Dropped secondary ElastiCenter.
                    
Starting RabbitMQ and Tomcat services..
Promoted secondary Elasticenter to primary Elasticenter. Check catalina logs for errors(if any).
  1. (On all the ElastiStor consoles) Modify the
    /usr/local/agent/listener/cbc.conf
    file by changing the IP address of ElastiCenter. Specify the IP address of secondary ElastiCenter.
  2. (On the ElastiStor console) Run the following command to restart all the
    python/cbc
    services on the Node:
    service -l | grep cbc | grep -v cbc_ec | xargs -I {} service {} onerestart

Also, ensure that the ElastiCenter is accessible to the Nodes with same status as prior to the promotion.

Note: During the promotion process, if a Node is offline or is powered down, then manually start the

cbc_monitor_hajobs

service and point it to the promoted ElastiCenter.

Backing up configurations

In Standalone environments

How to backup and restore ElastiCenter configuration

By default, CloudByte schedules a backup everyday. You can modify the settings using the following options in Global Settings:

  • devman.backup.files.retain
  • devman.backup.path
  • devman.backup.scheduling.interval.

But, in the case you need to do a manual backup, follow the steps:

Backing up
  1. In the ElastiCenter server, at command prompt, run the following:
    cd /usr/local/cb/bin/
  2. Run the following command to create backup file:
    ./backup.sh
    . The backup file is created in the /var/devman/backups (unless you modify the path). The following is a sample file name: backup_Jul26_1374815679.tar.bz2.
  3. Copy the file and store it elsewhere.
Restoring
  1. Copy the backed up TAR file to a location on ElastiCenter, for example /var.
  2. Go to the directory /usr/local/cb/bin and then run the following command:
    ./restore.sh <path of backedup file>
    . For example
     ./restore.sh /var/backup_Jul26_1374815679.tar.bz2
    .

The configuration is restored.

Do the following to collect the logs of the datacenter (both Nodes and ElastiCenter):

  1. Log in as root.
  2. In the ElastiCenter server, at command prompt, run the following command:
    cd /usr/local/cb/bin/
    .
  3. Run the following command:
    python logCollector.py
  4. At prompt, specify the root password for all Nodes from which you want to gather the logs.
How to backup and restore Node configuration Backing up

By default, CloudByte schedules a daily backup in the location

/cf/conf

CloudByte recommends you to move this backup daily to a safe location. Either schedule a cron job or manually copy the files from the conf directory.

Restoring
  1. Reinstall the Node with the same configurations (for example, same IP address, Hostname, and ElastiCenter details) as in the previous installation.
  2. Copy the following backed up files to the reinstalled Node:
    • config.xml
      file to the location
      /cf/conf/config.xml
    • haconfig.xml
      file to the location
      /cf/conf/haconfig.xml
    • ipmi.conf
      file to the location
      /usr/local/agent/listener/ipmi.conf
    • cbc_node_id
      file to the location
      /usr/local/agent/cbc_node_id
    • cbd_node_id
      file to the location
      /usr/local/agent/cbd_node_id
  3. After copying all the files, reboot the Node.
How to restore secondary ElastiCenter configuration

Reinstall ElastiCenter and then perform the procedures in the section Setting up the secondary ElastiCenter.

In EC-HA environments

In an EC-HA setup, when the primary Node goes offline, the other Node takes over as the primary. If the secondary Node goes offline, the Pools in the Node are taken over by the primary, however ElastiCenter configurations remain the same. For more details, see HA takeover and giveback scenarios

How to backup primary Node and the associated ElastiCenter configurations
  1. In the ElastiCenter server, at command prompt, run the following: cd /usr/local/cb/bin/
  2. Run the following command to create backup file: ./backup.sh. The backup file is created in the /var/devman/backups (unless you modify the path). The following is a sample file name: backup_Jul26_1374815679.tar.bz2.
  3. Copy the file and store it elsewhere.
  4. Run the following commands to back up the Node configurations:
    1. cp /usr/local/agent/cb* .
    2. cp -r /cf/conf .
How to backup and restore secondary Node and the associated ElastiCenter configurations
Backing up
Run the following commands to back up the Node configurations:
  1. cp /usr/local/agent/cb* .
  2. cp -r /cf/conf .
Restoring

In an EC- HA environment, you can restore only the secondary setup.

  1. Reinstall ElastiCenter HA using the Secondary ElastiCenter option. For details, see Installing ElastiCenter HA
  2. After the Node is online, login in to ElastiCenter.
  3. Check the EC-HA status (Admin panel > ElastiCenter HA).
  4. Ensure that both the Nodes are in sync with each other as shown in the following screen:
    Change-IP-in-appliance_21
  5. Note: In case the Nodes are not in sync (after 30 minutes), contact Technical Support.

  6. Copy the following backed up files to the reinstalled Node:
    • config.xml file to the location /cf/conf/config.xml
    • haconfig.xml file to the location /cf/conf/haconfig.xml
    • ipmi.conf file to the location /usr/local/agent/listener/ipmi.conf
    • cbc_node_id file to the location /usr/local/agent/cbc_node_id
    • cbd_node_id file to the location /usr/local/agent/cbd_node_id
  7. After copying all the files, reboot the Node.