Understanding ElastiStor and ElastiCenter Logs

This document describes the various log files generated by ElastiStor and ElastiCenter components and the information that you can derive from each log file.

ElastiCenter

Type/Module Log Name Location Description
Installation or Upgrade bsdinstall_log /tmp Status of
  • Packages installed
  • mySQL configuration
  • System configuration
nohup_devman_instalnohup_devman_installl /tmp Status of installing
  • Tomcat
  • Configuring keys
  • Configuring database
nohup_rabbitmq_mgmt_enable /tmp

Status of

  • RabbitMQ plug-in
nohup_rabbitmq_start /tmp

Status of

  • Starting RabbitMQ
Startup or Boot messages /var/log

OS startup information

  • Service initialization
startup.log /var/log

Status of

  • Services
dmesg.log /var/log

Boot log about

  • Hardware discovered
pci.info, message, if, dmesg.boot /var/log/ifdbug/debugSt<n> For every boot of the Node, contains the contents of /var/log/messages and the output of dmesg, pci, ifconfig on startup.

<n> represents the number of times the Node rebooted.
System events devd.log /var/log Information of
  • Create/Lost devices
auth.log /var/log Authentication logs
maillog /var/log Status of
  • Sending Emails from the Node
Server (Tomcat) catalina.out /var/log/elasticenter
  • Tomcat initialization
  • Exceptions
cbdevman.log /var/log/elasticenter
  • Processing of API requests
  • Monitoring data collection
  • Alerts
  • Reports
api.log /var/log/elasticenter API request and response
access_log.2014-01-22.txt /var/log/elasticenter HTTP logs:
  • URLs accessed
  • Response code
  • Response time
clusterhealthcheck.log /var/log/elasticenter HA Nodes monitoring
Configuration Datastore (MySQL) <hostname>.err /tpool/db/mysql/ Status of
  • Starting mySQL, queries
Communication Channel (RabbitMQ) rabbit@<hostname>.log /var/log/rabbitmq Status of
  • Different RabbitMQ queues
rabbit@<hostname>-sasl.log /var/log/rabbitmq Status of
  • Rabbit-MQ Server
High Availability (ElastiCenterHA) cbc_ec_healthcheck.log /var/log/cbc Monitors
  • Primary ElastiCenter
  • Secondary ElastiCenter
cbc_ec_primary.log /var/log/cbc
  • Checks the status of secondary ElastiCenter
  • Updates the database
cbc_ec_secondary.log /var/log/cbc
  • Tracks the status of primary ElastiCenter
  • Decides whether to takeover from primary ElastiCenter
cbc_ec_changemode.log /var/log/cbc Status of the primary ElastiCenter when it comes back online
slave.log /tmp Maintains details of the takeover process. For example, time to complete the process.

ElastiStor

Type/Module Log Name Location Description
Installation and Upgrade bsdinstall_log /tmp Status of
  • Packages installed
  • mySQL configuration
  • System configuration
  • VSM template creation
bsdinstall_log /tpool/mytenants/tenantbase/tmp/ Status of
  • Packages installed within VSM template
Startup or Boot messages /var/log
  • OS startup information
  • Service initialization
startmain.log /var/log/cbc Status of
  • Starting the required Node services
starttsm_<tsm uuid>.log /var/log/cbc Status of
  • Starting the VSM
stoptsm_<tsm uuid>.log /var/log/cbc Status of
  • Shutting down VSM during Giveback or Takeover
startup.log /var/log Status of
  • Services
  • Importing of Tenant Pool
dmesg.log /var/log Boot log about
  • Hardware discovered
pci.info, message, if, dmesg.boot /var/log/ifdbug/debugSt<n> For every boot of the Node, the file contains the contents of /var/log/messages and the output of dmesg, pci, ifconfig on startup.

<n> represents the number of times the Node rebooted.
nohup_cbc_hostdiscovery_start /tmp Status of
  • Node discovery agent
System Events devd.log /var/log Information of
  • Create/Lost devices
auth.log /var/log

Authentication logs

maillog /var/log Status of
  • Sending Emails from the Node
rclog /tmp Log of
  • Service startup attempts
Configuration cbc_storageconfiguration.log /var/log/cbc Logs related to
  • Storage configurations. For example, zpool ,VSM ,Storage Volumes creation, deletion and modifications
cbc_hostdetails.log /var/log/cbc Logs related to
  • The details of hardware configuration on the Node. For example, number and type of disks, network interface, FC target and initiators of Node
cbc_hostdiscovery.log /var/log/cbc Details of
  • Events while discovering the Node on ElastiCenter
jail_<jailid>_console.log /var/log VSM start/stop console logs
tenantlog /tmp Additional information of VSM creation
High Availability cbc_haconfiguration.log /var/log/cbc Logs related to Node State Change in ElastiCenter.
For example,
  • Logs related to services stop and Pool export while moving Node to Maintenance mode
  • Logs related to services start and Pool import while moving Node to Available mode
cbc_haservice.log /var/log/cbc Logs related to Peer Node Change State on ElastiCenter. For example,
  • Logs related to services stop and Pool export while Peer Node is moving back to available mode
  • Logs related to services start and Pool import while moving Peer Node to maintenance mode
cbc_helloservice.log /var/log/cbc Status of
  • Peer Node. For example, Peer node is reachable
cbc_ha.log /var/log/cbc Logs related to
  • devd events
  • All the events while takeover of Peer Node when it is down. For example, Peer Node power off, Peer Node down when storage lost on it
cbc_scsi_reservation.log /var/log/cbc Details of
  • SCSI reservation on disks of Pool imported on itself
cbc_haevents.log /var/log/cbc Events handled by
  • The ElastiCenter agent
carpmgmt.log /var/log/cbc Logs related to
  • CARP events
cbc_monitor_hajobs.log /var/log/cbc Logs related to HA activities. For example,
  • Moving Node to maintenance mode
  • Moving Node to available mode
cbc_touch_pool.log /var/log/cbc Logs related to
  • Touch events on Pools on the Node
hacarplog /tmp Logs related to
  • CARP down events
scsi.log /var/log/cbc Logs related to
  • Clearing SCSI reservation, registering and reserving the disks of Pool on the Node
fcmgmt.log /var/log/cbc Logs related to
  • Activation and deactivation of FC LUNs
Monitoring cbc_monitor_alerts.log /var/log/cbc Logs related to
  • Disk array enclosures
  • Power supply alerts
  • IPMI alerts
  • Temperature alerts
cbc_monitor_system.log /var/log/cbc Logs related to
  • Memory and CPU usage
  • Disk space usage
cbc_monitor_dataset_qos.log /var/log/cbc Information related to
QoS of Storage Volumes
cbc_monitor_dataset_storage.log /var/log/cbc Information related to space usage of Storage Volumes
cbc_monitor_tenant_services.log /var/log/cbc Logs related to Tenant QoS and services
cbc_monitor_pool_storage.log /var/log/cbc Logs related to Space usage of Pool
cbc_poolCap.log /var/log/cbc  
cbc_thinprov.log /var/log/cbc  
cbc_alerts.log /var/log/cbc Information related to Node alerts
cbc_monitor_disks.log /var/log/cbc Information related to bad disks detection service
Data Protection cbc_dataprotection.log /var/log/cbc Logs related to
  • Snapshots or Cloning
cbdp.log jexec <JID> tcsh

/var/log
Logs related to
  • Verifying the Backup / Migration / DR Transfer Job status. This Log saves the History of cbdpd instances
cbdpd.log jexec <JID> tcsh

/var/log
Logs related to
  • Verifying the Backup / Migration / DR Transfer Job status. Current state of the cpdpd instance
bkpd.log /var/log/ (On the receiving Node) Log contains
  • Transfer status from Source to Destination
CIFS smb.conf jexec <id> tcsh

/usr/local/etc/smb.conf
The smb.conf file is a configuration file for the Samba suite. It contains runtime configuration information of the Samba programs.

The complete description of the file format and possible parameters held within are here for reference purposes.
log.smbd jexec <id> tcsh

/var/log/samba/log.smbd
The smbd server daemon provides file sharing and printing services to Windows clients. In addition, it is responsible for user authentication, resource locking, and data sharing through the SMB protocol.
log.nmbd jexec <id> tcsh

/var/log/samba/log.nmbd
The nmbd server daemon understands and replies to NetBIOS name service requests such as those produced by SMB/CIFS in Windows based systems.
messages jexec <id> tcsh

/var/log/messages
Information of creation of CIFS share, mapping and accessing.
/home jexec <id> tcsh

/home
Contains users which are created for access of CIFS share.

This information is also stored in /etc/passwd log
iSCSI auth.conf /usr/local/etc/istgt/ (insde jail) Verify the authentication configuration
istgt.conf /usr/local/etc/istgt/ (insde jail) Verify the Manage ISCSI Volume configuration. For example, initiator group adnd portal group
messages /var/log Verify operations performed on ISCSI
istgtcontrol info inside jail <jexec <jail id> tcsh> Verify LUN connections
istgtcontrol list inside jal <jexec <jail id> tcsh> List all the SCSI LUNs present in VSM
istgt.core ls (inside jail) If istgt service is stopped, the core is dumped
Apache httpd-access.log, httpd-error.log, httpd-ssl_request.log    

Collect and download Logs

  1. In ElastiCenter, click Admin (at the top-right corner of the page).
  2. Click Troubleshooting from the drop-down list.
  3. log_info_troubleshooting

  4. In the Node for which you want to download the log files, click Download config.xml files icon (highlighted in the following screen).
  5. log_info_downloadxml

  6. Click System logs icon (highlighted in the following screen).log_info_download_systemlog