VMware vCloud Director and Cassandra DB

In the last week I had a customer appointment where we made an update of the VMware vCloud Director installation to the vCD Version 9.1. For this update we had to make a couple of steps and also a migration of the used MS SQL Database. Everything went smooth to the new versions and then des customer decided that he wanted to use the VM Performance monitoring for the vCD Customers.
The installation and configuration is documented in the VMware vCD Docs (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-55D8360A-B4F9-4CDC-8BDA-972C4F387D74.html). So, we started with the installation of the Cassandra cluster and the configuration of the vCD. After everything was configured the customers chose the option to not provide all performance metrics to the customers. Therefore you can limit the metric collection (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-983C566A-7DE5-4F06-9044-C979F8AE7C99.html). We created the /tmp/metrics.groovy file with the required input:


configuration {
metric("cpu.usage.average")
metric("cpu.usagemhz.average")
metric("cpu.usage.maximum")
metric("disk.used.latest") {
currentInterval=300
historicInterval=300
entity="VM"
instance=""
minReportingInterval=1800
aggregator="AVERAGE"
}
}

And after that we imported everything and everything looked good.
After a last restart of a cell we ran into an error on the cell:


2018-08-23 15:00:05,027 | INFO | Cell Application | AbstractCellApplication | Application startup event: Application startup complete. |
2018-08-23 15:00:05,027 | DEBUG | Cell Application | LifecycleStateManager | Transition complete: com.vmware.vcloud.common.main.LegacyCellApplication@7bac686b transition from STOPPED to STARTED via STARTING_FROM_STOPPED |
2018-08-23 15:00:05,132 | INFO | OSGI Delegator - listener dispatcher | BootstrapApplication | Cell startup completed in 1m 35s |
2018-08-23 15:00:05,381 | DEBUG | Spring Context: com.vmware.vcloud.ui-vcloud-webapp | OsgiBundleSpringContextFactory | Finished constructing ApplicationContext for Bundle: com.vmware.vcloud.ui-vcloud-webapp in 3 seconds |
2018-08-23 15:00:58,666 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 810fe296-828e-4ead-8354-e9209f1ef044, Broker URI: tcp://172.16.6.105:61616 |
2018-08-23 15:00:58,802 | DEBUG | .105:61616@48618, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#16 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:53,538 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.105:36820@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:58,672 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 6bc3326d-df7b-4f37-8d59-cd76f7394d47, Broker URI: tcp://172.16.6.106:61616 |
2018-08-23 15:01:58,704 | DEBUG | .106:61616@34410, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#18 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:02:01,647 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.106:59120@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 17:53:02,838 | WARN | processor-NetworkingWeb | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.dequeueElement(DefaultActivityQueueProcessor.java:313)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.access$100(DefaultActivityQueueProcessor.java:54)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor$1.run(DefaultActivityQueueProcessor.java:192)
2018-08-23 17:53:02,839 | WARN | processor-ValFabric | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
[.......]

The Cell wasn’t come up and also after a restart of the other Cell we hade the same issue on every Cell which was restarted. After a log of searching I figured out that the created configuration for the performance metrics produced this error. Problems / errors in the vCD Database stopped the start of the vCD Services.
After some search I found a Blog Post on the VMware Blogs (https://blogs.vmware.com/vcat/2017/11/virtual-machine-performance-metrics-vmware-vcloud-director-9-0.html) which didn’t indicate this problem but the solution from there also worked in my case.
The relevant Information’s are these lines:
2. Edit the content of the /tmp/metrics.groovy file to:

configuration {
}

3. Run the following command:

# cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy

After that I was able to restart my cells and everything went smooth again.