0

Nutanix Calm – a deeper look from an automation consultant perspective (Part 2)

This post will be the second post on Nutanix Calm and will build up on my first post which can be found here: http://www.vcoportal.de/2018/09/nutanix-calm-a-deeper-look-from-an-automation-consultant-perspective-part-1/ if you missed it.

 

In the last post we only created a virtual machine without any customization. In this post we will install additional packages and add an SSH Key for login to the VM.

Let’s go back to our previously created Blueprint. On the left site we have the tab “Package”. Choose this, give the Package a Name and use “Configure install”

 

What you will see is that beside the LinuxServer Icon a new Icon will open up. Here we choose “+ Task” to create some modifications during the VM build process.

 

On the left site you must configure some values. You provide a name, choose what you want to do (I want to execute a Shell Script), and take the Credentials you need to access the VM. In the script section you past your script (Linux Bash programming in my case) which you want to execute.

My script for example looks like this:


#!/bin/bash

cat <<'EOF' > /etc/resolv.conf

# Generated by NetworkManager

search vcoportal.de

nameserver 192.168.1.10

nameserver 192.168.1.20

EOF

yum -y update

yum -y install net-tools traceroute bind-utils

First, I declare my script as bash script (line 1). Then I change the DNS Server configuration (line 2 to 7) and update everything on my VM and then install some tools like dig, traceroute and so on for my virtual machine.

When you are finished, save the Blueprint and give it a try for a run.

In the Audit Section of the Application you can take a look at your installation and configuration. As you will see the new generated section InstallSoftware will also be there as point. You can see what the Output of the section is. When you click on the Small window in the market red rectangle you can pop up a bigger window with the output.

On the button you can also have a look at the created script.

In the next step we will insert an SSH Key. This key can be insert from the user wo execute the Blueprint. So, let’s go back to our Blueprint. Here we go the the “Service” Section from our Service. There we insert a Name for the Variable (in my case SSHKey) and a default value. It is important that you activate the “Runtime” checkbox. We want the SSHKey auf our user, not the default and so the user can insert “his” key.

 

When we are finished with the variable we add a “Task”. This task we use to insert the SSH Key into the authorized_keys field for a specific account (I am lacy so I use the root account. For security reason I would not recommend this in production….)

 

The script itself is also a Linux Bash script and easy.


#!/bin/bash

mkdir /root/.ssh

touch /root/.ssh/authorized_keys

echo "@@{SSHKey}@@" >> /root/.ssh/authorized_keys


 

In the first line we declare the script as a Linux Bash script. The second line create the necessary directory and the third line creates the file. In the last line we insert something in the create file……

Did you recommend the “@@{SSHKey}@@ in the last line? What’s that for a strange value?
In Calm variables which we use in script are market with “@@{VALUE}@@” so in our case that means we take the value of the before created Variable and replace the “@@{SSHKey}@@” field with the content.

One note why I used a new task and not placed the script in the first task we already did. From my point of view, I would recommend to create a new task for every goal which I want to reach. I personally thing that debugging is much easier in this case because we can see what went wrong (in which script). The next thing for me is, that even on script maybe doesn’t function like expected we still have the chance to access the VM and fix it during life time.

After we are done with our script, save the Blueprint and Launch the VM. When we are on the Blueprint just open the “Configuration” part and insert the SSH Key which you want to access the VM without password.

 

When everything works like expected you can login in your VM via SSH. If not, take a look in the Audit section.

As you can see the start with calm is very easy and there are a lot of things we can be done with calm.

During my “play around” with Calm I found some things which must be improved within Calm or limit Calm at the moment:

  • Calm is limited to Nutanix, vSphere-Cluster and Cloud Solutions. So, some customers still have to use a second solution for their automation need.
  • You cannot export created Blueprint and import them with all needed scripts, variables and images. With our easy Blueprint that is not a big issue but when you create a Blueprint with plenty of variables there is a “room” for mistakes and typos.
  • At the moment Calm has some limitations to work with for example ServiceNow and so on.

Some Ideas for Calm for future functions / extension:

  1. RFE: Blueprint Full Backup with images, scripts, cloudinit etc. à Provide a central repository in PC to make things re-usable: cloudinit, sysprep, scripts, images etc.
  1. RFE: DNS-Update Feature when VXLAN-DHCP from PE assigns IP-address à Autofill connection URL after application deployment (aka openshift) in the app description
  2. RFE: Integration of 3rd-Party config-mgmt tools like git, puppet, chef etc. in the GUI (dropdown menu, not via scripts) à Make existing scripts / tools / 3rd-party development re-useable
  3. RFE: Must-have: Native container integration to push CALM! No customer with an existing automation or configuration solution will switch to CALM without that feature.
  4. RFE: VM customization options with cloudinit for linux and sysprep for windows delivered as one-click/dropdown options in CALM (forgot that in the call)

 

If you can life with this limitations Calm is a good choice to start automation in your Nutanix environment.

0

Nutanix Calm – a deeper look from an automation consultant perspective (Part 1)

The last couple of days I played somewhat around with Nutanix Calm. Calm is the automation Solution from Nutanix based on the Acropolis. The official Calm statement from Nutanix can be found here:

What is Nutanix Calm?

Calm provides application automation and lifecycle management for the Nutanix and public clouds, as part of the Nutanix Enterprise Cloud Platform. Calm orchestrates the provisioning, scaling and management of applications across multiple environments to make the entire IT infrastructure more agile and application centric.

From Organization perspective calm works with Blueprint to organize the layers of the automation.

Also, here an official description:

Nutanix Calm models applications as Blueprints giving you a full end-to-end view of the application, its dependencies, and its requirements. From here you can quickly spin up multiple copies of an application and ensure that each one is done right each time.

Operations can be orchestrated, delegated, and repeated knowing each time that it’s done perfectly no matter who runs it. At the same you gain an impressive amount of visibility into who did what, when, and what the result was because Calm logs all actions and all logs from each action no matter where the application is deployed.

Blueprints can be published to end users based on their group or role, giving them the ability to instantly provision and use applications across clouds while ensuring policy is properly applied and logged. You can attach budgets and quotas to each team to track their unique spend or customize the cost of a particular application to control user behavior based on what applications you want them using.

 

A lot of customers are asking about Nutanix Acropolis and also for an automation solution. So, I decided to play a little bit around with calm to see how easy it is to archive first results with calm.

Bevor you can start with the development of a first Application you have to understand the different between a Blueprint and an Application.

In the Blueprint you declare the Number of virtual Machines, the “Automation” Part of the VMs and all other necessary things.

The Application is the running configuration of a Blueprint.

So, when you start with calm you have to create a Blueprint bevor you can create an Application. Let’s start with our first Blueprint which will be an easy on. The goal for this Blueprint is to create a running virtual Machine (Linux).

 

A Windows Open were you must provide a name for the Blueprint. A description is optional, but I would recommend this as a best practice.

 

After you are done with your Blueprint setup the Main Window of the Blueprint is opened.

 

Make yourself comfortable with the buttons and screen and then let’s create our first service.

The Services “hold” the virtual machine which offers a service for other virtual machines or user. A service could be a Load Balancer for example or a Webserver. When you create more advanced Blueprint (then this one we are doing now) you can build dependency between different services.

For now we only create one service.

On the left site of the screen we configure the necessary configuration. Create a Service Name, a Name for the Sub (Name is not the VM Name!), choose your Cloud (Nutanix or AWS for example) and your Operating System. I use Linux so that’s my choice.

 

On the lower part we provide a Name for the VM in our environment. Calm automatically adds some variables to provide unique names in our environment. And the Images we want to use as base for our deployment. If necessary you can change the Device Type and Device Bus. Note the “running men” nearby some of the fields. Here we are able to provide the user who execute the Workflow to provide his configurations variables for example a special VM Name. It is a good practice to provide default values for all parameter.

 

 

On the bottom of the field you configure CPU, Memory, additional Disks (VDISK) and the Network. If most network configurations are taken from the Nutanix default configuration so in my case I only need to insert an IP or use DHCP.

 

The last things to configure is the “Connection” Section. Here you must provide credentials to login to the base image. These credentials are used to configure the VM.

After that Save the Blueprint at give it a first try with “Lunch”

Now we must enter an Application Name and then let’s build the VM.

The build process starts automatically. Under the “Audit” log you can see what happens within your Blueprint and if it is successful.

Till now no rocket science and easy to go……

That’s it for now, in the next post we start to customize our VM…..

 

0

VMware vSphere Autodeployment (virtuallyGhetto)

Did I already mention that I am a huge fan of the Autodeploy Script from William Lam? William created a bunch of script which help you to deploy a nested VMware environment in an automated way. The environment itself is built with a VSAN based datastore.

You can find the Blog Post from William here:

vGhetto Automated vSphere Lab Deployment for vSphere 6.0u2 & vSphere 6.5

 

and his GIT repository with the scripts here:

https://github.com/lamw

When you look at the Blog Post from William you can see that he uses a single ESX Hosts for his deployment. In case that you maybe have more ESX host there is a small issue in case that you have a VSAN based cluster. For the deployment of a nested VSAN environment on a VSAN bases Cluster you must set a VSAN Parameter (not recommended for production usage!). William makes these settings only for one host as you can see in this script


if($datastore.Type -eq "vsan") {
My-Logger "VSAN Datastore detected, enabling Fake SCSI Reservations ..."
Get-AdvancedSetting -Entity $vmhost -Name "VSAN.FakeSCSIReservations" | Set-AdvancedSetting -Value 1 -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile
}

 

In my environment a have a four node cluster hosts with HA and DRS configured. In some cases, DRS kicks-in during the enrollment of the nested environment. When a virtual ESX Hosts ends up on a Hosts were the settings was node made, the deployment fails.

In my case I made an improvement (for my usage) to the script where I configure the Fake SCSI settings for all Hosts of a cluster.

 

if($datastore.Type -eq "vsan") {
$FakeVSANhosts = $cluster | Get-VMHost | ForEach-Object {
My-Logger "VSAN Datastore detected, enabling Fake SCSI Reservations for Host $_ ..."
Get-AdvancedSetting -Entity $_ -Name "VSAN.FakeSCSIReservations" | Set-AdvancedSetting -Value 1 -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile
}

 

So, if someone of you runs into the same problem, just change the lines in the script from William and everything should work like expected.

0

VMware vCloud Director and Cassandra DB

In the last week I had a customer appointment where we made an update of the VMware vCloud Director installation to the vCD Version 9.1. For this update we had to make a couple of steps and also a migration of the used MS SQL Database. Everything went smooth to the new versions and then des customer decided that he wanted to use the VM Performance monitoring for the vCD Customers.
The installation and configuration is documented in the VMware vCD Docs (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-55D8360A-B4F9-4CDC-8BDA-972C4F387D74.html). So, we started with the installation of the Cassandra cluster and the configuration of the vCD. After everything was configured the customers chose the option to not provide all performance metrics to the customers. Therefore you can limit the metric collection (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-983C566A-7DE5-4F06-9044-C979F8AE7C99.html). We created the /tmp/metrics.groovy file with the required input:


configuration {
metric("cpu.usage.average")
metric("cpu.usagemhz.average")
metric("cpu.usage.maximum")
metric("disk.used.latest") {
currentInterval=300
historicInterval=300
entity="VM"
instance=""
minReportingInterval=1800
aggregator="AVERAGE"
}
}

And after that we imported everything and everything looked good.
After a last restart of a cell we ran into an error on the cell:


2018-08-23 15:00:05,027 | INFO | Cell Application | AbstractCellApplication | Application startup event: Application startup complete. |
2018-08-23 15:00:05,027 | DEBUG | Cell Application | LifecycleStateManager | Transition complete: com.vmware.vcloud.common.main.LegacyCellApplication@7bac686b transition from STOPPED to STARTED via STARTING_FROM_STOPPED |
2018-08-23 15:00:05,132 | INFO | OSGI Delegator - listener dispatcher | BootstrapApplication | Cell startup completed in 1m 35s |
2018-08-23 15:00:05,381 | DEBUG | Spring Context: com.vmware.vcloud.ui-vcloud-webapp | OsgiBundleSpringContextFactory | Finished constructing ApplicationContext for Bundle: com.vmware.vcloud.ui-vcloud-webapp in 3 seconds |
2018-08-23 15:00:58,666 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 810fe296-828e-4ead-8354-e9209f1ef044, Broker URI: tcp://172.16.6.105:61616 |
2018-08-23 15:00:58,802 | DEBUG | .105:61616@48618, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#16 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:53,538 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.105:36820@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:58,672 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 6bc3326d-df7b-4f37-8d59-cd76f7394d47, Broker URI: tcp://172.16.6.106:61616 |
2018-08-23 15:01:58,704 | DEBUG | .106:61616@34410, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#18 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:02:01,647 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.106:59120@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 17:53:02,838 | WARN | processor-NetworkingWeb | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.dequeueElement(DefaultActivityQueueProcessor.java:313)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.access$100(DefaultActivityQueueProcessor.java:54)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor$1.run(DefaultActivityQueueProcessor.java:192)
2018-08-23 17:53:02,839 | WARN | processor-ValFabric | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
[.......]

The Cell wasn’t come up and also after a restart of the other Cell we hade the same issue on every Cell which was restarted. After a log of searching I figured out that the created configuration for the performance metrics produced this error. Problems / errors in the vCD Database stopped the start of the vCD Services.
After some search I found a Blog Post on the VMware Blogs (https://blogs.vmware.com/vcat/2017/11/virtual-machine-performance-metrics-vmware-vcloud-director-9-0.html) which didn’t indicate this problem but the solution from there also worked in my case.
The relevant Information’s are these lines:
2. Edit the content of the /tmp/metrics.groovy file to:

configuration {
}

3. Run the following command:

# cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy

After that I was able to restart my cells and everything went smooth again.

0

VMware NSX Manager – No backup possible

Last week I had some time so I decided to update my LAB NSX-Manager. My NSX installation is already pretty long in usage so I have a lot of upgrades done in the past months. At some point my scheduled Backup didn’t work anymore. Due the circumstance that I didn’t had the time to figure out why I didn’t investigate on that problem. For the Upgrade of the NSX-Manger a Backup is a relevant part of this process. So, I started to investigate on my backup problem:

In the appmgmt log I could found messages regading to my problem:


nsx-manager.vcoportal.de# show log appmgmt follow
2018-08-14 15:52:30.439 CEST  INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:104 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] start with Backup Restore settings BackupRestoreSettings [ftpSettings=FtpSettings [transferProtocol=FTP, hostNameIPAddress=10.10.121.150, port=21, userName=chris, password=*****, backupDirectory=/NSX, filenamePrefix=NSX_, passPhrase=*****, passiveMode=true, useEPRT=false, useEPSV=true], backupFrequency=null, excludeDataFor=[]]
2018-08-14 15:52:30.439 CEST  INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:131 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Adding entry for the key - notification.2
2018-08-14 15:52:30.443 CEST  INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:282 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Run backup script - Start
2018-08-14 15:52:56.959 CEST  INFO https-jsse-nio-443-exec-923 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:56.961 CEST  INFO https-jsse-nio-443-exec-923 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:56.959 CEST 2018', Status:'SUCCESS'
2018-08-14 15:52:57.788 CEST  INFO https-jsse-nio-443-exec-940 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:57.791 CEST  INFO https-jsse-nio-443-exec-940 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:57.788 CEST 2018', Status:'SUCCESS'
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:148 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Runtime error occured while taking backup
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:149 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"]
java.lang.NullPointerException: null
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) [classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) [spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at com.sun.proxy.$Proxy50.performBackup(Unknown Source) [?:?]
        at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
        at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-08-14 15:53:05.163 CEST  INFO https-jsse-nio-443-exec-936 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'unknown', ModuleName:'BACKUP_RESTORE', Operation:'BACKUP', Resource Name:'null', Time:'Tue Aug 14 15:53:05.160 CEST 2018', Status:'FAILURE' - DB backup NSX_15_52_30_2018_08_14 to server 10.10.121.150
2018-08-14 15:53:05.166 CEST  INFO https-jsse-nio-443-exec-936 VsmEventingService:73 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [SystemEvent] Time:'Tue Aug 14 15:53:05.163 CEST 2018', Severity:'Critical', Event Source:'Take Backup', Code:'9006', Event Message:'Backup of NSX Manager failed.', Module:'vShield DB Backup Restore'
2018-08-14 15:53:05.167 CEST  INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:152 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Deleting entry for the key - notification.2
2018-08-14 15:53:05.167 CEST ERROR https-jsse-nio-443-exec-936 VsmJsonErrorResolver:28 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" subcomp="manager"] Error discovered by JSON RPC
java.lang.NullPointerException: null
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) ~[classes/:?]
        at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) ~[spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at com.sun.proxy.$Proxy50.performBackup(Unknown Source) ~[?:?]
        at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) ~[classes/:?]
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) ~[jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
        at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Neither in the Internet nor in the VMware Knowledge Base I could find anything about this error. So, I searched more about this problem. In VMware NSX it is possible to get access to the “Linux System” behind VMware NSX. How the get this access is documented in the VMware KB Article Tech Support Access in NSX for vSphere 6.x (https://kb.vmware.com/s/article/2149630). There I worked my due the logs and found a message in the Backup Log:


+ reportErrorAndExit 'Unable to export rabbit.config for RabbitMQ.'
+ echo 'Unable to export rabbit.config for RabbitMQ.'

It is very likely linked to errors which can be  found in the rabbitmq log regarding the SSL cert –


[root@nsx-manager /var/log/rabbitmq]# tail -f rabbit\@localhost.log
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:37:08 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:39:27 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:53:05 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired

So the problem is a certifcate in the rabbitmq which is expired. First I tried myself to replace the certificates but ran in trouble so I had to recover my machine from backup. I made a Support by VMware and the GSS could help my with my problem.

Also I will not provide the solution here to fix the issue I hope you can use this information to check if you have the same problem. When you run into this issue I higly recommend to call the VMware Support for a production enviroment.