0

VMware vSphere Autodeployment (virtuallyGhetto)

Did I already mention that I am a huge fan of the Autodeploy Script from William Lam? William created a bunch of script which help you to deploy a nested VMware environment in an automated way. The environment itself is built with a VSAN based datastore.

You can find the Blog Post from William here:

vGhetto Automated vSphere Lab Deployment for vSphere 6.0u2 & vSphere 6.5

 

and his GIT repository with the scripts here:

https://github.com/lamw

When you look at the Blog Post from William you can see that he uses a single ESX Hosts for his deployment. In case that you maybe have more ESX host there is a small issue in case that you have a VSAN based cluster. For the deployment of a nested VSAN environment on a VSAN bases Cluster you must set a VSAN Parameter (not recommended for production usage!). William makes these settings only for one host as you can see in this script


if($datastore.Type -eq "vsan") {
My-Logger "VSAN Datastore detected, enabling Fake SCSI Reservations ..."
Get-AdvancedSetting -Entity $vmhost -Name "VSAN.FakeSCSIReservations" | Set-AdvancedSetting -Value 1 -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile
}

 

In my environment a have a four node cluster hosts with HA and DRS configured. In some cases, DRS kicks-in during the enrollment of the nested environment. When a virtual ESX Hosts ends up on a Hosts were the settings was node made, the deployment fails.

In my case I made an improvement (for my usage) to the script where I configure the Fake SCSI settings for all Hosts of a cluster.

 

if($datastore.Type -eq "vsan") {
$FakeVSANhosts = $cluster | Get-VMHost | ForEach-Object {
My-Logger "VSAN Datastore detected, enabling Fake SCSI Reservations for Host $_ ..."
Get-AdvancedSetting -Entity $_ -Name "VSAN.FakeSCSIReservations" | Set-AdvancedSetting -Value 1 -Confirm:$false | Out-File -Append -LiteralPath $verboseLogFile
}

 

So, if someone of you runs into the same problem, just change the lines in the script from William and everything should work like expected.

0

VMware vCloud Director and Cassandra DB

In the last week I had a customer appointment where we made an update of the VMware vCloud Director installation to the vCD Version 9.1. For this update we had to make a couple of steps and also a migration of the used MS SQL Database. Everything went smooth to the new versions and then des customer decided that he wanted to use the VM Performance monitoring for the vCD Customers.
The installation and configuration is documented in the VMware vCD Docs (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-55D8360A-B4F9-4CDC-8BDA-972C4F387D74.html). So, we started with the installation of the Cassandra cluster and the configuration of the vCD. After everything was configured the customers chose the option to not provide all performance metrics to the customers. Therefore you can limit the metric collection (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-983C566A-7DE5-4F06-9044-C979F8AE7C99.html). We created the /tmp/metrics.groovy file with the required input:


configuration {
metric("cpu.usage.average")
metric("cpu.usagemhz.average")
metric("cpu.usage.maximum")
metric("disk.used.latest") {
currentInterval=300
historicInterval=300
entity="VM"
instance=""
minReportingInterval=1800
aggregator="AVERAGE"
}
}

And after that we imported everything and everything looked good.
After a last restart of a cell we ran into an error on the cell:


2018-08-23 15:00:05,027 | INFO | Cell Application | AbstractCellApplication | Application startup event: Application startup complete. |
2018-08-23 15:00:05,027 | DEBUG | Cell Application | LifecycleStateManager | Transition complete: com.vmware.vcloud.common.main.LegacyCellApplication@7bac686b transition from STOPPED to STARTED via STARTING_FROM_STOPPED |
2018-08-23 15:00:05,132 | INFO | OSGI Delegator - listener dispatcher | BootstrapApplication | Cell startup completed in 1m 35s |
2018-08-23 15:00:05,381 | DEBUG | Spring Context: com.vmware.vcloud.ui-vcloud-webapp | OsgiBundleSpringContextFactory | Finished constructing ApplicationContext for Bundle: com.vmware.vcloud.ui-vcloud-webapp in 3 seconds |
2018-08-23 15:00:58,666 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 810fe296-828e-4ead-8354-e9209f1ef044, Broker URI: tcp://172.16.6.105:61616 |
2018-08-23 15:00:58,802 | DEBUG | .105:61616@48618, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#16 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:53,538 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.105:36820@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:58,672 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 6bc3326d-df7b-4f37-8d59-cd76f7394d47, Broker URI: tcp://172.16.6.106:61616 |
2018-08-23 15:01:58,704 | DEBUG | .106:61616@34410, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#18 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:02:01,647 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.106:59120@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 17:53:02,838 | WARN | processor-NetworkingWeb | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.dequeueElement(DefaultActivityQueueProcessor.java:313)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.access$100(DefaultActivityQueueProcessor.java:54)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor$1.run(DefaultActivityQueueProcessor.java:192)
2018-08-23 17:53:02,839 | WARN | processor-ValFabric | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
[.......]

The Cell wasn’t come up and also after a restart of the other Cell we hade the same issue on every Cell which was restarted. After a log of searching I figured out that the created configuration for the performance metrics produced this error. Problems / errors in the vCD Database stopped the start of the vCD Services.
After some search I found a Blog Post on the VMware Blogs (https://blogs.vmware.com/vcat/2017/11/virtual-machine-performance-metrics-vmware-vcloud-director-9-0.html) which didn’t indicate this problem but the solution from there also worked in my case.
The relevant Information’s are these lines:
2. Edit the content of the /tmp/metrics.groovy file to:

configuration {
}

3. Run the following command:

# cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy

After that I was able to restart my cells and everything went smooth again.

3

VMware NSX Manager – No backup possible

Last week I had some time so I decided to update my LAB NSX-Manager. My NSX installation is already pretty long in usage so I have a lot of upgrades done in the past months. At some point my scheduled Backup didn’t work anymore. Due the circumstance that I didn’t had the time to figure out why I didn’t investigate on that problem. For the Upgrade of the NSX-Manger a Backup is a relevant part of this process. So, I started to investigate on my backup problem:

In the appmgmt log I could found messages regading to my problem:


nsx-manager.vcoportal.de# show log appmgmt follow
2018-08-14 15:52:30.439 CEST  INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:104 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] start with Backup Restore settings BackupRestoreSettings [ftpSettings=FtpSettings [transferProtocol=FTP, hostNameIPAddress=10.10.121.150, port=21, userName=chris, password=*****, backupDirectory=/NSX, filenamePrefix=NSX_, passPhrase=*****, passiveMode=true, useEPRT=false, useEPSV=true], backupFrequency=null, excludeDataFor=[]]
2018-08-14 15:52:30.439 CEST  INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:131 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Adding entry for the key - notification.2
2018-08-14 15:52:30.443 CEST  INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:282 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Run backup script - Start
2018-08-14 15:52:56.959 CEST  INFO https-jsse-nio-443-exec-923 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:56.961 CEST  INFO https-jsse-nio-443-exec-923 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:56.959 CEST 2018', Status:'SUCCESS'
2018-08-14 15:52:57.788 CEST  INFO https-jsse-nio-443-exec-940 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:57.791 CEST  INFO https-jsse-nio-443-exec-940 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:57.788 CEST 2018', Status:'SUCCESS'
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:148 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Runtime error occured while taking backup
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:149 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"]
java.lang.NullPointerException: null
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) [classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) [spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at com.sun.proxy.$Proxy50.performBackup(Unknown Source) [?:?]
        at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
        at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-08-14 15:53:05.163 CEST  INFO https-jsse-nio-443-exec-936 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'unknown', ModuleName:'BACKUP_RESTORE', Operation:'BACKUP', Resource Name:'null', Time:'Tue Aug 14 15:53:05.160 CEST 2018', Status:'FAILURE' - DB backup NSX_15_52_30_2018_08_14 to server 10.10.121.150
2018-08-14 15:53:05.166 CEST  INFO https-jsse-nio-443-exec-936 VsmEventingService:73 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [SystemEvent] Time:'Tue Aug 14 15:53:05.163 CEST 2018', Severity:'Critical', Event Source:'Take Backup', Code:'9006', Event Message:'Backup of NSX Manager failed.', Module:'vShield DB Backup Restore'
2018-08-14 15:53:05.167 CEST  INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:152 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Deleting entry for the key - notification.2
2018-08-14 15:53:05.167 CEST ERROR https-jsse-nio-443-exec-936 VsmJsonErrorResolver:28 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" subcomp="manager"] Error discovered by JSON RPC
java.lang.NullPointerException: null
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) ~[classes/:?]
        at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) ~[spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at com.sun.proxy.$Proxy50.performBackup(Unknown Source) ~[?:?]
        at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) ~[classes/:?]
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) ~[jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
        at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Neither in the Internet nor in the VMware Knowledge Base I could find anything about this error. So, I searched more about this problem. In VMware NSX it is possible to get access to the “Linux System” behind VMware NSX. How the get this access is documented in the VMware KB Article Tech Support Access in NSX for vSphere 6.x (https://kb.vmware.com/s/article/2149630). There I worked my due the logs and found a message in the Backup Log:


+ reportErrorAndExit 'Unable to export rabbit.config for RabbitMQ.'
+ echo 'Unable to export rabbit.config for RabbitMQ.'

It is very likely linked to errors which can be  found in the rabbitmq log regarding the SSL cert –


[root@nsx-manager /var/log/rabbitmq]# tail -f rabbit\@localhost.log
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:37:08 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:39:27 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:53:05 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired

So the problem is a certifcate in the rabbitmq which is expired. First I tried myself to replace the certificates but ran in trouble so I had to recover my machine from backup. I made a Support by VMware and the GSS could help my with my problem.

Also I will not provide the solution here to fix the issue I hope you can use this information to check if you have the same problem. When you run into this issue I higly recommend to call the VMware Support for a production enviroment.

2

VMware View – Black Screen during login with Linux VM

The last couple of days I was playing around with a new View Pools based on Ubuntu Linux 18.04 VMs. After some initial test I created my working Image and made my pool for the users.
After a couple of days, I became a message from one of my users he could not login in the Linux Desktop. When he starts his Horizon Client he got a Black Screen and after a minute the connection was discard.
I started to investigate to the error and one of the first things I did was to looking in the Logfiles. First thing I did was to look into the installed packages which were different from my original Image. I noticed that the User installed Docker-CE in his virtual View Desktop to interact with a VMware Integrated Container (VIC) Server. For those which are not familiar with Docker, Docker need some devices (internal Network) to operate. This network is created during the installation. When I stopped the Docker Server and restart the viewagent the user could login again. After a reboot the login was not possible with the same Black Screen bevor. After these findings I looked at the logfiles from the viewagent-debug.log. There I found something in the Logs:


tail -f /var/log/vmware/viewagent-debug.log
018-08-28T18:50:53.911Z DEBUG [ComponentResponse] Reponse directed to:ID:/127.0.0.1/-2849395471335902840/75659/0
2018-08-28T18:50:53.912Z DEBUG [ComponentResponse] Message is -4836c1bb:1654679ccbd:-1e95 cn=9175d3da-3be6-4353-8b36-9804624e402a,ou=servers,dc=vdi,dc=vmware,dc=int cn=linux-vms,ou=server groups,dc=vdi,dc=vmware,dc=int ubuntu-vdi-001 172.17.0.1 XXXXXXXX thisIsAframeworkSSLAlgo thisIsAframeworkThumbprint f7911431-339e-47d3-a36f-7c27ac8874c2 XXXXXXXX BLAST ready 22443 172.17.0.1 XXXXXXXX

In the Log I could see that the Blast protocol was mapped to the wrong (the Docker network). So that was the reason which the Screen keep Black when the user tried to login. After some searching how to find the Blast protocol to the right network I found the solution in the VMware Community (https://communities.vmware.com/thread/591320). In short the solution is: That you configure the correct network (Subnet)in the /etc/vmware/viewagent-custom.conf file and there the parameter "Subnet=XXX.XXX.XXX.XXX/24" . After that you must restart the viewagent service ( service vieagent restart ). That’s all and also after a reboot of a system the login is possible.
Hope this helps if somebody has the same problem.

0

Honeypot as a Service (HaaS) Part 3

This is the third Post of the Haas Series…..

After we have everything up and running would should take a deeper look at the vRealize Log Insight.  There we get our notifications in the interactive analytics. In normal cases nobody will monitor the whole time the interactive analytics therefore we create a Dashboard. The create dashboard is only for your custom view. The log Insight documentation state it like this “You can add, modify, and delete dashboards in your Custom Dashboards space.”

So, in the first step we create a dashboard for your personal view. The creating is quite easy. So, if you have an alert in the interactive analytics you can create the Dashboard from there.

 

 

Just click in the New Dashboard Icon on the right site. In the Wizard you create a new dashboard. Provide a Name and if you want share the Dashboard with other Users in the environment.

I use the chart graph for my Dashboard. After you are finished you will find your Dashboard in on the Dashboard View under Shared Dashboards.

Quite Easy or? But would It be better to get an Alert Notification from Log Insight?

This is also very easy. When you go back in the Interactive Analytics just take the Alerts Button.

In the Windows we choose “Create Alert from Query”.

There you provide the required Details for the Alert. Be sure that you configured your SMTP Server before when you use E-Mail as alert notification.

That was cool so far, or?

In the last months I made a couple of NSX Implementations. NSX provides us some really cool feature like the possibility to move VMs into Quarantine to isolate them from the communication with other VMs. So why wo should not use these feature, to migrate the VM were the Access violation was made, into a quarantine location to research what is happening on this VM? When the access was done from a physical machine, we can also create a firewall rule which deny the access to the virtual environment? From my point of view this is security in an automated way.

In this blog Post I will not show how to install and configure the NSX Part of this series. There are a lot of useful information available.

To archive these goal, we can use a REST API Call to the vRO Server to call a workflow which we develop to migrate the VMs or create the firewall rule. One of the first things which we need, is the possibility to interact with NSX from the VMware Orchestrator. Therefore, will install the VMware Orchestrator NSX Plugin. The plugin can be found on the VMware Page:

https://my.vmware.com/web/vmware/details?downloadGroup=NSXV_VROPLUGIN_120&productId=417

We also need two additional Plugins The first plugin is needed to decrypt the BASE64 encrypt string from the Shim in vRO. The Plugin can be found here: https://communities.vmware.com/docs/DOC-24991

The next Plugin is not really necessary but from my point of view the JSON Implementation in vRO is not the best in the world…. therefore, I use the jsonPath for Orchestrator Plugin frim Soeldner-Consulting. The documentation and the Link to the VMware Solution Exchange can be found here:

jsonPath for Orchestrator

After we download the all plugins we can install them via the vRO Control center.

After we are finished with that, we have to create our Workflow that must start when an alert is triggered. The triggering can be done via email or REST API Call. I prefer the REST-API Call du the circumstance that this more flexible, but wait…….here we have a problem. The VMware vRealize Log Insight doesn’t offer any option to create a REST API Call. In the Web GUI only Webhook Notification is available. Therefore, we need a translation between the Webhook and the REST API Call which we need for the vRealize Orchestrator. As always there is an solution available…..Steve Flanders create a Shim which can act as Proxy for the Webhook notification in direction of the VMware vRealize Orchestrator. Steve has a GIT Repository with a good documentation about the different available versions and the installation. I will not go further on the installation as already everything is explained here: https://github.com/vmw-loginsight/webhook-shims

Another useful post can you find here:

https://blogs.vmware.com/management/2017/03/webhook-shims-now-available-on-docker-hub.html

After we installed and configured the Shim, we are ready to make our next steps.

I created a vRO Workflow which used as Input parameter the name of the Attacker VM. The workflow checks the Attacker VM against the configured Datacenter. When it is a virtual machine, then an NSX Security Profile is attached to the virtual machine and all communication is denied. When it is not a virtual machine, then an IPSet is created and this IPSet is added to a Security Group

For the workflow it is necessary, that the NSX Manager is configured within vRO. Also, that a security Policy exist which we can use in the workflow. Also, the Datacenter which host our virtual machines in our environment must be provided.

Now let’s have a look at the Workflow:

The Workflow consists from different areas. In the First Part of the Workflow (the yellow box) we have to decode the Base64 string and parse the VM Name from the Log Insight Message.

When we are finished with that, we check if the Attacker Machine is a virtual machine (the red box).  When the machine is a VM, we apply the security Group to the VM (the light brown box) which denies all communication via the Security Policy.

When the Attacker VM is not a virtual machine then the machine will be checked against DNS (the green box) to gather the IP. The DNS checking is done via vRO DNS query so the Orchestrator Appliance must be configured correctly with DNS Server. The DNS Check is done via Name and FQDN. When the name could be resolved and we have the IP Address, then an IPSet is created and the Attacker machine is applied to the security Policy. This only protects the virtual environment!

In all cases a email is created (the blue box) and send to an Administrator.

The Workflow itself has some Attributes which must be filled:

  1. Datacenter –> The Datacenter were to check against the Attacke Machine if it is a VMs
  2. SecurityGroupID –> The Securitygroup ID. The ID is not the name but should look like this: securitygroup-98765. If you don’t know the SecurityGroupID, you can Browse within vRO in the NSX Manager Connection. There you can find the correct name
  3. NSXManager –> Your corresponding NSX Manager (which must be configured before in vRO)
  4. smtpHost –> Your Mail Server
  5. fromName –>  Email Sender Name
  6. fromAddress –>  Email Sender Address
  7. toAddress –> Were the Mail should arrive
  8. DNSDomain –> Your DNS Name to check against FQDN
  9. Optional: SMTPUsername –> The Mail Server Username, if required
  10. Optional: SMTPPassword –> Password for the Mail Server user

For the Workflow itself there is room for optimizing. A loop to check against different Datacenter, or a loop to check against different Domain names……For me and this Demo the Workflow makes his job….

The Workflow can be found here: http://www.vcoportal.de/download/workflow/de.vcoportal.HaaS_.package

 

Honeypot as a Service (HaaS) Part 1 Link: http://wp.me/p7tsEp-C1
Honeypot as a Service (HaaS) Part 2 Link: http://wp.me/p7tsEp-Cb

Page 1 of 2512345...1020...Last »