2

VMware NSX Manager – No backup possible

Last week I had some time so I decided to update my LAB NSX-Manager. My NSX installation is already pretty long in usage so I have a lot of upgrades done in the past months. At some point my scheduled Backup didn’t work anymore. Due the circumstance that I didn’t had the time to figure out why I didn’t investigate on that problem. For the Upgrade of the NSX-Manger a Backup is a relevant part of this process. So, I started to investigate on my backup problem:

In the appmgmt log I could found messages regading to my problem:


nsx-manager.vcoportal.de# show log appmgmt follow
2018-08-14 15:52:30.439 CEST  INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:104 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] start with Backup Restore settings BackupRestoreSettings [ftpSettings=FtpSettings [transferProtocol=FTP, hostNameIPAddress=10.10.121.150, port=21, userName=chris, password=*****, backupDirectory=/NSX, filenamePrefix=NSX_, passPhrase=*****, passiveMode=true, useEPRT=false, useEPSV=true], backupFrequency=null, excludeDataFor=[]]
2018-08-14 15:52:30.439 CEST  INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:131 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Adding entry for the key - notification.2
2018-08-14 15:52:30.443 CEST  INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:282 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Run backup script - Start
2018-08-14 15:52:56.959 CEST  INFO https-jsse-nio-443-exec-923 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:56.961 CEST  INFO https-jsse-nio-443-exec-923 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:56.959 CEST 2018', Status:'SUCCESS'
2018-08-14 15:52:57.788 CEST  INFO https-jsse-nio-443-exec-940 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:57.791 CEST  INFO https-jsse-nio-443-exec-940 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:57.788 CEST 2018', Status:'SUCCESS'
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:148 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Runtime error occured while taking backup
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:149 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"]
java.lang.NullPointerException: null
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) [classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) [spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at com.sun.proxy.$Proxy50.performBackup(Unknown Source) [?:?]
        at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
        at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-08-14 15:53:05.163 CEST  INFO https-jsse-nio-443-exec-936 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'unknown', ModuleName:'BACKUP_RESTORE', Operation:'BACKUP', Resource Name:'null', Time:'Tue Aug 14 15:53:05.160 CEST 2018', Status:'FAILURE' - DB backup NSX_15_52_30_2018_08_14 to server 10.10.121.150
2018-08-14 15:53:05.166 CEST  INFO https-jsse-nio-443-exec-936 VsmEventingService:73 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [SystemEvent] Time:'Tue Aug 14 15:53:05.163 CEST 2018', Severity:'Critical', Event Source:'Take Backup', Code:'9006', Event Message:'Backup of NSX Manager failed.', Module:'vShield DB Backup Restore'
2018-08-14 15:53:05.167 CEST  INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:152 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Deleting entry for the key - notification.2
2018-08-14 15:53:05.167 CEST ERROR https-jsse-nio-443-exec-936 VsmJsonErrorResolver:28 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" subcomp="manager"] Error discovered by JSON RPC
java.lang.NullPointerException: null
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) ~[classes/:?]
        at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) ~[classes/:?]
        at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) ~[spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at com.sun.proxy.$Proxy50.performBackup(Unknown Source) ~[?:?]
        at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) ~[classes/:?]
        at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) ~[jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
        at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
        at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Neither in the Internet nor in the VMware Knowledge Base I could find anything about this error. So, I searched more about this problem. In VMware NSX it is possible to get access to the “Linux System” behind VMware NSX. How the get this access is documented in the VMware KB Article Tech Support Access in NSX for vSphere 6.x (https://kb.vmware.com/s/article/2149630). There I worked my due the logs and found a message in the Backup Log:


+ reportErrorAndExit 'Unable to export rabbit.config for RabbitMQ.'
+ echo 'Unable to export rabbit.config for RabbitMQ.'

It is very likely linked to errors which can be  found in the rabbitmq log regarding the SSL cert –


[root@nsx-manager /var/log/rabbitmq]# tail -f rabbit\@localhost.log
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:37:08 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:39:27 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:53:05 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired

So the problem is a certifcate in the rabbitmq which is expired. First I tried myself to replace the certificates but ran in trouble so I had to recover my machine from backup. I made a Support by VMware and the GSS could help my with my problem.

Also I will not provide the solution here to fix the issue I hope you can use this information to check if you have the same problem. When you run into this issue I higly recommend to call the VMware Support for a production enviroment.