Last week I had some time so I decided to update my LAB NSX-Manager. My NSX installation is already pretty long in usage so I have a lot of upgrades done in the past months. At some point my scheduled Backup didn’t work anymore. Due the circumstance that I didn’t had the time to figure out why I didn’t investigate on that problem. For the Upgrade of the NSX-Manger a Backup is a relevant part of this process. So, I started to investigate on my backup problem:
In the appmgmt log I could found messages regading to my problem:
nsx-manager.vcoportal.de# show log appmgmt follow
2018-08-14 15:52:30.439 CEST INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:104 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] start with Backup Restore settings BackupRestoreSettings [ftpSettings=FtpSettings [transferProtocol=FTP, hostNameIPAddress=10.10.121.150, port=21, userName=chris, password=*****, backupDirectory=/NSX, filenamePrefix=NSX_, passPhrase=*****, passiveMode=true, useEPRT=false, useEPSV=true], backupFrequency=null, excludeDataFor=[]]
2018-08-14 15:52:30.439 CEST INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:131 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Adding entry for the key - notification.2
2018-08-14 15:52:30.443 CEST INFO https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:282 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Run backup script - Start
2018-08-14 15:52:56.959 CEST INFO https-jsse-nio-443-exec-923 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:56.961 CEST INFO https-jsse-nio-443-exec-923 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:56.959 CEST 2018', Status:'SUCCESS'
2018-08-14 15:52:57.788 CEST INFO https-jsse-nio-443-exec-940 PAMAuthenticationProvider:95 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Successful login for admin
2018-08-14 15:52:57.791 CEST INFO https-jsse-nio-443-exec-940 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'10.10.120.210', ModuleName:'ACCESS_CONTROL', Operation:'LOGIN', Resource Name:'nsx\appliance-management', Time:'Tue Aug 14 15:52:57.788 CEST 2018', Status:'SUCCESS'
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:148 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Runtime error occured while taking backup
2018-08-14 15:53:05.151 CEST ERROR https-jsse-nio-443-exec-936 VsmServiceBackupRestoreExecutor:149 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"]
java.lang.NullPointerException: null
at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) [classes/:?]
at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) [classes/:?]
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) [spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at com.sun.proxy.$Proxy50.performBackup(Unknown Source) [?:?]
at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) [classes/:?]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2018-08-14 15:53:05.163 CEST INFO https-jsse-nio-443-exec-936 VsmAuditingService:94 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'admin', Originated Ip:'unknown', ModuleName:'BACKUP_RESTORE', Operation:'BACKUP', Resource Name:'null', Time:'Tue Aug 14 15:53:05.160 CEST 2018', Status:'FAILURE' - DB backup NSX_15_52_30_2018_08_14 to server 10.10.121.150
2018-08-14 15:53:05.166 CEST INFO https-jsse-nio-443-exec-936 VsmEventingService:73 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [SystemEvent] Time:'Tue Aug 14 15:53:05.163 CEST 2018', Severity:'Critical', Event Source:'Take Backup', Code:'9006', Event Message:'Backup of NSX Manager failed.', Module:'vShield DB Backup Restore'
2018-08-14 15:53:05.167 CEST INFO https-jsse-nio-443-exec-936 FilesystemDBKeyValueStore:152 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Deleting entry for the key - notification.2
2018-08-14 15:53:05.167 CEST ERROR https-jsse-nio-443-exec-936 VsmJsonErrorResolver:28 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" subcomp="manager"] Error discovered by JSON RPC
java.lang.NullPointerException: null
at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.runBackupScript(VsmServiceBackupRestoreExecutor.java:312) ~[classes/:?]
at com.vmware.vsmvam.support.backuprestore.service.impl.VsmServiceBackupRestoreExecutor.performBackup(VsmServiceBackupRestoreExecutor.java:139) ~[classes/:?]
at com.vmware.vsmvam.support.backuprestore.service.impl.BackupRestoreServiceImpl.performBackup(BackupRestoreServiceImpl.java:88) ~[classes/:?]
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) ~[spring-security-core-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at com.sun.proxy.$Proxy50.performBackup(Unknown Source) ~[?:?]
at com.vmware.vsmvam.support.backuprestore.facade.impl.BackupRestoreFacadeImpl.performBackup(BackupRestoreFacadeImpl.java:53) ~[classes/:?]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at com.googlecode.jsonrpc4j.JsonRpcServer.invoke(JsonRpcServer.java:503) ~[jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handleObject(JsonRpcServer.java:374) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handleNode(JsonRpcServer.java:283) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:220) [jsonrpc4j-0.26.jar:?]
at com.googlecode.jsonrpc4j.JsonRpcServer.handle(JsonRpcServer.java:205) [jsonrpc4j-0.26.jar:?]
at com.vmware.vsmvam.client.rpc.RpcController.facade(RpcController.java:50) [classes/:?]
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [servlet-api.jar:?]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [servlet-api.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at com.vmware.vsmvam.security.XsrfFilter.doFilter(XsrfFilter.java:79) [classes/:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:94) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.4.RELEASE.jar:3.2.4.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at com.vmware.vshield.rp.ProxyFilter.doFilter(ProxyFilter.java:75) [classes/:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.23]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.23]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:595) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.23]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [catalina.jar:8.5.23]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.23]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.23]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.23]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-coyote.jar:8.5.23]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.23]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.23]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-coyote.jar:8.5.23]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.23]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.23]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Neither in the Internet nor in the VMware Knowledge Base I could find anything about this error. So, I searched more about this problem. In VMware NSX it is possible to get access to the “Linux System” behind VMware NSX. How the get this access is documented in the VMware KB Article Tech Support Access in NSX for vSphere 6.x (https://kb.vmware.com/s/article/2149630). There I worked my due the logs and found a message in the Backup Log:
+ reportErrorAndExit 'Unable to export rabbit.config for RabbitMQ.'
+ echo 'Unable to export rabbit.config for RabbitMQ.'
It is very likely linked to errors which can be found in the rabbitmq log regarding the SSL cert –
[root@nsx-manager /var/log/rabbitmq]# tail -f rabbit\@localhost.log
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:37:08 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:39:27 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
=ERROR REPORT==== 14-Aug-2018::15:53:05 ===
SSL: certify: ssl_alert.erl:88:Fatal error: certificate expired
So the problem is a certifcate in the rabbitmq which is expired. First I tried myself to replace the certificates but ran in trouble so I had to recover my machine from backup. I made a Support by VMware and the GSS could help my with my problem.
Also I will not provide the solution here to fix the issue I hope you can use this information to check if you have the same problem. When you run into this issue I higly recommend to call the VMware Support for a production enviroment.