Spring Cloud Config Server迁移节点或容器化带来的问题-清一色财经

如果您跟我一样，目前正在使用Spring Cloud Config做为配置中心的话，本篇将来要描述的问题，强烈推荐了解和关注！因为这个问题目前存在于所有的版本中，还没有完全修复。

问题现象

为了说明下面的内容，我们可以先尝试重现一下问题：在一个测试环境中，将Spring Cloud Config的配置中心迁移到另外一个节点上，即配置中心的IP地址发生了变化。在完成迁移之后，我们会发现该环境下各个微服务应用的健康状态会变得时好时坏，并且在日志中会出现类似下面的报错：

2018-05-1317:01:28,569WARN[http-nio-9920-exec-1]org.springframework.cloud.config.client.ConfigServerHealthIndicator-Healthcheckfailed
java.lang.IllegalStateException:CouldnotlocatePropertySourceandthefailfastpropertyisset,failing
atorg.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:132)
atorg.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$FastClassBySpringCGLIB$$fa44b2a.invoke(<generated>)
atorg.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
atorg.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
atorg.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
atorg.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
atorg.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
atorg.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)
atorg.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153)
atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
atorg.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
atorg.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$EnhancerBySpringCGLIB$$3a43a1f4.locate(<generated>)
atorg.springframework.cloud.config.client.ConfigServerHealthIndicator.getPropertySource(ConfigServerHealthIndicator.java:54)
atorg.springframework.cloud.config.client.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:35)
atorg.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
atorg.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
atorg.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85)
atorg.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:177)
atorg.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:166)
atorg.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:143)
atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
atjava.lang.reflect.Method.invoke(Method.java:498)
atorg.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
atorg.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
atorg.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
atorg.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
atorg.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
atorg.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
atorg.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
atorg.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
atorg.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
atorg.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
atjavax.servlet.http.HttpServlet.service(HttpServlet.java:687)
atorg.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
atjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atcom.yonghui.feign.filter.RequestOriginFilter.doFilter(RequestOriginFilter.java:41)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
atcom.yonghui.rpc.feature.web.boot.RpcHolder4BootFilter.doFilterInternal(RpcHolder4BootFilter.java:29)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
atcom.yonghui.rpc.feature.web.boot.FeatureSupport4BootFilter.doFilterInternal(FeatureSupport4BootFilter.java:24)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
atorg.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
atorg.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
atorg.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
atorg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
atorg.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
atorg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
atorg.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
atorg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
atorg.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
atorg.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
atorg.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
atorg.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
atorg.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
atorg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
atjava.lang.Thread.run(Thread.java:748)
Causedby:org.springframework.web.client.ResourceAccessException:I/OerroronGETrequestfor"http://192.168.5.103:9010/config-server/test":Connectionrefused(Connectionrefused);nestedexceptionisjava.net.ConnectException:Connectionrefused(Connectionrefused)
atorg.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
atorg.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
atorg.springframework.web.client.RestTemplate.exchange(RestTemplate.java:539)
atorg.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:172)
atorg.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:93)
...95more
Causedby:java.net.ConnectException:Connectionrefused(Connectionrefused)
atjava.net.PlainSocketImpl.socketConnect(NativeMethod)
atjava.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
atjava.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
atjava.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
atjava.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
atjava.net.Socket.connect(Socket.java:589)
atjava.net.Socket.connect(Socket.java:538)
atsun.net.NetworkClient.doConnect(NetworkClient.java:180)
atsun.net.www.http.HttpClient.openServer(HttpClient.java:463)
atsun.net.www.http.HttpClient.openServer(HttpClient.java:558)
atsun.net.www.http.HttpClient.<init>(HttpClient.java:242)
atsun.net.www.http.HttpClient.New(HttpClient.java:339)
atsun.net.www.http.HttpClient.New(HttpClient.java:357)
atsun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
atsun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
atsun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
atsun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
atorg.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
atorg.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
atorg.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
atorg.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
...99more

可以看到类似上面的健康检查失败错误，但是并不是一直这样，这个环境下的微服务会出现时好时坏的情况，那么为什么会出现这种现象呢？

原因分析

从错误日志中我们可以发现一个非常关键的信息：I/O error on GET request for "http://192.168.5.103:9010/config-server/test"。

报错说明了微服务检查配置中心获取配置的连接是否畅通的时候出现了连接不上的情况，但是这个链接信息其实并不是当前配置中心的地址，而是我们迁移之前的配置中心的地址。

从健康检查的实现源码ConfigServerHealthIndicator中为入口去分析和调试，我们可以解答上面现象的两个疑问：

@Override
protectedvoiddoHealthCheck(Builderbuilder)throwsException{
PropertySource<?>propertySource=getPropertySource();
builder.up();
if(propertySourceinstanceofCompositePropertySource){
List<String>sources=newArrayList<>();
for(PropertySource<?>ps:((CompositePropertySource)propertySource).getPropertySources()){
sources.add(ps.getName());
}
builder.withDetail("propertySources",sources);
}elseif(propertySource!=null){
builder.withDetail("propertySources",propertySource.toString());
}else{
builder.unknown().withDetail("error","nopropertysourceslocated");
}
}
privatePropertySource<?>getPropertySource(){
longaccessTime=System.currentTimeMillis();
if(isCacheStale(accessTime)){
this.lastAccess=accessTime;
this.cached=locator.locate(this.environment);
}
returnthis.cached;
}

为什么会健康检查访问的还是老的配置中心地址？

真正导致健康检查失败的语句是getPropertySource中的 this.cached = locator.locate(this.environment);而这里的具体实现在org.springframework.cloud.config.client.ConfigServicePropertySourceLocator类中，具体实现如下：

@Override
@Retryable(interceptor="configServerRetryInterceptor")
publicorg.springframework.core.env.PropertySource<?>locate(
org.springframework.core.env.Environmentenvironment){
ConfigClientPropertiesproperties=this.defaultProperties.override(environment);
CompositePropertySourcecomposite=newCompositePropertySource("configService");
RestTemplaterestTemplate=this.restTemplate==null?getSecureRestTemplate(properties)
:this.restTemplate;
Exceptionerror=null;
StringerrorBody=null;
logger.info("Fetchingconfigfromserverat:"+properties.getRawUri());
...
}

可以看到，真正去访问的地址是直接从properties.getRawUri()获取的，它已经是一个固化的值，而不是通过服务发现机制来动态获取的。这就导致了当我们把配置中心做了迁移，或者直接部署在容器中出现重启的时候，IP发生变化，而所有的微服务还以为访问的是原来的配置中心地址，就会出现健康检查失败的问题，导致服务不可用的现象。

为什么健康检查时好时坏？

上面的问题会导致健康检查失败，但是这个服务并不是一直都不好，而是间断性的出现不健康。这主要还是健康检查时间中的机制导致，这里可以具体看ConfigServerHealthIndicator的getPropertySource函数，该方法执行的时候中间并不是每一次检查都会去访问配置中心（执行locator.locate(this.environment)方法），因此客户端的健康检查并不会每次都健康检查失败，从而出现了微服务健康检查时好时坏的情况。

如何解决

该问题目前也在官方的issue中被提出，还处于open状态，具体可见：https://github.com/spring-cloud/spring-cloud-config/issues/514

由于该问题目前并没有得到解决，虽然提交了一个PR，但是还有待完善以及提供一些测试，原本想完全处理好之后再写一篇文章，但是发现最近不少问过类似问题，所以索性先写一篇文章，提醒一下用户以及给一些相关的建议。

当前版本上不太容易通过扩展的方式去解决的这个问题，所以大家可以变通的去避免这个问题：

部署在虚拟机上而不是容器上，避免IP的变动
可以考虑关闭微服务队config客户端的健康检查，增加参数management.health.config.enabled=false；但是这个操作有一个弊端，虽然迁移不会引发服务时好时坏的问题了，但是如果有动态配置刷新需求，如果迁移了配置中心，刷新配置操作也是会失败的。

【本文为清一色专栏作者“翟永超”的原创稿件，转载请通过清一色联系作者获取授权】

戳这里，看该作者更多好文

©本文为清一色官方代发，观点仅代表作者本人，与清一色无关。清一色对文中陈述、观点判断保持中立，不对所包含内容的准确性、可靠性或完整性提供任何明示或暗示的保证。本文不作为投资理财建议，请读者仅作参考，并请自行承担全部责任。文中部分文字/图片/视频/音频等来源于网络，如侵犯到著作权人的权利，请与我们联系（微信/QQ:1074760229）。转载请注明出处：清一色财经