Friday, July 8, 2016

Microservices based Cloud Native Application - Part III

Preview:

This is the third post in the series of Microservices based application development.

The entire series could be found here:

Microservices based Cloud Native Application - Part I

Microservices based Cloud Native Application - Part II


Microservices based Cloud Native Application - Part III


Overview:


Continuing from previous posts, in this post, I'm going to write about a few challenges which I faced while implementing the Microservices and how did I address them. This might hopefully help other folks who might run into similar issues.


Challenges faced while implementing Microservices:


Issue 1:


While using Zuul API, I was getting the following exception, when the angular JS application, invoked the Zuul service.

com.netflix.zuul.exception.ZuulException: Forwarding error

at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:132)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:157) 
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:78)

We also got the following exception:
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server

Root Cause:

The root cause of both of the above exceptions were same:

The zuul server was failing to register with Eureka server as an Eureka Client.
After analyzing the logs, we found that while communicating with the Eureka Server, there was a mismatch in the API interface signatures. Looked like a version mismatch between Eureka client in Zuul and the Eureka server!

And it was indeed. 

Solution:

In the pom.xml of individual microservices, we were using Eureka clients as:

<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-eureka</artifactId>
            <version>1.0.6.RELEASE</version>
</dependency>
Clearly, this might lead to confusions and issues, if different microservices define different versions of Eureka client, than the one defined in Eureka server.

To fix this and to bring in consistency in Eureka versions in all microservices, we removed the versioning from individual pom's and introduced dependency management in the parent pom (the pom of the parent project for all microservices).

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>Brixton.RELEASE</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
As mentioned above, we used Brixton release and it automatically pulls the correct version of the dependencies defined in the child poms.

So the pom.xml of individual microservices for Eureka client will look like the below. Notice there is no version!

<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-eureka</artifactId>
</dependency>



Issue 2:

While inovking the Zuul APIs from the Angular JS application, the API calls were failing with CORS (Cross Origin Resource Sharing)issue as below:
"No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:63342' is therefore not allowed access".


Root Cause:

The reason for above issue is that, the angular app was running on a domain 'http://pronet-profile-web.cfapps.io/' and the Zuul App was running on a domain 'http://pronet-edge.cfapps.io/'. Notice that the sub domains are different. This was causing the CORS issue. 

Solution:

Add CORS filter in the Zuul server. Generally Spring recommends adding of an annotation "@CrossOrigin" on the spring boot microservice. However, this solution somehow did not work on the Zuul application (having @EnableZuulProxy annotation).

As an alternate fix, we added the the below filter to the Zuul application to enable CORS:

@Bean
  public CorsFilter corsFilter() {
      final UrlBasedCorsConfigurationSource urlBasedCorsConfigurationSource= new UrlBasedCorsConfigurationSource();
      final CorsConfiguration corsConfig = new CorsConfiguration();
      corsConfig.setAllowCredentials(true);
      corsConfig.addAllowedOrigin("*");
      corsConfig.addAllowedHeader("*");
      corsConfig.addAllowedMethod("OPTIONS");
      corsConfig.addAllowedMethod("HEAD");
      corsConfig.addAllowedMethod("GET");
      corsConfig.addAllowedMethod("PUT");
      corsConfig.addAllowedMethod("POST");
      corsConfig.addAllowedMethod("DELETE");
      corsConfig.addAllowedMethod("PATCH");
      urlBasedCorsConfigurationSource.registerCorsConfiguration("/**", corsConfig);
      return new CorsFilter(urlBasedCorsConfigurationSource);
  }
Note that you need to add the "OPTIONS" method as well.


Issue 3:

While inovking the Zuul APIs from the Angular JS application, the API calls were failing with the below error:
The 'Access-Control-Allow-Origin' header contains multiple values 'http://localhost:63342, http://localhost:63342', but only one is allowed.

Root Cause:
On analyzing this, we found that the individual microservices had a "@CrossOrigin" annotation. We already have a CORS filter on the Zuul server (Fix for Issue 2 above.)
 Adding another filter on the microservice layer, duplicates the CORS filter and hence we were gettting that error.

Solution:

 Removing the @CrossOrigin annotation from the individual microservices solved the issue.

Microservices based Cloud Native Application - Part II

Preview:

This is the second post in the series of Microservices based application development.
The entire series could be found here:


Overview:


Continuing from my previos post, I'm going to explain in detail, three concepts which are essential ingredients of a Microservices Architecture.
  1. Service Discovery
  2. API Gateway
  3. Circuit Breaker

Service Discovery:


In a Microservices environment, we will have multiple services and when the same is deployed in a Cloud Environment, we will  have multiple instances of each service.

In such a scenario, we need services to be self discover-able. This will help in two ways. 

First, when one service invokes another service, it needs to know the actual location where it is hosted and which instance it should point to.
Second, in a cloud environment, when we add/remove instances, other services need to know about this transparently.

Using a centralized Service Discovery will help us solve this problem. Spring Cloud Netflix provides a library called 'Eureka' which will allow services to register to and discover other services.

Following are some code snippets for Eureka client and Eureka Server:

Eureka client: 

Code:
The following annotation needs to be placed in all the microservices which would register into Eureka. (Note that I'm using Spring boot app).


@SpringBootApplication
@EnableAutoConfiguration
@EnableDiscoveryClient
public class AppMain extends SpringBootServletInitializer{

}

Configuration:
The microservice registers into Eureka with a specific name (or serviceId). This could be configured in a file bootstrap.yml.


bootstrap.yml:
server:
  port: 8090

spring:
  application:
    name: profile-details

In order to locate the Eureka server, the client needs to know the server details. This could be configured in a file application.yml.


eureka:
  instance:
    hostname: localhost
    leaseRenewalIntervalInSeconds: 10
    metadataMap:
      instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}}
  client:
    serviceUrl:
      defaultZone: ${vcap.services.eureka-service.credentials.uri:http://127.0.0.1:8761}/eureka/

Eureka server:

This is a Spring boot app with an annotation @EnableEurekaServer. 

Code:

@SpringBootApplication
@EnableEurekaServer
public class DiscoveryServerApplication {

 public static void main(String[] args) {
  SpringApplication.run(DiscoveryServerApplication.class, args);
 }
}

Configuration:
We can configure this application to be a Eureka server in application.yml

eureka:
  instance:
    hostname: localhost
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/



API Gateway:



A microservices based application can have multiple clients (for eg, Web, Mobile, Partner integrations etc). Note that each of the individual microservices can evolve on its own and can deploy with different versions for different clients. In such scenarios, it will be necessary to provide a centralized interface which will perform the routing and tranformation services required.

An API Gateway does exactly this. Spring cloud Netfix provides a library Zuul which acts as an API gateway.

To create an API gateway server, use the annotation @EnableZuulProxy on the spring boot app. Note that this application should also register itself with Eureka, since it has to locate other services which it would forward to.

Code:
@SpringBootApplication
@EnableDiscoveryClient
@EnableZuulProxy

public class EdgeServerApplication {
}


Configuration:
The following configs in application.yml will allow it to register with Eureka and add routing logic for request forwarding.

eureka:
  instance:
    hostname: localhost
    leaseRenewalIntervalInSeconds: 10
    metadataMap:
      instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}}
  client:
    serviceUrl:
      defaultZone: ${vcap.services.eureka-service.credentials.uri:http://127.0.0.1:8761}/eureka/ 
   
zuul:
  routes:    
    profile-skills: 
      path: /**/skill/**
      stripprefix: false
      serviceId: profile-skills
      
    profile-summary: 
      path: /**/summary/**
      stripprefix: false
      serviceId: profile-details



Note:

  • The angular web application will point to the edge server. So all microservice invocations from the angular app goes via edge server.
  • For eg, if the angular web app has to invoke profile-skills service, then it would invoke /<edge-server-host-url>/<something>/skill/<something>/.
    • The edge server would then apply the rule as in the above configuration, and forward it to the "profile-skills" microservice (It would use the host name map obtained from Eureka to resolve the actual host url)).
  • The "stripprefix" attribute in the above configuration would retain the prefix part of the url before the routing path (like before /skill or before /summary)


Circuit Breaker:



When we have a huge number of microservices (which we will, in a typical complex application), it is necessary for the services to be fault tolerant. Since it is common for services to fail in a cloud environment using commodity hardware, we need to design our services in a fault tolerant way.

Circuit Breaker is a pattern used in Microservices which will work pretty much like a Circuit breaker in electric circuits. When a service fails, it creates an open circuit, breaking the flow and to fix that temporarily, we define an alternative service implementation, which will kick in and close the circuit.

In our use case, Profile-Details service invokes Profile-Recommendation service. When Profile-Recommendation service fails, an alternative implementation kicks in which will just return a dummy default recommendation, so that the entire flow is not broken. Once the Profile-Recommendation service is back online, normal services will resume.


We use FeignClient (which is a Client side load balancer) along with Hysterix.

Code:
In the profile-details service, add this annotation to the spring boot app:

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients

public class ProfileDetailsApplication {
}


Now to invoke the Profile-Recommendation service, we need not know the host url of the Profile-Recommendation service service. We just need to know the service name (configured in bootstrap.yml of the Profile-Recommendation service). Since we use Eureka, the service will resolve the host url.


Create an interface as below to invoke an API in Profile-Recommendation service.

@FeignClient(name = "profile-recommendation" , fallback = RecommendationClientFallback.class)
public interface RecommendationFeignClient {
 @RequestMapping(method = RequestMethod.GET, value = "/api/profile/{userId}/recommendation")
 List getRecommendations(@PathVariable("userId") String userId);
}

The value of the name attribute is the service name which we are calling. The value of the  fallback attribute is an alternative implementation, in case the actual call fails.
Invoking another microservice from one is this simple!!!
Following is the fallback implemntation:



@Component
public class RecommendationClientFallback implements RecommendationFeignClient {
    @Override
    public List getRecommendations(String userId) {
     List recommdList = new ArrayList();
     ProfileRecommendation recommend = new ProfileRecommendation();
     recommend.setRecommendationText("This is a default recommendation!");
     recommdList.add(recommend);
     return recommdList;
    }
}

Thats it!! When profile-recommendation service fails, it would invoke the fallback and would return "This is a default recommendation!".