A bunch of us at work decided to become Mo Bro's and grow our Mo's. It's not pretty around the office. But it's for a good cause.
The Amazon Effect on Open Source
5 years ago
Wednesday, November 19, 2008
Friday, November 14, 2008
Adding Performance Monitoring to Web Service App
Were I work we just recently deployed a large system called Calypso. One of the key components we developed for integration with Calypso is the Web Service Engine (WSEngine). The WSEngine exposes the Service we created as Web Services.
Our WSEngine is based on CXF, Spring and Jetty embedded in a Calypso Engine. Engines are a stand alone processes which are managed by Calypso and passed Events from Calypso to perform their work.
During load testing our WSEngine performed quite well. But now in production we discover intermittently our clients are getting timeout errors and or connection refused errors. The purpose of my current work is to find the underlying cause and correct it. While we are here though, we should also put in place appropriate monitoring to pro-actively monitor performance and avoid recurrence.
As I said, the WSEngine exposes Services as SOAP WebServices using the CXF library. We implemented a set of services which translate external business functionality into internal API calls to achieve a certain result. Our Web Services are generated from the Java Interfaces we declared in our Service Layer.
In addition to calling core business services a given Web Service call must also be authenticated and authorized.
The primary client calling our WSEngine is webMethods. webMethods is our corporate ESB. Many of the business services are the result of webMethods agreggating results from calls to several systems. Therefore, we needed an ESB in the middle. Trouble is, we cannot see fully what is happening between the ESB and us.
So, to get to the underlying cause of performance issues, we need to measure each call to the WSEngine as well as performance of internal calls required to create our response.
As there are several layers into which performance logging can be injected, we need to find which one is most appropriate and then how to do it without much development or intrusion into the working code. Remember we are just live. We need to solve the problems now and not introduce a new development cycle risk, delay, etc...
The WSEngine is made of of the following layers:
Jetty logging can be turned on to give details of each http request. CXF has logging which can be turned on that would give the response time of each Web Service invocation. But neither of these would give the breakdown of the calls being made internally. For example, each WS call has a primary service call, but there is also authorization calls made as well. Therefore, it seems Spring is the most approrpriate layer.
Enter AOP
Interestingly, we have already used AOP to address an earlier cross cutting concern which was Security / Authorization. Now we have Performance Monitoring.
It didn't take much digging to find Spring has some built in facilities and people using AOP for Logging. One example is Kenan Sevindik's blog on Audit Logging at Service Level. This demonstrates using AOP to achieve an audit logging objective.
Ken DeLong's blog on Performance Monitoring with Spring AOP though is right on the money. His ideas look great if you have time to do some implementation work. Indeed I hope to implement the JMX based accumulation of performance facts for more proactive monitoring in the future. However, now I need something out of the box. So, I think JamonPerformanceMonitorInterceptor is the go.
This is good stuff
IBM Developer Works - Postcompilation instrumentation and performance monitoring as you would expect, well researched and documented. After a quick read, I feel good that AOP and interceptors is the way to go to get some Performance Monitoring in our app quickly.
Our WSEngine is based on CXF, Spring and Jetty embedded in a Calypso Engine. Engines are a stand alone processes which are managed by Calypso and passed Events from Calypso to perform their work.
During load testing our WSEngine performed quite well. But now in production we discover intermittently our clients are getting timeout errors and or connection refused errors. The purpose of my current work is to find the underlying cause and correct it. While we are here though, we should also put in place appropriate monitoring to pro-actively monitor performance and avoid recurrence.
A bit of Architecture
As I said, the WSEngine exposes Services as SOAP WebServices using the CXF library. We implemented a set of services which translate external business functionality into internal API calls to achieve a certain result. Our Web Services are generated from the Java Interfaces we declared in our Service Layer.
In addition to calling core business services a given Web Service call must also be authenticated and authorized.
Other Background
The primary client calling our WSEngine is webMethods. webMethods is our corporate ESB. Many of the business services are the result of webMethods agreggating results from calls to several systems. Therefore, we needed an ESB in the middle. Trouble is, we cannot see fully what is happening between the ESB and us.
Back to Performance Monitoring
So, to get to the underlying cause of performance issues, we need to measure each call to the WSEngine as well as performance of internal calls required to create our response.
As there are several layers into which performance logging can be injected, we need to find which one is most appropriate and then how to do it without much development or intrusion into the working code. Remember we are just live. We need to solve the problems now and not introduce a new development cycle risk, delay, etc...
The Layers
The WSEngine is made of of the following layers:
- CXF - for JavaWS binding
- Spring - IoC / AOP Layer
- Jetty - HTTP / Servlett Engine
Jetty logging can be turned on to give details of each http request. CXF has logging which can be turned on that would give the response time of each Web Service invocation. But neither of these would give the breakdown of the calls being made internally. For example, each WS call has a primary service call, but there is also authorization calls made as well. Therefore, it seems Spring is the most approrpriate layer.
Enter AOP
Interestingly, we have already used AOP to address an earlier cross cutting concern which was Security / Authorization. Now we have Performance Monitoring.
It didn't take much digging to find Spring has some built in facilities and people using AOP for Logging. One example is Kenan Sevindik's blog on Audit Logging at Service Level. This demonstrates using AOP to achieve an audit logging objective.
Ken DeLong's blog on Performance Monitoring with Spring AOP though is right on the money. His ideas look great if you have time to do some implementation work. Indeed I hope to implement the JMX based accumulation of performance facts for more proactive monitoring in the future. However, now I need something out of the box. So, I think JamonPerformanceMonitorInterceptor is the go.
For Further Reading on Interceptors and Performance Monitoring
This is good stuff
IBM Developer Works - Postcompilation instrumentation and performance monitoring as you would expect, well researched and documented. After a quick read, I feel good that AOP and interceptors is the way to go to get some Performance Monitoring in our app quickly.
Subscribe to:
Posts (Atom)
