Push vs Pull Interfaces

When implementing any large system and designing it's processes there comes a time you need to decide whether to send data to other systems (Push) or make it available for them to come and get (Pull). Looking at most of the systems around me, people have tended to design the push model. They have processes that as a last step send a file somewhere either for other systems to consume or in case of reports or csv files, for users to access.

Perhaps the reason behind the push model preference is that the sender system is the instigator. We wonder how the receiver will know the information is ready if we don't send it to them.

Push is fraught with issues
- security - to send a file to another's server sending system needs to get access to the target system. Once you have sent a file somewhere, you cannot be sure what sort of access control is imposed on it.
- retention and archival - once you have sent a file, you don't know how long it will be retained. If there are retention requirements, you need to keep a copy and that is likely duplicated at the target.
- issues on the target - target file servers frequently run out of space. They are generally managed by someone else. Problem resolution then means 2 teams are involved. The target support team to fix space issues. The sender support team to re-run the process which sends the file. In a pull model, generation and storage of output is in one team. Receiver is not tightly couples with Sender (who now becames Producer rather than Sender).

This is why I'm an advocate of the pull or self service model.

To address the issue of receiver not knowing when the data is available, you can use a notification mechanism. Notification mechanisms such as messaging or email do not have the same security and reliability issues as file transfer. Messages tend to be smaller. The sender only needs access to their sending mechanism and not the target. You don't need access to my inbox to send me an email. Space is rarely an issue because messaging systems tend to be monitored and small notifications tend not to be the straw that breaks the camels back. Large output files can often be the thing that fills the target system's space. And when there is an issue with sending a message, there is generally bounce / response.

Hopefully that helps you in designing your solutions.

Performance Monitoring and Capacity Planning

At work we've just finished our first month of live operation for a large new system.  Now that the teething problems are past, we need to be sure we understand how the system performs as it grows. 

I was hunting around for tools like Orca or Cacti which provide graphical presentation of Load vs Capacity trending over time.

This is interesting information, but it is only part of the question. While hunting I came across Adrian Cockroft's Capaciity Planning Blog. He shared an interesting concept he picked up at eBay called time-to-live which is a function of headroom and growth.

What's in a Name

For many years, I've wanted to start my own consulting firm. One hindrance has been the selection of a name. I think maybe I was putting too much thought into the meaning of the name. So, during a recent visit to my accountant, he told me to call it anything and get on with it. Sometimes we need a little push.

So, the name is Mussoorie House.

So what is in a name?

Mussoorie is the name of our house :) In Australia, it's very common for houses to have names on them. When I bought our house, it already had a name. Mussoorie is the Queen of the Hill Stations in the Himalya of India.

Apparently the previous owners of our house was stationed in India at the time they got married and they went to Mussoorie for their honeymoon. I'd say it must have made quite an impression on them.

So what's that go to do with Consulting?

Well - absolutely nothing. But I am trying to convince my wife Deniese to work in my company. She is a bit reluctant. So when she said, "Let's call it Mussoorie House." I of course agreed.

Well, that's where then name come's from. Perhaps my next trip to India will be tax deductable because I'll say we need to visit the namesake of the company :)

Wish me luck!

Mo Bro for the Boys

A bunch of us at work decided to become Mo Bro's and grow our Mo's.  It's not pretty around the office.  But it's for a good cause. 

Adding Performance Monitoring to Web Service App

Were I work we just recently deployed a large system called Calypso.  One of the key components we developed for integration with Calypso is the Web Service Engine (WSEngine).  The WSEngine exposes the Service we created as Web Services.

Our WSEngine is based on CXF, Spring and Jetty embedded in a Calypso Engine.  Engines are a stand alone processes which are managed by Calypso and passed Events from Calypso to perform their work.

During load testing our WSEngine performed quite well.  But now in production we discover intermittently our clients are getting timeout errors and or connection refused errors.  The purpose of my current work is to find the underlying cause and correct it.  While we are here though, we should also put in place appropriate monitoring to pro-actively monitor performance and avoid recurrence.

A bit of Architecture

As I said, the WSEngine exposes Services as SOAP WebServices using the CXF library.  We implemented a set of services which translate external business functionality into internal API calls to achieve a certain result.  Our Web Services are generated from the Java Interfaces we declared in our Service Layer. 

In addition to calling core business services a given Web Service call must also be authenticated and authorized. 

Other Background

The primary client calling our WSEngine is webMethods.  webMethods is our corporate ESB.  Many of the business services are the result of webMethods agreggating results from calls to several systems.  Therefore, we needed an ESB in the middle.  Trouble is, we cannot see fully what is happening between the ESB and us. 

Back to Performance Monitoring

So, to get to the underlying cause of performance issues, we need to measure each call to the WSEngine as well as performance of internal calls required to create our response.

As there are several layers into which performance logging can be injected, we need to find which one is most appropriate and then how to do it without much development or intrusion into the working code.  Remember we are just live.  We need to solve the problems now and not introduce a new development cycle risk, delay, etc...

The Layers

The WSEngine is made of of the following layers:

• CXF - for JavaWS binding


• Spring - IoC / AOP Layer


• Jetty - HTTP / Servlett Engine

Jetty logging can be turned on to give details of each http request.  CXF has logging which can be turned on that would give the response time of each Web Service invocation.  But neither of these would give the breakdown of the calls being made internally.  For example, each WS call has a primary service call, but there is also authorization calls made as well.  Therefore, it seems Spring is the most approrpriate layer.

Enter AOP
Interestingly, we have already used AOP to address an earlier cross cutting concern which was Security / Authorization. Now we have Performance Monitoring.

It didn't take much digging to find Spring has some built in facilities and people using AOP for Logging. One example is Kenan Sevindik's blog on Audit Logging at Service Level. This demonstrates using AOP to achieve an audit logging objective.

Ken DeLong's blog on Performance Monitoring with Spring AOP though is right on the money.  His ideas look great if you have time to do some implementation work.  Indeed I hope to implement the JMX based accumulation of performance facts for more proactive monitoring in the future.  However, now I need something out of the box.  So, I think JamonPerformanceMonitorInterceptor is the go.

For Further Reading on Interceptors and Performance Monitoring

This is good stuff
IBM Developer Works - Postcompilation instrumentation and performance monitoring as you would expect, well researched and documented. After a quick read, I feel good that AOP and interceptors is the way to go to get some Performance Monitoring in our app quickly.

North Island New Zealand

I'm trying to plan my trip to North Island New Zealand in January.  We've already go air tickets in and out of Auckland.  But that's it.  Now have to think about what to do.
I was told once not to miss the Bay of Islands.  I'd love to charter a yacht or just stay by the shore.  Also have always wanted to hire a caravan and just drive to nice places and stay.
We have some friends who will be in NZ because they're kiwis going home for Christmas.  So, we'll have to drop in on them.  But even so, it would be nice to have the caravan so we don't impose all 6 of us on them.
While planning, I found a really cool sight on Yahoo. Quite cool.  So, I'm going to try it out and see how it goes.

Thoughts - Rotorua has a luge that I bet the big kids (and I) will love.

Planning a Trip North

For a long time I've wanted to take a trip northwest of Sydney. North past Hunter, West past the Blue Mountains. We have some dear friends named Lynn and Jim Watt. Lynn's brother works the family farm out by Coonabarraban which is out there. It just happens to be the closest town to the Warrumbungles which are famous for an observatory and their remoteness.
This coming school holidays I have a week off and so nows the time. I think the quickest way to get there is drive north up the Pacific Hwy and then turn west past New Castle. But I'd rather drive out west of the blue mountains and drive north from there.

We've been out to Orange and Dubbo before, but I want to see the country on the other side of the mountains and north.

So we can swing out west through the Blue Mountains and maybe stop off in Mudgee. Mudgee is a country town famous for wine and like most Aussie country towns, a good time. I think the right way to experience the place would be to stay in a local Hotel. I found this place - Oriental Hotel - so perhaps we can book in advance. I bet it will be noisy because there is a pub downstairs. But that's really part of the charm.

From Mudgee it's a long drive north to Coonabarraban and the Warrumbungle National Park. Hopefully we can get a spot to camp out in the park. Here is the link to camping - Warrumbungle NP Camping. They don't require booking in advance, but I think that means you have to get there really early in the day. Better have a backup plan for accomodation. Maybe something in town.


seaside port macquarie resort - Google Search. That might be fun to stay at a pub style place