Performance Testing Center of Excellence: Part 2

Note: This article was originally posted on Loadtester.com, and has been migrated to the Northway web site to maintain the content online. This article has been modified for relevancy.

In part 1 of this series, we talked in general about the Performance Center product. Many of us out in the field who use LoadRunner are wondering if this is just a repackaging of old products to make them look new again, or if this a “revolution” as is claimed. We talked about the concept of a “Center of Excellence” and some of the qualities I had in my mind that defines such a thing in the realm of application performance. I’m happy to report that I have received some good feedback. This article will highlight some of that feedback.

Going back to my original question: What does Performance Center mean in terms of a suite of products? The first thing we have to do is separate the terms “Center of Excellence” from “Performance Center”. In the past these were used interchangeably , which may have been confusing. An “Center” is a product offering that would be used in a customer’s own Center of Excellence. Centers are considered to be product branding from a marketing aspect. By the vendor definition, an “Center” consists of 3 types of components:

• A Dashboard which provides summarized and real-time information as to the health of the area in question (PPM, QC, PC, BSM). This is done through performance indicators (KPIs) from a business perspective. Dashboards can be configured to provide the views that are needed by all levels of management – an Department Manager can see information specific to his/her area while an Executive’s view may encompass the entire enterprise.
• Technologies for automating and integrating critical business functions. This is the area that has been historically identified with and contains LoadRunner, Functional Test, etc. For the performance center this would be the individual product components.
• Foundation Technologies for collecting the data, performing functions, etc. that are used across modules and Centers.

At the 2003 Mercury Worldwide User Group conference, attendees learned the direction of LoadRunner and Test Center within the context of this new  Center. I see it as building additional tools into LoadRunner that you would have to get from 3rd party sources currently. Here is Mercury’s vision for the Performance Optimization Center (or POC) according to their web site (as of 12/01/2003):

The Application Delivery Dashboard and Foundation are common and mentioned above. We already know about general load and stress testing with LoadRunner, and production tuning with ProTune (this product is now deprecated). The Mercury Diagnostics refers to the Deep Diagnostics product (gained from the Performant acquisition). Diagnostics currently appears as an additional tab (with it’s own graphs) in LoadRunner which allow you to dig down to the code level for bottlenecks (method calls, memory leaks, etc). This is available only for J2EE, .NET and some ERP/CRM solutions. The Capacity Planning module has been deprecated.

Amnon Landan (Mercury CEO) says that Mercury has been very broad in the past, but now wants to become deep too. For instance, LoadRunner can reveal what the end user experience is easily. When isolating bottlenecks, usually only a general area for where to start looking is known. Diagnostics is an attempt to replace the need for third party application profilers (like Compuware DevPartner Studio or Intel’s VTune). I hope to see an additional module that contains a network sniffer, allowing the network layer to be looked at more closely at the time of load without using an additional tool such as NetMon. When you look at all modules planned and in existence, its obvious Mercury is trying to create a centralized performance tool used throughout the whole development lifecycle with the ability to get more granularity out of each layer (application, network, hardware). The benefit of having these modules within LoadRunner is the ability to correlate all metrics into a single “results” file. This makes for more powerful analysis and diagnostic troubleshooting of applications.

Now back to my original reason for this series of articles: Just because you have the entire POC product offering with every module, does not mean you have automatically have a Center of Excellence. Just as in the past, the tool makers assume you know what to do with the software to prevent it from being “shelf ware”. This time around, it has to be different.

HP/Mercury has a model for a Center Of Excellence. It is in use every day. If you have ever used the outsourced offerings of Protune Validation Service (formerly ActiveTest) or the Protune Delivery Service (formerly ActiveTune), you have interfaced with their internal Center. There are teams of people running projects (engagements) for multiple clients concurrently in a centralized framework, utilizing infrastructure in disparate places. They do performance testing and tuning inside and outside the firewall. They address performance at every layer and tier.

The model Mercury Interactive uses to operate and manage their own Center of Excellence should be the foundation for owners of the POC to begin implementing their own. I am sure Mercury has learned a few tips and tricks running multiple projects in a performance lab and has probably documented some best practices around how they use their tools internally. This is the missing piece of the equation that customers of their products needs and deserve. Perhaps you only have 10 projects a year and one LoadRunner lab. You can still provide a Center of Excellence on a smaller scale to your organization. If Mercury is truly committed to their customer base implementing a Center of Excellence, a detailed methodology for using the POC in this manner needs to be made available. I alluded to this last time. I have spoken with a few people within Mercury in recent days who have told me that this is their general direction.

Now that we have distinguished a Center of Excellence from the HP Performance Center product and marketing, it is time to talk about how a Center of Excellence focused on performance should be implemented. In my next article, I’ll try to do just that. As usual, your comments are appreciated.