Performance Testing Center of Excellence: Part 1

Note: This article was originally posted on Loadtester.com, and has been migrated to the Northway web site to maintain the content online.

In 2002, after reviewing the new launch of the Business Technology Optimization (BTO) initiative by Mercury at their Worldwide User Conference, I had the same feeling as when I first heard about .NET or “web services”. It was supposed to change the world, but no one could really explain what it was. I could parrot what I had heard, but it was not a revelation to me. This year, at the 2003 Mercury conference, the clouds lifted and I see the light. There were so many questions I had about where IT is going and what we can look forward to in the near future. Why aren’t web services and WI-FI bursting ahead like we all expected them to? What are the CIO’s struggling with today? I learned some of the answers to these questions.

I also gained an understanding about the direction Mercury is going with their products and why. They believe the market will continue on its current path of limited spending and strategic spending. New solutions will be scrutinized for ROI like never before. However, there is a great opportunity for people who are in the business of “optimization”. Those of us who make stuff work better, faster, lower the cost, and alleviate risk will be able to ride a wave of activity in the next 24 months. It’s a time period of optimization. CIO’s are saying to their IT departments, “Once you prove the ROI on the stuff we’ve already bought, then we’ll start spending some more on the latest, greatest…” And they are absolutely right to do so in my opinion.

Mercury’s BTO initiative focuses on the Optimization Centers they have recently announced. There were five announced in September (IT Governance, Quality, Performance, Business Availability, and Resolution). More will probably be announced in the near future. To keep the length of this column at bay, we will address the Performance Optimization Center. What does it mean to have a Performance Optimization Center? I believe this means developing a center of excellence where performance is addressed throughout the entire lifecycle by resources with specialized skills. This Center will use LoadRunner (and other testing tools) with a best practices philosophy around performance. Mercury is providing support by aligning their products with this model. What are the qualities of a Performance Optimization Center? Mercury hasn’t officially come out with the details as of my writing this article, but these are some traits I came up with on my own:

• Resources focused on performance with skills in performance methodology. These are experts in the tool sets they use, acting as “internal consultants”.
• A repeatable methodology established for performance testing that is able to handle any project, platform, or technology.
• Performance issues are addressed from the strategy phase to production monitoring. This would mean that they would assist developers in optimization of code and identify down to the source code where bottlenecks are. They can also do general load testing, capacity planning, and assist production in tuning their systems.
• The Center addresses performance at the infrastructure, network, and application level using Load Testing tools, network sniffers, application profilers, and capacity planning modeling.
• The Center acts as a central “hub” for all projects in the enterprise. When it is time for performance testing, this group is used as an internal service.
• There is a standardized set of tools, results, analysis, and reporting deliverables for each project regardless of platform or size.

I see this as a step in the right direction. Theresa Lanowitz of Gartner shared her “Magic Quadrant” in the general sessions of the Mercury conference. This is a 2 X 2 matrix with data points representing the various testing vendors. Companies are rated on their ability to deliver what they promise and also their leadership and vision in the industry. The best companies are in the top right corner. Currently the leader of the pack by a long shot is Mercury. It’s probably because Mercury is taking feedback from their customers and is changing their products to support how it is actually done by those in the daily trenches. The one area tool vendors have lacked for years is the best practices of how to properly use their tools in an immature QA organization. By placing a “best practices” library behind each of the phases of BTO (Strategize, Plan, Develop, Test, Deploy, Operate), the tools can used more effectively and QA analyst can ramp up productivity faster. Mercury’s recognition as a leader by Gartner is well deserved. Let the market ultimately decide who the winner is, but my bet is on the company in the top right corner of that Magic Quadrant.

In my next article, I’ll do a follow up to better define the Optimization Center with new information I’ve gathered. Do you agree with my traits of a Performance Optimization Center? Let me know your thoughts in a comment below.