Performance Testing Center of Excellence: Part 3

Note: This article was originally posted on Loadtester.com, and has been migrated to the Northway web site to maintain the content online.

In the past two sessions, I have defined one of the new product offerings from Mercury Interactive – the Performance Center product, and how it is used within the vision of Business Technology Optimization (BTO).

Here is the link to part 1 and here is the link to part 2.

Up to this point we have separated the Mercury product from the concept of a Center of Excellence (CoE). Now what about this term? What does it really mean for IT and how can you implement this into your organization?

Let’s look at this in a more practical way by using the healthcare industry as an example. You might go to the hospital for general help because you are not feeling well. After being examined, you are told that a heart transplant is needed. You have a special need. Another group takes over to perform this kind of operation. Their specific services are used based on the criteria of their specialty, experience, outcomes and efficiency in heart transplants. This activity is a “black box” function that a general doctor or even general surgeons don’t really have the expertise in. They allow this specialized group to “take over” for that function, and control is returned back when its over. This is how I would describe a CoE for the IT world. In theory you would be able to apply this to all areas of IT where a specialty is needed. A database administrators group is a good example, but a general web development group might not be. For the scope of this article we will continue to stick with performance testing.

You have no doubt heard about large companies that have many groups with their own QA testing. Reinventing the wheel is the norm. I have heard of a Fortune 100 having a QA group for each department in the same building on the same floor using totally different tools and different QA initiatives. Each group was off doing their own thing, trying to accomplish the same exact thing as the other group. A CoE would work for this company to minimize the duplication, and reduce the overall testing time with fewer resources.

Mercury says that they work with customers to help establish a “Centers of Excellence” model. This concept includes a set of rules and collaborative relationships within IT grouped around critical processes. Just like a heart transplant, when it comes time for performance testing, the project is released to this group who spins it through their performance testing methodology and spits it back out a polished high performance application. Because the process is solid and repeatable it gets faster, better, and easier each time it is done. Perhaps we should treat IT like a privately run health services group, at least in the realm of product assurance.

What should the CoE accomplish as a result of centralizing?

The most obvious thing is to bring together similar skill sets so that projects get accomplished faster and solutions are reused on various projects based on best practices instead of redoing things from scratch each time.

There are two important people involved in initiating a Center of Excellence. The first is the right person with the right skill set to begin doing the work. From a performance perspective, the right candidate has mastered the tools used for performance testing and has a full understanding of the performance management methodology behind the tools. Their success should be demonstrated on critical projects. The Center of Excellence is proved out with an army of one many times. They must set up the framework for the center so that adding another person is as simple as “plug and play”.

The second key person is their manager. A good manager must publish the success throughout the organization. They understand the value of performance testing and promote it every chance that comes along. Once other groups (i.e. project managers, etc..) begin to see the value, they will be drawn to use the service. Nothing causes recognition like solving that problem. A CoE begins with a person who is capable of solving the immediate need as well as the long term solution. For example, you get hired to help fix the performance on a mission critical application in a short period of time. You are able to do this by doing “emergency” speed load testing and isolating the area down to the code level, causing 500% performance increase in two weeks. Now managers are listening because they look good in this situation. Most managers will listen to ideas on how to prevent this kind of emergency in the future by testing early and often. If they brush aside these ideas and continue to move from emergency to emergency, is this the kind of place you really want to work, much less initiate a CoE?

An approach I have used in the past is to assume no performance management exists. Begin to talk to all of the parties who would be involved in the process of performance testing including project managers, leadership, operations, network engineers, developers, etc. Find out what they think they already have in place (you will most likely have many different views on this) and begin to build a picture of where the organization stands today. How are they doing performance testing today (if at all)? Are they focusing on fault management instead of end user experience? Are there only processes in place for being reactive instead of proactive? Once you get a feel for where the organization is you can know where to start.

Now let’s look at what it might be like to implement these ideas at a new company:

Phase One

Assuming you have to start at the beginning, you have to first prove out the ROI of performance testing. This is very easy from a risk mitigation standpoint (i.e. cost avoidance). I would start with the worst performing application and work from there. Nothing causes change like a problem. During this time you might be running only simple load and stress testing. Slowly you will be standardizing your processes, establishing relationships, and getting everyone accustomed to your deliverables. This can take six months to a year. Over this period of time, you will have created a mechanism for performance testing…a well oiled machine. You will be setting up a repeatable process for testing all apps no matter what their platform. You should also start some kind of scheduling system so that the status of projects, dates and times for testing, and other milestones are available to the project team. They will all know what you are working on and where you are at. Usually this is a web page or an online calendar which organizes testing dates for multiple projects. This allows you to manage expectations when project managers have a need for a performance test and want it immediately. They can check the calendar and see which project you are working on to determine which priority is higher (without having to call you).

Phase Two

After a few significant projects where your testing has made a significant improvement in the performance of application, or you have caught a serious flaw before it hit production, word will begin to get around. If the project managers believe that performance testing adds value, they will use the service. If your services are easy to engage and somewhat flexible, the world will beat your door down. You will have other groups and managers you’ve never heard of calling you to get on your calendar. There will come a point in time when you cannot handle all of the projects with just one person and one testing lab. As your area grows, you spend more and more time educating other areas how they can implement performance improvements in their area. Set up “lunch and learn” sessions with developers about testing their code in the development area using some free development stress tools. Talk to operations about how they are monitoring applications to ensure they not only get system resource information but business process response times. In this phase you begin to be a “draw” organization instead of a “push”. The key to knowing if you have the beginnings of a CoE is that people seek you out for projects instead of trying to avoid your service. When your function is not considered just a “check mark” on a project task list. This is a time of educating staff and management about the benefits of BTO and how your foundation for a Center of Excellence is working to align the IT organization with the business processes it supports.

Phase Three

This is when the leadership has seen the value of performance management and it is time to get serious. You have to be able to duplicate your success. With more resources (proper staffing, funds for hardware and software), you can begin to support all internal projects. You begin to standardize performance across all tiers and throughout the application lifecycle. You might document performance standards for how many megacycles each asp web page should stay under when executed, or how big graphics should be on any web page. You have your own repository for all load test results which can be searched by project and viewed online. Your deliverables are all online where anyone in the organization has access to them. A performance dashboard is provided to the CIO which shows how any project is performing anywhere in the lifecycle, and how it lines up with the business goals. Each project that has multiple versions tested over a period of time has a “scorecard” created from the output of load tests showing how they have gotten better over time by following your performance recommendations.

What responsibilities should not be part of the Center?

• Staffing/Resource Issues. The individuals that make up the Center should not have to deal with this. The groups manager should be handling this. By being able to accurately forecast how long things should take, it should be obvious if there is too much work. It becomes easy to spot a bottleneck. This is a good problem to have. It means others recognize the value added and are being drawn to the Center.
• People/Time conflicts. When methodologies and processes become established for the Center, one of the first obstacles is getting rid of the “backdoor” policies. Going through the back door is usually done because of a resource restrictions (usually time). It has to be demonstrated that repeatable processes will speed up the project in the long run while maintaining the kind of results the Center was founded on. If everyone wants to divert around the established processes, then the processes should be examined and changed. If excellence is not what the Center is producing, then obviously the methods are no good. When the methodology matches the IT goals, it will produce the kind of results that will cause people to take notice and want to use the people in the Center over and over again. If you begin backdoor policies, you will end up with frustrated employees and the Center will break down. A good manager of the Center should recognize this and respond quickly.
• Political Issues. When you find you are changing the entire process and methods that were set up to achieve the most efficient results simply because a particular manager is “hard to deal with”, you will never have a Center of Excellence. Here is an example: Manager Smith thinks his project is more important than Manager Jones. Although Mr. Smith used up all of his lab time troubleshooting (scope creep), he should be able to “bump” the project Mr. Jones has scheduled and add another week. The people in the Center should not be partial to any manager or project. They work based on scheduling and on a priority based system. Again, at this point the manager of the Center should take over and set the priority based on their discussions with Mr. Smith and Mr. Jones. The manager of the Center will decide if a project gets overridden or rescheduled. The process remains the same and the Center rolls on. To bog the member of the center in this situation only works against the processes in place. Members of the Center should not be confused as to what they should be working on and why. At the same time, the Center should be as responsive to individual customer needs and have some flexibility built in. The manager of the Center should be responsible for making this true without major deviations to how work is accomplished.

Much of this I have personally accomplished or witnessed in a large organization. Some of this would not apply to a small development shop, but most of it will in some scaled-down form. I hope this series of articles has helped to better describe how to use the CoE model to increase the efficiency of performance testing. I’d love to hear your thoughts in the comments section below.