What makes a good Performance Engineer?

Note: This article was originally posted on Loadtester.com, and has been migrated to the Northway web site to maintain the content online.

I was asked recently to describe in my own words what qualities a good performance engineer would have, as they were trying to make a decision on hiring a resource for their company. I came up with some stuff and thought it might be good to share it with you and offer it up as a discussion topic in the Forums for the site. Perhaps you can add your own input and help businesses make informed hiring choices. This is not for use with an official job description, but more about principles and concepts.

Technical Skills

You want someone with a general knowledge of the latest computing concepts and terminology. They should be experienced with things like installing an OS from scratch (Windows, Linux, etc…), putting their own PC’s and networks together. They might have to put their own test lab together so this is important.

Networking Essentials – You want a person who knows all about the OSI model, the TCP/IP stack, and how DNS, DHCP, WINS, Routers/Switches/Hubs, and all that stuff works. They might need to use network sniffers to determine a network bottleneck so they need to know what they are sniffing around for. When you have a connectivity issue between the Test Controller and the Load Generators you don’t want to have to hunt down a network guy before doing some troubleshooting (i.e. is it plugged in, is the IP address right, etc…).

Protocols – At a minimum, they need to be able to create test scripts easily for the protocols in your organization. It is better to have a wide variety of scripting experience with other protocols like Winsock, COM, HTTP, and Citrix. You never know what you might run into.

Although I don’t require a person to be a code junkie or a major developer type, they should be able to look at HTML, ASP, JSP, Java, C, and other code and grasp the concept of what is going on. This has more to do with scripting issues than finding code bottlenecks, but obviously the more they know the code, the more things that can be found.

SQL (queries, stored procedures, indexes, administration, replication, etc…). The database is on the short list of things that cause major bottlenecks in a complex application. It is really the job of a good DBA to find the tricky stuff that can cause problems, but if your performance tester has no clue what’s happening on the database and how to interact with it, they may miss a critical issue.

They should get “the big picture”. They should understand their role in the  software development lifecycle. They should know what the developers, PM’s, QA, and operations folks are up to and how to interface with them. You would be surprised the number of “technical” people who only see their little area and do not understand organizational impacts to other groups.

They should be fluid in the software testing products your organization has chosen to use. If they have mastered one it is pretty easy to learn others, but the best thing is to choose someone with a year of real world experience in that particular product (more for a sr. person).

Non-Technical Skills

As important as any technical skill is, a performance engineer should understand performance testing, performance tuning, and capacity planning methodologies and processes common to the industry. It is not enough to know only which buttons make the test execute.

They should have a set of best practices and documentation for testing, tuning, and planning which they can use to customize based on the companies initiatives. An entry level position would be the exception. They will need to build this on the companies dime.

A good engineer is always a consultant, even if the client is “internal”. If they do not treat everyone as a client, you are asking for constant battles between them and every development team. Your resource should be flexible, cool under pressure, and respectful.

The goal of a performance team is to make everyone in the “pre-production” area to look good by rolling out applications that “rock” in performance. In return, when things run fast (even those poor in every other aspect), everyone loves the performance team. This is way to market the team. This means the resources must have good communication skills and come across as helpful instead of being seen as a roadblock for the project.

I prefer to bring in people of passion. I look for those who are constantly trying to broaden their knowledge. When they come across a project with a protocol they have never scripted in, they get excited because it is chance to learn something new. They are talking to other performance engineers in other companies, networking, and building a good support system. They attend social events, user groups, and conferences.

Other simple things: A person who actually calls people back when they say they will. Those make every effort to be of help even at their own expense. People willing to share knowledge. Those who know when to take the lead and when to follow. These are things to look for with any resource in general.

Summary

I’ve been fortunate to meet some really good performance engineers. The ones who are full time consultants stay busy because they all have similar qualities when it comes to work ethic. I think their non-technical skills take them farther than their technical skills. Do you have other suggestions for the qualities of a good performance engineer? If so, post it on our Forums in the General section and let’s talk about it.