Northway Interviews Perfbyte’s James Pulley

James Pulley Interview

Many people in the IT world know James Pulley as a thought leader regarding performance testing. We recently had a chance to catch up with James and ask him some questions about his views on the performance testing industry, as well as his involvement in the online show Perfbytes.

Northway: How did you get involved with performance testing?

James: On April 1, 1996 I joined Mercury Interactive as  pre-sales system’s engineer.   Prior to that, I had worked for several other vendors, including Microsoft and Banyan Systems.  My background included work on Operating systems, developer support, databases and network infrastructure.   In short, my foundation skills and background were almost ideal for work as a performance tester.    Pre-sales at Mercury was a jack-of-all-trades role when I was there.  I had wide exposure to the functional automation, test management, performance and other applications offered by Mercury at the time as a part of proof of concept efforts and sales demonstrations.   I was in that role for about a year and a half.

In August of 1998, I joined Mercury’s Professional services group.  It was in this group that I became more specialized in the performance role.    When I left Mercury in late 2000, I decided that I would spend my time exclusively working on the performance side of the fence.

For a short period of time between my sales role and my professional services roles at Mercury, I worked for a supercomputing infrastructure company which built HPPI switches as well as a company which built network performance testing software.

Northway: What is the biggest change you have seen in the last few years with regards to performance testing in IT (good or bad)?

James: How much space will you allow me on your site Scott? The trends are pretty much on the “bad” side of things:

First, companies are not hiring for foundation level skills.  Being a performance tester requires a very broad technology and skills base, from architecture through development, project management, testing, systems analysis, etc…   People just are not hiring for the foundational skills or recognizing the value of these skills.   Instead they are taking manual functional testers or automated functional testers and throwing them into the role of a performance tester thinking, “You’re a tester….this is a tool…..go for it” and then expecting to have at least a positive ROI.  This positive ROI just doesn’t happen anymore,  and is leading to the decimation of value for performance testing practices.  Providing someone with poor aim a machine gun and a performance tester with poor testing, architecture and development skills both have predictable results – lots of mayhem and collateral damage

Secondly, companies aren’t training.   This is related to tools, process and even shoring up of foundation skills (see above).  Here is the tool – Go.

Third, there is very little mentoring these day.  This is essential to bring someone up to speed in a new field.  We would never think of putting a green electrician or plumber on the job without supervision and inspection of work but this is expected and normal with new performance testers.

Vendors and Customers going for the lowest cost provider independent of skills.  Granted, it’s hard to figure out who is effective and who is not when all of the providers are supplying the same marketing messages.  It doesn’t help that the purchasing agents making the decision are being compensated on the “per hour rate that they save” without an additional check against the quality of that hour delivered.

Northway: You’ve been known to moderate several online forums. What is your biggest frustration concerning this role?

James: There are two core frustrations.  There are five questions which get asked over and over again.  No one researches previous answers.    The other is a recent trend termed “shotgun posting”.  Invariably, these are new users without training or a mentor who are being asked to deliver in the performance testing role.   Instead of researching an issue (core attribute of a performance tester: research skills) or asking in one location for an answer, a question is simultaneously asked in four, five, or as many as a dozen locations at once.

This creates an administrative burden on the cleanup of the orphaned messages with no answers.  It also creates a headache for those users who actually do research the answers to questions for it becomes impossible to know where to look for a response, which response is correct and even if a full answer is located in any one location.  There are clearly occasions where the partial genesis an an answer can be found in three or four answers in different locations.  It is possible that the original poster has assembled the answer from the posts, but never have I observed any shotgun poster follow up with a full answer to anywhere when the solution is identified.   This is impacting the value of all of the forums where this is occurring, making the forums less valuable to all participants.

Northway: How did your show “Perfbytes” get started?

James: What is the biggest challenge in our profession?  How do you grow the next generation of performance testers which have high value?  How do you get information into their hands to make them successful?   How do you get them to think their way through the problem at hand and visualize a solution?   PerfBytes came out of a series of Tuesday morning breakfast events|rants|discussions at IHOP in Beaverton, Oregon with myself, Mark Tomlinson, Carlos Chidiac, Rex Black, Howard Chorney, John Wishon and a few others in different combinations on different weeks.

Mark and I decided in late 2012 to continue these Tuesday morning conversations in the form of the PerfBytes broadcast.   We want to enable those in the profession to be more effective in their role, independent of tool.  You will notice that the broadcasts are fairly tool agnostic, with a heavier concentration on processes and methods.   This is a very deliberate decision.  The same holds for the takeaways at the end of the show, a handful of items to remember and to keep you on track.

There is a financial side to this effort as well. We cannot raise the average rates for the entire profession until we also can raise the average value of services delivered.   A more effective performance engineer|tester naturally produces higher value output.   This leads to higher compensation in recognition of that value.   This is a direct effort to counter the low value present in the majority of performance testing projects today and the constant downward pressure on rates related to that low value delivered.  Admittedly this is a long-term play and the success of this will remain unknown for years.

Northway: The Perfbytes “News of the Damned” segment seems to be your favorite. Are you just trying to run down a company who has experienced a public performance challenge? Or is there a great purpose for highlighting these stories?

James: There are several reasons for the “News of the Damned” Segment:

a)  You often hear, “What’s the harm….” when people outside of performance testing either want to skip the effort or diminish the impact of a defect and go to production anyway.   These stories would normally disappear into the noise of the general news of the day and they provide direct examples of “…the potential harm…”   And they keep coming and coming.  Failure under load is not some nebulous risk, this is something that happens daily.

b) Patterns.  There are common patterns of load resulting in failure:  There are common solutions.  This is an educational opportunity.   We’re trying to emphasize this aspect harder now that we have broken out this segment almost as an independent show.

c) Admit it, even without the commentary the actual circumstances of some of these stories are just unbelievably funny – Steve Harvey taking out a lingerie shop, Prancercize, Paltrow announcing her separation as “conscious uncoupling” and taking out a website called ‘Goop.’  You can’t make this stuff up.

d) Mark was able to nail the intro endorsement of all time.   I am not sure how he did it.  I am totally frightened of what it may cost him.  It scares the willies out of me every time I hear it, but it’s just right for the topic.

Northway: What is the best advice you can give to someone taking on the role of a performance engineer?

James: Performance Testing, independent of tool, is micro scale development.   You will need every skill that would be needed in any other development context, requirements, QA, code development, maintenance, etc…   If you are weak in foundational skills, then you will struggle in the profession for years. Always have a mentor.  I have several who are still helping me to grow in my various roles. Build your support system.  You need to have sounding boards, technical peers you can call or email when you come across the “that’s wierd…” stuff.   I have been known to pick up the phone and ring a professional colleague who works for a direct competitor but who is an expert in a particular field where I have encountered “weird.”   I take those calls as well.   Just this past weekend I had two calls from New Zealand – Some servers were not happy in the Shire. Read voraciously on systems architecture and public failures of systems. Be the calm one in the crises.  A lot of our work is performed when trying to diagnose issues in prod.   Passions are high.  Hours are long.   The pizza gets cold fast.   Remain calm and always be able to articulate what you are doing, why, and for the next two steps as well. Join Toastmasters (or equivalent).  The ability to communicate to a wide audience of technical and non technical types is key in what we do. Learn to Laugh and do it often.  This is a very sober job, dealing often with dry subject matter.   Find what makes you laugh: instant stress relief.  Give back.  Nothing forces a more complete understanding of an issue or technology than mentoring another person in the same. I know that the above violates the PerfBytian “five takeways” by adding a few more.  Hopefully the additional items are of value to you and your readers.

Northway: Thank you for your time James. As always, we get some good takeaways from the conversation.