Performance Test Measurement Categories

User --> System --> Business

1. User Behavior Workflows, Think Time/Pacing, Volume Profile (Vuser Running Mix)

2. User Activity Sessions, Hits, Connections

3. System Activity Messages, Thread Counts, Queue Depths

4. System Behavior CPU, Memory, Disk ,Network, Data

5. Business Productivity Transaction Rate, Transaction Volume

6. Business Experience Response Times, Errors

Business Effectiveness ??? 

Reasons for IT Performance Testing

5 Distinct Times and Reasons to Performance Test

1. Vendor Trials ..... Try
   
2. System Design and Tuning ..... Try, Tune
   
3. System Test and Approval ..... Try, Tune, Troubleshoot
   
4. Environment Hardening and Right Sizing ..... Try, Tune, Troubleshoot
   
5. Production Problem Troubleshooting ..... Try, Troubleshoot

From zTest

LoadTestTryTuneTroubleshoot.vsd

Hagar and Load Testing

From zTest

Load Test Meaningful

More to come, but how about ...

• Try
• Tune
• Troubleshoot
  

From zTest

1. Anticipate the Awesome
2. Prove the Powerful
3. Improve the Infirm
4. Weed out the Whimpy
   

From zTest

Making Test Meaningful

It Matters!

We test because conditions matter!

We Test What We Care About

I'm definitely care when

• Gmail is slow!
  
• My van does not accelerate properly after going in the shop

I REALLY CARE!

Being Helpful

Test is all about providing helpful information and knowledge. Like when you see someone getting into their car in a parking lot, and say:

"Did you know your tire is almost flat?"
"Why no, thanks! I'll take care of that!"

Classic: Know --> Feel --> Do

Test --> Helpful Knowledge

Balanced SWOT

From Strategy Management

http://ventures.powweb.com/business_guide/crosscuttings/swot_analysis.html

SWOT Analysis at SMARTDRAW

Positive - Negative
Internal External

Strength Weakness / Opportunity Threat

SWOT Market Analysis

SmartDraw includes thousands of professional-looking diagrams like this that you can easily edit and make your own.

From
SmartDraw.com

Put it through its paces

Let's see what this baby can do ...

http://idioms.thefreedictionary.com/put+through+its+paces

put someone/something through their paces

to test the ability or skill of a person or system. This contest will really put you guys through your paces. Frank took the car for a drive through the mountains and really put it through its paces.

Wikipedia and Software Testing

It looks like a rather good addressing of the topic.

http://en.wikipedia.org/wiki/Software_testing

A test is a monitored ...

A test is ... a monitored ...

* action/reaction = cause/effect = stimulus/response

* application/effect

* inspection/condition

Defining Test - Examples

Student taking SATs

Test

Students takes the test in English, Math, etc.

Assess

Grade the tests
Compare Test Score to University or College Standard

Discern

If student clears the SAT Gate, then consider other issues to determine if they should be accepted

Road Test of Car

Test

Potential buyer drives car around

Assess

Subjective impression of handling, acceleration, breaking, etc.

Discern

If car 'passes' subjective assessment, consider more significantly on price, etc.

Load Test of Software System

Test

Run appropriate volume of virtual users in a given activity mix against the system and capture multiple data points

Assess

Assess the performance of the system against predefined criteria

Discern

Can we expect to be successful if we move forward with the system as is,
OR do we need to improve the system
OR do we need to seriously rethink this approach.

Test Process and Definition

Test - Assess - Discern

(Discover --> Determine --> Decide)

Test/Discover

• to act on, investigate/inspect, apply/employ
• an actual item, in whole or in part,
• in order to discover/reveal/disclose
• its actual behavior, condition/status, impact/effect

Assess/Determine

with the intent to

compare the actual results to a predefined standard/expected result

Discern/Discuss/Decide

enabling a more informed decision on the path/course to success.

--
Tom

Business Needs Test

BUSINESS NEEDS TEST - Thu 8/16/2007

1. Definition: A business produces beneficial products, services and experiences.
2. The business retains this privilege and authority as long as it abides by the moral, ethical and written laws of the land.
3. An authority determines what conditions will be produced.
   
   
4. Work must be done to produce these conditions.
5. Definition: Work is mental or physical activity which produces something of value.
6. Something of value must possess and exhibit specific conditions, static and dynamic.
   
   
7. Test occurs in the context of an authority determining how/whether to proceed.
   
8. Test provides information about a particular item.
   
9. Definition: A test is a focused disclosure of the actual condition of a particular item.

From Google Doc, Thoughts on Test

Vision for IT

Vision defined ...

What the world looks like when your product or service is valuable.


Vision for IT

Our enterprise is a more
effective, profitable, and pleasant place
because of our IT systems.

An IT system, just like any other tool, must satisfy all three aspects of soundness as defined by Vitruvius

• Firm - firmitas - Strong & Durable
  
• Functional - utilitas - Useful
  
• Friendly - venustas - Beautiful
  

Vision for Performance Testing

Greater Disclosure

Test

Disclose the actual condition, behavior, or effect of a particular item

Assess

Compare
... the actual to
... a predetermined requirement or standard

Discern

Identify the wisest steps to take next to achieve greatest success.

The Real Purpose of Testing

The real purpose of testing ... is to create information.

James Bach nailed it when he wrote, "The ultimate reason testers exist is to provide information that others on the project use to create things of value."

Reference

Better Software Magazine - July/August 2006
Proving Our Worth - Quantifying the Value of Testing by Lee Copeland, p. 34

Vision for Performance Testing

What does the world look like when we succeed?

Our client has hard data
on how their IT system performs
before it goes live!

Vision for IT