LISA - Test Pacing. Capgemini India Private Limited. July Prepared by: Srikumar Bose

Transcription

1 Capgemini India Private Limited July 2013 Prepared by: Srikumar Bose

2 1 Table of Contents 1. Test Pacing An Introduction Overview Stair Step Pattern (Load Patterns) Test Pacing Calculation Sample Pacing Runs Conclusion References Author s Biography...9

3 2 1. Test Pacing An Introduction Test Pacing enables end users to limit how fast test cases are executed. Pacing acts as a kind of think-time between test cases. The pace of transactions is computed using the formula: Pace Time in Seconds / (Transactions / Virtual Users) With a pacing of 50 test cases in 5 minutes using 10 Virtual Users, the pace per test case (Ptc) is Ptc = Time / (Trxn s / Vu s) Ptc = (5 minutes * 60 seconds/minute) / (50 test cases/10 Virtual Users) = (300/5) = 60 seconds (i.e.), one test case should be executed every 60 seconds. Pacing is, fundamentally, a braking system and, just as important, not an acceleration system. Pacing will not cause the Virtual Users to run tests at a faster pace. Pacing will not cause more Virtual Users to be started to satisfy the pace. Pacing will cause the Virtual Users to wait longer before they run a test again. Pacing only works with these load patterns: Immediate Ramp Run N Times Stair Step

4 3 2. Overview Pacing provides a way to limit the average number of transactions (tests) hitting the System under Test. For example, If you have pacing set at 50 transactions per second (tps), which means that you don t want LISA tool to run faster than 50 tests per second. This limit is set across the population of Virtual Users. If you have 150 Steady State Virtual User s, then 50 tps / 150 vu, that's tps per VU. With that, if a VU runs the test in less than seconds, pacing will cause the VU to sleep a bit between test runs. If a test takes longer than seconds to run, pacing is ignored, since LISA is staying under the limit of 50 tps. Pacing is limited to these load patterns because the load pattern must be able to estimate the number of steady state Virtual User s it provides. The pacing calculation depends on the number of steady state Virtual Users. It is important to notice that pacing is determined by the Steady State Load (which is also the period when the most Virtual Users are running). If you run a Stair Step Load pattern or a similar pattern, the pacing on the first step will be lower than you might expect. The idea of pacing is to throttle how many test cases are executed when the maximum numbers of Virtual Users are reached. While the numbers of Virtual Users are ramping up, pacing is not increased to hit the test cases/minute target. If LISA did that, it would mean more test cases are being executed per step, which cancels out the effect which we are trying to achieve with a ramp up time. The idea of ramp up time is to slowly increase the load on the system. For LISA, the load is based on the number of test cases hitting the System Under Test. If during ramp up, LISA decides to execute more test cases because the number of Virtual Users running at that time we re keeping up with the pace, we would just end up running at the maximum load on the first step -- there would be no ramp up. When you are trying to predict how pacing will behave, determine the number of test cases/minute of each Virtual User. If my pacing works out to 3 Test Cases per Minute per Virtual User then, when 1 Virtual User is running, I'll see 3 TC/minute hit the SUT. When 5 Virtual Users are running, I'll see 15 TC/minute hit the SUT. When I have 10 Virtual Users running, I'll see 30 TC/minute hit the SUT.

5 4 3. Stair Step Pattern (Load Patterns) The Stair Step pattern introduces Virtual Users (instances) to the system in well-defined steps rather than all at once. You specify the total number of steady state Users, the ramp up and ramp down times, and the number of steps for the ramp. To configure this pattern: Steady State Instances: The maximum number of Virtual Users to run at steady State. Number of Steps: The number of steps to use to reach max number of Virtual Users. The number of Virtual Users to introduce at each step is then: Steady State Instances divided by Number of Steps. Ramp Up Time: The time period over which Virtual Users are added to get to the Steady State. The time interval between steps is then: Ramp Up Time divided by Number of Steps. Steady State Time: The time period of the meaningful test run. Ramp Down Time: The time period over which Virtual Users are removed. Since The tests will run to completion after a "stop" request, the ramp down times is approximate. You can enter a time followed by h for hours, m for minutes, s for seconds (default). For example, if you have Steady State Instance = 10 Number of Steps = 5 Ramp Up Time = 1 m Steady State Time = 10 m Ramp Down Time = 30 s There are 10 instances and 5 steps, so 2 Virtual Users (10instance/5 steps) will be added each step. The ramp up time is 1(60sec) minute, so Virtual Users will be added every 12 seconds (Ramp up time 60sec/5 steps). The test will run in Steady State for 10 minutes. Two Virtual Users will be removed approximately every 6 (30/5) seconds. Number of Virtual Users to be included in each step = Steady State Instances/Number of Steps. pace per test case = pace time in seconds/ (# of tests cases/# of Virtual Users)

6 5 4. Test Pacing Calculation I seconds or 30 minutes with 150 transactions. So 1800/150 = 12 transactions/second. II. For example, If total transaction = 400,000 per day. Per hour = 400,000/6 = 66,666. (Let s assume 6 hours) For half-an-hour = 66,666/2 = 33,333 in 30 minutes. 33,333 = 30 minutes * 14 scripts * X transactions (14 is the # of scripts in the project. depending on how many one has, it has to be changed accordingly). Therefore X = 80 transactions/minute. Consider 1 User executes 20 transactions in a minute. So to complete 80 transactions, number of Users required would be 80/20 = 4 Users. If the test pacing is 42 transactions per minute and if number of Users is 3, then number of cycles required in 30 minutes would be 42 * 30 = 1260 cycles by 3 Users. So for 1 User, number of cycles would be 1260/3 = 420 (Note: To find out the number of cycles, divide the number of transactions by the number of Users)

7 6 5. Sample Pacing Runs Sample test results, using 4 test cases, each test case containing one NoOp step and no data set. In the suite, tests ran in parallel. Suite - 4 Tests, parallel execution Staging Document - RunNTimes; 2vu, 15min, pacing 2000 tc/15min Results of 8000 test runs Per Test Counts , 1992, 1994, 1989 Suite - 4 Tests, parallel execution Staging Document - RunNTimes; 4vu, 15min, pacing 2000 tc/15min Results of 8000 test runs Per Test Counts - As long as the CPU can keep up with the pacing, individual tests within a suite will hit their pacing target. If the load overtakes the CPU, pacing accuracy is off and is unpredictable. By unpredictable, I mean the Percentage-error for each test fluctuates. Some tests may be dead-on with their pacing while another test's pacing is off by 50%. Pacing acts as brake to limit the load on the SUT. Pacing does not act as an accelerator to increase the load. Internally, pacing only affects the lag time between each cycle of a test run. Thus, pacing will determine how long a Virtual User waits before running a test again. Let s imagine the load pattern is a Stair-Step Load pattern, ramping up from 10 Virtual Users to 100 Virtual Users, then back down to 10 Virtual Users. Let s also say pacing is set at 10 test cases/minute. On the first iteration of the test cycle, 10 Virtual Users execute the test 1 time each. If the average time to execute 1 test case is 5 seconds, these first 10 Virtual Users will finish their first iteration in about 5 seconds. The pacing algorithm will figure out that, with 10 Virtual Users running, and each taking 5 seconds to run 1 test, that the lag time needs to be 55 seconds. Thus, after 55 seconds, each Virtual User will be told to execute the test again. This will keep the pace of 10 tests/minute, since there are 10 Virtual Users executing the test 1 time within 1 minute. Let s say one iteration of test case takes 2 minutes. With the initial 10 Virtual Users of the load pattern, there aren't enough Virtual Users running to keep up the pace of 10 tests/minute. Pacing will set the lag time between test iterations of 0 milliseconds - as soon as a Virtual User finishes 1 iteration, the next iteration can begin. Pacing will not cause more Virtual Users to be fired up. If there are not enough Virtual Users running to keep up with the pacing, then there just are not. Once there are 20 Virtual Users running, things will be on pace - 20 Virtual Users running 20 tests in 2 minutes averages to 10 tests/minute. With 20 Virtual Users, the lag time between test iterations is still 0 milliseconds. When the load increases to 40 Virtual Users, a lag time finally moves above 0ms. 40 Virtual Users executing 40 iterations in 2 minutes average to 20 tests/minute. That's 2X over the pace, so the Virtual Users need to be slowed down: each Virtual User needs to sleep 1 minute between each test iteration. With 40 Virtual Users, a test only needs to be executed every few minutes to hit that pace. When the load increases to 80 Virtual Users, pacing will slow them down to wait 2 minutes between executions. The load pattern determines how many Virtual Users are running and how many iterations of the test case are executed per Virtual User, while pacing controls how much time elapses between when a Virtual User finishes one iteration of a test case and starts the next iteration.

8 7 6. Conclusion LISA generates very efficient load for its overhead, it has easier ways to pace the transactions of a test case to allow for variable "user think time" or steadily distribute them over a time period. While this functionality was already available in LISA, it is now dynamically paced in a much simpler automated fashion, giving testers the feel of "acceptance testing" with high load and efficiency possibilities.

9 8 7. References

10 9 8. Author s Biography Srikumar Bose has nearly 10 years of experience in Insurance, Cards as well as Retail Banking Domains. Worked extensively in CICS, DB2, Visionplus, Cardpac, e-banking & Guidewire and is currently working in AMFAM account for P&C Insurance with Guidewire v Performed as an SME for E-banking, Automation Expert (QTP & LISA) as well as provided Data Support & Environment Support along with Quality Assurance / User Acceptance / Base / Compliance / Product Upgrade / System Testing / ETL Testing & Data Migration Testing for GE UK (Hyderabad Unit), FDI (Chennai Unit), RBS [Gurgaon, London (UK) & Chennai Units] and CNA (Chennai Unit) Projects in Capgemini India Pvt. Ltd. Expertise in CICS, TSO, File Aid (DB2), Oracle SQL, UNIX & JCL. Extensive knowledge in Batch processing, report generations & processing s, Running Jobs using Control Cards as well as ETL process.