Baseline Comparison

Defining a test run as a baseline helps testers make sure that the application performance remains stable as the code changes. Once a baseline is defined, subsequent test runs are compared to the baseline.

With baselines, you can easily identify degradations and bottlenecks, find the related changes in code, and quickly take actions to resolve issues.

You can select a test run as a baseline against which subsequent runs are being compared. In addition, subsequent test runs can automatically be marked as failed if the performance degrades compared to the baseline. BlazeMeter helps you automate the decision-making process, by allowing you to configure the test failure criteria using the baseline, so that test runs that significantly deviate from the baseline are automatically marked as “Failed”.

• Example Scenario
• Baselines and Multi-tests
• Set a Report as a Baseline
• Remove a Baseline
• Set a Baseline from Report
• Review Test Runs Compared Against the Baseline
• Identify Changes in Application Performance
• Define Failure Criteria Using the Baseline

Example Scenario

Create a test and run it. Set the generated test report as a baseline. Based on the report, you can make changes to improve the system performance as needed. Run the test again and compare the new report to the previous report. If the results from the new report are better, set the new report as a baseline and also set the failure criteria against the baseline. Run the test again. If performance degrades compared to the baseline, the test will fail automatically.

Baselines and Multi-tests

You can select a specific test run within a multi-test scenario and define it as the baseline for future comparisons.

All subsequent test runs within the multi-test scenario will automatically be compared to the defined baseline. The comparison data will be accessible in the Request Stats tab of the test report.

Set a Report as a Baseline

Follow these steps:

1. Go to the Performance tab, click Tests and select a test from the drop down list.
   The test window opens.
2. Click the History tab.

3. Choose the report that you want to set as a baseline and click Set as Baseline. If you need more info on the reports to make a decision, expand the report row to see the report summary.

   

The report is set as a baseline. Subsequent test runs will be compared to this report.

Remove a Baseline

1. Go to the Performance tab, click Tests and select a test from the drop down list.
   The test window opens.
2. Click the History tab.

3. To remove the baseline, just click the ‘x’ next to the Baseline button.

   If the baseline was used in Failure Criteria, they will need to be redefined.

   

   Or, when you click Set as Baseline for a different Report from your list, your current baseline will be removed and replaced with the new one you just selected.

   

   

Set a Baseline from Report

Alternatively, you can set a baseline from the Report itself.

Follow these steps:

1. Go to the Performance tab, click Reports and select a report from the drop-down list.
   A window with report details opens.
2. In the top right corner below the report name, click Set Report as a Baseline.
   
   

The report is set as a baseline. Subsequent test runs will be compared to this report.

To remove the baseline, just click Baseline and it will revert back to Set Report as a Baseline. Subsequent test runs will not be compared to this report.

If the baseline was used in Failure Criteria, they will need to be redefined.

Review Test Runs Compared Against the Baseline

Follow these steps:

1. Go to the Performance tab, click Tests, and select a test from the drop-down list.
   The test window opens.
2. Click the History tab.
3. Click one of the test reports. Make sure the test had a baseline defined when the selected report was executed.
   The page with the report details opens. Review the report compared to the baseline.
   

4. Click Summary. The KPIs show the change in comparison to the baseline.

   

5. Click Request Stats. To compare metrics to the baseline report, select an option from the Baseline Comparison dropdown.

   

   Select Change from Baseline to view the change in percentages from the baseline.

   

   Select Actual Baseline values to view the change in actual value terms.

   

6. Click Failure Criteria. You will see this tab only if failure criteria were defined against the baseline. If the failure criteria were defined using the baseline, you will see that the thresholds originated from the baseline.

   

7. Click Original Test Configuration. You can see the details of the baseline used for the current report. The baseline for the current report is the baseline that was defined in the test at the time this report was executed. If another baseline was selected after the execution of the current report - it will not affect the baseline of the report.

   

the baseline comparison data will be calculated and displayed only once the test run ended.

Identify Changes in Application Performance

Follow these steps:

1. Go to the Performance tab, click Tests, and select a test from the drop-down list.
   The test window opens. Make sure the test has a baseline defined.
2. Click the Trend Charts tab.

You can see the trend charts graphs that are compared against the baseline.

Define Failure Criteria Using the Baseline

When you set a report as a baseline, you can also set the failure criteria against the baseline. If performance degrades compared to the baseline in a subsequent test, the test will fail automatically.

Follow these steps:

1. Go to the Performance tab, click Tests, and select a test from the drop-down list.
   The test reports window opens.
2. Click the Configuration tab and scroll down to the Failure Criteria section.
3. Toggle the button for Failure Criteria on.

4. Check the box for Use from baseline.

   

5. You can:
   • Set the threshold for each failure criteria from the selected baseline.
   • Define failure criteria as a percentage of the baseline result.
     If you use Baseline to define the threshold, the threshold will be visible after saving.
     

Subsequent tests are compared against the baseline with these failure criteria. If performance degrades compared to the baseline, the test will fail automatically.