Baseline Comparison

Defining a test run as a baseline helps testers make sure that the application performance remains stable as the code changes. Once a baseline is defined, subsequent test runs are compared to the baseline.

With baselines, you can easily identify degradations and bottlenecks, find the related changes in code, and quickly take actions to resolve issues.

You can select a test run as a baseline against which subsequent runs are being compared. In addition, subsequent test runs can automatically be marked as failed if the performance degrades compared to the baseline. BlazeMeter helps you automate the decision-making process, by allowing you to configure the test failure criteria using the baseline, so that test runs that significantly deviate from the baseline are automatically marked as “Failed”.

  • For more information about how to fail tests against the baseline when running from Taurus and for baseline-based Taurus failure criteria in BlazeMeter UI, see Failure Criteria.
  • For customers on accounts with unlimited data retention, baselines will not be archived. View plans or contact your account manager for more information.

Example Scenario

Create a test and run it. Set the generated test report as a baseline. Based on the report, you can make changes to improve the system performance as needed. Run the test again and compare the new report to the previous report. If the results from the new report are better, set the new report as a baseline and also set the failure criteria against the baseline. Run the test again. If performance degrades compared to the baseline, the test will fail automatically.

Baselines and Multi-tests

You can select a specific test run within a multi-test scenario and define it as the baseline for future comparisons.

All subsequent test runs within the multi-test scenario will automatically be compared to the defined baseline. The comparison data will be accessible in the Request Stats tab of the test report.

Unlike single tests, multi-test scenarios do not support the defining of baseline-based failure criteria. Failure criteria for multi-tests are applied on individual test runs and not on the multi-test.

Set a Report as a Baseline

Follow these steps:

  1. Go to the Performance tab, click Tests and select a test from the drop down list.
    The test window opens.
  2. Click the History tab.
  3. Choose the report that you want to set as a baseline and click Set as Baseline. If you need more info on the reports to make a decision, expand the report row to see the report summary.

    no baseline set

The report is set as a baseline. Subsequent test runs will be compared to this report.

baseline set

Remove a Baseline

  1. Go to the Performance tab, click Tests and select a test from the drop down list.
    The test window opens.
  2. Click the History tab.
  3. To remove the baseline, just click the ‘x’ next to the Baseline button.

    If the baseline was used in Failure Criteria, they will need to be redefined.

    test 1 baseline x button

    Or, when you click Set as Baseline for a different Report from your list, your current baseline will be removed and replaced with the new one you just selected.

    set as baseline

    test 2 baseline

Set a Baseline from Report

Alternatively, you can set a baseline from the Report itself.

Follow these steps:

  1. Go to the Performance tab, click Reports and select a report from the drop-down list.
    A window with report details opens.
  2. In the top right corner below the report name, click Set Report as a Baseline.
    test report

The report is set as a baseline. Subsequent test runs will be compared to this report.
baseline

To remove the baseline, just click Baseline and it will revert back to Set Report as a Baseline. Subsequent test runs will not be compared to this report.

If the baseline was used in Failure Criteria, they will need to be redefined.

Review Test Runs Compared Against the Baseline

Follow these steps:

  1. Go to the Performance tab, click Tests, and select a test from the drop-down list.
    The test window opens.
  2. Click the History tab.
  3. Click one of the test reports. Make sure the test had a baseline defined when the selected report was executed.
    The page with the report details opens. Review the report compared to the baseline.
  4. Click Summary. The KPIs show the change in comparison to the baseline.

    original test configuration

  5. Click Request Stats. To compare metrics to the baseline report, select an option from the Baseline Comparison dropdown.

    baseline comparison not displayed

    Select Change from Baseline to view the change in percentages from the baseline.

    baseline comparison change from baseline

    Select Actual Baseline values to view the change in actual value terms.

    baseline comparison actual baseline values

  6. Click Failure Criteria. You will see this tab only if failure criteria were defined against the baseline. If the failure criteria were defined using the baseline, you will see that the thresholds originated from the baseline.

    failure criteria tab

  7. Click Original Test Configuration. You can see the details of the baseline used for the current report. The baseline for the current report is the baseline that was defined in the test at the time this report was executed. If another baseline was selected after the execution of the current report - it will not affect the baseline of the report.

    original test configuration

the baseline comparison data will be calculated and displayed only once the test run ended.

Identify Changes in Application Performance

Follow these steps:

  1. Go to the Performance tab, click Tests, and select a test from the drop-down list.
    The test window opens. Make sure the test has a baseline defined.
  2. Click the Trend Charts tab.

You can see the trend charts graphs that are compared against the baseline.

Define Failure Criteria Using the Baseline

When you set a report as a baseline, you can also set the failure criteria against the baseline. If performance degrades compared to the baseline in a subsequent test, the test will fail automatically.

Follow these steps:

  1. Go to the Performance tab, click Tests, and select a test from the drop-down list.
    The test reports window opens.
  2. Click the Configuration tab and scroll down to the Failure Criteria section.
  3. Toggle the button for Failure Criteria on.
  4. Check the box for Use from baseline.

    set failure criteria

  5. You can:
    • Set the threshold for each failure criteria from the selected baseline.
    • Define failure criteria as a percentage of the baseline result.
      If you use Baseline to define the threshold, the threshold will be visible after saving.

Subsequent tests are compared against the baseline with these failure criteria. If performance degrades compared to the baseline, the test will fail automatically.