In Perforce Autonomous Testing, policies are optional. Describe your failure criteria in natural language. For more information what metrics you can test, see Performance testing KPIs in BlazeMeter.
To write policies, either provide all criteria in your chat message, or start with what's important and fill in the details as part of the guided conversation. The assistant helps you write unambiguous and clear policies by asking clarifying questions.
Examples - Test your response time
What you can say to define global response time policies:
- We need to establish all connections in no more than 300 ms on average
- At least half the requests must come back in under 120 ms
- Only less than 5% of requests can take longer than 500 ms
- average latency below 300 ms
- The system needs to respond in less than 2 secs
Examples - Test your throughput or resource usage
What you can say to define global throughput policies:
- Fail a test that triggers over 1000 responses
- The app must be able to handle 150 successful requests per second
- Priority is to log over 50,000 successful hits/sec during the test
- Fail if something pushes over 1.2 MB per second in response data
- I expect data transfer at roughly 2 MB/sec in a successful test
For more information on policies, see Performance testing KPIs in BlazeMeter.
Example - Run test without policies
As a new user, you evaluate the need for policies:
User: I have no policies, can we just run the Login test?
Assistant:
Yes, you can absolutely run a test without defining any policies. Policies are optional and serve as an extra layer of validation for your tests. Without policies, your tests will still run and report if they were able to complete all the steps (like clicking buttons and navigating pages), but they won't automatically validate these other quality attributes.
User: To be fair I don't even know what policies are here
Assistant:
In Perforce Autonomous Testing, policies are collections of rules that your test results are measured against. They help you automatically check for things that go beyond simple pass/fail actions.
Do you want me to help you create your first policy with a few simple rules?
User: No, just run the Login test.
Example - Fulfill high-level requirements
Let the assistant guide you to fulfill project-wide test requirements.
User: How do I test whether all pages still load fast in a new build?
Assistant:
To make sure your pages load quickly, you can create a performance policy. This policy will define the performance rules for your application, such as page load times, API response times, and more.
I can help you create a "Page Load Performance Policy" with some standard rules to get you started, for example:
-
Page load time should be under 3 seconds
-
API response times should be less than 500ms
-
All images and resources should load in under 2 seconds
Do you want me to create the "Page Load Performance Policy" with these rules?
User: ok