Here are few guidelines that can help you make a rational decision while selecting a test case for automation:
What to Automate?
- Repetitive / Important – Test cases those are repetitive & relatively unchanging. Check if the test case is going to be tested all the time / most of the time? – This also gives an indication of the importance of the test case. Don’t automate a test if you are only going to run it once and forget about it. While you decide to automate a test, take a step back and think how often you are likely to execute this test? Is it really worth the effort to automate the test?
- Stability – Check if the functionality to be automated is stable. If the module of the application is not stable enough, the effort in automation is wasted to an extent.
- Simplicity & Relevancy – In many cases your ROI will be better by implementing a lot of simple tests rather than few complicated tests so begin by doing a group of related simple tests and their related infrastructure. Choose a group that has more in common with others. It also includes scenarios that take a lot of effort and time when performing manual testing but are quick to automate and execute.
- Regression tests – Generally these are the ideal candidates for automation and available when the functionality has ‘matured’ & you have a safe level of confidence in the stability. For such candidates, automation is easier to develop & tests are more reliable (false failures are less probable).
- Deterministic – Scenarios that will yield pass/fail without false results, or – Scenarios that have standard results where you know that X input may only result in Y output. Choose the test cases with highly predictable conditions and results & that can provide a good measure of confidence in stability and quality
- Data driven scenarios – where same function needs to be validated with lots of different data such as using different user accounts or using different privileges and roles (e.g. login)
- Complexity – Tests that tend to cause human error, for e.g. a TC that has complex steps or calculations.
- High Risk – Frequently used functionality that introduces high risk conditions.
- Manual Execution Not Possible – Scenarios that are impossible to perform such as load testing scenarios
- Heterogeneous Configuration – Tests that run on several different hardware or software platforms and configurations.
- Risk based Approach – Use risk based approach. To create an automated script, it requires time and effort, and if aim for “Automating Every Test”, we will require lot of resource and time, which in many cases is not possible. Instead use a Risk based approach to determine which tests should be automated. To get the most value out of automation, only automate the most important business cases and scenarios.
What not to Automate?
- Test cases that must be validated subjectively by humans such as Usability.
- Complex Functionality where test automation may be a major challenge in terms of time & cost
- Don’t automate Test cases whose actions are distributed across multiple (third party) systems where validation is difficult to achieve while manual execution is faster and reliable to perform.
- Check if scenario requires human to watch and observe when the test is being run? Is human intervention mandatory or optional? If it requires human intervention then it is not an ideal candidate.
- It is better not to automate verification of presence/state/location/etc. of controls and static text.
- Look & Feel Scenarios – It’s very time-consuming and unreliable to try and “digitize” look-and-feel for verification. Scripts may check existing controls and text but what if something new appears depending on data condition? You can’t code for something you don’t know about yet.
- Another note to bear in mind is that not all tests can be automated. Some tests are very complex in nature and require many downstream system checking and can be inconsistent. In these cases, it is best to leave these checks for manual testing.