Deprecating manual test cases

The document describes the whole process of deprecating manual test cases in the project, from the review of tests concerning the service in question to moving no longer useful test cases to Dead. The process consists of five steps, and they are the following:

Review of tests concerning a given service

In order to determine which manual test cases concerning a given service can be moved to Dead, one first has to perform a review of all tests, both manual and automated, concerning the service in question. An example of a such review is available e.g. at the Review of Requisition Regression Testing page. This work can be performed by one or more developers (as developers have access both to automated and manual tests) or by a developer and one or more testers, as the latter can analyze the test cases. A preferable solution is a cooperation between a tester and a developer.

First, one needs to verify what features are already covered with what type of tests, whether they are tested in a proper manner (i.e. whether they should be tested manually or automatically), whether they are the most important ones and related to the happy path (most of those are verified by the test cases with the CoreRegression label), or whether they are edge cases, and whether they need to be tested at all. Guidance on what features should be verified with what type of tests can be found in the Testing Guide and the Regression testing guide and improvements documents.

Then, one has to determine whether all features pertaining to a given service are covered with tests. This can be achieved by comparing the existing tests with the features available on all screens related to the service in question. Of course, in order to be able to check it, the familiarity with the service is indispensable. If any of them are missing, one needs to determine if they are essential features or edge cases, and how they should be tested.

Having gathered all these data, one has to create a list of manual test cases that can be merged, as well as those that can serve as a basis for automated tests. The above-mentioned page, concerning the Requisition service, can be perceived as an exemplar when performing the review for other services.

Finally, after the creation of a draft version of the review, one should hold a workshop with the rest of the team in order to discuss its results, and to consult them in case of any doubts, notably regarding whether a given feature should be tested at all and if so, how, if it is not clear even after reading the above-mentioned documents. After the workshop, the final version of the page with the review should be prepared.

Cleanup of tests concerning the service

After the review of the service’s test coverage, one has to implement its results, i.e.:

• move all test cases verifying features that aren’t vital to Dead;
• combine the manual test cases that can be merged;
• create the tickets for the creation of automated tests based on suitable test cases (i.e. functional tests);
• create the tickets for the creation of other automated tests (i.e. contract and integration tests).

Creation of automated tests based on test cases

The developer that is to create the functional test should base the scenarios contained in it on suitable manual test case(s) mentioned in the ticket concerning the creation of the automated test. Testers can also write the scenarios but usually, they are written by developers, as they have access to automated tests on a regular basis. Yet if the tester has no other tasks assigned, they can write the test steps and scenarios for the functional test. Anytime, they can assist the developer in the process of writing them.

Verification of the manual test case’s coverage

After the developer’s functional test passes code review, it should be assigned to the tester with the QA status. Thereupon, the tester has to compare the test steps in the functional test with those contained in the test case(s) on which the automated test was based. The test case(s) will be mentioned in the ticket concerning the creation of the functional test.

Moreover, the tester has to verify whether the automated test’s test steps meet the acceptance criteria, e.g. whether only users with suitable rights can approve the requisition that the functional test concerns. If the steps in both the automated and the manual test(s) match, and if the former meets all acceptance criteria, the tester should move the ticket concerning the creation of the functional test to Done and assign it to the developer that worked on it. When any of the acceptance criteria is not met, the tester has to write a comment informing the developer on the issue noticed. If the issue turns out to be a valid one, and e.g. not an outdated criterion or a mistake, the ticket has to be assigned to the developer with the In Progress status.

The same case is with missing test steps in the functional test that were verifying some vital features in the test case. In such a situation, the issue also has to be raised to the developer and depending on the result, as sometimes e.g. the missing significant feature is covered by another functional test, the ticket has to either be moved to In Progress or to Done.

In case of doubts whether something should be verified by means of a functional/manual test, or whether it needs to be verified at all, it’s advisable to consult the above-mentioned Testing Guide and the Regression testing guide and improvements documents.

Moving manual test cases to Dead

After analyzing the test steps contained in the functional test and those in the test case(s) on which the former were based, and determining that they match, the tester should move the latter to Dead and write a comment in the ticket with the test case that it was moved to Dead because a functional test covering the feature that the manual test was verifying had been created, including a link to the automated test in it. One also has to link the ticket with the test case to the one concerning the creation of the automated test. If the test case that was moved to Dead contains the CoreRegression label, one should remove it, so that the test cases that are no longer valid do not appear in test cycles concerning regular or release candidate regression testing, causing confusion.

Having moved the test case(s) on which the functional test was based to Dead, the tester should browse through the test cases concerning the same component in order to check whether any other manual tests can also be moved to Dead or whether some steps contained in them can be deleted, as they are now covered by the automated test. Sometimes, it is possible to delete test steps in other test cases than the one on which the functional test had been based, as some of them concern very frequent actions, such as e.g. requisition approval. Of course, the test steps can only be removed if the test case will still make sense after their removal. If not, the steps should be kept as they are, despite them being covered by a functional test.

After removing the steps, the tester has to write a comment in the ticket with the test case that e.g. the steps concerning requisition approval had been deleted because now they are covered by a functional test. The ticket’s status should normally be left as To Do, unless after the deletion of the test steps, the test case contains only steps that are already covered by other manual or automated tests. It may also happen that the remaining steps concern matters that do not need to be checked at all (e.g. they concern the verification of names of labels in a table or other elements of the UI), or that they concern actions that can be verified by another existing test case. In all of the aforementioned situations, the ticket with the test case should be moved to Dead; only in the latter case, one in addition has to move the remaining test steps to some other valid manual test and leave a comment in both tickets (one concerning the just moved to Dead test case, the other concerning the test to which the steps were moved), stating that the test case was moved to Dead and that the remaining test steps were moved to the other manual test and are now verified by it.