Overview

Currently we have a single migration test that verify if all migrations from all services were written correctly. We could see that this test did not catch some issues in migrations:

The following document will try to show how the migration tests could be written inside the given service to make sure that all migrations wrote by developers are correct.

Purpose

Each OpenLMIS service that use database to store data use Flyway as a database version control tool. Each time when database need to be changed a developer write a migration to modify the database to needed shape. When a service starts up Flyway verify the current version of database and applies only missing migrations. Thanks that we don't have to worry about that some migrations will be applied several times on the same database. Currently all of our migrations are loaded on empty database so all of them are correct. The problems show up when implementators try to migrate from the current OpenLMIS to the newest version. From time to time we get a information that they have issues related with migrations. The purpose is to find a good way to make sure that migrations are written correctly and they will work on the production.

Idea

Each service will contain additional step in the build process that will execute migration/flyway tests. Those tests should not be executed together with integration tests because the purpose of tests are different. Also there could be some issues that the database is recreated in migration tests. A configuration for the step can be found below. All tests need to be in the src/integration-test directory in the org.openlmis.{service_name}.migration package.

task flywayTest(type: Test) {
    testClassesDir = sourceSets.integrationTest.output.classesDir
    classpath = sourceSets.integrationTest.runtimeClasspath
    testLogging {
        events "passed", "skipped", "failed"
        exceptionFormat = 'full'
    }

    include '**/migration/**'
}

task integrationTest(type: Test) {
    testClassesDir = sourceSets.integrationTest.output.classesDir
    classpath = sourceSets.integrationTest.runtimeClasspath
    testLogging {
        events "passed", "skipped", "failed"
        exceptionFormat = 'full'
    }
    mustRunAfter test
    environment 'BASE_URL', "http://localhost"

    exclude '**/migration/**'
}

How write tests?

Those tests can be probably written is several ways but here we will describe two of them:

• single test for all migrations
• a single test for a migration or small set of migrations

Single test for all migrations

In this appoach we would have a single test that verify migrations are correct. When a developer add a new migration, he/she will need to update the existing test to handle new migrations. Advantages of this approach is that we have a single test class and in common cases it would be easy to add a new migration check - we would probably need to add a check to a list of checks and everything else will be done automatically. Unfortuently the list of disadvantages contains things like breaking the rule of single responsibility or after some time it would be hard to understand the test because number of steps and checks will be large. For example in the reference data service we have about 79 migrations and this would be the number of steps and checks in the test. Also becuase there will be single test, it has to be written in way that would support future extensions because in some cases the developer will need additional check or before migration step and adding it to the exising test could take some time and issues.

A single test for a migration or small set of migrations

In this approach we would have a single test for single migration or small set of migrations that are close to each other. When a developer add a new migration(s), he/she will need to create a new test - and probably extends it by base version to not create a code duplication. Advantages of this approach is that each test represents a single migration, it is separate from other tests so we don't have to worry that one test will break something in another one. This approach respects single resposibility rule and creating custom tests should be easy - that does not match default pattern. Disadvantages contain things like a lot of test classes, more difficult to find a proper test class name - we can't name it ResourceMigrationTest because what if in the future there will be additional migrations for the given resource?

Examples of service migration tests can be found under this link.

How create/get data for a test?

To verify that migrations are written correctly we need data on which those migrations will be executed. The data should be created in the way that they contain a typical and edge cases. For example if we copy values from one table to another by some field (where part of SQL statement) we should make sure that the field will contain different data. 

Additional data migrations

In this approach data are pushed to database by additional migrations defined for tests. Those migrations will not be present in the final JAR file. In this case it is easy to add a lot of data because we simple create a INSERT statements. Unfortunetly, those migrations will be threated in the same way as standard migrations so if in some place we put data, those data will be visible by all next migrations. Another issue can be with database contraint for example as a someone who create data/test I don't have to know that a resource with the given field already exist in the database and when I put exactly the same resource I could get an error message because of unique constraint. Tests will be harder to read because on the first look a developer will not know from where data are and why some verification checks are written in the given way.

Data generated by tests

In this approach data are generated in test before the test migration will be applied to database. Thanks this tests are easier to read because a developer will know what data are pushed to database and why some verification checks are written in the given way. Unfortunetly, in this case we will need to create additional classes/methods to put data into database. Also we can't use domain classes because they represent the final version of database and we need data in certain structure.