Are we using demo-data for contract tests?
Background
We want to improve the process of writing contract tests to have more valuable contract tests. We also don't want to compromise doing demos of the system. The way contract tests load data must be improved in order to make them easier to extend and maintain - if we decide on not using demo data for contract tests, we will still have to think up with an alternative in order to have the contract tests more valuable.
Current state
Currently contract tests use their own set of data on which contract tests are run. This includes users, role assignments, facilities, requisition groups, supervisory nodes etc. This data is currently stored as hard to main sql data files (Link to GitHub directory). Editing these files is hard, coming up with extensions to them is hard, reading them is hard. Because of this, creating contract tests is a painful chore and makes writing a contract a task on itself - unlike writing unit/integration tests is pretty much perceived as part of the original task. In order to make our contract tests easier to extend going forward, with the hope that this will increase our coverage, we must take some steps to sanitize this data and make it more manageable.
A fairly easy to achieve way of doing this is starting to use our regular demo data as the base for contract testing. There are several pros to this, however care must be taken in order to not create undesired effects, such as causing issues to people giving demos of OpenLMIS.
Test data is reloaded before each scenario.
Pros & Cons
- Known set of data - developers are more familiar with it.
- You can always review the test data on our test server - you don't have look through data files or spin up the contract tests us a distro
- Demo data is developed along features - you won't have to create your set up for contract tests you are writing, which should allow you to focus on the more important part of the test
- Demo data is reviewed by more people including business stakeholders - this is data is closer to real use cases then what the devs come up with for their tests
- Demo data will be used by contract tests - broken/invalid demo data will be far less likely and can get caught in contract-tests
- Since the data is duplicated, there is less chance that contract test can catch something that does not come in regular testing
- This can prompt devs to make changes to the demo data for the sake of contract tests and might impact demos - we should limit such changes
- Adding demo data can backfire in contract tests
LOE
Switching contract tests to use demo-data should be easy - some changes need to be made to compose files in order to load the demo data when starting and the code that is currently responsible for loading the contract test data. The bigger part of this task will be going through our existing contract tests and switching them to use the existing data (primarily ID switching). There shouldn't be an issue with data missing since demo data is more robust and provides the more complex hierarchies and users - the tests will have to be adjusted to use the new data.
Currently data in contract tests is reloaded before each scenario runs - this is something we probably would want to continue having in order to run tests in isolation. With how we currently load demo data, this might not be obvious. However here is a possible solution that is easy to achieve - after the distro is ready, but before running the first test, take a snapshot (dump) of the database. Then revert to that database before each test scenario. That way demo data lives in the services, but contract tests keep reloading it.
To summarize - we will need to create two tickets: one for switching to demo data and handling data reloads, the second one to rewrite actual contract tests to use new ids.
Guidelines
- Avoid adding demo data specifically for the sake of the contract tests. Contract tests should create their own objects to play with, such as requisitions. Demo data should considered more a foundation to work with, providing a working set of primarily ref data.
- First of all limit any changes to demo data in contract tests, but if that happens, make sure that these changes are not retained for other tests. Ideally we will have a solution in place to guarantee this.
Action items
OpenLMIS: the global initiative for powerful LMIS software