CI

Spec

https://goo.gl/1uvs4a

Example

A typical pipeline for a service:

http://build.openlmis.org/view/Referencedata-pipeline/

How tos

How to run contract tests in CI?

How it’s configured in jenkins:

http://build.openlmis.org/view/Requisition-pipeline/job/OpenLMIS-requisition-contract-test/configure

Use this one script to run contract tests for different services by passing parameter:

https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/run_contract_tests.sh

Each contract test suite would need its own compose file to define which container is needed:

https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/docker-compose.requisition-service.yml

Use gradle waitFor -Pcontainers=x,y,z to wait for http server in containers before start testing:

https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/build.gradle#L36

The same way could be used by developers locally as well.

How to write contract tests?

Refer to this doc:

https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/README.md

How to configure deployment job to run only after all previous steps pass?

Take the requisition service deployment job for example:

http://build.openlmis.org/view/Requisitoin-pipeline/job/OpenLMIS-requisition-deploy-to-test/configure

It uses the “Fan-in” plugin to make sure the deployment job is only triggered after contract tests, erd generation and sonar checks all pass.

How is db credentials and other sensitive information handled during auto deployment?

Please refer to this doc:

https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/README.md

The content of the .env file for test and uat is identical with the default one for now, but they are not referring to the one in github, you can change them under “/var/lib/jenkins/credentials” if you have ssh access.

How to set up a Jenkins Slave machine?

The jenkins slave machine is initialized with this script:

http://build.openlmis.org/configure

(at the bottom of the page, in the “init script” text box)

This script will be ran to create a new EC2 instance when “Provision via AWS” is clicked on this page: http://build.openlmis.org/computer/

Or when there are more jobs in the queue than the current executors can handle.

The slaves auto launched by Jenkins are configured to have 32G disks, if that becomes too small in the future, this option can be changed in “Block device mapping” field in this page http://build.openlmis.org/configure

How to publish erd documents?

ERD documents are published as a zip file in the Jenkin job’s artifact, and also published to apache web server for online browsing.

Take the auth service for example.

This is the job configuration:

http://build.openlmis.org/job/OpenLMIS-auth-erd-generation/configure

The latest zip file: http://build.openlmis.org/job/OpenLMIS-auth-erd-generation/lastSuccessfulBuild/artifact/erd-auth.zip

The online browsing url:

http://ci.openlmis.org/erd-auth/

These urls follow a pattern, if you substitute the word auth with another service’s name, the url will point to the generated erd documents of that service.

Note: in the erd generation job’s configuration, the “Restrict where this project can be run” option is checked. This is because the erd generation jobs produce external artifacts, which is the files under “/var/www/html”. The Jenkins slave machines have no access to that, so the erd generation jobs are only allowed to run on the master node of Jenkins.

There is a new ticket working on publishing docs: https://openlmis.atlassian.net/browse/OLMIS-673

This may be replaced by that, please refer to the ticket for latest information.

How to Version your component and publish to Docker Hub

Versioning a component is the responsibility of the component. This version should be set in a revision controlled file at or near the root of your repository. It's recommended that a Java Properties format be used for this version file as that's what CI can use. If you'd prefer to generate this property file as part of the build that's also acceptable. It's strongly recommended that the versioning scheme used for the component follows the versioning scheme for the project.

On the CI server, the main build job of a component is responsible for building and publishing. In order to publish your component under this version, configure the build job:

add a step before publishing to DockerHub and after the build that injects environment variables from the version property file for your component
use the environment variable that will be created from that property file that is the version as the tag in the DockerHub publish step.

Example: Requisition Service build job using gradle.properties (which has serviceVersion property) injecting ${serviceVersion} and tagging this in DockerHub.

CD

Spec

https://goo.gl/Wl3Md6

(Revised)

Example

http://elb-test-env-swarm-683069932.us-east-1.elb.amazonaws.com:8888/auth/

https://elb-test-env-swarm-683069932.us-east-1.elb.amazonaws.com/auth/

(Test env)

http://uat-env-elb-1284865908.us-east-1.elb.amazonaws.com/auth/

https://uat-env-elb-1284865908.us-east-1.elb.amazonaws.com/auth/

(UAT env)

How tos

How to Provision a deployment environment?

There are three docs under this directory:

https://github.com/OpenLMIS/openlmis-deployment/tree/master/provision

They cover provision for single host or swarm, with EIP or ELB.

How to configure deployment job?

How it’s configured in jenkins:

http://build.openlmis.org/view/Requisition-pipeline/job/OpenLMIS-requisition-deploy-to-test/configure

Note that deployment jobs need to be run on the master/jenkins server. This is due to a security plugin we're using that will help keep the hosts secure. See OLMIS-1226 - Getting issue details... STATUS for more information.

Use this one script to re-deploy different services by passing parameter:

https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/test_env/deploy_to_test_env.sh

(Apart from these two links, also refer to the provision docs for how to enable jenkins to access the remote deployment target machine)

How to wipe data during redeployment?

Go to: http://build.openlmis.org/job/OpenLMIS-notification-deploy-to-test/build?delay=0sec

(example of redeploying notification service, there are jobs for redeploying other services as well)

By default, when the deployment job is triggered automatically by a upstream job, it keeps the data. Wiping data is done by manually trigger the job, as the image above show, you can choose the “wipe” option to do so.

The above example is for test env, it’s the same thing for UAT env as well.

How to backup kept data?

Please refer to this doc:

https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/README.md#how-to-backup-persisted-data

How to deploy a versioned component

Jenkins uses the docker-compose files to know which components/services to deploy as well as which versions of those to use from DockerHub. Once your component is versioned and tagged as such in DockerHub, you may use that version in the deployment's docker-compose.

Currently there are three deployment files:

Blue, used by developers and the official reference distribution file. The other 2 files follow this one. https://github.com/OpenLMIS/openlmis-blue/blob/master/docker-compose.yml
Test, the test environment that deploys test.openlmis.org. https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/test_env/docker-compose.yml
UAT, the UAT environment that deploys uat.openlmis.org. https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/uat_env/docker-compose.yml

Deploying a version in this file looks like:

...
requisition:
  image:openlmis/requisition:3.0.0-beta-SNAPSHOT
...

Note at the top of a docker-compose file is a "version: 2" statement. This version describes the version of the docker-compose format to use and is not an OpenLMIS version. For more information see the Docker Compose File documentation.

How demo data is automatically deployed to the Test and UAT environments

The docker-compose files are also where demo data is automatically loaded by each service when it starts up. Each service has its own demo data (if it exists) stored in its Docker image, under the /demo-data directory. To have the service load the demo data, the JAVA_OPTS environment variable should include a setting for spring.jpa.properties.hibernate.hbm2ddl.import_files, which points to the demo-data.sql file. Below is an example of what the deployment files would contain:

...
requisition:
  environment:
    JAVA_OPTS: '-Dlogging.config=/logback.xml -Dspring.jpa.properties.hibernate.hbm2ddl.import_files=/bootstrap.sql,file:///demo-data/data.sql'
...

If this setting provides a list of sql files, those files will be executed right after the Hibernate auto-generation of the DDL, in order they are specified. Note, there is an entry for bootstrap.sql, which is the loading of bootstrap data. If you are changing this setting for a service, make sure the bootstrap.sql is still specified, and specified first.

Note: once the CI/CD process moves away from having Hibernate auto-generate the DDL, this demo data auto-load process will need to be changed as well. In that case, bootstrap data will be part of the migration scripts, while seed and demo data will be loaded using a different setting.

Even though the *deploy-to-test and *deploy-to-uat* Jenkins jobs trigger the automatic loading of demo data to the Test and UAT environments, the configuration is not in the Jenkins jobs themselves. The configuration can be found in the openlmis-deployment GitHub repo (https://github.com/OpenLMIS/openlmis-deployment.git), under deployment.

The Reference Distribution (blue) is designed to load in the demo data that is stored in JSON and built within each service. For more, see the OpenLMIS-Blue README and its utils/demo-data.sh script.