CI/CD documentation index
- Pengfei Cui (Deactivated)
- Brandon Bowersox-Johnson
- Chongsun Ahn (Unlicensed)
CI
Spec
Example
A typical pipeline for a service:
http://build.openlmis.org/view/Referencedata-pipeline/
How tos
How to run contract tests in CI?
How it’s configured in jenkins:
http://build.openlmis.org/view/Requisition-pipeline/job/OpenLMIS-requisition-contract-test/configure
Use this one script to run contract tests for different services by passing parameter:
https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/run_contract_tests.sh
Each contract test suite would need its own compose file to define which container is needed:
Use gradle waitFor -Pcontainers=x,y,z to wait for http server in containers before start testing:
https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/build.gradle#L36
The same way could be used by developers locally as well.
How to write contract tests?
Refer to this doc:
https://github.com/OpenLMIS/openlmis-contract-tests/blob/master/README.md
How to configure deployment job to run only after all previous steps pass?
Take the requisition service deployment job for example:
It uses the “Fan-in” plugin to make sure the deployment job is only triggered after contract tests, erd generation and sonar checks all pass.
How is db credentials and other sensitive information handled during auto deployment?
Please refer to this doc:
https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/README.md
The content of the .env file for test and uat is identical with the default one for now, but they are not referring to the one in github, you can change them under “/var/lib/jenkins/credentials” if you have ssh access.
How to set up a Jenkins Slave machine?
The jenkins slave machine is initialized with this script:
http://build.openlmis.org/configure
(at the bottom of the page, in the “init script” text box)
This script will be ran to create a new EC2 instance when “Provision via AWS” is clicked on this page: http://build.openlmis.org/computer/
Or when there are more jobs in the queue than the current executors can handle.
The slaves auto launched by Jenkins are configured to have 32G disks, if that becomes too small in the future, this option can be changed in “Block device mapping” field in this page http://build.openlmis.org/configure
How to publish erd documents?
ERD documents are published as a zip file in the Jenkin job’s artifact, and also published to apache web server for online browsing.
Take the auth service for example.
This is the job configuration:
http://build.openlmis.org/job/OpenLMIS-auth-erd-generation/configure
The latest zip file: http://build.openlmis.org/job/OpenLMIS-auth-erd-generation/lastSuccessfulBuild/artifact/erd-auth.zip
The online browsing url:
http://ci.openlmis.org/erd-auth/
These urls follow a pattern, if you substitute the word auth with another service’s name, the url will point to the generated erd documents of that service.
Note: in the erd generation job’s configuration, the “Restrict where this project can be run” option is checked. This is because the erd generation jobs produce external artifacts, which is the files under “/var/www/html”. The Jenkins slave machines have no access to that, so the erd generation jobs are only allowed to run on the master node of Jenkins.
There is a new ticket working on publishing docs: https://openlmis.atlassian.net/browse/OLMIS-673
This may be replaced by that, please refer to the ticket for latest information.
How to Version your component and publish to Docker Hub
Versioning a component is the responsibility of the component. This version should be set in a revision controlled file at or near the root of your repository. It's recommended that a Java Properties format be used for this version file as that's what CI can use. If you'd prefer to generate this property file as part of the build that's also acceptable. It's strongly recommended that the versioning scheme used for the component follows the versioning scheme for the project.
On the CI server, the main build job of a component is responsible for building and publishing. In order to publish your component under this version, configure the build job:
- add a step before publishing to DockerHub and after the build that injects environment variables from the version property file for your component
- use the environment variable that will be created from that property file that is the version as the tag in the DockerHub publish step.
Example: Requisition Service build job using gradle.properties (which has serviceVersion property) injecting ${serviceVersion} and tagging this in DockerHub.
CD
Spec
(Revised)
Example
http://elb-test-env-swarm-683069932.us-east-1.elb.amazonaws.com:8888/auth/
https://elb-test-env-swarm-683069932.us-east-1.elb.amazonaws.com/auth/
(Test env)
http://uat-env-elb-1284865908.us-east-1.elb.amazonaws.com/auth/
https://uat-env-elb-1284865908.us-east-1.elb.amazonaws.com/auth/
(UAT env)
How tos
How to Provision a deployment environment?
There are three docs under this directory:
https://github.com/OpenLMIS/openlmis-deployment/tree/master/provision
They cover provision for single host or swarm, with EIP or ELB.
How to configure deployment job?
How it’s configured in jenkins:
Note that deployment jobs need to be run on the master/jenkins server. This is due to a security plugin we're using that will help keep the hosts secure. See
OLMIS-1226
-
Getting issue details...
STATUS
for more information.
Use this one script to re-deploy different services by passing parameter:
(Apart from these two links, also refer to the provision docs for how to enable jenkins to access the remote deployment target machine)
How to wipe data during redeployment?
Go to: http://build.openlmis.org/job/OpenLMIS-notification-deploy-to-test/build?delay=0sec
(example of redeploying notification service, there are jobs for redeploying other services as well)
By default, when the deployment job is triggered automatically by a upstream job, it keeps the data. Wiping data is done by manually trigger the job, as the image above show, you can choose the “wipe” option to do so.
The above example is for test env, it’s the same thing for UAT env as well.
How to backup kept data?
Please refer to this doc:
How to deploy a versioned component
Jenkins uses the docker-compose files to know which components/services to deploy as well as which versions of those to use from DockerHub. Once your component is versioned and tagged as such in DockerHub, you may use that version in the deployment's docker-compose.
Currently there are three deployment files:
- Blue, used by developers and the official reference distribution file. The other 2 files follow this one. https://github.com/OpenLMIS/openlmis-blue/blob/master/docker-compose.yml
- Test, the test environment that deploys test.openlmis.org. https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/test_env/docker-compose.yml
- UAT, the UAT environment that deploys uat.openlmis.org. https://github.com/OpenLMIS/openlmis-deployment/blob/master/deployment/uat_env/docker-compose.yml
Deploying a version in this file looks like:
... requisition: image:openlmis/requisition:3.0.0-beta-SNAPSHOT ...
Note at the top of a docker-compose file is a "version: 2" statement. This version describes the version of the docker-compose format to use and is not an OpenLMIS version. For more information see the Docker Compose File documentation.
How demo data is automatically deployed to the Test and UAT environments
The docker-compose files are also where demo data is automatically loaded by each service when it starts up. Each service has its own demo data (if it exists) stored in its Docker image, under the /demo-data directory. To have the service load the demo data, the JAVA_OPTS environment variable should include a setting for spring.jpa.properties.hibernate.hbm2ddl.import_files, which points to the demo-data.sql file. Below is an example of what the deployment files would contain:
...
requisition:
environment:
JAVA_OPTS: '-Dlogging.config=/logback.xml -Dspring.jpa.properties.hibernate.hbm2ddl.import_files=/bootstrap.sql,file:///demo-data/data.sql'
...
If this setting provides a list of sql files, those files will be executed right after the Hibernate auto-generation of the DDL, in order they are specified. Note, there is an entry for bootstrap.sql, which is the loading of bootstrap data. If you are changing this setting for a service, make sure the bootstrap.sql is still specified, and specified first.
Note: once the CI/CD process moves away from having Hibernate auto-generate the DDL, this demo data auto-load process will need to be changed as well. In that case, bootstrap data will be part of the migration scripts, while seed and demo data will be loaded using a different setting.
Even though the *deploy-to-test and *deploy-to-uat* Jenkins jobs trigger the automatic loading of demo data to the Test and UAT environments, the configuration is not in the Jenkins jobs themselves. The configuration can be found in the openlmis-deployment GitHub repo (https://github.com/OpenLMIS/openlmis-deployment.git), under deployment.
The Reference Distribution (blue) is designed to load in the demo data that is stored in JSON and built within each service. For more, see the OpenLMIS-Blue README and its utils/demo-data.sh script.
Monitors
Monitor for docker hosts(the EC2 instances):
(Test env)
(UAT env)
Monitor for containers
http://uat-env-elb-1284865908.us-east-1.elb.amazonaws.com:3000/dashboard/db/test-docker-host
http://uat-env-elb-1284865908.us-east-1.elb.amazonaws.com:3000/dashboard/db/uat-docker-host
There is a read only user, name: view_only, password: Nn24AQnG8Nxwd7oj
If you need admin credentials, see the credentials part of this doc.
How to set up docker container monitor?
Please refer to this doc: https://github.com/OpenLMIS/openlmis-deployment/blob/master/monitoring/README.md
And use the one liner script to set up.
Credentials
Items listed below have been sent to Josh:
Pem key for jenkins slaves
Pem key for test and uat env
Docker certs for remotely deployment
Password for docker monitor
Repositories
For both contract test repo https://github.com/OpenLMIS/openlmis-contract-tests and deployment repo https://github.com/OpenLMIS/openlmis-deployment, the team Developers and Soldevelo have been given write access.
The team called owners-oldgithubperm has been given admin access.
OpenLMIS: the global initiative for powerful LMIS software