Jenkins administration tips & tricks

This page is intended to hold information about Jenkins administration. Each time a bigger problem is encountered, learnings should be listed here to avoid the same problems in the future.

EC2 nodes

• Do not attempt to create EC2 instances manually in AWS Console - EC2 Jenkins plugin creates them on its own
• Avoid using "provision via AWS" button to spawn new EC2 instances - in the past this caused issues starting them up automatically (preferred way is to just set the instance capacity and they will spawn when needed)
• Whenever updating an init script that stands up EC2 instance, always update it for all existing instances and also for the template (Manage Jenkins → Configure system → Amazon EC2 → AMIs)

Jenkins disk space

• Jenkins running out of space? Try clearing the workspace directory if there are no builds running on master. Sometimes the jobs cannot clear it after builds and this takes up space.
• Check what occupies most of the space with "ncdu /" (might take several minutes)
• If logs are taking too much space, check if log rotation is performed correctly
  • logs should be compressed
  • rotation frequency or file size should be adjusted to expected log volume - consult logrotate documentation for configuration adjustments - https://linux.die.net/man/8/logrotate
  • on instances with centos global config can be found in /etc/logrotate.conf individual configs are here /etc/logrotate.d
  • logrotate is started by cron once a day. If you need immediate results, run it manually
• Check space taken by docker with "docker system df" depending on the output, try to clean images, stopped containers or local volumes
  • When cleaning docker make sure no job is currently running in Jenkins
• Jenkins Jobs directory might grow big. It should not be cleared, as it contains builds history

Upgrading Jekins

1. Take a backup of the Jenkins directory - /var/lib/jenkins or take a snaphost of the whole volume (named Jenkins-snapshot) via the AWS console (Elastic Block Store → Snapshots→ Create snapshot)
2. Stop Jenkins (via web console or via terminal - service jenkins stop) - make sure no bulds are running first!
3. Get the latest Jenkins war (https://jenkins.io/download/) (LTS recommended) and replace it in /usr/lib/jenkins/jenkins.war
4. Sometimes some additional steps may be required - the upgrade guide will list them: https://jenkins.io/doc/upgrade-guide/
5. Start jenkins again - service jenkins start
6. Make sure that after a few minutes Jenkins is running at http://build.openlmis.org/ and that everything looks stable
7. In case of any problems you can consult the logs (Scalyr or /var/log/jenkins/jenkins.log - as superuser)

How to update Transifex API token for Jenkins build jobs - for core modules?

1. Create file with your credentials on your computer.
2. Within this file add TX_TOKEN with API Token obtained from Transifex app.
3. Go into Jenkins instance and sign in.
4. Select 'Manage Jenkins'.
5. Go into 'Security' tab and select 'Manage credentials'
6. Click 'Jenkins' under the Stores scoped to 'Jenkins' heading.
7. Click 'Global credentials (unrestricted)' under the System heading.
8. Click Add credentials.
9. Select type: 'Secret file', add 'id' of this credential, select scope: 'System (Jenkins and nodes only)', add description to indicate the purpose of this credential.
10. Update jenkinsfile in every UI modules in 'build' section. Please update 'id' of the credential.
11. Run Jenkins job responsible for particular building UI module to check if works.
12. See logs of Jenkins job - you should see information about push/pull operations.

How to update Transifex API token for Jenkins build jobs - for implementations?

1. Go into config repository of particular implementation.
2. Add your token as TX_TOKEN env variable into one of the particular files. It could be test.env or production.env - depends where you want to deploy changes: either test, dev or production one.
3. Update version of dev-ui module (>=9.0.5) in the same repository or in UI one (depends on implementations).
4. Run Jenkins job responsible for building UI app.
5. See logs of Jenkins job - you should see information about push/pull operations.