This page is intended to hold information about Jenkins administration. Each time a bigger problem is encountered, learnings should be listed here to avoid the same problems in the future.
EC2 nodes
- Do not attempt to create EC2 instances manually in AWS Console - EC2 Jenkins plugin creates them on its own
- Avoid using "provision via AWS" button to spawn new EC2 instances - in the past this caused issues starting them up automatically (preferred way is to just set the instance capacity and they will spawn when needed)
- Whenever updating an init script that stands up EC2 instance, always update it for all existing instances and also for the template (Manage Jenkins → Configure system → Amazon EC2 → AMIs)
Jenkins disk space
- Jenkins running out of space? Try clearing the workspace directory if there are no builds running on master. Sometimes the jobs cannot clear it after builds and this takes up space.
Upgrading Jekins
- Take a backup of the Jenkins directory - /var/lib/jenkins or take a snaphost of the whole volume (named Jenkins-snapshot) via the AWS console (Elastic Block Store → Snapshots→ Create snapshot)
- Stop Jenkins (via web console or via terminal - service jenkins stop) - make sure no bulds are running first!
- Get the latest Jenkins war (https://jenkins.io/download/) (LTS recommended) and replace it in /usr/lib/jenkins/jenkins.war
- Sometimes some additional steps may be required - the upgrade guide will list them: https://jenkins.io/doc/upgrade-guide/
- Start jenkins again - service jenkins start
- Make sure that after a few minutes Jenkins is running at http://build.openlmis.org/ and that everything looks stable
- In case of any problems you can consult the logs (Scalyr or /var/log/jenkins/jenkins.log - as superuser)