Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Update worker node AMI and provision another Jenkins node

Description

Our build queue appears to be getting longer and longer which ties it up and makes developers wait. We should update our worker node AMI to be the latest instance and tools and then provision more worker nodes to drain the build queue faster.

Note: I don't think this will appreciably increase AWS costs as it's the same amount of work to be done, just more will be done in parallel. Unused worker nodes will shut off when they're not needed.

EC2 Plugin page: https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin

AC:

  • Update the worker node AMI to have the latest OS/tools and the latest instance (M5.large at time of writing).

  • Provision 2 more worker nodes with 2 executors each.

  • Turn the node off after 5 min of inactivity

  • Use the BillTo: OpenLMIS tag

  • Use the Type: BuildInf tag

  • Use the name: OpenLMIS-CI-Node OpenLMIS-CI-N (where N is some serial integer)

  • Don't use the word "slave" anywhere

Open Question

  • should we instead provision more nodes and limit them to 1 executor? This would be more expensive but we might see fewer random resource contention issues.

  • Should we use spot instances?

Confluence content

mentioned on

QAlity Plus - Test Management

Checklists

Activity

Mateusz Kwiatkowski 
July 11, 2018 at 8:52 PM

thanks, i think that is the last thing that we were concerned about, I think that if there will be no concerns with new nodes I will close this ticket Tomorrow.

Sebastian Brudziński 
July 11, 2018 at 3:01 PM

I now also added contract tests to the pipeline throttle category. We should no longer be building contract tests and backend pipeline on the same node, as it caused conflicts.

Sebastian Brudziński 
July 5, 2018 at 5:19 PM

please take a look whether all the nodes start up/shut down properly and if other acc are met (the only one that isn’t is the naming of the instance, but this is not a concern I believe)

Sebastian Brudziński 
June 21, 2018 at 11:12 AM

I think we can keep at least one node as is (so on-demand) and perhaps try to have another one as a spot instance.

Josh Zamor 
June 21, 2018 at 8:40 AM

Okay, lets start with 2 executers per node and we'll revise if we see resource contention leading to job failure.

I'm glad you found that article , it's the one I had in mind when I wrote the open question. Does this mean you have no concerns activating this feature of the plugin? I'd like to be sure we're not slowing builds down waiting for spot instances or losing control of our costs. Both seem unlikely from what I've seen so far. Do you agree?

Otherwise yes, this can be moved to todo.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Story Points

Original estimate

Time tracking

2d 3h logged1h remaining

Sprint

Fix versions

Priority

Parent

Time Assistant

Created June 12, 2018 at 5:56 PM
Updated July 13, 2018 at 8:02 AM
Resolved July 13, 2018 at 8:02 AM