Nginx latency

Description

There is some latency introduced, under any sort of load, between Nginx and back-end services.

Want to time box this to... a day.

Activity

Show:
Brandon Bowersox-Johnson
August 8, 2017, 12:27 AM
Edited

I just reviewed the code changes. It appears the main work here is adding Keep-Alive support to speed up NGINX (reduce latency). I also see that a change was made to allow access log buffering. (Update: See comments below for corrections and additions to this...)

Josh Zamor
August 8, 2017, 12:30 AM

Actually the biggest increase likely came from tcp_nodelay. The others are useful, but not something that's making huge improvements atm.

Josh Zamor
August 8, 2017, 12:32 AM

While looking into this, I discovered that there is a seperate performance issue that occurs due to ephemeral port exhaustion. Given that this is not the original latency issue, and the issue only crops up in specific stress tests I've tried to change the test to avoid it and filed a ticket:

Josh Zamor
August 8, 2017, 12:38 AM

Docker hub built nginx v4, bumped deployments and ref distro. Moving back to review for ticket.

Josh Zamor
August 8, 2017, 3:45 PM

A note on what appears to have been the delay, and what the solution is.

The ~0.2 latency introduced on many of the tests (preventing us from writing AC much lower than this) was not having an Nginx setup with TCP_NODELAY on. This article explains what we now have (sendfile, tcp_nodelay and tcp_nopush in our nginx.conf): https://t37.net/nginx-optimization-understanding-sendfile-tcp_nodelay-and-tcp_nopush.html

At most Nginx now seems to be incurring about ~40-60ms latency, however this may be an inaccuracy of reporting from the Spring Boot side, and so this time may actually be spent in the Service and not network.

Done

Assignee

Josh Zamor

Reporter

Josh Zamor

Labels

Components

Sprint

None

Fix versions

Priority

Minor