There is some latency introduced, under any sort of load, between Nginx and back-end services.
Want to time box this to... a day.
I just reviewed the code changes. It appears the main work here is adding Keep-Alive support to speed up NGINX (reduce latency). I also see that a change was made to allow access log buffering. (Update: See comments below for corrections and additions to this...)
Actually the biggest increase likely came from tcp_nodelay. The others are useful, but not something that's making huge improvements atm.
While looking into this, I discovered that there is a seperate performance issue that occurs due to ephemeral port exhaustion. Given that this is not the original latency issue, and the issue only crops up in specific stress tests I've tried to change the test to avoid it and filed a ticket:
Docker hub built nginx v4, bumped deployments and ref distro. Moving back to review for ticket.
A note on what appears to have been the delay, and what the solution is.
The ~0.2 latency introduced on many of the tests (preventing us from writing AC much lower than this) was not having an Nginx setup with TCP_NODELAY on. This article explains what we now have (sendfile, tcp_nodelay and tcp_nopush in our nginx.conf): https://t37.net/nginx-optimization-understanding-sendfile-tcp_nodelay-and-tcp_nopush.html
At most Nginx now seems to be incurring about ~40-60ms latency, however this may be an inaccuracy of reporting from the Spring Boot side, and so this time may actually be spent in the Service and not network.