It is 8:03. The daily prompt went out, and for three seconds there is more traffic than the two app servers can take. The extra requests are turned away, and the app gives up after 300 ms and retries at once, up to three tries.
The rush ends with an error for about one request in eight.
Workload
1 400 rps for most of the 15 seconds, jumping to 4 000 rps for three seconds, 90% reads / 10% writes.
Your move
The servers and the database are fixed. Select the client and change how it retries, then run again.