Question posted in 
I have an Ubuntu 22 VM running in GCP. I have about 850 legacy IIoT devices talking to the server via Apache to a php script. The keep alive is set to 10 1/2 minutes. The IIoT devices are talking at least every ten minutes. At the start of the php script, a call is made to GCP’s secret manager (via Google’s API). Every 10 minutes, (basically) every request to secret manager fails. The exception shows one of two failure messages.
cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for XXX
cURL error 55: Send failure: Broken pipe (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for XXX
Things I have tried:
I changes the keep-alive to 10 hours since the timing between the two was suspicious. It had no effect.
I read about an Ubuntu 22 issue with openssl 3.0.X, but I was under the impression that was a server side (GCP) error. I tried different libcurl versions (7.81.0-1ubuntu1.7, 7.81.0-1ubuntu1.14, 7.81.0-1ubuntu1.15, 8.5.0) with no effect.
I split the devices between two VMs (400 on one and 450 on the other). And the problem went away. Checking my GCP quota limits and I am not at even 1% usage. I do see higher request counts when the errors are occurring, but no where close to the 90k per minute GCP limit (https://cloud.google.com/secret-manager/quotas#request-rate-quotas). Picking a random 30 minute window:
12:00 782
12:01 1623
12:02 762
12:03 148
12:04 140
12:05 150
12:06 144
12:07 161
12:08 161
12:09 603
12:10 748
12:11 1648
12:12 762
12:13 216
12:14 133
12:15 107
12:16 112
12:17 89
12:18 71
12:19 624
12:20 784
12:21 1628
12:22 720
12:23 160
12:24 103
12:25 125
12:26 120
12:27 105
12:28 61
12:29 536
Edit: Here are the requests for this morning on the 450 server. There are higher request and there are no failures:
08:00 954
08:01 52
08:02 63
08:03 47
08:04 57
08:05 35
08:06 61
08:07 77
08:08 53
08:09 326
08:10 963
08:11 71
08:12 92
08:13 81
08:14 50
08:15 65
08:16 96
08:17 116
08:18 132
08:19 2242
08:20 2089
08:21 331
08:22 91
2
Answers
The error code "empty reply from server" (cURL error 52) indicates that the client received a zero-length response, meaning no HTTP headers or content, only a closed TCP connection with no HTTP payload. This issue is typically server-related and occurs when the libcurl library does not receive any response from the server, even after sending the request. It can be caused by various factors and this error should be approached from the server-side.
It is also important to differentiate between an "empty response" and "no response."
See this document to know more about curl (52) empty reply from server
Here’s another post related to curl error 52 empty reply from server with different causes and fixes
What is the curl error 52 "empty reply from server"?
According to the Google documentation for Idle Connections to GCE:
Google recommends the following to resolve this:
Set operating system TCP keep-alive parameters to a time frame of less than 10 minutes. This ensures that at least one packet is sent within the time frame.
Ensure applications that open TCP connections do so with the
SO_KEEPALIVEoption enabled.I am unsure why splitting the traffic between two VMs would resolve the issue. Perhaps after adding another VM, your devices are sending packets within that 10 min window.
If you have VPC Flow Logs enabled, you should be able to see more granular traffic to and from the instance.