I have two apache servers setup on two separate physical machines. My current setup is:
Apache 1 (Reverse Proxy) <===> Apache 2
Both apache server versions are Apache/2.4.29 (Ubuntu)
running on Ubuntu 18.04.4 LTS
and their /etc/apache2/apache.conf
files are identical.
Apache 1 sites-enabled config:
<VirtualHost *:80>
ServerName subdomain.domain.tld
ServerAlias www.subdomain.domain.tld
ServerAdmin [email protected]
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyRequests off
ProxyPreserveHost On
ProxyPass /maintenance_page !
ProxyPass / http://[apache2-ip-address]:27300/
ProxyPassReverse / http://[apache2-ip-address]:27300/
</VirtualHost>
Apache 2 sites-enabled config:
<VirtualHost *:27300>
ServerName subdomain.domain.tld
ServerAlias www.subdomain.domain.tld
ServerAdmin [email protected]
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorDocument 400 /notfound.html
ProxyRequests off
ProxyPreserveHost on
</VirtualHost>
If I directly hit http://[apache2-ip-address]:27300/
from the web browser the apache server landing page comes up fine. If I enter http://subdomain.domain.tld
into the browser I get a proxy error:
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
I logged a trace on both apache servers. Apache server 2 is receiving the proxied request from apache server 1 and is returning a 200 status response perfectly fine to apache server 1. The flow breaks at apache server 1 where I am seeing the following logs:
[Sat Jul 11 20:34:08.671267 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(3075): AH00962: HTTP: connection complete to [apache2-ip-address]:27300 ([apache2-ip-address])
[Sat Jul 11 20:34:08.671333 2020] [core:trace6] [pid 32275:tid 140388069250816] core_filters.c(525): [remote [apache2-ip-address]:27300] core_output_filter: flushing because of FLUSH bucket
[Sat Jul 11 20:34:08.677508 2020] [proxy_http:error] [pid 32275:tid 140388069250816] (104)Connection reset by peer: [client xx.xxx.xxx.xx:39014] AH01102: error reading status line from remote server [apache2-ip-address]:27300
[Sat Jul 11 20:34:08.677575 2020] [proxy_http:debug] [pid 32275:tid 140388069250816] mod_proxy_http.c(1324): [client xx.xxx.xxx.xx:39014] AH01105: NOT Closing connection to client although reading from backend server [apache2-ip-address]:27300 failed.
[Sat Jul 11 20:34:08.677624 2020] [proxy:error] [pid 32275:tid 140388069250816] [client xx.xxx.xxx.xx:39014] AH00898: Error reading from remote server returned by /
[Sat Jul 11 20:34:08.677681 2020] [proxy:debug] [pid 32275:tid 140388069250816] proxy_util.c(2192): AH00943: HTTP: has released connection for ([apache2-ip-address])
[Sat Jul 11 20:34:08.677724 2020] [http:trace3] [pid 32275:tid 140388069250816] http_filters.c(1128): [client xx.xxx.xxx.xx:39014] Response sent with status 502, headers:
Things I’ve tried, from few other discussions I could find online, are the following changes to apache server 1 sites-enabled config :
SetEnv proxy-initial-not-pooled 1
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
ProxyTimeout 600
ProxyPass / http://[apache2-ip-address]:27300/ timeout=600
ProxyPass / http://[apache2-ip-address]:27300/ nocanon
I’ve pretty much bruteforced the situation with several combinations of the above settings, but nothing seems to work. Any help is appreciated.
An additional check I ran is, if I run a nodejs application or python flask service on the same machine as either apache servers and proxy the service using ProxyPass / http://localhost:[port]/
, the setup works properly. So both apache servers are running fine and are able to proxy services on their respective localhosts. Whatever is breaking has to do with the communication between the two apache servers.
UPDATE : Upon further triaging using curl
with a networking person, the issue seems to be that the org firewall is only allowing inbound traffic to apache server 2 and blocking outbound traffic which may be causing 502 errors on apache server 1. This didn’t seem like the issue up until I realized that my laptop was VPN’ed into the org network all along while testing and apache server 1 is sitting outside the org network. If this turns out to be the issue it’s going to be a real bummer.
2
Answers
Adding following parameter in the http.conf file solves my issue of "proxy: error reading status line from remote server":
I go the reference from Apache URL https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html
Note: restart http server and try again.
In my case, an error with the database’s connection was triggering this Apache’s Reverse Proxyng error.