skip to Main Content

Since a couple of days ago, I’m getting some errors on my server. I use CentOS 6.5 with Parallels 12.0.18, Apache server to serve dynamic content and Nginx as proxy to serve static content.

At first, I was getting the following error:

[error] 29951#0: *5138862 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 89.7.24.108, server: , request: "GET /page/2/ HTTP/1.1", upstream: "http://ip:7080/page/2/", host: "domain.es", referrer: "http://domain.es/"

Then, I changed some configuration, like increasing MaxClients on my “httpd.conf” file and this lines to my /etc/nginx/conf.d/timeout.conf file:

proxy_connect_timeout       600;
proxy_send_timeout          600;
proxy_read_timeout          600;
send_timeout                600;

It all seemed to be working fine until I got the same errors again, along with a new one:

[error] 15228#0: *130292 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 89.130.25.154, server: domain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:7080/", host: "domain.com"

I have two different websites on the same server. That’s why you see two different hosts in there.

Here’s the problem: when I got these errors, I got a “502 Bad Gateway” and the server becomes so slow that I can’t even log in using the SSH terminal. I can only fix it temporally by resetting the httpd service.

I know there are other topics similar to this one, but all I found were problems with PHP-FPM, which I don’t use.

Here’s my Nginx configuration file:
user nginx;
worker_processes 16;

error_log  /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
server_names_hash_max_size 2048;
server_names_hash_bucket_size 512;

server_tokens off;

include    mime.types;
default_type  application/octet-stream;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout  10;

# Gzip on
gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_buffers 4 32k;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
gzip_disable "MSIE [1-6].";

# Other configurations
ignore_invalid_headers on;
client_max_body_size    8m;
client_header_timeout  3m;
client_body_timeout 3m;
#send_timeout     3m;
connection_pool_size  256;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
request_pool_size  4k;
output_buffers   4 32k;
postpone_output  1460;

# Cache most accessed static files
open_file_cache          max=10000 inactive=10m;
open_file_cache_valid    2m;
open_file_cache_min_uses 1;
open_file_cache_errors   on;

# virtual hosts includes
include "/etc/nginx/conf.d/*.conf";
}

Here’s my Nginx vhost file:
server {
listen ip:80 default_server;

server_name domain.es;
server_name www.domain.es;
server_name ipv4.domain.es;

client_max_body_size 128m;

root "/var/www/vhosts/domain.es/httpdocs";
access_log "/var/www/vhosts/system/domain.es/logs/proxy_access_log";
error_log "/var/www/vhosts/system/domain.es/logs/proxy_error_log";

if ($host ~* ^www.domain.es$) {
    rewrite ^(.*)$ http://domain.es$1 permanent;
}

location / {
    proxy_pass http://82.194.74.41:7080;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    access_log off;
}

location @fallback {
    proxy_pass http://ip:7080;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    access_log off;
}

location ~ ^/plesk-stat/ {
    proxy_pass http://ip:7080;
    proxy_set_header Host             $host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
    access_log off;
}

location ~ ^/(.*.(ac3|avi|bmp|bz2|css|cue|dat|doc|docx|dts|exe|flv|gif|gz|htm|html|ico|img|iso|jpeg|jpg|js|mkv|mp3|mp4|mpeg|mpg|ogg|pdf|png|ppt|pptx|qt|rar|rm|swf|tar|tgz|txt|wav|xls|xlsx|zip))$ {
    access_log off;
    expires 7d;
    add_header Cache-Control public;

 try_files $uri @fallback;
}

include "/var/www/vhosts/system/domain.es/conf/vhost_nginx.conf";
}

And some of the configuration vars I use with Apache (httpd.conf):

<IfModule prefork.c>
StartServers      14
MinSpareServers    8
MaxSpareServers   14
ServerLimit      1000
MaxClients       1000
MaxRequestsPerChild  2000
</IfModule>

Thank you very much in advance!

4

Answers


  1. It seems your Apache is more busy than your Nginx. When Nginx get some requests but Apache can’t handle, you get ‘502 Bad Gateway’, which meas Apache refused to work for Nginx.

    Try decrease ‘worker_connections’ and ‘worker_processes’ in Nginx and increase ‘MaxClients’, ‘ServerLimit’

    Make sure
    worker_connections * worker_processes < MaxClients < ServerLimit

    Login or Signup to reply.
  2. In my case, a php extension is missing, after I turned it off, it recovered! Check /var/log/messages to see if there is any segfault.

    Login or Signup to reply.
  3. In a case I saw recently this error was being logged for 95% of requests but both the web server and the apps server were almost idle with only a few connections open. The other 5% were successful.

    It turned out that our friendly networking colleagues had configured an IPS device between the web and apps server and it started clawing back and dropping connections when traffic increased because it thought there was a brute force attack happening. A false-positive. So nothing to do with Nginx or Apache, but reconfiguring the IPS fixed the issue.

    Login or Signup to reply.
  4. Trace your code to detect if you have any calls like "opendir" without "closedir".

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search