Rails + Nginx + Certbot: 422 Errors on Logins

Chakron
November 26, 2021
311 views
0 votes
2 Answers

I have a Rails 6 website running on Elastic Beanstalk (Amazon Linux 2). I successfully implemented a process to use Certbot to generate an SSL certificate, and when I visit my website everything is working correctly. However, when I try to log in to my user console (using Devise), I receive 422 errors.

Rails Production Log

W, [2021-11-26T17:55:17.528942 #22645]  WARN -- : [6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] HTTP Origin header (https://example.com) didn't match request.base_url (http://example.com)
I, [2021-11-26T17:55:17.529316 #22645]  INFO -- : [6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Allocations: 565)
F, [2021-11-26T17:55:17.530195 #22645] FATAL -- : [6ef6bfd6-6d78-4ded-90df-a9472e0d40f6]
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6]
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/request_forgery_protection.rb:211:in `handle_unverified_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/request_forgery_protection.rb:243:in `handle_unverified_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] devise (4.8.0) lib/devise/controllers/helpers.rb:255:in `handle_unverified_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/request_forgery_protection.rb:238:in `verify_authenticity_token'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:427:in `block in make_lambda'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:199:in `block in halting'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:512:in `block in invoke_before'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:512:in `each'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:512:in `invoke_before'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:115:in `block in run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actiontext (6.1.4.1) lib/action_text/rendering.rb:20:in `with_renderer'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actiontext (6.1.4.1) lib/action_text/engine.rb:59:in `block (4 levels) in <class:Engine>'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:126:in `instance_exec'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:126:in `block in run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:137:in `run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/abstract_controller/callbacks.rb:41:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/rescue.rb:22:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/notifications.rb:203:in `block in instrument'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/notifications.rb:203:in `instrument'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/instrumentation.rb:33:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activerecord (6.1.4.1) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/abstract_controller/base.rb:165:in `process'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionview (6.1.4.1) lib/action_view/rendering.rb:39:in `process'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal.rb:190:in `dispatch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal.rb:254:in `dispatch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/route_set.rb:33:in `serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/mapper.rb:49:in `serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/journey/router.rb:50:in `block in serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/journey/router.rb:32:in `each'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/journey/router.rb:32:in `serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/route_set.rb:842:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] warden (1.2.9) lib/warden/manager.rb:34:in `catch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] warden (1.2.9) lib/warden/manager.rb:34:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/etag.rb:27:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/conditional_get.rb:40:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/head.rb:12:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/cookies.rb:689:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:98:in `run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/rack/logger.rb:37:in `call_app'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/rack/logger.rb:26:in `block in call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/tagged_logging.rb:99:in `block in tagged'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/tagged_logging.rb:37:in `tagged'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/tagged_logging.rb:99:in `tagged'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/rack/logger.rb:26:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/request_id.rb:26:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/method_override.rb:24:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/runtime.rb:22:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/executor.rb:14:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/host_authorization.rb:92:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/engine.rb:539:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/configuration.rb:249:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/request.rb:77:in `block in handle_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/thread_pool.rb:340:in `with_force_shutdown'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/request.rb:76:in `handle_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/server.rb:447:in `process_client'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/thread_pool.rb:147:in `block in spawn_thread'

Puma Log (most recent lines, nothing relevant in here as far as I can tell)

[22565] - Worker 0 (PID: 22643) booted in 5.67s, phase: 0
[22565] - Worker 1 (PID: 22645) booted in 5.68s, phase: 0

Nginx Access Log

99.83.42.176 - - [26/Nov/2021:17:55:10 +0000] "GET /users/sign_in HTTP/1.1" 200 7754 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" "-"
99.83.42.176 - - [26/Nov/2021:17:55:17 +0000] "POST /users/sign_in HTTP/1.1" 422 0 "https://example.com/users/sign_in" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" "-"

Nginx Error Log

2021/11/26 17:54:57 [notice] 22658#22658: signal process started
2021/11/26 17:54:57 [warn] 22626#22626: conflicting server name "localhost" on 0.0.0.0:80, ignored
2021/11/26 17:54:57 [warn] 22626#22626: conflicting server name "_" on 0.0.0.0:80, ignored

I’ve messed with a few options in nginx.conf but still haven’t been able to track down why this is happening. This is my nginx.conf file, with the certbot-embedded changes:

#Elastic Beanstalk Nginx Configuration File

user                    nginx;
error_log               /var/log/nginx/error.log warn;
pid                     /var/run/nginx.pid;
worker_processes        auto;
worker_rlimit_nofile    65874;

events {
    worker_connections  1024;
}

http {

    upstream appserver {
        server unix:/var/run/puma/my_app.sock;
    }

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    include       conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
        default     "upgrade";
    }

    server {
        listen 80;
        server_name www.example.com example.com localhost _;
        return 301 https://$host$request_uri;
    }

    server {
        client_header_timeout 60;
        client_body_timeout   60;
        keepalive_timeout     60;
        gzip                  on;
        gzip_comp_level       4;
        gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;

        server_name www.example.com example.com localhost _; # managed by Certbot

        listen 443 ssl; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

        location appserver {
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_redirect off;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header X-Forwarded-Proto https;
            proxy_pass http://appserver;
            access_log /var/log/nginx/access.log;
            error_log /var/log/nginx/error.log;
        }

    }

    server {

        if ($host = www.example.com) {
            return 301 https://$host$request_uri;
        } # managed by Certbot


        if ($host = example.com) {
            return 301 https://$host$request_uri;
        } # managed by Certbot


        listen        80 ;
        server_name www.example.com example.com localhost _;
        return 404; # managed by Certbot
    }
}

Has anyone faced this problem, and know how to make sure nginx handles everything as https?

Not sure if this in related, but in the production config I have:

config.force_ssl = false

If I switch it to ‘true’, nothing loads at all. Not sure why, shouldn’t that be ‘true’?

Answers

Chosen as BEST ANSWER
- Chakron
- November 27, 2021 at 10:26 pm
- 0 votes
0
@jamesc's suggestion ultimately led me to a solution. Elastic Beanstalk's AL2 Nginx confix file includes a webapp.conf file with the following block:
```
location @proxy {
    proxy_pass http://my_app; # match the name of upstream directive which is defined above
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
```
That wasn't passing through the full set of headers, and was resulting in invalid requests. I updated my nginx.conf file to provide an updated location block based on this thread: https://github.com/rails/rails/issues/22965.
```
location / {
    proxy_pass        http://my_app;
    proxy_set_header  Host $host;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Forwarded-Ssl on;
    proxy_set_header  X-Forwarded-Port $server_port;
    proxy_set_header  X-Forwarded-Host $host;
}
```

(Edit)

If you are going to use ssl then using certbot is an ok solution to use but you should no longer serve http requests so you need to set your server block that listens on port 80 to redirect to an ssl block i.e. a block that listens on port 443.
This doesn’t look like it is setup to act as a reverse proxy to your rails server either, there should be puma or unicorn or whatever rails server you are running declarations. so none of this really makes sense.

You have a few options set in your config that I am unfamiliar with plus you seem to have the sites_available options mixed in with the nginx.conf file, this is ok but it is not a normal nor very configurable setup, however, because of the few options you have that I am not familiar with I’m not going to comment on your options, I’m just going to make a suggestion that you tidy up your server blocks so that the block listening on port 80 redirects to the port 443 block. You will need to open up your firewall to allow port 443. If using ufw then there is an nginx app you can set otherwise I’ll leave you to figure out or ask in a different question how to setup your specific firewall for SSL on port 443.

Anyway, besides the above you should change your port 80 http block to look like this

server {
  #no longer default server, all port 80, http requests, should be redirected to https port 443
    listen 80;
    server_name www.example.com example.com localhost _;
    # really should be your production server name i.e. an.ip.address a_proper_domain.com a_proper_domain.co.uk www.a_proper_domain.com www.a_proper_domain.co.uk #provide a list of domains you are listening for separated by spaces
    #redirect to the https server block straight away, you do not want to have a site serving mixed https and http content it is a massive potential security hole
    return 301 https://$host$1$request_uri;
}

So that your config looks like this

#Elastic Beanstalk Nginx Configuration File

user                    nginx;
error_log               /var/log/nginx/error.log warn;
pid                     /var/run/nginx.pid;
worker_processes        auto;
worker_rlimit_nofile    65874;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    include       conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
        default     "upgrade";
    }

server {
  #no longer default server, all port 80, http requests, should be redirected to https port 443
    listen 80;
    server_name www.example.com example.com localhost _;
    # really should be your production server name i.e. an.ip.address a_proper_domain.com a_proper_domain.co.uk www.a_proper_domain.com www.a_proper_domain.co.uk #provide a list of domains you are listening for separated by spaces
    return 301 https://$host$1$request_uri;
}

    server {
        access_log    /var/log/nginx/access.log main;

        client_header_timeout 60;
        client_body_timeout   60;
        keepalive_timeout     60;
        gzip                  off;
        gzip_comp_level       4;
        gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;

    server_name www.example.com example.com localhost _; # managed by Certbot

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot



}

    server {
    if ($host = www.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen        80 ;
    server_name www.example.com example.com localhost _;
    return 404; # managed by Certbot

}}

Usually your capistrano deployment scripts will take care of setting up the nginx config for you if using capistrano 3 gem with capistrano3 puma gem if you are using puma as your rails server, there are also capistrano nginx gems as well but if using puma in production then I highly recommend using the correct configuration with your nginx.config file including the sites_enabled folder and setting up your specific server requirements in a separate file that lives in the appropriate location e.g. /etc/nginx/sites_available folder and gets symlinked to the sites_available folder. This is the standard approach that most server admins or dev ops would expect the setup to look like.

In your SSL server block you should have settings for your puma server and a typical setup is something like this

  location @name_of_your_puma_socket {
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_redirect off;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-Forwarded-Proto https; # this is your issue I think
    proxy_pass http://name_of_your_puma_socket;
    # limit_req zone=one;
    access_log #set the path to your nginx.access log
    error_log # set the path to your nginx.error.log;
  }

Please signup or login to give your own answer.

Click here to cancel reply.