Rails 5 + MongoDB issue in production – CentOS

I’ve just moved a Rails 5.2.2 app to production server and i’m having problem with MongoDB, none of the queries work. Here some info:

CentOS 7
Ruby 2.6.3
Rails (5.2.2)
Mongoid (7.0.2)

$ mongo --version
MongoDB shell version v4.0.9
git version: fc525e2d9b0e4bceff5c2201457e564362909765
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
    distmod: rhel70
    distarch: x86_64
    target_arch: x86_64

MongoDB authentication is enabled, user with proper permissions has been created, the DB is not empty (was restored from a local copy), here’s the production section of the mongoid.yml file:

production:
  clients:
    default:
      database: mydb_production
      hosts:
        - localhost:27017
      username: myuser
      password: "mypass"
      options:
        auth_mech: :scram
        auth_source: admin

I keep seeing these topology debug messages in the production log:

[rails_root]/log/production.log

D, [2019-05-27T20:01:58.272760 #27444] DEBUG -- : MONGODB | EVENT: #<TopologyOpening topology=Unknown[]>
D, [2019-05-27T20:01:58.272934 #27444] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-05-27T20:01:58.273748 #27444] DEBUG -- : MONGODB | EVENT: #<TopologyChanged prev=Unknown[] new=Unknown[localhost:27017]>
D, [2019-05-27T20:01:58.273848 #27444] DEBUG -- : MONGODB | Topology type 'Unknown' changed to type 'Unknown'.
D, [2019-05-27T20:01:58.274028 #27444] DEBUG -- : MONGODB | EVENT: #<ServerOpening address=localhost:27017 topology=Unknown[localhost:27017]>
D, [2019-05-27T20:01:58.274091 #27444] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-05-27T20:01:58.365308 #27444] DEBUG -- : MONGODB | EVENT: #<ServerDescriptionChanged address=localhost:27017 topology=Single[localhost:27017] prev=#<Mongo::Server:Description:0x70049014837980 config={} average_round_trip_time=> new=#<Mongo::Server:Description:0x70048970460940 config={"ismaster"=>true, "maxBsonObjectSize"=>16777216, "maxMessageSizeBytes"=>48000000, "maxWriteBatchSize"=>100000, "localTime"=>2019-05-28 00:01:58 UTC, "logicalSessionTimeoutMinutes"=>30, "minWireVersion"=>0, "maxWireVersion"=>7, "readOnly"=>false, "ok"=>1.0} average_round_trip_time=0.088107942>>
D, [2019-05-27T20:01:58.365596 #27444] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-05-27T20:01:58.365843 #27444] DEBUG -- : MONGODB | EVENT: #<TopologyChanged prev=Unknown[localhost:27017] new=Single[localhost:27017]>
D, [2019-05-27T20:01:58.366024 #27444] DEBUG -- : MONGODB | Topology type 'Unknown' changed to type 'Single'.

connecting to the console, any attempt at reading or writing records throws the following error:

$ rails c production
Running via Spring preloader in process 3131
Loading production environment (Rails 5.2.2)
2.6.3 :001 > User.first
Traceback (most recent call last):
        1: from (irb):1
NoMethodError (undefined method `encode' for nil:NilClass)

Any idea what’s going on here? Connecting to the db either with a GUI remotely or via the mongo command line i can read the collections fine so i’m assuming it’s a problem between rails and mongo, not mongo itself.

EDIT

Full stack trace of the error, seems to me that mongoid can’t authenticate with MongoDB, no idea why though, i’m using the same configurations has when connecting via GUI or command line…

begin; User.first; rescue => e; puts e.backtrace.join("n"); end
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/auth/user.rb:96:in `encoded_name'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/auth/scram/conversation.rb:334:in `first_bare'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/auth/scram/conversation.rb:277:in `client_first_message'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/auth/scram/conversation.rb:184:in `start'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/auth/scram.rb:71:in `login'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection.rb:284:in `block in authenticate!'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server.rb:361:in `handle_auth_failure!'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection.rb:282:in `authenticate!'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection.rb:135:in `connect!'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connectable.rb:78:in `ensure_connected'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection_base.rb:97:in `deliver'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection.rb:299:in `deliver'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection_base.rb:90:in `dispatch'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/operation/shared/executable.rb:42:in `block in dispatch_message'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server/connection_pool.rb:123:in `with_connection'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/server.rb:334:in `with_connection'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/operation/shared/executable.rb:41:in `dispatch_message'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/operation/shared/executable.rb:36:in `get_result'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/operation/shared/executable.rb:24:in `execute'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/operation/shared/executable_transaction_label.rb:27:in `execute'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/operation/shared/op_msg_or_find_command.rb:28:in `execute'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/collection/view/iterable.rb:82:in `send_initial_query'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/collection/view/iterable.rb:42:in `block in each'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/retryable.rb:43:in `read_with_retry'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongo-2.8.0/lib/mongo/collection/view/iterable.rb:40:in `each'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/query_cache.rb:227:in `each'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/contextual/mongo.rb:258:in `first'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/contextual/mongo.rb:258:in `block in first'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/contextual/mongo.rb:531:in `try_cache'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/contextual/mongo.rb:256:in `first'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/contextual.rb:20:in `first'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/mongoid-7.0.3/lib/mongoid/findable.rb:148:in `first'
(irb):1:in `irb_binding'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/workspace.rb:85:in `eval'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/workspace.rb:85:in `evaluate'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/context.rb:385:in `evaluate'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:493:in `block (2 levels) in eval_input'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:647:in `signal_status'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:490:in `block in eval_input'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/ruby-lex.rb:246:in `block (2 levels) in each_top_level_statement'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/ruby-lex.rb:232:in `loop'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/ruby-lex.rb:232:in `block in each_top_level_statement'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/ruby-lex.rb:231:in `catch'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb/ruby-lex.rb:231:in `each_top_level_statement'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:489:in `eval_input'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:428:in `block in run'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:427:in `catch'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:427:in `run'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/irb.rb:383:in `start'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/railties-5.2.2/lib/rails/commands/console/console_command.rb:64:in `start'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/railties-5.2.2/lib/rails/commands/console/console_command.rb:19:in `start'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/railties-5.2.2/lib/rails/commands/console/console_command.rb:96:in `perform'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/railties-5.2.2/lib/rails/command/base.rb:65:in `perform'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/railties-5.2.2/lib/rails/command.rb:46:in `invoke'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/railties-5.2.2/lib/rails/commands.rb:18:in `<main>'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:83:in `register'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:291:in `block in require'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:257:in `load_dependency'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/activesupport-5.2.2/lib/active_support/dependencies.rb:291:in `require'
/var/www/webtsh/bin/rails:9:in `<main>'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
/home/tsh/.rvm/gems/ruby-2.6.3/gems/bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:50:in `load'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/home/tsh/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'