Oracle Linux 7.2
I updated the SSL Certificate for a particular domain in the httpd.conf file.
Then I did a ‘sudo service httpd restart’, and it hung for a long time (over a minute), then errored out. Now, I can’t restart httpd, and I don’t see any hints in ‘systemctl status httpd.service’ or ‘journalctl -xe’.
Here’s what I’m seeing:
[oracle@secure-web-server-dvl ~]$ sudo service httpd start
Redirecting to /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
[oracle@secure-web-server-dvl ~]$ systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2019-01-04 16:22:31 EST; 37s ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 3299 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
Process: 3297 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
Main PID: 3297 (code=exited, status=1/FAILURE)
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: Starting The Apache HTTP Server...
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com kill[3299]: kill: cannot find process ""
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: httpd.service: control process exited, code=exited status=1
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: Failed to start The Apache HTTP Server.
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: Unit httpd.service entered failed state.
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: httpd.service failed.
[oracle@secure-web-server-dvl ~]$ journalctl -xe
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 1 has been created for the user oracle.
--
-- The leading process of the session is 3259.
Jan 04 16:22:15 secure-web-server-dvl.nitssolutions.com systemd[1]: Started Session 1 of user oracle.
-- Subject: Unit session-1.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-1.scope has finished starting up.
--
-- The start-up result is done.
Jan 04 16:22:15 secure-web-server-dvl.nitssolutions.com systemd[1]: Starting Session 1 of user oracle.
-- Subject: Unit session-1.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-1.scope has begun starting up.
Jan 04 16:22:15 secure-web-server-dvl.nitssolutions.com sshd[3259]: pam_unix(sshd:session): session opened for user oracle by (uid=0)
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com sudo[3281]: oracle : TTY=pts/0 ; PWD=/home/oracle ; USER=root ; COMMAND=/sbin/service httpd st
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com polkitd[618]: Registered Authentication Agent for unix-process:3282:11555 (system bus name :1.14
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com kill[3299]: kill: cannot find process ""
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: httpd.service: control process exited, code=exited status=1
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: Unit httpd.service entered failed state.
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com systemd[1]: httpd.service failed.
Jan 04 16:22:31 secure-web-server-dvl.nitssolutions.com polkitd[618]: Unregistered Authentication Agent for unix-process:3282:11555 (system bus name :1.
Help?
2
Answers
Ok, I figured this out. Our SSL certificate had expired, so I renewed it. When I replaced the cert in the various VirtualHost sections of the httpd.conf file, I forgot to also replace the key for the SSL cert. That caused the problem. When I went back and updated the keys for the corresponding SSL cert, Apache started right up.
I think it's a bug that when this happens, there is no trace of root cause in the error messages. I was lucky to figure it out relatively quickly.
There must be a process (like a web request that is still processing) attached to httpd which is causing it not to restart.
I see in your error message that process with ID 3299 is trying to be killed, but is failing.
Check to see if this process is running using
or using
See man ps
This will show you manual for viewing a snapshot of the current processes
If you find that the process is still running then you can kill it manually, using command
Using the -9 flag will prevent any process from blocking the kill.
See man kill
This should allow you to start the httpd service again.
WARNING: Manually killing a processing can corrupt configuration files.
Make a back up of your config files for your apache/httpd server first!
This solution is a bit unorthodox, but as a last resort, sometimes you need to kill a rouge process that is being blocked, manually.