I am working on a program that sends logs with syslog()
. Then I configured rsyslog
service to save logs in a file under Linux
. Most of the time this process works normally. But sometimes, some of the logs are not sent to rsyslog
. Instead, I can watch them when I use the journalctl -f -u Myservice
command. I am using the Debian Jessie
version of Linux
. Do you have any idea what is the problem and how to solve it?
3
Answers
Finally, I found a good answer. In the
rsyslog
config file in we can add different methods to get the messages. I had many filters with thecontains
filter. this significantly reduced the performance since it needs to completely search the string for the filter value. Then I found that thestartswith
filter is far better in searching for a value in a string. I changed the message structure so that I can use thestartswith
filter. then I changed thersyslog
filter tostartswith
. now the performance is much better and no message sent to the journal. the syntax is like this::msg, startswith, "val" # instead of (:msg, contains, "val")
This totally depends on whether or not the
rsyslog
service is running. Please check ifrsyslog
is running when you face the issue.Loosely speaking, with systemd, the logs are managed as follow:
Process calls syslog()
–writes in–>/dev/log = /run/systemd/journal/dev-log
–read by–>systemd-journald
–forwards to–>/run/systemd/journal/syslog
–read by–>rsyslogd
Let’s look at it in more details…
/lib/systemd/system/systemd-journald-dev-log.socket is the systemd socket unit to capture the messages from /dev/log:
In the above socket unit, there are two important things:
Hence any process calling
syslog()
actually writes into /dev/log synonymous of /run/systemd/journal/dev-log. As systemd-journald reads from this socket, this makes it capture the logs of all the processes writing into /dev/log. But systemd implements a mechanism to forward those logs to any "registered" service.There is a syslog.socket unit which sets up the /run/systemd/journal/syslog socket:
The corresponding syslog.service is triggered afterwards. The latter is actually a symbolic link onto the actual syslog service (e.g. rsyslogd or syslog-ng). Here is an example, where it is a symbolic link onto rsyslog.service:
The content of the latter service, executes
rsyslogd
daemon:We can verify its activation looking at the status of the syslog service (field "TriggeredBy"):
The startup messages above shows that
rsyslogd
is passed the unix socket /run/systemd/journal/syslog (file descriptor id 3). This is managed by the imuxsock module ofrsyslogd
. This is indeed part of the file descriptors opened byrsyslogd
process:The configuration of
systemd-journald
decides if what is read from /run/systemd/journal/dev-log is forwarded or not to /run/systemd/journal/syslog:The above commented line means that the default is "yes".
To answer the OP’s question
If some of the logs sent by the processes are not seen in
rsyslogd
‘s output but seen in thesystemd-journald
‘s output, this may mean thatrsyslogd
is not started early enough compared to the application. So, the latter’s logging are read bysystemd-journald
but are not forwarded torsyslogd
. The latter starts in thesystemd
‘s multi-user.target whereas systemd-journald is launched at the very beginning of the system startup.To verify that, the result of
systemctl status syslog.service
displays the activation date. Compare it to the activation date of systemd-journald.service and the one of the application.A synchronization key like "After=syslog.service" may be missing in the service file of the application.
Further references
https://www.freedesktop.org/wiki/Software/systemd/syslog/