Understanding Linux Logging
System logs are essential to understand the timeline of events for any working system, and when issues arise, logs are an invaluable troubleshooting tool.
Linux is a popular operating system that is widely used on servers, desktops, and other devices. One of the key features of Linux is its system of logging. linux logging allows administrators to keep track of important events and messages, baselining, system audits, ascertaining SLA adherence, monitoring, and troubleshooting.
In this article, we will explain how Linux logging works, and how it can be used to manage and troubleshoot a Linux system.
A) rsyslog
Syslog was the first logging project that was available in 1980’s at that time it was very simple, Syslog-ng later extended syslog with additional features and capabilities. In 2004 Rsyslog enhanced syslog with more capabilities, Rsyslog is mostly used today and it provides a system of logging that records the events that happen on the operating system.
The system logs are by default stored in the /var/log directory, further, each application or system service is supposed to have its own log file. For example, the HTTPD web server will have a log file called /var/log/httpd/access.log that contains a record of all incoming requests to the web server.
In addition to the system logs, rsyslog also provides a way for applications to generate their own logs using the syslog system calls. This allows applications to record important events and messages, and send them to the syslog service for storage and processing.
rsyslog service status
we can check the status of the rsyslog service by running the command systemctl status rsyslog.service
● rsyslog.service – System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-01-13 17:49:02 IST; 2 days ago
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 727 (rsyslogd)
Tasks: 3 (limit: 48490)
Memory: 5.7M
CPU: 6.562s
CGroup: /system.slice/rsyslog.service
└─727 /usr/sbin/rsyslogd -n
Jan 13 17:49:01 midas.upspiroffice.com systemd[1]: Starting System Logging Service…
Jan 13 17:49:02 midas.upspiroffice.com rsyslogd[727]: $WorkDirectory: /var/lib/rsyslog can not be accessed, probably does>
Jan 13 17:49:02 midas.upspiroffice.com rsyslogd[727]: [origin software=”rsyslogd” swVersion=”8.2102.0-105.el9″ x-pid=”727>
Jan 13 17:49:02 midas.upspiroffice.com systemd[1]: Started System Logging Service.
If not installed, you can install rsyslog using yum, and start and enable the service
rsyslog configuration
The rsyslog service is configurable, and administrators can specify the types of events that should be logged, as well as where and how the logs should be stored. Rsyslog can be configured to run as a client and server at the same time.
/etc/rsyslog.conf is the default configuration for the rsyslog
rsyslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* –/var/log/maillog
Some default System Log Files (linux logging)
- /var/log/message – as can be seen from above configuration it will have all the logs except mail, authpriv and cron. it is mostly the first file to be analyzed when faced with any issues.
- /var/log/secure – all the messages related to security and authentication events, if you have to find who logged into the server or who tried but failed to login to the server, this is the log file to refer
- /var/log/mail – for logs related to mail server
- /var/log/cron – for logs relate to cron
- /var/log/boot.log – for logs related system startup.
Analyzing logfiles System Admins use different Linux commands like grep, tail, head, more, less, and cat to search and filter the various logfiles, depending on what they are looking for.
Log Severity Levels
- Emergency – System is unusable
- Alert – Action must be taken immediately
- Critical – Critical conditions
- Error – Error conditions
- Warning – Warning conditions
- Notice – Normal but significant condition
- Informational – Informational messages
- Debug – Debug-level messages
Customizing rsyslog.conf
You can edit the rsyslog.conf to have your custom configuration., but direct edit the file is not recommended
Recommended Method
To add any user configuration rsyslog.d directory is available, and it is added in the default configuration to load all config files from this directory
# Include all config files in /etc/rsyslog.d/
include(file=”/etc/rsyslog.d/*.conf” mode=”optional”)
So the user can create their custom config files and put in this directory, which will be automatically read by the service, without changing the default settings.
Using rsyslog with CLI
Rsyslog not only supports most of the programming languages
It also supports the CLI command to log messages. Let’s test whether our rsyslog is up and running.
B) journalctl
journalctl is a command-line utility that can be used to view and manage the logs collected by the systemd system and service manager in Linux. It allows system admins to view these logs as well as filter and searrch specific logs for specific services, and also provides ability to only log messages for a specific time period.
To use journalctl, administrators can simply run the journalctl command with no options, which will display all of the logs collected by systemd.
[root@midas ~]# journalctl
Jan 13 23:18:52 localhost kernel: microcode: microcode updated early to revision 0x28, date = 2019-11-12
Jan 13 23:18:52 localhost kernel: Linux version 5.14.0-160.el9.x86_64 (mockbuild@x86-05.stream.rdu2.redhat.com) (gcc (GCC>
Jan 13 23:18:52 localhost kernel: The list of certified hardware and cloud instances for Red Hat Enterprise Linux 9 can b>
Jan 13 23:18:52 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-160.el9.x86_64 root=UUID=2db362f1-e8>
Jan 13 23:18:52 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: ‘x87 floating point registers’
Jan 13 23:18:52 localhost kernel: x86/fpu: Supporting XSAVE feature 0x002: ‘SSE registers’
Jan 13 23:18:52 localhost kernel: x86/fpu: Supporting XSAVE feature 0x004: ‘AVX registers’
Jan 13 23:18:52 localhost kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Jan 13 23:18:52 localhost kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using ‘standard’ forma>
Jan 13 23:18:52 localhost kernel: signal: max sigframe size: 1776
Jan 13 23:18:52 localhost kernel: BIOS-provided physical RAM map:
met, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.
Using Journalctl, As journalctl does not kepp different logfiles as in the case of Syslog, system administrators can use various options and flags to search, filter, and extract relevant information.
For example,
- -u flag can be used to show only logs from a specific service,
- -b flag can be used to show only logs from the current boot.
- -n for last n number of log entries, by default 10 are shown
- -p err for priority error and above
- –since ‘yyyy-mm-dd hh:mm:ss’ to limit the output based on time
journalctl -u httpd, will show all of the logs for httpd service
Jan 14 12:22:50 midas.upspiroffice.com systemd[1]: Starting The Apache HTTP Server…
Jan 14 12:22:51 midas.upspiroffice.com httpd[3328]: Server configured, listening on: port 80
Jan 14 12:22:51 midas.upspiroffice.com systemd[1]: Started The Apache HTTP Server.
Jan 15 00:00:03 midas.upspiroffice.com systemd[1]: Reloading The Apache HTTP Server…
Jan 15 00:00:03 midas.upspiroffice.com systemd[1]: Reloaded The Apache HTTP Server.
Jan 15 00:00:03 midas.upspiroffice.com httpd[3328]: Server configured, listening on: port 80
Systemd journal stores the logs in structured and indexed library file called journal
From RHEL8, the journal is stored in /run/log by default and its contents by default are cleared after reboot, we have to configure journalctl to store the logs from older sessions.
journalctl will show full system logs when run by root user starting from oldest log entry,
It highlights important log messages, and warning as bold text while errors in red color.
Configuration File /etc/systemd/journald.conf
#Defaults can be restored by simply deleting this file and all drop-ins.
# Use ‘systemd-analyze cat-config systemd/journald.conf’ to display the full config.
# See journald.conf(5) for details.
[Journal]#Storage=auto#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
Comparison with syslog
- syslog is an older logging service, it is a centralized service that collects log messages from various sources on the system, journalctl is a newer tool that is used to view and manage the logs collected by the systemd system and service manager.
- syslog logs are typically stored in text files in the /var/log directory, and each application or service has its own log file. users have to use tools like grep, text editor search, etc to filter and analyze the logs. journalctl, stores its logs in a binary journal file, and provides features such as indexed search, automatic rotation, filter logs by time and service,
- syslog is still widely used in Linux, but newer distributions are moving towards journalctl and it is likely to become the more commonly used tool in the future.
C) dmesg
dmesg is a command-line utility in Linux that allows administrators to view the kernel’s log buffer.
This allows system admins to view messages about the system hardware and software including the hardware initialization messages, system errors, and warnings at the time of system booting up.
To use dmesg, administrators can simply run the dmesg command with no options, which will display the entire contents of the kernel’s log buffer.
dmesg logs are extremely useful to identify issues that may have cropped up during booting and these errors are not reflected in syslogs, though analyzing dmesg logs may need special skills and experience.
Here is an example of using dmesg to view the kernel’s log buffer:
[ 0.000000] microcode: microcode updated early to revision 0x28, date = 2019-11-12
[ 0.000000] Linux version 5.14.0-160.el9.x86_64 (mockbuild@x86-05.stream.rdu2.redhat.com) (gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2), GNU ld version 2.35.2-24.el9) #1 SMP PREEMPT_DYNAMIC Thu Aug 25 23:06:03 UTC 2022
[ 0.000000] The list of certified hardware and cloud instances for Red Hat Enterprise Linux 9 can be viewed at the Red Hat Ecosystem Catalog, https://catalog.redhat.com.
[ 0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-160.el9.x86_64 root=UUID=2db362f1-e876-4b12-9e1c-185da3103150 ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=a4d08e34-154f-4c78-b318-04bd2dec58c3 rhgb quiet
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: ‘x87 floating point registers’
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: ‘SSE registers’
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: ‘AVX registers’
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using ‘standard’ format.
[ 0.000000] signal: max sigframe size: 1776
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000d9c5dfff] usable
Summary
Understanding the above logs and the ability to analyze them is an essential skill for anyone providing support on Linux be it system administrators, application support engineers, or even application developers, The proficiency in analyzing the logs helps to identify and isolate the issues, predict system behavior and prevent any negative event.
..
..
Author
Nikhil Kumar Ahluwalia
Founder Upspir
Nikhil has over 16years of experience in the industry, before starting UPSPIR he was heading the support, services, and delivery function at Ameyo/Exotel as vice-president, and he has vast experience in managing and leading technical support and delivery teams, With Upspir he aims to take his passion of developing and mentoring people to next level by sharing his experiences and learning with those who want to build their career in the tech support domain.