[SOLVED] CentOS7: systemd-journald[4372]: Failed to open system journal: Not a directory.

Sum1 · 03-15-2021, 03:49 PM

I have a CentOS 7 server acting as a Samba Active Directory Domain Controller.

Suddenly today - about half of the Windows (7/8/10) clients joined to the domain can not access fileshares and the other server.

DNS is working correctly and everyone can access the internet.
All clients can authenticate to the domain.

I thought it might be related to hardware so I cloned the CentOS 7 server to a different ssd hard drive and ran it from a different server. The same result --- only about half of the Windows clients can access the fileshares.

Even Connecting to the server via ssh was very slow and the terminal would hang when trying to run 'dmesg' or 'htop'.

When I run the dmesg on a local machine terminal I see a flood of the following messages:

systemd-journald[4372]: Failed to open system journal: Not a directory.

They appear in bursts of 20 or so, every 3 seconds.

I cannot find much on this message and do not know what to try to alleviate the congestion on the server and regain access for LAN clients.

I would appreciate any help or suggestions. Thank you for reading.

Gad · 03-16-2021, 02:46 AM

If it suddenly stopped working is there anything that has changed or anything that might come to mind that would cause the sudden issue?

Although rebooting is considered a temporary fix have you tried it and was the result the same?

There is also a similar thread here: https://bbs.archlinux.org/viewtopic.php?id=198683

Sounds like this is a production machine so you may need to set a maintenance windows and a failback plan if necessary in case anything else breaks in the process.

berndbausch · 03-16-2021, 06:43 AM

By default, the systemd journal writes to a directory named /run/log/journal/. Many administrators want to have a persistent journal and move it to /var/log/journal. Of course, it can be configured to be located elsewhere.

You need to find out where systemd-journald wants to write to. My guess is that this directory is missing. Why? Hard to tell, but start this way. Perhaps it's sufficient to create the directory and/or restart systemd-journald.

Whether this has anything to do with your real problem, and how to solve your real problem, is a different question.

Sum1 · 03-16-2021, 05:12 PM

Thank you both for your thoughtful replies.

Ok --- 36 hours later all is well again.
The TL;DR: Hardware failure. 48 Port switch is dying/degrading.

A little more explanation:
I bought a supplemental 24 port switch and linked it to the 48 port.
I plugged all the patch cables associated with users desktops experiencing performance problems, into the 24 port switch.
Immediately thereafter, all those moved to the 24 port were operational and able to access fileshares, print services, and consistent internet access.

So, my systemd_journald observations had nothing to do with the file-server problem.

Now that I can breathe again after restoring normal operations, I can experiment with satisfying systemd's need to write/log to a particular location.

Thank you for the guidance!

Cheers

Gad · 03-17-2021, 01:22 AM

Glad to hear you managed to sort out the challenge

Sum1 · 03-17-2021, 01:33 PM

Quote:

Originally Posted by berndbausch

By default, the systemd journal writes to a directory named /run/log/journal/. Many administrators want to have a persistent journal and move it to /var/log/journal. You need to find out where systemd-journald wants to write to. My guess is that this directory is missing. Perhaps it's sufficient to create the directory and/or restart systemd-journald.

BBausch, you're thought sounded way too reasonable to ignore.
Had a few moments over lunch and took the risk of deleting the "journal" file resting in /var/log.
Then

Code:

~$root@host#  mkdir journal

Code:

systemctl restart systemd-journald

The following sub-directory was created -

Code:

/var/log/journal/57cc5zzzzzzzzzzzzzzzzzzzzzzzzzzzz

Containing -

Code:

drwxr-xr-x 2 root root 4.0K Mar 17 14:14 .
drwxr-xr-x 3 root root   45 Mar 17 14:14 ..
-rw-r----- 1 root root 128M Mar 17 14:14 system@2deeasdfzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.journal
-rw-r----- 1 root root 128M Mar 17 14:14 system@2deeasdfzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.journal
-rw-r----- 1 root root 128M Mar 17 14:14 system@2deeasdfzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.journal
-rw-r----- 1 root root  16M Mar 17 14:25 system.journal

No more bursty irritable complaints from systemd-journald, either.

I cannot read what's in the system.journal as it appears to be hashed output.

Thankyou, BBausch!