"at" service not running...

rnturn · 06-05-2015, 12:09 AM

...or is it?

I tried to toss something in to execute in batch today and started getting error messages about the atd daemon not running. So I issued:

Code:

# systemctl -a | grep atd
atd.service   loaded failed   failed   Execution Queue Daemon
# systemctl enable atd.service
# systemctl start atd.service
# systemctl -a | grep atd
atd.service   loaded failed   failed   Execution Queue Daemon

For some time I was issuing the command:

Code:

ps -ef | grep -v grep | grep atd

and getting nothing returned, i.e., atd was not running.

After waiting a few minutes and issuing the systemctl commands again, "ps -ef" showed an atd process. However, "systemctl -a | grep atd" is still showing the same thing as shown above: "... loaded failed failed ...".

Anyone got any ideas why initially systemctl wasn't starting atd? And why it now appears to be running even though systemctl says there's been a failure (of something)?

Environment: OpenSUSE 12.2 (yeah, I need to do that upgrade)

Any information that'll help me understand what's going on will be welcomed.

TIA...

--
Rick

joec@home · 06-05-2015, 11:17 AM

I saw this once before long ago, but rather obscure so I do not know if it is related or not. The at command is commonly used by hackers so some of the older security gurus will lock down the at function in the pam authentication. Check the /etc/pam.d/atd or if that does not exist the /etc/pam.d/other configurations? If /etc/pam.d/atd is missing then I would suggest to create one rather than messing with /etc/pam.d/other as that might cause other security problems down the road.

rnturn · 06-06-2015, 12:35 AM

Quote:

Originally Posted by joec@home

...some of the older security gurus will lock down the at function in the pam authentication.

In this case, I am that older security guru. Well, at least I'm the older guy in charge of the system in question. (Any guruness on my part is a matter of opinion.) The PAM configuration hasn't been changed, i.e., it's being used with the original, out-of-the-box settings. I did notice some errors in /var/log/messages that seem to point to something odd in apparmor:

Code:

2015-06-04T16:33:08-05:00 mysys atd[31499]: unknown option
2015-06-04T16:33:08-05:00 mysys systemd[1]: atd.service: main process exited, code=exited, status=1
2015-06-04T16:33:08-05:00 mysys systemd[1]: Unit atd.service entered failed state.
.
.
.
2015-06-04T16:34:52-05:00 mysys atd[31615]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted
2015-06-04T16:34:52-05:00 mysys atd[31614]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted
2015-06-04T16:34:52-05:00 mysys atd[31613]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted
2015-06-04T16:34:52-05:00 mysys atd[31614]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted
2015-06-04T16:34:52-05:00 mysys atd[31615]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted
2015-06-04T16:34:52-05:00 mysys atd[31613]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted

(Of course, systemd had to be involved. Ugh.)

I don't recall if that flurry of messages were due to my stacking up several "at" jobs (after I started noticing that "ps -ef" was showing an "atd" process) so I created a real simple batch script:

Code:

#!/bin/bash

logger "Message produced by at job."

which resulted in the following messages in /var/log/messages:

Code:

2015-06-05T22:15:00-05:00 mysys atd[24141]: pam_apparmor(atd:session): Unknown error occurred changing to users hat: Operation not permitted
2015-06-05T22:15:00-05:00 mysys root: Message produced by at job.

I'm not sure just what "operation" wasn't being permitted but it looks as though the "at" job did, indeed, run.

In case anybody wants to know: There is no "/etc/at.allow" file and "/etc/at.deny" contains the usual suspects ("root", "bin", "daemon", etc.) which, according to the man pages, is the default set up that allows anyone to submit jobs via "at".

Anyone got any ideas about this?

joec@home · 06-06-2015, 01:36 PM

I'm grasping at straws here but seems we are in the same boat. I do recall that I have seen some distros have problems with at compared to others but not certain if SUSE was one of them. Seen too many different distros over the years. Maybe get the version number of that specific at command and search for errors by version number?

rnturn · 06-06-2015, 08:13 PM

Quote:

Originally Posted by joec@home

Maybe get the version number of that specific at command and search for errors by version number?

V3.1.8 on OpenSUSE 12.2 and V3.1.15 on 13.2. Not a radical change in versions. Something's different, though. I don't see the odd messages about "hats" on the 13.2-based system

I've run across some discussions and bug reports about date handling but nothing (so far) about "at"/"apparmor" or "at"/"systemd" interactions.

What's weird is that -- after some initial errors that looked like systemd was having trouble starting the service -- things are working even though every "at" job execution throws that "couldn't change hat" error message into /var/log/messages. Even disabling "apparmor" from YaST hasn't eliminated the error message disappear when I run an "at" job. I haven't rebooted, though I may in a while after I finish up some other work that's in progress. I doubt a reboot's going to do much. Looks like it's time to do the upgrade. Probably take far less time than trying to make this work on 12.2.

joec@home · 06-06-2015, 09:47 PM

Last straw I might suggest is to turn off the auditd service to see off it is being blocked there. Beyond that, it is obviously a PAM issue somewhere. I'm not certain, but I want to say that some distros have at disabled by default as a security precaution and you have to manually build the allow tables? I remember something like that, but can not think of any good reference.