runqueue: Skipping queue run -- load average too high

tonj · 09-24-2017, 10:59 AM

I'm running a centos 6 server (CentOS release 6.9 (Final)) with a really stubborn problem in sendmail (sendmail-8.14.4-9.el6_8.1.x86_64) which is that all mail is being queued and nothing sent. The entry in /var/log/maillog is:

Quote:

and all mail sent ends in

Quote:

stat=queued

. I've been trawling google on this but can't work out what's wrong. The command mailq gives:

Quote:

/var/spool/mqueue is empty
Total requests: 0

thanks for any help.

bathory · 09-24-2017, 03:55 PM

Hi,

Quote:

runqueue: Skipping queue run -- load average too high

Before anything else, you must find out what process is causing the high load and try to resolve the problem.

In the meantime you can increase QueueLA and RefuseLA options in sendmail.cf and restart sendmail. You need to uncomment the following 2 lines and increase the 2 values a little:

Code:

# load average at which we just queue messages
O QueueLA=8

# load average at which we refuse connections
O RefuseLA=12

Regards

syg00 · 09-24-2017, 09:30 PM

A high loadavg is not necessarily a problem. At least not one that the sendmail devs should be mindlessly using to throttle their application. Especially if they think it equates to runq. That's a *nix mindset from the last millenium.

tonj · 09-25-2017, 02:46 AM

thankyou bathory, I in sendmail cf I have O QueueLA=400 and O RefuseLA=400 but still the mail is queued. I've also run top and the process using the most cpu and mem is zoneminder and it's only around 10% cpu and 1% mem. Thanks for any further advice.

syg00 · 09-25-2017, 03:16 AM

cat /proc/loadavg

tonj · 09-25-2017, 03:32 AM

thankyou syg00

Quote:

# cat /proc/loadavg
157.23 157.41 157.34 1/599 32589

what does that mean?

syg00 · 09-25-2017, 03:58 AM

See the manpage for proc.
Those first 3 numbers are the loadavg numbers you see in other tools. They are unusually high, and consistent - so you've had a high load for (quite) a while. Try this

Code:

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 ~ /[RD]/) {print; count++} } END {print "Total: "count}'

Maybe just post the top 20 lines or so if they are all the same.

bathory · 09-25-2017, 04:17 AM

Quote:

Originally Posted by tonj

thankyou bathory, I in sendmail cf I have O QueueLA=400 and O RefuseLA=400 but still the mail is queued. I've also run top and the process using the most cpu and mem is zoneminder and it's only around 10% cpu and 1% mem. Thanks for any further advice.

You should definitely look for what is eating your server resources. So follow syg00 instructions

Regards

tonj · 09-25-2017, 04:45 AM

you were right syg00, this is the output from that command:

Quote:

# top -b -n 1 | awk '{if (NR <=7) print; else if ($8 ~ /[RD]/) {print; count++} } END {print "Total: "count}'
top - 12:38:20 up 123 days, 4:40, 1 user, load average: 157.08, 157.10, 157.10
Tasks: 479 total, 1 running, 478 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.8%us, 0.8%sy, 0.0%ni, 91.0%id, 1.3%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 5451868k total, 5271708k used, 180160k free, 884296k buffers
Swap: 5586940k total, 49848k used, 5537092k free, 3142388k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
694 root 20 0 15296 1388 800 R 5.7 0.0 0:00.03 top
676 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
679 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.00 packagekitd
682 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.01 packagekitd
863 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.01 packagekitd
866 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
869 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.01 packagekitd
1052 root 20 0 77820 2960 2412 D 0.0 0.1 0:00.01 packagekitd
1055 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.02 packagekitd
1058 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
1237 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.01 packagekitd
1240 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
1243 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
1488 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
1491 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
1494 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.01 packagekitd
1680 root 20 0 77820 2964 2412 D 0.0 0.1 0:00.01 packagekitd
1683 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.00 packagekitd
1686 root 20 0 77820 2968 2412 D 0.0 0.1 0:00.01 packagekitd

syg00 · 09-25-2017, 05:01 AM

Yet another reason to hate PackageKit. I regularly have to scrape disk space back from it.
In all likelihood those aren't affecting the rest of your system - except sendmail of course. Only a reboot will get rid of them.

tonj · 09-25-2017, 08:38 AM

thanks and a big breakthrough...
I did the reboot but got a different error message in the maillog which was:

Quote:

stat=Deferred: Connection refused by [127.0.0.1]

after a bit of digging around on google I discovered I didn't have sendmail-cf in my system so I installed it, restarted sendmail and suddenly loads of emails starting flushing out.
What puzzles me is if there were loads of queued emails why did mailq show 0?

bathory · 09-25-2017, 11:17 AM

Quote:

after a bit of digging around on google I discovered I didn't have sendmail-cf in my system so I installed it, restarted sendmail and suddenly loads of emails starting flushing out.

And how did you get

Quote:

Originally Posted by tonj

in sendmail cf I have O QueueLA=400 and O RefuseLA=400

Quote:

What puzzles me is if there were loads of queued emails why did mailq show 0?

Probably mail was still in clientmqueue. Use the following to see the mail in that queue

Code:

mailq -Ac

tonj · 09-26-2017, 02:57 AM

ok thanks, that command will come in handy for future problems like this.