I ran a read-only self-check on my own production box and it found three things

vradarvradar
in Technical

Anyone got a service quietly dying and respawning while every dashboard stays green?

I wrote a read-only self-check this morning and ran it against my own production box. Mostly to see whether the script had a syntax error in it. It found three things.

A unit sitting in failed that I had never once looked at. No password-strength rule in pam at all, meaning the system itself would accept a one-character password (SSH here is key-only so it isn't an open hole, but it has no business being absent). And one gap on the login path that I'm scheduling a fix for. That third one I'm keeping to myself, for the obvious reason that spelling it out is handing over a map to my own front door.

That's the box belonging to the guy who looks at other people's boxes for a living. So I don't believe any box is clean.

Here's the shortest piece of it. Read-only, no root, writes nothing, calls nowhere:

uptime -p
[ -d /run/systemd/system ] || echo "CANNOT CHECK: no systemd on this box"
systemctl list-units --type=service --state=running --no-legend 2>/dev/null | awk '{print $1}' | while read -r u; do n=$(systemctl show "$u" -p NRestarts --value 2>/dev/null); case "$n" in ''|*[!0-9]*) continue;; esac; [ "$n" -gt 0 ] && echo "$n restarts  $u"; done | sort -rn

Three outcomes, not two:

CANNOT CHECK means draw no conclusion: no systemd, so the tool is blind, not your box clean. No lines at all means genuinely clean. Any line is the number of times that service died and came back, counted from exactly the uptime on line one. Without the uptime the number has no unit, and people will invent one.

I'd bet a few of you get a three-digit number you've never seen before.

Run it and drop the output here, I'll read all of them.

Comments

  • DigitalCloudDigitalCloud New Member

    Nice smoke test. One blind spot: because the first command limits the list to --state=running, it will not show a service that is currently failed — including the exact first issue you found.

    I would add a separate check first:

    systemctl --failed --type=service --no-legend

    Then use the restart counter for services that recovered automatically. For anything with a non-zero count, systemctl show "$unit" -p Result -p ExecMainStatus -p NRestarts adds useful context before checking the journal. Still fully read-only, but it separates “failed now” from “failed and recovered.”

  • vradarvradar

    You're right, and it's worse than a blind spot. The failed unit I found came from a separate check, so the snippet as posted cannot reproduce my own first finding. Running systemctl --failed first is the fix. The Result and ExecMainStatus pair is the part I would keep: a unit with Restart=on-failure reads active while its counter climbs, and that is the one nobody looks at.

Sign In or Register to comment.