From 879edbdfb0aff68fc7ccc9b30b9424f08cb1538a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Padez?= Date: Mon, 24 Aug 2026 20:43:40 +0000 Subject: [PATCH] boot hang: "is networkd running" was the wrong question MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This section did nothing on the machine it was written for, and the machine was losing two minutes on every boot the whole time. Both stacks can be ACTIVE at once. Here NetworkManager owned enp1s0 and wlp2s0, systemd-networkd was up and managing not one of eighty links, and systemd-networkd-wait-online still burned its full timeout — the journal has `Timeout occurred while waiting for network connectivity`, 18:31:05 to 18:33:05. The old predicate required networkd to be INACTIVE, so it reported "nothing to fix — networkd is in charge" and moved on. What decides it is whether networkd CONFIGURED a link, which networkctl will say: a link it owns reads `configured`, one it ignores reads `unmanaged`. If every link is unmanaged the wait can never be satisfied, and masking is safe. A missing networkctl counts as "yes it does". An unanswerable question must not read as permission to mask — the failure mode on the other side of this is a VPS that comes up with no network and no ssh. `network_manager_name` had the same blind spot and printed "NetworkManager" for the both-running case, which is exactly the case that was being got wrong. It now describes the machine instead of picking a winner. Verified against this box: configures-a-link no, spurious yes, reported as "NetworkManager (networkd is up but configures nothing)". Masked here by hand; NetworkManager-wait-online still provides network-online.target, so nothing that orders after the network lost its ordering. Co-Authored-By: Claude Opus 5 Officer-Session: de3ed4e6-f88a-429c-94e6-a760336ba83b --- scripts/setup/machine-setup/lib/system.sh | 59 +++++++++++++++++--- scripts/setup/machine-setup/machine-setup.sh | 14 +++-- 2 files changed, 59 insertions(+), 14 deletions(-) diff --git a/scripts/setup/machine-setup/lib/system.sh b/scripts/setup/machine-setup/lib/system.sh index 01915d81..d7cd308c 100644 --- a/scripts/setup/machine-setup/lib/system.sh +++ b/scripts/setup/machine-setup/lib/system.sh @@ -466,32 +466,73 @@ disable_sleep() { # systemd-networkd actually manages the network — a server or cloud image, via # cloud-init and netplan — it does that in milliseconds and is load-bearing. # -# Where NetworkManager owns the network instead, systemd-networkd runs nothing, -# but the wait-online unit is still enabled and waits for a link that will never -# be configured. It gives up after its full timeout, on every boot. +# Where NetworkManager owns the network instead, systemd-networkd configures +# nothing, but the wait-online unit is still enabled and waits for a link that +# will never come up. It gives up after its full timeout, on every boot. # # So the fix is masking one unit, and only on the second stack. Do NOT # `systemctl disable --now systemd-networkd` to achieve the same thing: on a # networkd-managed box that brings it up with no network on the next boot, no # ssh, and nothing but the provider's rescue console. +# +# ── "is networkd running" is the wrong question ── +# +# It was the test here until 2026-08-24, and it is why this section did nothing +# on the machine it was written for. Both stacks can be ACTIVE at once: on the +# dev box NetworkManager owned enp1s0 and wlp2s0, systemd-networkd was up and +# managing not one of eighty links, and wait-online still burned its full two +# minutes on every boot — `Timeout occurred while waiting for network +# connectivity`, at 18:31:05 to 18:33:05, in the journal. +# +# What matters is whether networkd CONFIGURED anything, which `networkctl` will +# say. A link it owns reads `configured`; a link it is ignoring reads +# `unmanaged`. If every link is unmanaged, the wait cannot ever be satisfied and +# masking it is safe. WAIT_ONLINE_UNIT=systemd-networkd-wait-online.service +# Reported to the operator, so it has to describe the machine rather than pick a +# winner: both daemons running with networkd idle is the exact case this section +# used to get wrong, and printing "NetworkManager" for it hid that. network_manager_name() { - if systemctl is-active --quiet NetworkManager.service 2>/dev/null; then + local nm=no nd=no + systemctl is-active --quiet NetworkManager.service 2>/dev/null && nm=yes + systemctl is-active --quiet systemd-networkd.service 2>/dev/null && nd=yes + + if [[ $nm == yes && $nd == yes ]]; then + if networkd_configures_a_link; then + echo "both — networkd configures at least one link" + else + echo "NetworkManager (networkd is up but configures nothing)" + fi + elif [[ $nm == yes ]]; then echo "NetworkManager" - elif systemctl is-active --quiet systemd-networkd.service 2>/dev/null; then + elif [[ $nd == yes ]]; then echo "systemd-networkd" else echo "neither — unclear" fi } -# Is the wait actually pointless here? NetworkManager in charge and networkd not. -# Anything else, including "cannot tell", is left alone. +# Does networkd actually configure a link on this machine? +# +# `networkctl list --no-legend` puts the SETUP state last on each line. Anything +# other than unmanaged/pending/n-a means networkd owns that link and the wait is +# real work. Missing networkctl counts as "yes, it does" — an unanswerable +# question must not be read as permission to mask. +networkd_configures_a_link() { + command -v networkctl &>/dev/null || return 0 + networkctl list --no-legend 2>/dev/null | awk '{ print $NF }' | + grep -qvE '^(unmanaged|pending|n/a)$' +} + +# Is the wait actually pointless here? NetworkManager in charge, and networkd +# either absent or configuring nothing. Anything else, including "cannot tell", +# is left alone. wait_online_is_spurious() { - systemctl is-active --quiet NetworkManager.service 2>/dev/null && - ! systemctl is-active --quiet systemd-networkd.service 2>/dev/null + systemctl is-active --quiet NetworkManager.service 2>/dev/null || return 1 + systemctl is-active --quiet systemd-networkd.service 2>/dev/null || return 0 + ! networkd_configures_a_link } # What that unit actually cost on this boot, straight from systemd's own diff --git a/scripts/setup/machine-setup/machine-setup.sh b/scripts/setup/machine-setup/machine-setup.sh index 745785cf..37ac3edb 100755 --- a/scripts/setup/machine-setup/machine-setup.sh +++ b/scripts/setup/machine-setup/machine-setup.sh @@ -1170,18 +1170,22 @@ elif ! skip; then echo "" info "Boot hang — a unit that can add two minutes to every boot" echo " On a machine where NetworkManager owns the network, systemd-networkd" - echo " runs nothing — but systemd-networkd-wait-online is still enabled, and" - echo " waits for a link that will never come up. It gives up after its full" - echo " timeout, every single boot." + echo " configures nothing — but systemd-networkd-wait-online is still" + echo " enabled, and waits for a link that will never come up. It gives up" + echo " after its full timeout, every single boot." + echo "" + echo " Note both daemons can be running at once. What decides it is whether" + echo " networkd CONFIGURED a link, not whether it is up." echo "" echo " network managed by: $(network_manager_name)" echo " that unit took: ${WAIT_ONLINE_TIME:-it did not run} on this boot" if ! wait_online_is_spurious; then echo "" - echo " systemd-networkd is live here, so that unit is doing real work." + echo " systemd-networkd configures a link here, so that unit is doing" + echo " real work." echo " Masking it would be how this machine comes up with no network." - SUMMARY+=("Boot hang: nothing to fix — networkd is in charge") + SUMMARY+=("Boot hang: nothing to fix — networkd configures a link") else echo "" if is_role dev; then