PRODENG-3442: fall back to forced swarm dissolution when uninstall-ucp times out

[james-nesbitt]

Jira: https://mirantis.jira.com/browse/PRODENG-3442

Problem

The uninstall-ucp bootstrapper deploys ucp-uninstall-agent as a global Swarm service and waits ~2 minutes (hardcoded) for every node to acknowledge completion. On large clusters or hosts with cold image caches (fresh CI runners) the deadline is missed, causing Reset() to fail even though the cluster and infrastructure are otherwise healthy.

Observed in CI:

• smoke-modern (MKE 3.9.2, 7 nodes): all nodes missed the deadline
• smoke-windows (MKE 3.8.8, Win2025 worker): Win2025 missed the deadline
• smoke-legacy (MKE 3.8.8, 6 Linux nodes): passes cleanly — confirms size/platform dependency

The timeout is internal to the MKE container; there is no --timeout flag on uninstall-ucp.

Fix

MKE documents the manual recovery path for this exact error:

"Remove the ucp-uninstall-agent and ucp-uninstall-agent-win services from a swarm manager, then force each node to leave the swarm."

UninstallMKE.Run() now detects the specific "Uninstalling UCP took too long" message and automatically executes that recovery:

1. Remove the stuck ucp-uninstall-agent / ucp-uninstall-agent-win services from the leader (best-effort).
2. Force all non-leader nodes to leave the swarm in parallel (per-node failures logged as warnings, not fatal).
3. Force the leader to leave last (hard failure if this fails).

All other uninstall-ucp errors continue to propagate as hard failures unchanged. The UninstallMCR phase that follows handles MCR cleanup on each host regardless of how the swarm was dissolved.

Changes

• pkg/product/mke/phase/uninstall_mke.go — captures Bootstrap output (not just error); isUninstallTimeout(output string) detects the timeout from the output stream; dissolveSwarm() fallback
• pkg/mcr/mcr.go — DrainNode no-ops when NodeID is empty (node already left the swarm after dissolution); also removes a pre-existing duplicate drainCmd execution
• pkg/product/mke/phase/uninstall_mke_test.go — unit tests for isUninstallTimeout

Copilot review fixes

Copilot identified that after dissolveSwarm() succeeds, UninstallMCR would call DrainNode which runs docker node update --availability drain <empty> — failing because the node is no longer in a swarm. Fixed in DrainNode: an empty NodeID is now treated as a no-op.

Copilot also surfaced that the initial isUninstallTimeout implementation checked the Bootstrap error string, but the timeout message "Uninstalling UCP took too long" is emitted at error level by MKE (not fatal), so it only appears in Bootstrap's output string. Fixed by capturing the output and checking it instead.

CI results

Run	Result	Notes
25912008675	✅ passed	After Copilot fix #1 (DrainNode guard) — fallback still did not fire (wrong detection)
25922740499	✅ passed	After Copilot fix #2 (output detection) — fallback fired and completed correctly

Confirmed from run 25922740499 logs:

MKE uninstall-ucp: Uninstalling UCP took too long!
[ssh] 44.192.39.182:22: force-leaving swarm
[ssh] 3.80.1.151:22: force-leaving swarm
[ssh] 32.197.224.118:22: force-leaving swarm
[ssh] 44.197.188.102:22: force-leaving swarm
[ssh] 100.48.222.71:22: force-leaving swarm
[ssh] 32.192.34.185:22: force-leaving swarm
[ssh] 32.197.48.207:22: force-leaving swarm (leader)
[ssh] 32.197.48.207:22: swarm dissolved; continuing with MCR uninstall

Reset() returned nil — no non-fatal warn, full reset completed successfully.

[Copilot]

Pull request overview

Adds an automatic recovery path for uninstall-ucp timeouts during MKE reset by detecting the known timeout error and force-dissolving the Swarm so the reset process can continue.

Changes:

• Detect uninstall-ucp “took too long” failures via isUninstallTimeout().
• Add dissolveSwarm() fallback that removes stuck uninstall-agent services and forces nodes to leave the swarm (leader last).
• Add unit tests for the timeout detector.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
pkg/product/mke/phase/uninstall_mke.go	Adds timeout detection and forced swarm dissolution fallback when uninstall-ucp times out.
pkg/product/mke/phase/uninstall_mke_test.go	Adds unit tests for isUninstallTimeout().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.