From 375e880f162404bde0c0870a061598e97d44304a Mon Sep 17 00:00:00 2001 From: Komh Date: Sat, 2 May 2026 16:47:14 +0000 Subject: [PATCH 1/2] [configure] Inspecting kubelet Garbage Collection and Eviction Configuration on ACP Nodes --- ...and_Eviction_Configuration_on_ACP_Nodes.md | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md diff --git a/docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md b/docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md new file mode 100644 index 000000000..83e06fd03 --- /dev/null +++ b/docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md @@ -0,0 +1,112 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- + +# Inspecting kubelet Garbage Collection and Eviction Configuration on ACP Nodes +## Issue + +Operators ask three related questions: + +- Has garbage collection been explicitly configured on this cluster, or is it running with defaults? +- What are the effective eviction thresholds for the nodes, and where did they come from? +- How can I see what the kubelet is actually using, as opposed to what some configuration object claims it should be using? + +These are the right questions to ask before tuning — editing the node configuration without first knowing the running values leads to changes that either silently do nothing (wrong pool) or produce a surprising outcome (stacked on top of an unnoticed explicit value). + +## Resolution + +There are three complementary inspection paths. Pick the one that matches the question being asked: the first is "what is the kubelet actually doing right now", the second is "what is on disk on the node", the third is "what is the node configuration CR committing to render". They should agree; when they disagree, that discrepancy is itself the finding. + +### Option 1 — Ask the kubelet what it is running + +The authoritative answer lives inside the kubelet process itself and is exposed by the `/configz` endpoint on every node. Query it through the API server proxy and extract only the garbage-collection and eviction-related fields: + +```bash +NODE=worker1.cluster.example.com + +kubectl get --raw "/api/v1/nodes/${NODE}/proxy/configz" \ + | jq '.kubeletconfig + | .kind = "KubeletConfiguration" + | .apiVersion = "kubelet.config.k8s.io/v1beta1"' \ + | jq '. | { + evictionHard, + evictionSoft, + evictionSoftGracePeriod, + evictionPressureTransitionPeriod, + imageMinimumGCAge, + imageGCHighThresholdPercent, + imageGCLowThresholdPercent + }' +``` + +A field that comes back as `null` or absent means the default is in effect (it has not been overridden). A typical default shape is: + +```text +{ + "evictionHard": { + "imagefs.available": "15%", + "memory.available": "100Mi", + "nodefs.available": "10%", + "nodefs.inodesFree": "5%" + }, + "evictionSoft": null, + "evictionSoftGracePeriod": null, + "evictionPressureTransitionPeriod": "5m0s", + "imageMinimumGCAge": "2m0s", + "imageGCHighThresholdPercent": 85, + "imageGCLowThresholdPercent": 80 +} +``` + +### Option 2 — Look at `kubelet.conf` on the node filesystem + +The kubelet renders its final configuration to `/etc/kubernetes/kubelet.conf`. Use a debug container chrooted to the host to read it directly: + +```bash +kubectl debug node/${NODE} -- cat /etc/kubernetes/kubelet.conf +``` + +The relevant fields are the same set shown above: `evictionHard`, `evictionSoft`, `evictionSoftGracePeriod`, `evictionPressureTransitionPeriod`, `imageMinimumGCAge`, `imageGCHighThresholdPercent`, `imageGCLowThresholdPercent`. If a value shows up here but not in the live `/configz`, the node has been reconfigured but not restarted. + +### Option 3 — Inspect the rendered node configuration object + +If the cluster manages node configuration through ACP's node configuration CR (under `configure/clusters/nodes`, or via the **Immutable Infrastructure** extension product), the committed-but-not-yet-rendered state lives in that CR, not on the node. Locate the rendered configuration for the relevant pool, find the file entry for `/etc/kubernetes/kubelet.conf`, and decode it — its contents use the same Ignition-style `data:` URI convention (base64 or URL-encoded inline content): + +```bash +# Adjust the selector/resource names to match the ACP node-config CR in use. +POOL=worker +kubectl get "${POOL}" -o json \ + | jq -r '.spec.configuration.files[] + | select(.path == "/etc/kubernetes/kubelet.conf") + | .contents.source' \ + | awk -F',' '{ print $2 }' \ + | base64 -d 2>/dev/null \ + | jq '{ + evictionHard, + evictionSoft, + evictionSoftGracePeriod, + evictionPressureTransitionPeriod, + imageMinimumGCAge, + imageGCHighThresholdPercent, + imageGCLowThresholdPercent + }' +``` + +If the `contents.source` uses URL encoding rather than base64, swap the `base64 -d` step for a URL-decode (for example `python3 -c 'import sys,urllib.parse; sys.stdout.write(urllib.parse.unquote(sys.stdin.read()))'`). + +This path is the one to use when reading from a support bundle / diagnostic archive: the live kubelet isn't reachable, but the rendered node configuration is preserved verbatim in the archive. + +## Diagnostic Steps + +If the three paths disagree, the disagreement is the answer: + +- **Option 3 shows a value, Option 2 agrees, Option 1 does not** — configuration has been committed and written to disk but the kubelet has not reloaded; confirm the kubelet service has been restarted on the target node since the change landed. +- **Option 1 shows a value, Option 3 is empty** — the value is the built-in kubelet default; no operator has overridden it. +- **Option 1 and Option 3 both show values but they differ** — there is another layer (a node-scoped override, a second pool matching the same node) that is winning. Inspect labels on the node and the selector in the node configuration CR(s) to find the duplicate. + +The kubelet emits `EvictionThresholdMet` and `ImageGCFailed` events on the node when these thresholds fire; `kubectl get events -A --field-selector involvedObject.name=${NODE}` surfaces them and is often the fastest way to confirm the eviction settings are being enforced as expected. From 09993686cf4cf6c39058e336b520748d62a0c26f Mon Sep 17 00:00:00 2001 From: Komh Date: Sun, 17 May 2026 02:40:41 +0000 Subject: [PATCH 2/2] [configure] Inspect kubelet garbage-collection and eviction settings on an ACP node --- ...on_and_eviction_settings_on_an_ACP_node.md | 75 ++++++++++++ ...and_Eviction_Configuration_on_ACP_Nodes.md | 112 ------------------ 2 files changed, 75 insertions(+), 112 deletions(-) create mode 100644 docs/en/solutions/Inspect_kubelet_garbage_collection_and_eviction_settings_on_an_ACP_node.md delete mode 100644 docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md diff --git a/docs/en/solutions/Inspect_kubelet_garbage_collection_and_eviction_settings_on_an_ACP_node.md b/docs/en/solutions/Inspect_kubelet_garbage_collection_and_eviction_settings_on_an_ACP_node.md new file mode 100644 index 000000000..884f9ebf9 --- /dev/null +++ b/docs/en/solutions/Inspect_kubelet_garbage_collection_and_eviction_settings_on_an_ACP_node.md @@ -0,0 +1,75 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- + +# Inspect kubelet garbage-collection and eviction settings on an ACP node + +## Issue + +Operators need to confirm which image garbage-collection and eviction thresholds the kubelet is currently using on a given ACP node — for example to debug pods being evicted under disk or memory pressure, or to verify that a change to the on-disk kubelet config has actually taken effect. The relevant settings live in the kubelet `KubeletConfiguration` object and cover a fixed set of fields: `imageGCHighThresholdPercent`, `imageGCLowThresholdPercent`, `imageMinimumGCAge`, `evictionHard`, `evictionSoft`, `evictionSoftGracePeriod`, and `evictionPressureTransitionPeriod`. + +These names and semantics come from upstream kubelet; ACP ships the kubelet unmodified, so the field shape is identical to what is documented for Kubernetes itself rather than something ACP-specific. + +## Root Cause + +On ACP the kubelet has no cluster-wide configuration CRD: there is no apiserver-managed custom resource that wraps the per-node kubelet config and reconciles it onto disk. Each kubelet reads its config directly from the on-disk file on its node, and any change to GC or eviction settings has to be made by editing that file on the node itself — there is no cluster-scoped object to `kubectl edit`. + +On the Ubuntu-based ACP node the kubelet's main configuration file lives at `/var/lib/kubelet/config.yaml`. GC and eviction values appear inline in that YAML; fields whose on-disk value is the sentinel zero (`0s`) or `null` are replaced at runtime with the kubelet's built-in defaults and only the substituted values become visible through the live kubelet. + +## Resolution + +On ACP install package v4.3.0-online (Ubuntu 22.04 nodes, Kubernetes v1.34.5, kubelet `cgroupDriver=systemd`), the supported way to read the in-effect kubelet GC and eviction values is the standard upstream apiserver-to-kubelet `configz` proxy, which still works unchanged on ACP. A request against `/api/v1/nodes//proxy/configz` returns a JSON body whose `.kubeletconfig` object carries the same GC and eviction fields listed above. + +A typical query for one node, filtered down to the GC and eviction fields, follows the standard upstream form: + +```bash +NODE= +kubectl get --raw "/api/v1/nodes/${NODE}/proxy/configz" \ + | jq '.kubeletconfig | { + evictionHard, + evictionSoft, + evictionSoftGracePeriod, + evictionPressureTransitionPeriod, + imageMinimumGCAge, + imageGCHighThresholdPercent, + imageGCLowThresholdPercent + }' +``` + +When the on-disk values (rather than the live, in-effect values) are needed — for instance to confirm which fields a node operator has explicitly overridden versus which are still defaults — the second supported probe is to read `/var/lib/kubelet/config.yaml` directly on the node. + +On a freshly installed v4.3.0-online cluster with Kubernetes v1.34.5 on Ubuntu 22.04, `configz` reports the following baseline values, which match the stock upstream kubelet defaults rather than an ACP-specific override: + +| Field | Default value | +| --- | --- | +| `evictionHard.memory.available` | `100Mi` | +| `evictionHard.nodefs.available` | `10%` | +| `evictionHard.nodefs.inodesFree` | `5%` | +| `evictionHard.imagefs.available` | `15%` | +| `evictionHard.imagefs.inodesFree` | `5%` | +| `evictionHard.pid.available` | `10%` | +| `imageMinimumGCAge` | `2m0s` | +| `imageGCHighThresholdPercent` | `85` | +| `imageGCLowThresholdPercent` | `80` | +| `evictionPressureTransitionPeriod` | `5m0s` | + +Because the kubelet on ACP is configured by the on-disk YAML file directly, the supported probes for the in-effect values are limited to the live `configz` proxy view and the per-node file at `/var/lib/kubelet/config.yaml`; there is no cluster-scoped resource to query for a rendered view. + +## Diagnostic Steps + +To establish a baseline before changing anything, capture the live configuration from the apiserver `configz` proxy for the node in question; the JSON returned is the authoritative view of what the kubelet is using at that moment: + +```bash +NODE= +kubectl get --raw "/api/v1/nodes/${NODE}/proxy/configz" \ + | jq '.kubeletconfig' +``` + +If a value reported by `configz` does not match what an operator expects to see, compare against the on-disk file. The on-disk YAML and the live `configz` view will agree on any field that carries an explicit non-sentinel value; fields whose on-disk value is `0s` or `null` are replaced at runtime with the kubelet's built-in defaults, so `configz` will show the substituted value rather than the literal `0s` / `null` recorded on disk. + +If a step in an unrelated workflow expects a cluster-scoped kubelet-config object to query, no such object exists on ACP; the two probes above (`configz` for live values, `/var/lib/kubelet/config.yaml` for on-disk values) are the only supported surfaces for inspecting GC and eviction settings on a node. diff --git a/docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md b/docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md deleted file mode 100644 index 83e06fd03..000000000 --- a/docs/en/solutions/Inspecting_kubelet_Garbage_Collection_and_Eviction_Configuration_on_ACP_Nodes.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -kind: - - How To -products: - - Alauda Container Platform -ProductsVersion: - - 4.1.0,4.2.x ---- - -# Inspecting kubelet Garbage Collection and Eviction Configuration on ACP Nodes -## Issue - -Operators ask three related questions: - -- Has garbage collection been explicitly configured on this cluster, or is it running with defaults? -- What are the effective eviction thresholds for the nodes, and where did they come from? -- How can I see what the kubelet is actually using, as opposed to what some configuration object claims it should be using? - -These are the right questions to ask before tuning — editing the node configuration without first knowing the running values leads to changes that either silently do nothing (wrong pool) or produce a surprising outcome (stacked on top of an unnoticed explicit value). - -## Resolution - -There are three complementary inspection paths. Pick the one that matches the question being asked: the first is "what is the kubelet actually doing right now", the second is "what is on disk on the node", the third is "what is the node configuration CR committing to render". They should agree; when they disagree, that discrepancy is itself the finding. - -### Option 1 — Ask the kubelet what it is running - -The authoritative answer lives inside the kubelet process itself and is exposed by the `/configz` endpoint on every node. Query it through the API server proxy and extract only the garbage-collection and eviction-related fields: - -```bash -NODE=worker1.cluster.example.com - -kubectl get --raw "/api/v1/nodes/${NODE}/proxy/configz" \ - | jq '.kubeletconfig - | .kind = "KubeletConfiguration" - | .apiVersion = "kubelet.config.k8s.io/v1beta1"' \ - | jq '. | { - evictionHard, - evictionSoft, - evictionSoftGracePeriod, - evictionPressureTransitionPeriod, - imageMinimumGCAge, - imageGCHighThresholdPercent, - imageGCLowThresholdPercent - }' -``` - -A field that comes back as `null` or absent means the default is in effect (it has not been overridden). A typical default shape is: - -```text -{ - "evictionHard": { - "imagefs.available": "15%", - "memory.available": "100Mi", - "nodefs.available": "10%", - "nodefs.inodesFree": "5%" - }, - "evictionSoft": null, - "evictionSoftGracePeriod": null, - "evictionPressureTransitionPeriod": "5m0s", - "imageMinimumGCAge": "2m0s", - "imageGCHighThresholdPercent": 85, - "imageGCLowThresholdPercent": 80 -} -``` - -### Option 2 — Look at `kubelet.conf` on the node filesystem - -The kubelet renders its final configuration to `/etc/kubernetes/kubelet.conf`. Use a debug container chrooted to the host to read it directly: - -```bash -kubectl debug node/${NODE} -- cat /etc/kubernetes/kubelet.conf -``` - -The relevant fields are the same set shown above: `evictionHard`, `evictionSoft`, `evictionSoftGracePeriod`, `evictionPressureTransitionPeriod`, `imageMinimumGCAge`, `imageGCHighThresholdPercent`, `imageGCLowThresholdPercent`. If a value shows up here but not in the live `/configz`, the node has been reconfigured but not restarted. - -### Option 3 — Inspect the rendered node configuration object - -If the cluster manages node configuration through ACP's node configuration CR (under `configure/clusters/nodes`, or via the **Immutable Infrastructure** extension product), the committed-but-not-yet-rendered state lives in that CR, not on the node. Locate the rendered configuration for the relevant pool, find the file entry for `/etc/kubernetes/kubelet.conf`, and decode it — its contents use the same Ignition-style `data:` URI convention (base64 or URL-encoded inline content): - -```bash -# Adjust the selector/resource names to match the ACP node-config CR in use. -POOL=worker -kubectl get "${POOL}" -o json \ - | jq -r '.spec.configuration.files[] - | select(.path == "/etc/kubernetes/kubelet.conf") - | .contents.source' \ - | awk -F',' '{ print $2 }' \ - | base64 -d 2>/dev/null \ - | jq '{ - evictionHard, - evictionSoft, - evictionSoftGracePeriod, - evictionPressureTransitionPeriod, - imageMinimumGCAge, - imageGCHighThresholdPercent, - imageGCLowThresholdPercent - }' -``` - -If the `contents.source` uses URL encoding rather than base64, swap the `base64 -d` step for a URL-decode (for example `python3 -c 'import sys,urllib.parse; sys.stdout.write(urllib.parse.unquote(sys.stdin.read()))'`). - -This path is the one to use when reading from a support bundle / diagnostic archive: the live kubelet isn't reachable, but the rendered node configuration is preserved verbatim in the archive. - -## Diagnostic Steps - -If the three paths disagree, the disagreement is the answer: - -- **Option 3 shows a value, Option 2 agrees, Option 1 does not** — configuration has been committed and written to disk but the kubelet has not reloaded; confirm the kubelet service has been restarted on the target node since the change landed. -- **Option 1 shows a value, Option 3 is empty** — the value is the built-in kubelet default; no operator has overridden it. -- **Option 1 and Option 3 both show values but they differ** — there is another layer (a node-scoped override, a second pool matching the same node) that is winning. Inspect labels on the node and the selector in the node configuration CR(s) to find the duplicate. - -The kubelet emits `EvictionThresholdMet` and `ImageGCFailed` events on the node when these thresholds fire; `kubectl get events -A --field-selector involvedObject.name=${NODE}` surfaces them and is often the fastest way to confirm the eviction settings are being enforced as expected.