-
Notifications
You must be signed in to change notification settings - Fork 18
[kb] Cold VM Migration Hangs Because NetworkPolicy in the Target Namespace Blocks the virt-v2v Pod #560
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jing2uo
merged 2 commits into
main
from
kb/2026-04/cold-vm-migration-from-vmware-hangs-at-c
May 14, 2026
Merged
[kb] Cold VM Migration Hangs Because NetworkPolicy in the Target Namespace Blocks the virt-v2v Pod #560
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
96 changes: 96 additions & 0 deletions
96
..._Hangs_Because_NetworkPolicy_in_the_Target_Namespace_Blocks_the_virt_v2v_Pod.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| --- | ||
| kind: | ||
| - Troubleshooting | ||
| products: | ||
| - Alauda Container Platform | ||
| ProductsVersion: | ||
| - 4.1.0,4.2.x | ||
| --- | ||
|
|
||
| # Cold VM Migration Hangs Because NetworkPolicy in the Target Namespace Blocks the virt-v2v Pod | ||
| ## Issue | ||
|
|
||
| A cold VM migration from VMware into ACP virtualization (KubeVirt), | ||
| driven by the Alauda Build of Forklift Operator, hangs at the | ||
| conversion-progress step. The `forklift-controller` logs repeat | ||
| `Failed to update conversion progress`, and the migration's | ||
| `virt-v2v` pod in the target namespace logs that it is serving but | ||
| seeing no consumer reach it. | ||
|
|
||
| ## Root Cause | ||
|
|
||
| At the conversion step, the per-VM `virt-v2v` pod is launched in the | ||
| **target namespace** (the namespace where the imported VM will live) | ||
| and exposes the VM's XML on port `8080`. The `forklift-controller` | ||
| runs in `konveyor-forklift` and polls that endpoint to read | ||
| conversion progress. | ||
|
|
||
| If the target namespace has a `NetworkPolicy` that denies ingress by | ||
| default (or restricts ingress to a label set that does not include | ||
| the Forklift namespace), the controller's TCP connection to the | ||
| `virt-v2v` pod's `:8080` is dropped. The migration pod is healthy in | ||
| isolation; the controller cannot observe its progress, so it never | ||
| advances past the conversion step. | ||
|
|
||
| ## Resolution | ||
|
|
||
| Add a `NetworkPolicy` to the **target namespace** allowing ingress on | ||
| port `8080` from the `konveyor-forklift` namespace: | ||
|
|
||
| ```yaml | ||
| apiVersion: networking.k8s.io/v1 | ||
| kind: NetworkPolicy | ||
| metadata: | ||
| name: allow-forklift-controller | ||
| namespace: <target-namespace> | ||
| spec: | ||
| podSelector: {} # apply to virt-v2v pods (and any others) | ||
| policyTypes: | ||
| - Ingress | ||
| ingress: | ||
| - from: | ||
| - namespaceSelector: | ||
| matchLabels: | ||
| kubernetes.io/metadata.name: konveyor-forklift | ||
| ports: | ||
| - protocol: TCP | ||
| port: 8080 | ||
| ``` | ||
|
|
||
| Apply, then re-run the migration plan. The controller's polling reaches | ||
| `virt-v2v` and conversion progress advances normally. | ||
|
|
||
| If existing policies in the target namespace use a different selector | ||
| shape (named labels, allow-all-ingress with a deny-list, etc.), add a | ||
| single allow rule equivalent to the above; the rule is purely | ||
| additive. | ||
|
|
||
| ## Diagnostic Steps | ||
|
|
||
| Confirm where the controller and the conversion pod live: | ||
|
|
||
| ```bash | ||
| kubectl -n konveyor-forklift get deploy forklift-controller | ||
| kubectl -n <target-namespace> get pod -l job-name -o wide # the virt-v2v pod is a Job-spawned pod | ||
| ``` | ||
|
|
||
| Confirm a NetworkPolicy is in fact blocking the path: | ||
|
|
||
| ```bash | ||
| kubectl -n <target-namespace> get networkpolicy | ||
| # Pick the policy and check its ingress.from — if no namespaceSelector | ||
| # admits `konveyor-forklift`, that is the block. | ||
| ``` | ||
|
|
||
| Check the controller logs for `Failed to update conversion progress` | ||
| or connection-refused / connection-timeout messages against the target | ||
| pod IP: | ||
|
|
||
| ```bash | ||
| kubectl -n konveyor-forklift logs deploy/forklift-controller | \ | ||
| grep -E 'conversion progress|virt-v2v' | tail -20 | ||
| ``` | ||
|
|
||
| A timeout from the controller's IP to the virt-v2v pod's IP confirms | ||
| the policy block. A successful `:8080` response after the policy is in | ||
| place confirms the fix. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🌐 Web query:
In Kubernetes NetworkPolicy, doespodSelector: {}select all pods in the policy namespace, and what is the recommended least-privilege pattern for narrowing access to a specific workload?💡 Result:
Yes. In a Kubernetes NetworkPolicy,
spec.podSelector: {}(empty label selector) matches all pods in the policy’s namespace, so the policy applies to every pod there [1][2]. Least-privilege pattern to narrow access to a specific workload: 1) Use a “default deny” NetworkPolicy that selects all pods in the namespace (e.g.,podSelector: {}) and sets the relevantpolicyTypes(commonlyIngressand/orEgress) with no allow rules, so nothing is permitted by default [3]. 2) Create a second NetworkPolicy that selects only the target workload pods (e.g.,podSelector.matchLabelsfor that app/deployment) and then explicitly allow only the required sources/destinations by usingingress.from/egress.topeers with bothnamespaceSelector(if needed) andpodSelector.matchLabels(for the specific client/workload pods) [1][3]. Example (narrow ingress to a specific backend app): - Default deny ingress for the namespace (select all pods): apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress namespace: my-namespace spec: podSelector: {} policyTypes: - Ingress - Then allow only from the specific frontend pods to the specific backend pods: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: my-namespace spec: podSelector: matchLabels: app: backend policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend This approach is least-privilege because the first policy isolates everything, and the second policy opens only the minimum explicitly required communication path to/from precisely labeled pods, rather than applying broadly with an empty selector [1][3].Citations:
Scope the allow policy to virt-v2v pods only (least privilege).
Line 47 uses
podSelector: {}, which allowskonveyor-forkliftto reach port 8080 on all pods in the target namespace, not just migration pods. Narrow this to virt-v2v pod labels to match Kubernetes least-privilege best practices.Suggested doc change
📝 Committable suggestion
🤖 Prompt for AI Agents