Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

354 changes: 177 additions & 177 deletions architecture/custom-vm-runtime.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -605,7 +605,7 @@ The gateway reaches the sandbox exclusively through the supervisor-initiated `Co

- **Create**: The VM driver process allocates a sandbox-specific rootfs from its own embedded `rootfs.tar.zst`, injects an explicitly configured guest mTLS bundle when the gateway callback endpoint is `https://`, then re-execs itself in a hidden helper mode that loads libkrun directly and boots the supervisor.
- **Networking**: The helper starts an embedded `gvproxy`, wires it into libkrun as virtio-net, and gives the guest outbound connectivity. No inbound TCP listener is needed — the supervisor reaches the gateway over its outbound `ConnectSupervisor` stream.
- **Gateway callback**: The guest init script configures `eth0` for gvproxy networking, prefers the configured `OPENSHELL_GRPC_ENDPOINT`, and falls back to host aliases or the gvproxy gateway IP (`192.168.127.1`) when local hostname resolution is unavailable on macOS.
- **Gateway callback**: The guest init script configures `eth0` for gvproxy networking, seeds `/etc/hosts` so `host.openshell.internal` resolves to the gvproxy gateway IP (`192.168.127.1`), preserves gvproxy's legacy `host.containers.internal` / `host.docker.internal` DNS answers, prefers the configured `OPENSHELL_GRPC_ENDPOINT`, and falls back to those aliases or the raw gateway IP when local hostname resolution is unavailable on macOS.
- **Guest boot**: The sandbox guest runs a minimal init script that starts `openshell-sandbox` directly as PID 1 inside the VM.
- **Watch stream**: Emits provisioning, ready, error, deleting, deleted, and platform-event updates so the gateway store remains the durable source of truth.

Expand Down
8 changes: 8 additions & 0 deletions crates/openshell-driver-vm/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,13 @@ libloading = "0.8"
tar = "0.4"
zstd = "0.13"

# smol-rs/polling drives the BSD/macOS parent-death detection in
# procguard via kqueue's EVFILT_PROC / NOTE_EXIT filter. We could use
# it on Linux too (via epoll + pidfd) but sticking with
# nix::sys::prctl::set_pdeathsig there keeps the Linux path a single
# syscall with no helper thread.
[target.'cfg(any(target_os = "macos", target_os = "ios", target_os = "freebsd", target_os = "netbsd", target_os = "openbsd", target_os = "dragonfly"))'.dependencies]
polling = "3.11"

[lints]
workspace = true
7 changes: 0 additions & 7 deletions crates/openshell-driver-vm/Makefile

This file was deleted.

46 changes: 33 additions & 13 deletions crates/openshell-driver-vm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,19 +31,15 @@ Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside th

## Quick start (recommended)

`start.sh` handles runtime setup, builds, codesigning, and environment wiring. From the repo root:

```shell
crates/openshell-driver-vm/start.sh
mise run gateway:vm
```

or equivalently:

```shell
make -C crates/openshell-driver-vm start
```

First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy and `mise run vm:rootfs -- --base` builds the embedded rootfs. Subsequent runs are cached. State lives under `target/openshell-vm-driver-dev/` (SQLite DB + per-sandbox rootfs + `compute-driver.sock`).
First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy and `mise run vm:rootfs -- --base` builds the embedded rootfs. Subsequent runs are cached. To keep the Unix socket path under macOS `SUN_LEN`, `mise run gateway:vm` and `start.sh` default the state dir to `/tmp/openshell-vm-driver-dev-$USER-port-$PORT/` (SQLite DB + per-sandbox rootfs + `compute-driver.sock`) unless `OPENSHELL_VM_DRIVER_STATE_DIR` is set.
The wrapper also prints the recommended gateway name (`vm-driver-port-$PORT` by default) plus the exact repo-local `scripts/bin/openshell gateway add` and `scripts/bin/openshell gateway select` commands to use from another terminal. This avoids accidentally hitting an older `openshell` binary elsewhere on your `PATH`.
It also exports `OPENSHELL_DRIVER_DIR=$PWD/target/debug` before starting the gateway so local dev runs use the freshly built `openshell-driver-vm` instead of an older installed copy from `~/.local/libexec/openshell` or `/usr/local/libexec`.

Override via environment:

Expand All @@ -53,10 +49,33 @@ OPENSHELL_SSH_HANDSHAKE_SECRET=$(openssl rand -hex 32) \
crates/openshell-driver-vm/start.sh
```

Run multiple dev gateways side by side by giving each one a unique port. The wrapper derives a distinct default state dir from that port automatically:

```shell
OPENSHELL_SERVER_PORT=8080 mise run gateway:vm
OPENSHELL_SERVER_PORT=8081 mise run gateway:vm
```

If you want a custom suffix instead of `port-$PORT`, set `OPENSHELL_VM_INSTANCE`:

```shell
OPENSHELL_SERVER_PORT=8082 \
OPENSHELL_VM_INSTANCE=feature-a \
mise run gateway:vm
```

If you want a custom CLI gateway name, set `OPENSHELL_VM_GATEWAY_NAME`:

```shell
OPENSHELL_SERVER_PORT=8082 \
OPENSHELL_VM_GATEWAY_NAME=vm-feature-a \
mise run gateway:vm
```

Teardown:

```shell
rm -rf target/openshell-vm-driver-dev
rm -rf /tmp/openshell-vm-driver-dev-$USER-port-8080
```

## Manual equivalent
Expand All @@ -78,16 +97,17 @@ codesign \
--force -s - target/debug/openshell-driver-vm

# 4. Start the gateway with the VM driver
mkdir -p target/openshell-vm-driver-dev
mkdir -p /tmp/openshell-vm-driver-dev-$USER-port-8080
target/debug/openshell-gateway \
--drivers vm \
--disable-tls \
--database-url sqlite:target/openshell-vm-driver-dev/openshell.db \
--database-url sqlite:/tmp/openshell-vm-driver-dev-$USER-port-8080/openshell.db \
--driver-dir $PWD/target/debug \
--grpc-endpoint http://host.containers.internal:8080 \
--ssh-handshake-secret dev-vm-driver-secret \
--ssh-gateway-host 127.0.0.1 \
--ssh-gateway-port 8080 \
--vm-driver-state-dir $PWD/target/openshell-vm-driver-dev
--vm-driver-state-dir /tmp/openshell-vm-driver-dev-$USER-port-8080
```

The gateway resolves `openshell-driver-vm` in this order: `--driver-dir`, conventional install locations (`~/.local/libexec/openshell`, `/usr/local/libexec/openshell`, `/usr/local/libexec`), then a sibling of the gateway binary.
Expand All @@ -97,7 +117,7 @@ The gateway resolves `openshell-driver-vm` in this order: `--driver-dir`, conven
| Flag | Env var | Default | Purpose |
|---|---|---|---|
| `--drivers vm` | `OPENSHELL_DRIVERS` | `kubernetes` | Select the VM compute driver. |
| `--grpc-endpoint URL` | `OPENSHELL_GRPC_ENDPOINT` | — | Required. URL the sandbox guest calls back to. Use a host alias that resolves to the gateway's host from inside the VM (gvproxy answers `host.containers.internal` and `host.openshell.internal` to `192.168.127.1`). |
| `--grpc-endpoint URL` | `OPENSHELL_GRPC_ENDPOINT` | — | Required. URL the sandbox guest calls back to. Use a host alias that resolves to the gateway's host from inside the VM (`host.containers.internal` comes from gvproxy DNS; the guest init script also seeds `host.openshell.internal` to `192.168.127.1`). |
| `--vm-driver-state-dir DIR` | `OPENSHELL_VM_DRIVER_STATE_DIR` | `target/openshell-vm-driver` | Per-sandbox rootfs, console logs, and the `compute-driver.sock` UDS. |
| `--driver-dir DIR` | `OPENSHELL_DRIVER_DIR` | unset | Override the directory searched for `openshell-driver-vm`. |
| `--vm-driver-vcpus N` | `OPENSHELL_VM_DRIVER_VCPUS` | `2` | vCPUs per sandbox. |
Expand Down
23 changes: 20 additions & 3 deletions crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
set -euo pipefail

BOOT_START=$(date +%s%3N 2>/dev/null || date +%s)
GVPROXY_GATEWAY_IP="192.168.127.1"

ts() {
local now
Expand Down Expand Up @@ -72,6 +73,20 @@ tcp_probe() {
fi
}

ensure_host_gateway_aliases() {
local hosts_tmp="/tmp/openshell-hosts.$$"

if [ -f /etc/hosts ]; then
grep -vE '(^|[[:space:]])host\.openshell\.internal([[:space:]]|$)' /etc/hosts > "$hosts_tmp" || true
else
: > "$hosts_tmp"
fi

printf '%s host.openshell.internal\n' "$GVPROXY_GATEWAY_IP" >> "$hosts_tmp"
cat "$hosts_tmp" > /etc/hosts
rm -f "$hosts_tmp"
}

rewrite_openshell_endpoint_if_needed() {
local endpoint="${OPENSHELL_ENDPOINT:-}"
[ -n "$endpoint" ] || return 0
Expand All @@ -92,7 +107,7 @@ rewrite_openshell_endpoint_if_needed() {
return 0
fi

for candidate in host.containers.internal host.docker.internal 192.168.127.1; do
for candidate in host.openshell.internal host.containers.internal host.docker.internal "$GVPROXY_GATEWAY_IP"; do
if [ "$candidate" = "$host" ]; then
continue
fi
Expand Down Expand Up @@ -163,18 +178,20 @@ DHCP_SCRIPT
if ! udhcpc -i eth0 -f -q -n -T 1 -t 3 -A 1 -s "$UDHCPC_SCRIPT" 2>&1; then
ts "WARNING: DHCP failed, falling back to static config"
ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true
ip route add default via 192.168.127.1 2>/dev/null || true
ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true
fi
else
ts "no DHCP client, using static config"
ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true
ip route add default via 192.168.127.1 2>/dev/null || true
ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true
fi

if [ ! -s /etc/resolv.conf ]; then
echo "nameserver 8.8.8.8" > /etc/resolv.conf
echo "nameserver 8.8.4.4" >> /etc/resolv.conf
fi

ensure_host_gateway_aliases
else
ts "WARNING: eth0 not found; supervisor will start without guest egress"
fi
Expand Down
Loading
Loading