5.5.26

[scottyhq]

Attendees: Eric, Anthony, Rob, Scott

Location: eScience Seminar Room

Agenda:

• ~5 min: voluntary updates and/or introductions
• ~10 min: @egagli to share a little bit of code and a figure for feedback
• ~40 min: co-working time on topics of interest
• ~5 min: update discussion thread with updates and todos

Notes:

• people are welcome to linger in the eScience studio and continue to work!

[scottyhq]

Notes:

• Follow up from last week. Why not use s3 links / work in a AWS jupyterhub? Short answer: want to use local vscode app & ssh connection for familiar development experience with copilot / claude code.

@aaarendt @egagli
Here are docs on how to do that: https://github.com/huskysar/nisar-examples/blob/main/JupyterHubs.md

seems maybe some additional steps might be needed for windows WSL?... https://stackoverflow.com/questions/60150466/can-i-ssh-from-wsl-in-visual-studio-code

[aaarendt]

Connecting local VS Code to CryoCloud via VS Code Tunnels

A guide for connecting your local VS Code Desktop to a CryoCloud JupyterHub pod
using Microsoft's Remote - Tunnels feature, as an alternative to the
jupyter-sshd-proxy + websocat SSH approach documented for VEDA and CryoCloud.

This approach also works for NASA VEDA and other 2i2c-managed pangeo-style
JupyterHubs — the only thing that changes is the hub URL you launch your pod from.

Why tunnels instead of SSH-over-websocket?

The current jupyter-sshd-proxy approach (see
VEDA's ssh docs
and the NISAR JupyterHubs notes)
tunnels SSH inside a WebSocket carrying your hub API token. It works, but it has
several rough edges:

• Requires websocat installed locally and a hand-edited ~/.ssh/config.
• Requires the pod's container image to ship a working jupyter-sshd-proxy
  (the default CryoCloud image historically has not).
• The hub's own docs flag it as experimental and not supported long-term.
• SSH sessions are spawned outside JupyterHub's process tree and therefore do
  not inherit the IRSA environment variables (AWS_ROLE_ARN,
  AWS_WEB_IDENTITY_TOKEN_FILE, etc.) that grant S3 access. This is why the
  NISAR doc carries a ~/.bashrc snippet that scrapes /proc/1/environ to
  reconstruct them, plus a 00-env.py IPython startup script for kernels.
• Token expiry, pod restarts, and proxy hiccups leave stale SSH config entries
  pointing at dead pods.

VS Code Tunnels sidestep all of this. The code tunnel process runs as a child
of the JupyterHub-spawned notebook server, so it inherits IRSA env vars
naturally — no /proc/1/environ hack required. The remote process reaches
outbound to a Microsoft relay over an authenticated WebSocket; your local VS
Code connects to that relay by tunnel name. There are no inbound ports, no SSH
keys, no ~/.ssh/config edits, and no per-image proxy dependency.

Prerequisites

• A working CryoCloud account and the ability to launch a server at
  https://hub.cryointhecloud.com.
• VS Code Desktop installed locally.
• A GitHub or Microsoft account (you'll authenticate both ends of the tunnel
  with the same account).

Step 1: Launch a CryoCloud server and open a terminal

1. Go to https://hub.cryointhecloud.com and start a server with whatever
   resource profile you need. The default works fine for setup.
2. Once JupyterLab loads, open File → New → Terminal.
3. You should see a prompt like jovyan@jupyter-<yourname>:~$. This shell is
   running inside your pod — that's where the tunnel needs to live.

Step 2: Install the VS Code CLI

The code CLI is a single static binary. Drop it in ~/bin:

mkdir -p ~/bin
cd /tmp
curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' -o vscode_cli.tar.gz
tar -xzf vscode_cli.tar.gz
mv code ~/bin/
chmod +x ~/bin/code
export PATH="$HOME/bin:$PATH"
code --version

Note: Use the cli-alpine-x64 build. It's statically linked against
musl and runs cleanly across the pangeo / cryocloud base images regardless
of glibc version. Don't substitute the linux-x64 build.

code --version should print three lines (version, commit, architecture). If
it does, the binary is good.

Persist the PATH change so future sessions pick it up automatically:

echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc

Step 3: Start the tunnel

code tunnel --accept-server-license-terms --name cryocloud-$(whoami)

The --name flag is the label you'll see in your local VS Code's Remote
Explorer. It must be unique within your GitHub/Microsoft account, lowercase,
no spaces. Adjust as you like.

You'll see output similar to:

*
* Visual Studio Code Server
*
To grant access to the server, please log into https://github.com/login/device
and use code XXXX-XXXX

Open that URL in your laptop's browser, paste the code, and approve. Back in
the pod terminal you'll then see:

Open this link in your browser https://vscode.dev/tunnel/cryocloud-yourname
[timestamp] [info] Tunnel name: cryocloud-yourname

Once you see Tunnel name: the tunnel is up. Leave this terminal tab
open for now — the tunnel dies if this foreground process exits.
Step 6 covers how to make it survive.

Step 4: Connect from local VS Code

On your laptop:

1. Install the Remote - Tunnels extension (publisher ms-vscode, exact
   name "Remote - Tunnels"). If you already have the "Remote Development"
   extension pack, you have it.
2. Click the Accounts icon at the bottom-left of VS Code → Sign in with
   GitHub to use Remote Tunnels. Use the same GitHub account you
   authenticated the CLI with in step 3 — that account linkage is what pairs
   the two ends.
3. Open the Command Palette (Cmd+Shift+P / Ctrl+Shift+P) →
   Remote-Tunnels: Connect to Tunnel → pick cryocloud-yourname from the
   list.
4. VS Code reloads into a remote window. The bottom-left status bar should
   now read Tunnel: cryocloud-yourname.
5. File → Open Folder → /home/jovyan (or whatever subfolder you work
   in — typically a git repo clone).

First connect takes 20-60 seconds while VS Code Server installs into
~/.vscode-server on the pod side. Subsequent connects are near-instant.

Step 5: Verify IRSA / S3 access works

This is the payoff over the SSH approach. Open a terminal inside the remote
VS Code window (Ctrl+`) and run:

env | grep -E 'AWS_ROLE_ARN|AWS_WEB_IDENTITY|AWS_REGION'

Under tunnels these should be populated automatically — no ~/.bashrc hack,
no /proc/1/environ scraping. Then confirm AWS access:

aws sts get-caller-identity

You should see the assumed role ARN (something like
arn:aws:sts::...:assumed-role/...) with no additional configuration.
This means boto3, rasterio, xarray, etc. will pick up credentials
automatically for S3 reads.

Step 6: Keep the tunnel running

The foreground process in step 3 dies when its terminal closes, taking the
tunnel with it. Pick one of these:

Option A: nohup (quick, per-pod)

# Ctrl+C the foreground tunnel from step 3 first, then:
nohup code tunnel --accept-server-license-terms --name cryocloud-$(whoami) > ~/tunnel.log 2>&1 &
disown

The tunnel survives terminal closure but dies when the pod is culled or
restarted (CryoCloud culls idle pods after roughly an hour on the default
profile). You'll need to re-run this each time you launch a fresh pod. Tail
~/tunnel.log if anything misbehaves.

Option B: code tunnel service install (probably won't work in z2jh pods)

code tunnel service install --accept-server-license-terms --name cryocloud-$(whoami)

This registers the tunnel under systemd-user. JupyterHub pods generally
don't run systemd, so expect this to fail with a "systemd not found" error.
If it works, great; if not, fall back to Option A.

Option C: Ask the hub admins for a postStart hook (long-term fix)

The clean long-term solution is to have the hub itself drop the binary and
start the tunnel automatically when each pod boots, via a z2jh
singleuser.lifecycleHooks.postStart hook. That makes tunnels a hub-wide
feature rather than a per-user workaround.

For CryoCloud, file an issue at
https://github.com/CryoInTheCloud/cryocloudwebsite or raise it in the
community Slack.

For VEDA, the equivalent is the
veda-jh-deployment repo
or the veda-docs repo.

Disconnecting

• From local VS Code: File → Close Remote Connection, or just close
  the window. The tunnel keeps running on the pod and you can reconnect later
  without re-authenticating.
• From the pod side: kill the code tunnel process (pkill -f 'code tunnel')
  or stop the server from the JupyterHub control panel. When the pod terminates,
  the tunnel goes offline cleanly and disappears from your local VS Code's
  Remote Explorer.

Caveats

• Account binding. Each tunnel is tied to one GitHub or Microsoft account.
  You can't share a tunnel between teammates — each person runs their own.
• Microsoft relay dependency. Traffic is end-to-end encrypted, but the
  relay is operated by Microsoft. If your project has a "no third-party SaaS
  in the data path" policy, raise this with your project lead before adopting.
• Quotas. Microsoft enforces undocumented per-account limits on
  concurrent tunnels and total data transfer. For normal interactive use you
  won't hit them.
• Doesn't replace pod launch. Tunnels solve the connect step; you still
  need to start a pod first. See the Pod-launch automation
  section for how to drive that from the command line so you never need to
  open the hub UI either.
• Browser fallback. If you're on a machine without local VS Code,
  https://vscode.dev → "Connect to Tunnel" gives you the same environment
  in a browser tab.

Pod-launch automation (no-browser workflow)

Once tunnels are working, the natural next step is to skip the JupyterHub web
UI entirely. The NISAR doc shows that JupyterHub exposes a REST API for
spawning pods (POST /hub/api/users/<user>/servers/). Combined with the
auto-start tunnel below, your daily flow becomes a single command on your
laptop: pod boots, tunnel registers, you connect from VS Code.

One-time pod-side setup: auto-start the tunnel

Open a JupyterLab terminal once to do this initial setup. Everything goes
into your NFS home directory (/home/jovyan), so it persists across pod
restarts.

# Install the CLI (same as Step 2 above; skip if you've done it)
mkdir -p ~/bin
cd /tmp
curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' -o vscode_cli.tar.gz
tar -xzf vscode_cli.tar.gz
mv code ~/bin/
chmod +x ~/bin/code

# Add an auto-start guard to ~/.bashrc
cat >> ~/.bashrc << 'EOF'

# Auto-start VS Code tunnel if not already running
export PATH="$HOME/bin:$PATH"
if [ -x "$HOME/bin/code" ] && ! pgrep -f "code tunnel" > /dev/null 2>&1; then
    nohup "$HOME/bin/code" tunnel --accept-server-license-terms \
        --name "$(hostname | sed 's/jupyter-//')" \
        > ~/tunnel.log 2>&1 &
    disown
fi
EOF

Authenticate the CLI once now (~/bin/code tunnel --accept-server-license-terms --name $(hostname | sed 's/jupyter-//')) so your GitHub device-code grant is
cached in ~/.vscode-cli/. You can Ctrl+C immediately after the tunnel
registers — the credential cache persists on NFS, so future auto-starts won't
prompt.

One-time local setup: get a JupyterHub API token

From the hub UI (one time only): visit
https://hub.cryointhecloud.com/hub/token, generate a long-lived token, and
stash it somewhere safe — your OS keychain, a .env file with chmod 600,
or a password manager. The NISAR doc has equivalent URLs for VEDA
(hub.openveda.cloud/hub/token).

Local launch script

Drop this into your laptop's ~/.zshrc or ~/.bashrc:

launch-cryocloud() {
    export JHUB_URL=https://hub.cryointhecloud.com
    export JHUB_TOKEN="${CRYOCLOUD_TOKEN:?set CRYOCLOUD_TOKEN in your env}"
    export JHUB_USER=<your-username>           # edit this
    local vm="${1:-mem_4_gb}"                  # default profile, override as arg

    echo "Requesting pod ($vm)..."
    curl -sS -X POST \
        "$JHUB_URL/hub/api/users/$JHUB_USER/servers/" \
        -H "Authorization: token $JHUB_TOKEN" \
        -H "Content-Type: application/json" \
        -d "$(printf '{"profile":"cpu-only","image":"01-python","resource_allocation":"%s"}' "$vm")" \
        -o /dev/null -w "HTTP %{http_code}\n"

    echo "Waiting for pod to be ready..."
    while true; do
        local ready
        ready=$(curl -sS "$JHUB_URL/hub/api/users/$JHUB_USER" \
            -H "Authorization: token $JHUB_TOKEN" | jq -r '.servers[""].ready // false')
        [ "$ready" = "true" ] && break
        sleep 3
    done

    echo "Pod ready. Tunnel will register with the relay in ~10-20s."
    echo "In VS Code: Cmd+Shift+P → Remote-Tunnels: Connect to Tunnel → cryocloud-$JHUB_USER"
}

stop-cryocloud() {
    export JHUB_URL=https://hub.cryointhecloud.com
    export JHUB_TOKEN="${CRYOCLOUD_TOKEN:?set CRYOCLOUD_TOKEN in your env}"
    export JHUB_USER=<your-username>           # edit this
    curl -sS -X DELETE \
        "$JHUB_URL/hub/api/users/$JHUB_USER/servers/" \
        -H "Authorization: token $JHUB_TOKEN" \
        -o /dev/null -w "HTTP %{http_code}\n"
}

Daily flow:

$ export CRYOCLOUD_TOKEN=<from-your-keychain>
$ launch-cryocloud mem_29_gb
Requesting pod (mem_29_gb)...
HTTP 201
Waiting for pod to be ready...
Pod ready. Tunnel will register with the relay in ~10-20s.
In VS Code: Cmd+Shift+P → Remote-Tunnels: Connect to Tunnel → cryocloud-yourname

For a GPU pod on CryoCloud, swap the -d payload in launch-cryocloud to
'{"profile":"gpu","image":"pytorch"}' per the NISAR doc. For VEDA, change
JHUB_URL to https://hub.openveda.cloud and the payload to use VEDA's
choose-your-environment-and-resources profile name.

The honest catch: bashrc only fires when something sources it

The auto-start guard above lives in ~/.bashrc, which only runs when an
interactive bash shell starts. A freshly-spawned JupyterHub pod does not
automatically start a login shell — it runs jupyter-server as PID 1 and
that's it. So after launch-cryocloud reports the pod is ready, you need
one thing to source bashrc and trigger the auto-start.

Pick whichever fits your team:

Workaround A — JupyterLab-terminal trigger (least invasive).
After the pod is ready, hit the hub URL (https://hub.cryointhecloud.com)
in your browser once and open a terminal in JupyterLab. That sources bashrc,
the tunnel starts, then you close the browser tab and connect from local VS
Code. You're using the browser, but only as a one-shot trigger — your actual
work happens in local VS Code.

Workaround B — Wake bashrc via the existing SSH path.
If your team already has the jupyter-sshd-proxy SSH path working, run
ssh hub.cryointhecloud.com 'source ~/.bashrc' from your laptop right after
launch-cryocloud. Ironic, but it works: SSH starts a login shell, bashrc
fires, tunnel comes up, you disconnect SSH and connect via tunnel. You can
fold this into the launch script.

Workaround C — Curl the hub's terminal-spawn endpoint.
JupyterLab has a REST endpoint for creating terminado sessions
(POST /user/<user>/api/terminals with the user's API token). Hitting this
spawns a shell inside the pod, sources bashrc, and triggers the auto-start
without you ever loading the UI. This is the cleanest scriptable option but
needs a per-user JupyterLab token (different from the hub token) — ask the
admins or pull it from the JupyterLab settings UI once.

Workaround D — Hub-side postStart hook (real long-term fix).
Ask the CryoCloud / VEDA admins to add this to the z2jh config:

singleuser:
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - |
            if [ -x "$HOME/bin/code" ] && ! pgrep -f "code tunnel" > /dev/null; then
              nohup "$HOME/bin/code" tunnel --accept-server-license-terms \
                --name "$(hostname | sed 's/jupyter-//')" \
                > "$HOME/tunnel.log" 2>&1 &
            fi

This runs inside the pod's process tree as a child of the notebook server,
so IRSA env vars are inherited correctly, and it fires automatically on every
pod start. No bashrc trigger needed. This is the version worth proposing to
2i2c once your team has confirmed the workflow works.

Suggested rollout for your team

1. Get everyone working with the manual flow (Steps 1-6 above).
2. Layer on the local launch script + bashrc auto-start with workaround A or B.
3. Once you have a few weeks of usage data, file an issue with the hub
   (CryoCloud: https://github.com/CryoInTheCloud/cryocloudwebsite; VEDA:
   https://github.com/nasa-impact/veda-jh-deployment) proposing the
   postStart hook — workaround D — with a writeup of why it cleans up
   several pain points the existing SSH path has.

Comparison with the SSH approach

	SSH-over-websocket (jupyter-sshd-proxy)	VS Code Tunnels
Local prerequisites	websocat, ~/.ssh/config edits, SSH keys in ~/.ssh/authorized_keys	Remote - Tunnels extension
Pod-side requirement	Image must ship jupyter-sshd-proxy	code CLI (single binary, no system packages)
Authentication	JupyterHub API token + SSH public key	GitHub / Microsoft account
IRSA env vars in session	Not inherited; requires /proc/1/environ hack	Inherited automatically
Kernel env vars	Requires ~/.ipython/profile_default/startup/00-env.py	Inherited automatically
Hub support status	Experimental on VEDA	Not yet officially adopted, but works without hub-side changes
Inbound ports	None (tunneled over hub HTTPS)	None (outbound WebSocket to relay)

References

• VS Code Remote - Tunnels documentation
• VEDA: How to ssh into the VEDA JupyterHub (the SSH approach)
• VEDA: How to use VS Code on the VEDA JupyterHub (browser-based code-server)
• NISAR examples — JupyterHubs notes
• jupyter-sshd-proxy (Yuvi Panda)