Remote development

The thin client development pattern transforms any device into a terminal window to a powerful remote workstation or server. Instead of running resource-intensive tools locally, a laptop or tablet connects via SSH to a remote machine where the actual development environment runs. Sessions persist on the remote machine across disconnects, network changes, and even local reboots. This page covers the networking tools (Mosh, SSH port forwarding) and session management (Zellij) that make this workflow practical.

The thin client pattern

In a thin client workflow:

1. A workstation or server hosts the development environment with all necessary tools, source code, and running services.
2. A lightweight client device connects via terminal over a mesh VPN.
3. Sessions persist on the remote machine regardless of the client's connectivity state.
4. The developer disconnects and reconnects without losing state.

This pattern aligns with Keystone's device role paradigm. A workstation configured with keystone.os and keystone.terminal provides the full development environment. A laptop configured as a thin client needs only an SSH client and Mosh to access that environment. The workstation handles compilation, testing, and long-running processes; the laptop provides a window into that work.

Benefits of the thin client approach include the ability to work from underpowered devices, a consistent environment regardless of local hardware, sessions that survive network transitions and sleep cycles, and powerful hardware available on demand without duplicating development setups across machines.

Mesh VPN connectivity

A mesh VPN such as Headscale (self-hosted) or Tailscale provides the network layer that connects the thin client to the workstation regardless of physical location. Both devices join the same tailnet and receive stable IP addresses that do not change as the devices move between networks. DNS names (via MagicDNS) allow connecting by hostname rather than IP address.

VPN server setup is outside the scope of this page. Keystone's server module includes Headscale support for self-hosted deployments. The key requirement is that the thin client and workstation can reach each other by hostname or tailnet IP.

Mosh: resilient remote shell

Mosh (mobile shell) maintains persistent connections that survive network changes, sleep, and roaming between Wi-Fi and cellular networks. It replaces SSH as the transport for interactive terminal sessions.

How Mosh differs from SSH

| Characteristic | SSH | Mosh | | ------------------------- | -------------------------- | ---------------------------- | | Transport | TCP | UDP | | Network changes | Connection dies | Survives automatically | | High-latency links | Laggy typing | Local echo, feels responsive | | Intermittent connectivity | Requires stable connection | Handles drops gracefully |

Mosh initiates an SSH connection to authenticate, starts a mosh-server process on the remote host, then switches to UDP for the actual session. Local predictions make typing feel instant even on high-latency connections. Close the laptop on home Wi-Fi, open it at a different location, and Mosh reconnects automatically.

NixOS configuration

On the workstation or server (the machine being connected to):

{ pkgs, ... }: {
  environment.systemPackages = [ pkgs.mosh ];

  # Open Mosh UDP ports
  networking.firewall.allowedUDPPortRanges = [
    { from = 60000; to = 61000; }
  ];
}

On the client side, install Mosh via home-manager:

{ pkgs, ... }: {
  home.packages = [ pkgs.mosh ];
}

Usage

# Connect to remote host
mosh user@workstation

# Specify SSH port if non-standard
mosh --ssh="ssh -p 2222" user@workstation

SSH port forwarding

SSH port forwarding makes services running on the remote workstation accessible as if they were local. This is essential for web development workflows where a browser on the thin client needs to reach a dev server running on the workstation.

Local port forwarding

Forward a remote port to localhost on the thin client:

# Forward remote port 3000 to local port 3000
ssh -L 3000:localhost:3000 user@workstation

# Forward remote PostgreSQL to local
ssh -L 5432:localhost:5432 user@workstation

# Multiple forwards in a single connection
ssh -L 3000:localhost:3000 -L 5432:localhost:5432 user@workstation

After establishing the tunnel, localhost:3000 on the thin client reaches the dev server on the workstation.

Persistent forwarding with SSH config

Configure persistent forwards in ~/.ssh/config or via home-manager:

{ pkgs, ... }: {
  programs.ssh = {
    enable = true;
    matchBlocks = {
      workstation = {
        hostname = "workstation.local";
        user = "developer";
        localForwards = [
          { bind.port = 3000; host.address = "localhost"; host.port = 3000; }
          { bind.port = 5432; host.address = "localhost"; host.port = 5432; }
          { bind.port = 8080; host.address = "localhost"; host.port = 8080; }
        ];
      };
    };
  };
}

Then connect with a single command:

ssh workstation

Dynamic forwarding (SOCKS proxy)

Route all traffic through the remote machine:

ssh -D 1080 user@workstation

Configure the browser to use localhost:1080 as a SOCKS5 proxy. All browsing then occurs from the workstation's network perspective.

Combining Mosh with port forwarding

Mosh does not support port forwarding directly. Use SSH for the tunnel and Mosh for the interactive session in separate terminals:

# Terminal 1: SSH tunnel (leave running)
ssh -N -L 3000:localhost:3000 workstation

# Terminal 2: Mosh session
mosh workstation

For persistent tunnels that auto-reconnect, use autossh:

autossh -M 0 -N -L 3000:localhost:3000 workstation

Or create a systemd user service for always-on port forwarding:

systemd.user.services.workstation-tunnel = {
  Unit.Description = "SSH tunnel to workstation";
  Service = {
    ExecStart = "${pkgs.autossh}/bin/autossh -M 0 -N workstation";
    Restart = "always";
  };
  Install.WantedBy = [ "default.target" ];
};

Zellij session resumption

Zellij is a terminal multiplexer that runs on the remote workstation. Sessions persist independently of the SSH or Mosh connection. This is the key enabler for true thin client development: all tabs, panes, and running processes remain exactly as they were between connections.

Named sessions

Create a session per project on the remote machine:

zellij -s project-name

Detaching and reattaching

Detach from a session without closing it:

Ctrl+o then d

The session continues running. All panes, tabs, and processes remain active.

Reattach from any subsequent connection:

# List available sessions
zellij list-sessions

# Attach to a specific session
zellij attach project-name

# Attach to the last active session
zellij attach

Session lifecycle

# Attach to existing session or create a new one
zellij attach myproject || zellij -s myproject

# Kill a session when truly finished
zellij kill-session project-name

# Kill all sessions
zellij kill-all-sessions

Practical daily workflow

Starting work

# 1. Connect to workstation
mosh workstation

# 2. Attach to (or create) project session
zellij attach myproject || zellij -s myproject

# 3. Work in the persistent workspace
# All tabs, panes, and running processes are present

During the day

Work normally in the Zellij session. Start dev servers, run tests, and edit code. Everything runs on the remote machine. The thin client is simply a viewport.

Switching networks

Close the laptop lid or let the network drop. Open the laptop on a different network. Mosh reconnects automatically. Run zellij attach to resume the session. Everything is exactly as it was left.

End of day

# Option 1: Detach and disconnect cleanly
Ctrl+o then d    # detach from Zellij
exit              # close Mosh connection

# Option 2: Close the terminal directly
# Mosh session ends, Zellij session persists on the workstation

The next day, reconnect and attach. The workspace is unchanged.

Workstation-side NixOS configuration

A complete workstation configuration for hosting thin client connections:

{ pkgs, ... }: {
  # SSH server
  services.openssh = {
    enable = true;
    settings.PasswordAuthentication = false;
  };

  # Mosh
  programs.mosh.enable = true;

  # Development tools (also available via keystone.terminal)
  environment.systemPackages = with pkgs; [
    zellij
    helix
    git
    lazygit
  ];
}

When using Keystone modules, enabling keystone.terminal on the workstation provides Zellij, Helix, Zsh, and other development tools. See Terminal development environment for the full tool configuration.

Clipboard integration

Modern terminals support OSC 52 escape sequences for clipboard synchronization between the local machine and the remote session. When using a terminal emulator that supports OSC 52 (such as Ghostty), copying text in the remote Zellij session automatically places it on the local clipboard. No additional configuration is typically required.

Troubleshooting

Mosh will not connect

Verify UDP ports 60000-61000 are open on the workstation:

# On the workstation
sudo iptables -L -n | grep 60000

If using NixOS firewall, ensure the Mosh port range is allowed in the configuration.

Session not persisting

Ensure Zellij is running on the remote machine, not locally. The session persists where Zellij runs. If Zellij is running on the thin client, closing the terminal closes the session.

Port forward not working

# Check if port is already in use locally
lsof -i :3000

# Verify SSH connection includes the forward
ssh -v workstation 2>&1 | grep "Local forward"

Wake-on-LAN

Start the workstation remotely if it is powered off:

nix-shell -p wol
wol AA:BB:CC:DD:EE:FF

This requires the workstation's network interface to support Wake-on-LAN and the MAC address to be known.

See also

• Terminal development environment -- Tools used in remote sessions (Helix, Zsh, Zellij, Ghostty)
• Architecture -- Device role paradigm and how the thin client pattern fits into Keystone's design