Machines and SSH

What a machine is

• A machine is a host: a computer that runs herdr (D43), This Mac or a host the app reaches with ssh. Each machine has its own icon, because the title row's machine button shows only the icon (Machine button).
• A machine runs several herdr sessions (default, pi-herd, …), each one herdr server. The sessions of one machine share one SSH connection (ControlMaster) and nothing else: each session has its own events, streams and state folder.
• A window shows one machine. Sessions open as pages: its sidebar has one page per session the user opened in that window (Sidebar). A window with nothing to restore opens no page: the user picks a machine's session (Windows).

The machine list

• The app keeps its own list of remote machines (D54): name, SSH target, icon and id, in UserDefaults. The machine menu lists This Mac first, then the list. Settings ▸ Machines lists them too (Settings).
  • The list is JSON under the machines key. A run launched with -session, -defaultSession or -machines (the e2e tests, a run against a throwaway session) never reads or writes it: it takes -machines <json> or no machines, so a test never reaches the user's hosts.
  • A window remembers its machine's id; -machine <id> picks it at launch.
  • Removing a machine also removes its browser data store and its approved browser origins (Profiles).
• This Mac is always in the list, with a laptop icon when the Mac has an internal battery, else a desktop icon.
• A machine's sessions come from herdr session list --json, run on that machine. The Open Session panel lists them (Open Session). herdr on the Mac is needed only for This Mac's sessions, not for remote machines.

The SSH connection

• SSH is the system ssh. The user's ~/.ssh/config, agent, keys, ProxyJump, YubiKey and Tailscale SSH work unchanged. The app never writes ~/.ssh: only ssh itself writes known_hosts.
• One ControlMaster per host, started on purpose, once, by the first run that finds none (SSHExec in HerdrKit):

  ssh -M -N -f -o ControlPersist=10m -o ServerAliveInterval=5 \
      -o ServerAliveCountMax=3 -o ConnectTimeout=10 \
      -o ForwardAgent=no -o ForwardX11=no -o ClearAllForwardings=yes \
      -o StreamLocalBindMask=0177 -o LogLevel=ERROR -o BatchMode=yes -S ~/Library/Caches/ovvi/<h8>-<pid> <target>
  ssh -F /dev/null -S ~/Library/Caches/ovvi/<h8>-<pid> -O check <target>    # its pid

  • With -f, ssh goes to the background once it is authenticated and the socket serves, so the start's exit status says whether it worked. Its stderr (merged into stdout) is ssh's reason when it did not.
  • LogLevel=ERROR: that reason is ssh's own error lines only. The server's login banner and ssh's notices ("Permanently added … to the list of known hosts") are INFO, so they never show on the Disconnected page. So is "Connection closed by …": a start that ends with no text says "The connection to ‹target› closed".
  • The relay's session (below), which carries every later run, uses -F /dev/null -o ControlMaster=no -o BatchMode=yes -o ProxyCommand=/usr/bin/false -S ~/Library/Caches/ovvi/<h8>-<pid>. Without a master it fails at once instead of opening their own connection and prompting again. They send none of the user's configured forwards to the master. With a master, ssh never reads the target's config, so an alias works with -F /dev/null (checked on OpenSSH 10.3).
  • Before each run the app checks that the master lives with kill(pid, 0), not a process: the pid comes from -O check once per start. A master that is gone (killed, or ended by its keepalive) is started again by the next run. Its socket stays behind when it was killed, and a new master does not bind over it, so the app removes it first. A master that outlived an app that crashed is never used, as its socket has that app's pid: it may still hold that app's sessions. It ends 10 min after its last one.
  • Starting the master first means one password, OTP or key prompt per host. With ControlMaster=auto, parallel first calls each prompt. Starts are one at a time per host; runs that need a master wait for the start in flight.
  • <h8> is 8 hex characters of a hash of the SSH target, and <pid> is the app's process id. Each running app has its own masters: a second Ovvi (a test's, another build) removes no socket and ends no master of this one. The folder is per user, mode 0700, with no spaces. The path stays under the 104-byte socket limit for user names up to 44 characters, with ssh's 17-character temporary suffix. The app makes the folder and the askpass script (below) once per launch, so a cache cleaner that deletes ~/Library/Caches/ovvi breaks remote connects until Ovvi starts again.
  • Forwarding is off, so a machine cannot reach the Mac's SSH agent or ports through the app's long-lived connection. The app's own -O forward calls (herdr's socket, the browser's SOCKS) still work; they forward from the Mac to the machine only.
  • A dead link: ssh's own keepalive (ServerAliveInterval=5, ServerAliveCountMax=3) ends the master within about 15 s after the network goes (a network change, sleep, a server restart). The next run starts a new one. The app runs no -O exit of its own on a failed check.
  • On quit the app runs ssh -F /dev/null -S <ctl> -O exit -- <target> for each master.
• The relay: every run of a remote machine (commands, every terminal stream) goes through one SSH session. A server may allow only a few sessions per connection (MaxSessions; a server with one-time passcodes can refuse the third), and a second connection would ask for a second passcode. The first run on a new master starts that session; it runs a Perl script (Relay.script in HerdrKit, sent in the envelope; nothing is installed). It spawns each run as sh -c <command> and carries every run's stdin, stdout, exit status and stderr over its own stdin and stdout, as lines:
  • the app sends o <id> <command hex>, w <id> <line>, c <id> (close stdin) and k <id> (SIGKILL by pid, then close stdin);
  • each run's writes wait in its own buffer, so a run that reads nothing never holds up the others; its sh runs the command with exec;
  • the relay sends l <id> <line>, and x <id> <status> <stderr hex> once a run's output ended and it was reaped.
  • herdr's terminal streams and the commands' output are lines, so this needs no other framing; one process writes them, so lines of two runs never mix.
  • A line passes byte for byte, NUL bytes too, so the NUL-separated (-z) listings of the plugin cards work over SSH.
  • When its stdin ends (the app quit, the connection dropped) it exits, and every run's stdin closes with it, so every run ends (Running commands).
  • Perl is on nearly every Unix (Debian's perl-base is essential, macOS has it). The relay greets with ovvi-relay 1 within 10 s. On a machine where it does not, every run fails with "Ovvi needs Perl on ‹target›: its relay did not start", and the next run tries again. The page shows Reconnecting with that text under it.
  • herdr's own transports do not fit: herdr --remote runs one remote-client-bridge session, but it speaks herdr's internal binary client protocol; herdr --machine opens one session per request, as remote-api-bridge does.
• herdr's API socket: requests and the events stream do not take the relay. The master forwards a local socket to herdr's socket on the machine (-O forward -L), and the app connects to it as it connects to This Mac's (Requests).
• Automatic starts never prompt. Every start a run makes uses BatchMode=yes. How it fails decides what follows:
  • ssh needs an answer (its text has "Permission denied", "Host key verification failed", "passphrase" or "IDENTIFICATION HAS CHANGED"): the machine is blocked. Runs fail at once with ssh's text and start no ssh, and the pages show Disconnected with Connect (States).
  • Anything else (no network, the host down, a refused connection): runs fail at once until the next try, 2, 4, 8, 16, then every 30 s, and the pages show Reconnecting.
• Prompts: only Connect and a new machine's first window start a master that may ask. That start sets SSH_ASKPASS to a small script the app writes beside the sockets (mode 0700) and SSH_ASKPASS_REQUIRE=force. ssh runs the script with ssh's own stdin and stderr, which are the app's pipes: the script prints the question, base64 on one line after a marker, to stderr and reads the answer from stdin. The app shows the question in a dialog of the window that connects and writes the answer to ssh's stdin. Nothing is stored. A key or agent that works needs no question, so the start asks only what ssh asks.
  • A new host key shows ssh's text with its fingerprint and asks yes or no; Connect answers yes, and ssh writes known_hosts itself.
  • Every answer is hidden: a password, a passphrase, a one-time passcode or any other code. Only the host key question has no field.
  • The page shows Connecting from the click until the session answers, except while a question is up (States). A start that asks has no time limit (a 2FA push may wait a minute after its answer), so a login that hangs without asking shows Connecting until ssh gives up.
  • Cancel ends the start; the page shows Disconnected.
  • A none question (a security key's touch) gets no dialog.
  • Why the app's own dialog and not osascript: it belongs to the window that asked, follows the app's look, and an e2e test can answer it.
• herdr on the machine: one run of the remote login shell gives its path, once per master (herdr on each machine). A missing herdr shows "herdr is not installed on ‹machine›" with the install command and Install herdr, which installs it over the connection (Install herdr).
• New spaces, and new tabs with no folder to take, pass cwd ~, which herdr reads as the machine's home, so the app never sends This Mac's home to a remote machine.
• Dropped files (and Insert Path's file) are copied to the machine over the relay, one run per file, and the copies' paths are pasted (Drag and drop).
• Browser pages of a remote machine go through a SOCKS proxy on the Mac: the app's own front on a loopback port, then ssh's forward on the master, ssh -F /dev/null -S <ctl> -O forward -D 127.0.0.1:<P> <target>. It runs from the machine's first browser card to its last (-O cancel -D), and forwards again on a new master, or when its port stops answering. A forward that fails because the master does not answer (ssh -O check) starts a new master: a dead master's pid can belong to a new process (Remote localhost).
• Plugin cards of a remote machine read, save, list and watch files and run git over the relay. There is no Open With, a file over 4 MiB opens read only, and a card that shows checks for changes every 2 s instead of file system events (Remote machines).
• Chat panes of a remote machine work as on This Mac: the launch line, the side files and every follower run on the machine, over the relay (Chat panes). OpenCode's chat needs curl on the machine. The others need only POSIX tools.
• Local Network privacy: an ssh started by the app counts as the app, so a LAN target fails with "No route to host"; Tailscale addresses are not affected.

Open sessions

• A session is open while it has a page in at least one window. Each open session has one HostStore, shared by every window that shows it (Architecture). The app keeps one registry of them, by machine and session name; a page takes its session's store from it, made on first use. Detach Session disconnects the store.
• An open session keeps its control plane open (one event stream plus short calls), so notifications and the Dock cover every open session in every window.
• A session with no page has no store, no snapshot reads and no event stream. A machine with many sessions costs nothing until the user opens them.
• Terminals stream only for the tabs shown in a window (What streams).

Health check

One loop per open session checks that it answers.

• The check is session.snapshot, the read the app needs anyway. It carries herdr's version and catches any event the app missed.
• It runs every 30 s while the session is healthy. A failed read switches it to every 2 s, so a session started again comes back at once.
• A session counts as live only while the check answers. While it fails, its cards gray out.
• On a remote machine a failed check runs over a new connection when the old one is gone: the next run starts a new ControlMaster (ssh's keepalive ends a dead one, The SSH connection).
• Automatic retries never prompt (The SSH connection): a host that needs an answer shows Disconnected with Connect and stops retrying.
• When a check succeeds after a failure, the app opens the event stream and the terminal streams again: the same path as the first connect (Checks on connect).

SSH sessions

• A remote machine uses one SSH session, the relay's (The SSH connection), whatever it streams. herdr's forwarded API socket and the browser's SOCKS forward open channels on the connection, not sessions. This Mac uses no SSH.

Checks on connect

1. session.snapshot. Its errors say what is wrong: herdr missing (the install command), herdr too old, or the server not running (herdr on each machine). If the server is not running:
   • This Mac: a Start button. A new window's first page starts its session this way when it is stopped, and so does each page of a window that macOS restores. In the Open Session panel, a stopped session or a new name starts the same way. A server the app started gets its first space in $HOME; a server that already runs is left alone. The app starts the server as a one-shot launchd job (D47). launchd, not the app, is the server's parent and "responsible process": macOS privacy prompts never name Ovvi, the server has its own process group, and it outlives the app.
     • launchctl bootstrap gui/<uid> (the user's GUI domain) loads the job dev.ovvi.app.herdr.<S>, which runs <herdr path> --session <S> server (the path from the login shell).
     • RunAtLoad without KeepAlive: once herdr session stop ends the server, launchd never runs it again (a launchctl submit job restarts it).
     • ProcessType Interactive: launchd's default throttles the CPU and I/O of the server and every agent in it.
     • stderr goes to ~/Library/Logs/Ovvi/<S>.log.
     • The job gets the user's environment keys from the login shell (PATH, HOME, SSH_AUTH_SOCK, XDG_*; herdr on each machine, D47), so the server and the CLI agree on paths.
     • An older job of the label is booted out first (a label loads only once). A running server, one that answers a snapshot, is never booted out: the app uses it.
   • A remote machine: the same Start button runs sh -c 'nohup <herdr path> --session S server </dev/null >/dev/null 2>&1 &' over SSH and polls session.snapshot as This Mac's start does. A window opened for a remote machine opens its default session without starting it; a window that macOS restores connects, then starts each of its pages' sessions that is stopped.
2. Open the event stream. Its subscription_started reply counts as an event, so the app reads the snapshot once more and misses nothing between the two steps.

What it looks like: Add a machine, States.