AXP-Resurection/docs/001-host-runbook.md

# 001 host runbook — `rom-dev01`

Living notes for feature `001-development-environment`.

## Access (laptop → host)

| Item | Value | Notes |
|------|-------|-------|
| Project name | `rom-dev01` | Guest hostname set 2026-09-13 |
| DNS / IP | **`a16.ssystems.de`** (preferred) / `a15.ssystems.de` → `49.13.104.83` | |
| SSH port | **37037** | `:22` closed |
| Admin | `root` (**default**) | `ssh a16` / `a16.ssystems.de` / `a16-root` |
| Build user | `android-build` (uid/gid **1000**) | `ssh a16-build` (also `rom-dev01`, `a15`) |
| Laptop private key | `~/.ssh/id_ed25519_rom_dev01` | also YubiKey PIV in `authorized_keys` |
| Client SSH | `docs/ssh/config` | Include from `~/.ssh/config` |
| Web UI | `https://a16.ssystems.de/` | nginx + wildcard + HTTP Basic |

### OpenSSH / Cursor

```sshconfig
Host a16 a16.ssystems.de a16-root
  HostName 49.13.104.83
  User root
  Port 37037
  IdentityFile ~/.ssh/id_ed25519_rom_dev01
  IdentitiesOnly yes

Host a16-build
  HostName 49.13.104.83
  User android-build
  Port 37037
  IdentityFile ~/.ssh/id_ed25519_rom_dev01
  IdentitiesOnly yes
```

Verify:

```bash
ssh a16 'whoami; hostname'                # root @ rom-dev01
ssh a16-build 'whoami'                    # android-build
```
### SSH password auth

Effective `sshd -T`: **`passwordauthentication no`** (Ubuntu
`/etc/ssh/sshd_config.d/60-cloudimg-settings.conf` overrides
`PasswordAuthentication yes` in main `sshd_config`). Pubkey-only in practice.
Do **not** edit Salt-/cloud-managed sshd unless required; current effective
policy already matches the desired end state. YubiKey PIN ≠ SSH password.

## Coexistence with `saltenv=base`

If a Salt-/base-managed file conflicts with project needs: **stop and report**.
Do not silently replace Salt-managed settings.

Observed: `salt-minion` **inactive**; many company pubkeys already in
`/root/.ssh/authorized_keys` (base provision). Project added:
`id_ed25519_rom_dev01` + YubiKey PIV for `root` and `android-build`.

Rollback SSH key: remove the ed25519 line from `authorized_keys` on host;
delete laptop `~/.ssh/id_ed25519_rom_dev01` if retiring.

## Account model

| Account | Role |
|---------|------|
| `root` | packages, users, services, firewall observation |
| `android-build` | all Android builds, `/srv/rom` writes, Spec Kit meta tree |
| future Gitea/Semaphore | least privilege; **never** build Android as root |

## Paths

| Path | Purpose | Mode (approx) |
|------|---------|----------------|
| `/home/android-build/sunfish-a16-rom` | Meta Spec Kit tree | user-owned |
| `/srv/rom/src` | Android source trees | `android-build` |
| `/srv/rom/references` | Reference archives | `android-build` |
| `/srv/rom/tooling` | Host tooling — **not** Spec Kit checkout | `android-build` |
| `/srv/rom/ccache` | `CCACHE_DIR` (50G) | `android-build` |
| `/srv/rom/artifacts` | Artifact handoff to laptop | `android-build` |
| `/srv/rom/logs` | Preflight / toolchain / smoke logs | `android-build` |
| `/srv/rom/secrets` | Restricted secrets | `700 root:root` |

## Preflight summary (2026-09-13)

Full log: `/srv/rom/logs/preflight/preflight-latest.txt`

- Ubuntu 24.04.4 LTS / Linux 6.8.0-136-generic
- 8 vCPU (host CPU E3-1275 v5), **62 GiB** RAM, **32 GiB** swap (`/swapfile`)
- Disk: **1.9T** ext4 on `/` (~2T vda)
- NIC: `ens18` only — `49.13.104.83/29`, default via `49.13.104.81` (public-only)
- Firewall: ufw inactive; iptables ACCEPT (no flush)
- Listening: SSH `0.0.0.0:37037`, systemd-resolved localhost DNS
- apt: Ubuntu noble archives

## Toolchain (US2)

- Package log: `/srv/rom/logs/toolchain-packages.txt` (LOS wiki sunfish list +
  `openjdk-17-jdk`, `python-is-python3`)
- `repo` in `~/bin/repo` (launcher 2.65)
- `CCACHE_DIR=/srv/rom/ccache`, `USE_CCACHE=1`, size **50G**, compression on
- Env in `android-build` `.profile` / `.bashrc`
- Smoke: `/srv/rom/logs/toolchain-smoke.txt`
- **FR-007 / build parallelism (use the HW):** guest has **8 vCPU / ~62 GiB /
  32 GiB swap**. Default Android builds: **`mka … -j8`** (match vCPU count).
  Do **not** leave conservative `-j4` unless diagnosing OOM. Watch `load` /
  `available` RAM; if linking OOMs, drop to `-j6` temporarily — not below
  without a written reason. Host-side: Proxmox CPU steal can waste cycles;
  prefer dedicated/pinned vCPUs when practical (outside this repo).
  Wrapper: `/srv/rom/logs/003-build/run-brunch.sh` (`-j8`).
- **nsjail / build sandbox (Ubuntu 24.04):** AppArmor
  `kernel.apparmor_restrict_unprivileged_userns=1` put nsjail into profile
  `unprivileged_userns` and DENY `mount`/`setgid` → Soong message
  `Build sandboxing disabled due to nsjail error`. Fix (scoped, not global
  sysctl): `/etc/apparmor.d/nsjail-android-build` (`userns` +
  `flags=(unconfined)` for
  `/srv/rom/src/lineage/prebuilts/build-tools/linux-{x86,arm64}/bin/nsjail`).
  Evidence: `/srv/rom/logs/nsjail-apparmor-fix.txt`. Re-probe on next Soong
  start (current `mka bacon` already started with sandbox off).

## Thin client / artifacts (US5)

- Cursor Remote SSH: Host `rom-dev01` →
  `/home/android-build/sunfish-a16-rom` as `android-build`
- Dummy rsync: host `/srv/rom/artifacts/dummy-001.txt` → laptop
  `Workspaces/sunfish-a16-rom/.local-artifacts/` (gitignored path preferred)
- Pixel USB: **adb/fastboot on laptop only**; no USB passthrough to VM

## Gitea / Ansible / Semaphore

Independent of SSYSTEMS Salt/GitLab. All on `rom-dev01`.

| Service | Bind | Public URL |
|---------|------|------------|
| nginx | `0.0.0.0:80/443` | TLS + HTTP Basic gate |
| Gitea 1.27.3 | `127.0.0.1:3000` | https://a16.ssystems.de/ |
| Semaphore 2.19.12 | `127.0.0.1:3001` | https://a16.ssystems.de/semaphore/ |
| Ansible | host CLI | core 2.16.3 |

- Site: `/etc/nginx/sites-available/a16.ssystems.de`
- TLS: company wildcard `*.ssystems.de` (Sectigo) at
  `/etc/nginx/ssl/ssystems.de/{fullchain,privkey}.pem`
  Source (laptop, not in this repo):
  `buildout-saar-docker/salt/srv/files/etc/pki/tls/{certs,private}/wildcard_.ssystems.de-*`
- HTTP Basic: `/etc/nginx/.htpasswd` — credentials on host only:
  `/srv/rom/secrets/web-htpasswd.txt` (user `sunfish`)
- Unit: `gitea.service` — data `/var/lib/gitea`, config `/etc/gitea/app.ini`
  (`ROOT_URL=https://a16.ssystems.de/`)
- Unit: `semaphore.service` — data `/var/lib/semaphore`, config `/etc/semaphore/config.json`
  (`web_host=https://a16.ssystems.de/semaphore`, `web_root=/semaphore`)
  nginx: `location /semaphore/` → `proxy_pass` **without** trailing URI
  (must preserve `/semaphore` prefix or UI assets 404 → blank page)
- App secrets (host only, never Git): `/srv/rom/secrets/gitea-admin.txt`,
  `/srv/rom/secrets/semaphore-admin.txt`
- Admin user (both apps): `sunfish-admin` (after nginx Basic)
- Git transport: **HTTPS via nginx** (Gitea still `DISABLE_SSH=true`; no public `:2222`)
- Repos (private): `sunfish-admin/{sunfish-a16-rom,sunfish-rom-manifest,sunfish-rom-build,sunfish-rom-patches}`
- Automation tree: `/home/android-build/sunfish-rom-build` (not inside Spec Kit meta)
- Semaphore project `sunfish-rom` template `env-smoke` clones **canonical**
  `sunfish-rom-build` from Gitea and runs `ansible/smoke.yml` unprivileged
- Evidence: `/srv/rom/logs/ansible-smoke.txt`, `/srv/rom/logs/semaphore-smoke.txt`
- Off-VM mirrors (laptop): `/home/aventura/Archives/sunfish-rom-mirrors/*.git`
- No SSH tunnels needed for UIs anymore

### Accounts (reminder)

- **root**: install packages/services only
- **android-build**: Android builds, meta + automation working trees
- **git** / **semaphore**: service users for Gitea / Semaphore
- Do **not** build Android as root (LOS/AXP practice)

## Isolation checklist

- [x] Public NIC only (`ens18` /29) — no internal SSYSTEMS NIC required
- [x] App backends remain loopback (`:3000` / `:3001`); only nginx is public
- [x] Workflow uses public DNS `a16.ssystems.de` + SSH port 37037 (no VPN)
- [x] Gitea/Semaphore behind nginx HTTPS + HTTP Basic (`htpasswd`)

## Off-VM mirror

- Laptop: `/home/aventura/Archives/sunfish-rom-mirrors/*.git`
- Host: `/srv/rom/artifacts/gitea-mirrors/` (token stays on host; laptop only rsyncs)

## Handoff

**`001` complete → proceed to `002` (AXP reference capture).**  
Do **not** start LineageOS `repo sync` / build as part of `001`.