AXP-Resurection/docs/001-host-runbook.md

001 host runbook — rom-dev01

Living notes for feature 001-development-environment.

Access (laptop → host)

Item	Value	Notes
Project name	rom-dev01	Guest hostname set 2026-09-13
DNS / IP	a16.ssystems.de (preferred) / a15.ssystems.de → 49.13.104.83
SSH port	37037	:22 closed
Admin	root (default)	ssh a16 / a16.ssystems.de / a16-root
Build user	android-build (uid/gid 1000)	ssh a16-build (also rom-dev01, a15)
Laptop private key	~/.ssh/id_ed25519_rom_dev01	also YubiKey PIV in authorized_keys
Client SSH	docs/ssh/config	Include from ~/.ssh/config
Web UI	https://a16.ssystems.de/	nginx + wildcard + HTTP Basic

OpenSSH / Cursor

Host a16 a16.ssystems.de a16-root
  HostName 49.13.104.83
  User root
  Port 37037
  IdentityFile ~/.ssh/id_ed25519_rom_dev01
  IdentitiesOnly yes

Host a16-build
  HostName 49.13.104.83
  User android-build
  Port 37037
  IdentityFile ~/.ssh/id_ed25519_rom_dev01
  IdentitiesOnly yes

Verify:

ssh a16 'whoami; hostname'                # root @ rom-dev01
ssh a16-build 'whoami'                    # android-build

SSH password auth

Effective sshd -T: passwordauthentication no (Ubuntu /etc/ssh/sshd_config.d/60-cloudimg-settings.conf overrides PasswordAuthentication yes in main sshd_config). Pubkey-only in practice. Do not edit Salt-/cloud-managed sshd unless required; current effective policy already matches the desired end state. YubiKey PIN ≠ SSH password.

Coexistence with saltenv=base

If a Salt-/base-managed file conflicts with project needs: stop and report. Do not silently replace Salt-managed settings.

Observed: salt-minion inactive; many company pubkeys already in /root/.ssh/authorized_keys (base provision). Project added: id_ed25519_rom_dev01 + YubiKey PIV for root and android-build.

Rollback SSH key: remove the ed25519 line from authorized_keys on host; delete laptop ~/.ssh/id_ed25519_rom_dev01 if retiring.

Account model

Account	Role
root	packages, users, services, firewall observation
android-build	all Android builds, /srv/rom writes, Spec Kit meta tree
future Gitea/Semaphore	least privilege; never build Android as root

Paths

Path	Purpose	Mode (approx)
/home/android-build/sunfish-a16-rom	Meta Spec Kit tree	user-owned
/srv/rom/src	Android source trees	android-build
/srv/rom/references	Reference archives	android-build
/srv/rom/tooling	Host tooling — not Spec Kit checkout	android-build
/srv/rom/ccache	CCACHE_DIR (50G)	android-build
/srv/rom/artifacts	Artifact handoff to laptop	android-build
/srv/rom/logs	Preflight / toolchain / smoke logs	android-build
/srv/rom/secrets	Restricted secrets	700 root:root

Preflight summary (2026-09-13)

Full log: /srv/rom/logs/preflight/preflight-latest.txt

• Ubuntu 24.04.4 LTS / Linux 6.8.0-136-generic
• 8 vCPU (host CPU E3-1275 v5), 62 GiB RAM, 32 GiB swap (/swapfile)
• Disk: 1.9T ext4 on / (~2T vda)
• NIC: ens18 only — 49.13.104.83/29, default via 49.13.104.81 (public-only)
• Firewall: ufw inactive; iptables ACCEPT (no flush)
• Listening: SSH 0.0.0.0:37037, systemd-resolved localhost DNS
• apt: Ubuntu noble archives

Toolchain (US2)

• Package log: /srv/rom/logs/toolchain-packages.txt (LOS wiki sunfish list + openjdk-17-jdk, python-is-python3)
• repo in ~/bin/repo (launcher 2.65)
• CCACHE_DIR=/srv/rom/ccache, USE_CCACHE=1, size 50G, compression on
• Env in android-build .profile / .bashrc
• Smoke: /srv/rom/logs/toolchain-smoke.txt
• FR-007 / build parallelism (use the HW): guest has 8 vCPU / ~62 GiB / 32 GiB swap. Default Android builds: mka … -j8 (match vCPU count). Do not leave conservative -j4 unless diagnosing OOM. Watch load / available RAM; if linking OOMs, drop to -j6 temporarily — not below without a written reason. Host-side: Proxmox CPU steal can waste cycles; prefer dedicated/pinned vCPUs when practical (outside this repo). Wrapper: /srv/rom/logs/003-build/run-brunch.sh (-j8).
• nsjail / build sandbox (Ubuntu 24.04): AppArmor kernel.apparmor_restrict_unprivileged_userns=1 put nsjail into profile unprivileged_userns and DENY mount/setgid → Soong message Build sandboxing disabled due to nsjail error. Fix (scoped, not global sysctl): /etc/apparmor.d/nsjail-android-build (userns + flags=(unconfined) for /srv/rom/src/lineage/prebuilts/build-tools/linux-{x86,arm64}/bin/nsjail). Evidence: /srv/rom/logs/nsjail-apparmor-fix.txt. Re-probe on next Soong start (current mka bacon already started with sandbox off).

Thin client / artifacts (US5)

• Cursor Remote SSH: Host rom-dev01 → /home/android-build/sunfish-a16-rom as android-build
• Dummy rsync: host /srv/rom/artifacts/dummy-001.txt → laptop Workspaces/sunfish-a16-rom/.local-artifacts/ (gitignored path preferred)
• Pixel USB: adb/fastboot on laptop only; no USB passthrough to VM

Gitea / Ansible / Semaphore

Independent of SSYSTEMS Salt/GitLab. All on rom-dev01.

Service	Bind	Public URL
nginx	0.0.0.0:80/443	TLS + HTTP Basic gate
Gitea 1.27.3	127.0.0.1:3000	https://a16.ssystems.de/
Semaphore 2.19.12	127.0.0.1:3001	https://a16.ssystems.de/semaphore/
Ansible	host CLI	core 2.16.3

• Site: /etc/nginx/sites-available/a16.ssystems.de
• TLS: company wildcard *.ssystems.de (Sectigo) at /etc/nginx/ssl/ssystems.de/{fullchain,privkey}.pem Source (laptop, not in this repo): buildout-saar-docker/salt/srv/files/etc/pki/tls/{certs,private}/wildcard_.ssystems.de-*
• HTTP Basic: /etc/nginx/.htpasswd — credentials on host only: /srv/rom/secrets/web-htpasswd.txt (user sunfish)
• Unit: gitea.service — data /var/lib/gitea, config /etc/gitea/app.ini (ROOT_URL=https://a16.ssystems.de/)
• Unit: semaphore.service — data /var/lib/semaphore, config /etc/semaphore/config.json (web_host=https://a16.ssystems.de/semaphore, web_root=/semaphore) nginx: location /semaphore/ → proxy_pass without trailing URI (must preserve /semaphore prefix or UI assets 404 → blank page)
• App secrets (host only, never Git): /srv/rom/secrets/gitea-admin.txt, /srv/rom/secrets/semaphore-admin.txt
• Admin user (both apps): sunfish-admin (after nginx Basic)
• Git transport: HTTPS via nginx (Gitea still DISABLE_SSH=true; no public :2222)
• Repos (private): sunfish-admin/{sunfish-a16-rom,sunfish-rom-manifest,sunfish-rom-build,sunfish-rom-patches}
• Automation tree: /home/android-build/sunfish-rom-build (not inside Spec Kit meta)
• Semaphore project sunfish-rom template env-smoke clones canonical sunfish-rom-build from Gitea and runs ansible/smoke.yml unprivileged
• Evidence: /srv/rom/logs/ansible-smoke.txt, /srv/rom/logs/semaphore-smoke.txt
• Off-VM mirrors (laptop): /home/aventura/Archives/sunfish-rom-mirrors/*.git
• No SSH tunnels needed for UIs anymore

Accounts (reminder)

• root: install packages/services only
• android-build: Android builds, meta + automation working trees
• git / semaphore: service users for Gitea / Semaphore
• Do not build Android as root (LOS/AXP practice)

Isolation checklist

• Public NIC only (ens18 /29) — no internal SSYSTEMS NIC required
• App backends remain loopback (:3000 / :3001); only nginx is public
• Workflow uses public DNS a16.ssystems.de + SSH port 37037 (no VPN)
• Gitea/Semaphore behind nginx HTTPS + HTTP Basic (htpasswd)

Off-VM mirror

• Laptop: /home/aventura/Archives/sunfish-rom-mirrors/*.git
• Host: /srv/rom/artifacts/gitea-mirrors/ (token stays on host; laptop only rsyncs)

Handoff

001 complete → proceed to 002 (AXP reference capture).
Do not start LineageOS repo sync / build as part of 001.