AXP-Resurection/docs/009-fortify-diag-review-20260927.md

Status: historical. Do not use as execution order. Resume: 009-handoff-20261008.md.

FORTIFY diagnostic review — 2026-09-27

Conclusion

The boot failure remains unresolved. The latest capture does not exclude fortify_panic, and does not establish a watchdog reset. The diagnostic channel has an early-boot blind spot and its documented panic validation admits a stale marker. Fix these experimental controls before interpreting another empty dump.

Review covered the Cursor transcript, local patch scripts and capture plan, remote kernel sources/build artifacts, saved diag2 dump and installed LAB modules. No flashing, rebooting, live source changes or new builds were performed in this review.

Experiment history and what it establishes

1. Original defconfig bisects also changed bp4a to bp2a (security patch and platform version regression). Those experiments did not isolate hardening.
2. Corrected ksync packaging retained GOODMAKE's boot environment. Combo4 boots with DMESG restriction, slab freelist hardening and allocation/free clearing.
3. FORTIFY required a Clang port. Exported overflow BUG stubs were removed; pass_object_size was aligned to the wrappers' object-size mode 0.
4. The audio/WLAN overflow-helper link failures were localized to specific functions. Explicit array-capacity guards resolved the targeted compilation failures. Successful linking does not establish runtime correctness of the port, or prove those drivers caused the subsequent boot failure.
5. Bounds FORTIFY boots still failed. Diagnostics then added caller printing, stack dumping and a plaintext marker in the alternate ramoops region.
6. Plain-channel iterations tested mapping, probe overwriting, a panic notifier and SysRq writes. These are useful channel experiments, but the recorded plain7 result is insufficient to validate fresh panic-time writes.
7. Latest diag2 (20260927T032419Z-bisect-ksync-fortify-diag) failed to boot; recovery used combo4. No readable failure record has been obtained.

Confirmed diagnostic defects

1. Capture starts too late to exclude an early FORTIFY failure

apply_fortify_diag.py maps the alternate region only at the successful end of ramoops_probe. Remote fs/pstore/ram.c:1270 performs the mapping; ramoops_init is a postcore_initcall at line 1365. The writer immediately returns if axp_alt_va is null (patch script lines 105–106).

Remote init/main.c calls setup_arch at 565 and parse_early_param at 579, before rest_init at 738 and the later do_basic_setup / do_initcalls path. Disassembly of the current FORTIFY vmlinux contains branches to fortify_panic:

parse_early_param: ffffff8012290974 -> ffffff8011ac186c
start_kernel:     ffffff8012290fb8 -> ffffff8011ac186c
setup_arch:       ffffff8012296750 -> ffffff8011ac186c

These are possible failure paths, not evidence that any executed. They demonstrate that an instrumented FORTIFY failure can occur while the persistent writer is unavailable. A SysRq test from a fully booted system cannot validate that interval.

2. Plain7's positive control can accept an old marker

The capture plan records sysrq-x + sysrq-c followed by survival of the magic AXPDIAG1, sometimes without the complete body. SysRq-x already writes that magic. The next probe explicitly preserves it. Therefore magic survival alone does not prove that SysRq-c or the panic notifier wrote anything new.

Require a distinct per-attempt identifier and a complete panic-specific record, different from the pre-panic marker. Validate the exact failed-boot → fastboot → rescue-image → dump sequence, not just a normal reboot into plain7.

3. The durable record follows operations that may fail

The injected fortify_panic first calls pr_emerg and dump_stack, then writes the persistent mark (patch script lines 36–39). A failure or recursion in those operations could prevent the mark. The durable body contains only the helper name, not the caller address printed to the unavailable console. Subsequent panic-notifier and pstore writes also target the same slots and can overwrite the first record.

Record the first failure before formatted output; include a raw caller address, event type and attempt identifier. Preserve the first record. Use a minimal writer independent of the fortified helpers under investigation. This still requires a correctly reserved/mapped region available at the tested boot stage; moving the call alone does not fix defect 1.

Additional checks

• Latest local boot SHA-256 matches the remote published boot: a5be43617770205bb9a2d2ad5989ca566ad9713134e7210c585f02fd7bdea554.
• Current build configuration enables FORTIFY, Clang LTO, Clang CFI and MODVERSIONS.
• Read-only comparison of all 41 installed vendor .ko files against current remote Module.symvers: 3,182 version entries, zero CRC mismatches, zero missing symbols. This checks symbol version compatibility only; it does not prove all runtime ABI/CFI behavior is correct or identify the crashing code.
• LAB was recovered and running combo4 kernel #16 with sys.boot_completed=1.
• Saved diag2 access-ramoops.bin is 4 MiB and contains none of AXPDIAG0, AXPDIAG1, FORTIFY, DBGC or Oops. Measured byte entropy is approximately 7.99996 bits/byte. This does not uniquely distinguish encryption, overwritten contents or other high-entropy data, nor prove no panic occurred.
• bootreason=reboot observed after rescue is not a preserved trace of the original failure and cannot exclude a panic in the earlier attempt.

Next experiment requirements

1. Preserve combo4 and archive each diagnostic patch, config, vmlinux and hashes.
2. Validate fresh, distinguishable failure records with an otherwise booting control. Test persistence across the same recovery sequence used in practice.
3. Cover the early interval explicitly: use a board-supported early output or correctly reserved early persistent writer, with initialization milestones. Do not insert ordinary ioremap calls into arbitrary early/panic context.
4. Validate an intentionally triggered diagnostic event at the boot stage being tested. A late SysRq control alone is insufficient.
5. Only then repeat FORTIFY with a first-failure record containing a caller and symbolize it using the exact vmlinux. If no reliable early channel is achievable, use controlled localization experiments and label them as such; do not promote them to a working or complete hardening port.

Do not add overflow stubs, suppress fortify_panic or declare watchdog/CFI the cause solely because the current capture is empty.