An autonomous system that crashes and stays down isn’t autonomous — it’s a demo. Keeping Sentinel running reliably required a layer of infrastructure that isn’t glamorous but is absolutely essential.
The health watchdog monitors the system continuously. If a component stops responding, it attempts a silent self-heal — restarting the affected service without human intervention. If the self-heal fails, it sends a Signal notification to my phone. I find out about problems before users do.
Compose locking solved a subtle problem with the test runners. The validation suite, red team runner, and overnight benchmark all manipulate the container stack. If two runners overlap — which happens when you’re running things in parallel — they can corrupt each other’s state. Each runner now acquires a lock before touching compose, and releases it when done.
Config verification runs at the start of every test suite. Before executing a single test, the runner checks that the compose config matches expectations: correct image tags, correct environment variables, correct port mappings. A misconfigured test environment produces misleading results, and I’d rather fail fast than debug phantom failures.
The Signal notification system ties it all together. Test suites report their status — pass, fail, invalid — via Signal with timing information. Overnight runs that finish at 3am send their results to my phone. I wake up knowing whether the system is healthy.
None of this is technically interesting. But it’s the difference between a project that runs when I’m watching and a project that runs when I’m not.