fix(agent): verify upgrades and support OpenRC lifecycle

This commit is contained in:
sagitchu
2026-11-09 08:57:30 +08:00
parent fadc2b6c12
commit 051d8b0365
12 changed files with 1057 additions and 73 deletions
@@ -0,0 +1,27 @@
# Agent upgrade checksum fail-closed
## Scope
Fix the panel-issued UpgradeAgent flow only. Do not change Alpine support, installer scripts, release artifacts, production state, or publish changes.
## Observed defect
`handleUpgradeAgent` treats checksum fetch errors, non-200 responses and body read errors as success and emits `校验通过`. An absent checksum URL bypasses verification entirely. The panel already supplies a checksum URL for individual and batch upgrades, and the release workflow generates GNU sha256sum files.
## Intended behavior
- Require a nonblank checksum URL before downloading an upgrade.
- Abort before chmod, backup, or restart on fetch/status/read/format/hash failure; remove the staged binary.
- Only emit verification success after a complete, valid SHA256 record matches the downloaded bytes.
- Accept one bare 64-hex SHA256 or a single GNU sha256sum record (text or binary marker), case-insensitive hex and surrounding whitespace/CRLF. Reject empty, truncated, nonhex, multiline and oversized payloads. Do not treat an ambiguous multi-file manifest as a checksum.
- Preserve architecture substitution and current installation flow. This is integrity checking, not a signature or an independent guarantee against a compromised download source.
## Verification plan
User-confirmed seam: actual UpgradeAgent command handling, loopback HTTP fixtures, isolated binary paths and an intercepted restart action. Assert error/response and progress, untouched old executable/backup, absence of staged bytes and no restart on every invalid checksum condition. Valid records should reach the intercepted restart with exact downloaded bytes.
Use red-green slices, then race tests, module checks, build and independent scope review. No push, PR, merge, tag or deployment is authorized for this task.
## Review scope
User confirmed review against rc8 commit `fadc2b6c122df99140e0c9cbd481a1214d17d175`, covering all changes for this task and this plan. Run independent Standards and Spec reviews after the implementation is ready.
@@ -0,0 +1,29 @@
# Agent panel upgrade and rollback on OpenRC
## Scope
Repair the agent UpgradeAgent/RollbackAgent paths on Alpine/OpenRC without regressing systemd. Preserve the pending checksum fail-closed repair. Do not change live nodes, install components, publish or deploy.
## Evidence
In a disposable cached Alpine 3.22.5 linux/amd64 container, the actual handlers fail launching `systemd-run` after successful download and verification, and also when rolling back an existing backup. Installer scripts already support OpenRC, but their CI does not exercise these command handlers.
## Safety requirements
- Prefer OpenRC on Alpine, including when a systemctl shim exists; refuse missing required manager tools.
- Keep systemd's independent transient unit behavior to survive stopping the agent cgroup.
- Launch an independent OpenRC restart helper that survives termination of the original agent, without installing extra utilities.
- Use the selected manager for stop/start; do not run systemd legacy-service cleanup on OpenRC.
- Retain validated upgrade staging and backup behavior, and do not consume the rollback backup.
- Reject setup/launch errors before claiming restart was launched. Report asynchronous-completion limits honestly.
- Preserve existing command response, request ID and progress contracts.
## Proposed tests and review
User-confirmed boundaries: real UpgradeAgent/RollbackAgent command routing, isolated files, systemd/OpenRC tools as test doubles, no-manager rejection and launch/stop/replacement failures. Use cached Alpine to execute the helper with rc-service doubles, not a real OpenRC installation. Explicitly distinguish this from service-manager integration testing.
User-confirmed review baseline is fadc2b6, covering both pending fixes and plans. No commits, push, PR, merge, release or deployment.
## Publication authorization
After the local implementation, verification and independent reviews were complete, the user authorized submitting a PR, merging if checks pass, and publishing the next tag. Earlier no-publication limits above describe the implementation phase. Deployment to live nodes remains outside scope.
@@ -0,0 +1,32 @@
# Agent checksum upgrade validation
## Scope and baseline
Reviewed local changes against `fadc2b6c122df99140e0c9cbd481a1214d17d175` (rc8), as confirmed by the user. Tests use the actual UpgradeAgent command router and WebSocket response path, loopback HTTP, temporary executable/backup paths, and an intercepted restart action. No real installer or service process is launched.
## Reproduction and fix
Four red-to-green slices reproduced missing URL, checksum HTTP 404, ambiguous multiple records, and oversized checksum acceptance. Initially the command returned success, emitted verification/install progress, and reached the restart substitute. The corrected flow returns an unsuccessful UpgradeAgentResponse before installation and removes staged bytes.
Additional cases cover blank URL, invalid request URL, connection interruption, truncated HTTP body despite a valid-looking hash, empty/whitespace checksum, short/nonhex/mismatched hash, invalid GNU marker, absent filename, garbage, NUL and embedded newline. Valid fixtures cover bare digest, GNU text and binary records, uppercase hex, CRLF and surrounding whitespace. Both URL architecture placeholders and response request IDs are checked. Failed cases preserve old executable and backup; valid cases reach the intercepted restart with exact `abc` bytes and the independent known SHA256 fixture.
## Checks
- Initial socket baseline: passed.
- UpgradeAgent targeted tests and race tests: passed.
- Socket package race tests, three runs: passed (2.009s after cleanup).
- go-gost/x: `go test ./...` and `go vet ./...`: passed.
- go-gost: `go test ./...`, `go vet ./...`, and build to a temporary output: passed.
- `git diff --check`: passed.
- All listed module checks and the build were rerun after cleanup and passed.
- Go toolchain: 1.26.10 darwin/arm64.
## Independent review
Standards: no hard violations or blockers. One optional duplicate cleanup observation was addressed by consolidating staging removal in the defer.
Spec: no actionable deviations found. Missing checksum, fetch/status/read/format/hash errors stop before success and installation; supported SHA256 formats and architecture substitution are retained. No unrelated installer, Alpine, release or deployment changes.
## Limits
This verifies the agent command and integrity gate, not actual systemd installation or live-node deployment. The fix must be installed on an agent before that agent gains the new behavior; updating the panel alone cannot retrofit old agents. Hash comparison is not a signature and does not protect against a compromised source supplying both artifact and matching checksum. No push, PR, merge, tag or deployment was performed. HTTP request timeout and installer/platform redesign remain outside this repair.
@@ -0,0 +1,43 @@
# OpenRC agent upgrade and rollback validation
## Baseline and scope
User-approved review baseline: fadc2b6c122df99140e0c9cbd481a1214d17d175. Includes the pending checksum fail-closed repair (071) and manager-aware upgrade/rollback (072). No publication, deployment, or new software installation.
## Reproduction
A disposable existing Alpine 3.22.5 linux/amd64 image on Mac mini Docker reproduced both failures: actual upgrade and rollback handlers could not find systemd-run. The upgrade used a valid checksum. Source inspection confirmed install.sh supports OpenRC but the two agent handlers did not. Existing installer CI did not test those command handlers.
## Regression coverage
Actual UpgradeAgent/RollbackAgent routing and WebSocket responses cover manager selection, Alpine preference over systemctl shims, missing manager, rejected submission, backup preservation, atomic rollback staging, request IDs and all checksum cases from 071. Script execution checks stop/replace/start order, stop failure, replacement failure recovery, directory targets and no systemd legacy cleanup on OpenRC.
Linux-only integration executes the real detached helper. An rc-service double sends SIGTERM to the original test-agent's entire process group. The test verifies that the original process receives SIGTERM while the helper survives, replaces the binary with the rollback copy, preserves the backup and invokes the start command.
## Environment and checks
- Mac mini: Go 1.26.10 darwin/arm64.
- Cached Alpine 3.22.5 linux/amd64 under Docker emulation; no package installation. Tests use loopback HTTP and service-command doubles, with no host service mounts.
- Full socket tests and focused race tests: passed.
- Alpine manager/upgrade/rollback/checksum/helper-survival suite: passed.
- go-gost/x full `go test ./...`, `go vet ./...`: passed.
- go-gost `go test ./...`, `go vet ./...`, build to temporary output: passed.
- `git diff --check`: passed.
## Limitations
Real OpenRC/systemd service integration was not run. The detached process test models the installed OpenRC PID/process-group stopping behavior, not custom cgroup policies. Successful command response acknowledges helper launch or systemd submission, not completed asynchronous service restart. No automatic downgrade is attempted if starting the replaced executable fails. Live nodes were not changed. HTTP request timeouts, release/deployment, and retrofitting already installed agents are outside this local repair.
## Independent review
Standards: no documented violations or blockers. Optional one-line legacy restart-script wrapper cleanup was deferred to preserve the existing focused legacy-service regression interface.
Spec: no blocking implementation deviation. Reviewer requested explicit OpenRC process-launch-failure coverage; follow-up validation is tracked below.
Follow-up: explicit OpenRC exec-failure coverage passed for both routed commands. A Linux-only, explicitly opted-in disposable Alpine test temporarily hides only the container's /bin/sh symlink and restores it afterward. It uses production Alpine detection, executable lookup and cmd.Start; both commands reject ENOENT, preserve the live executable and expected backup, and remove staging. No runtime test abstraction was added. The test requires Alpine and Docker markers and is skipped by default.
Reproduction of the opt-in case (only a disposable container; never mount host /bin): build the socket test binary with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go test -c -o /tmp/flvx-072-socket.test ./socket` from go-gost/x, then run `docker run --rm --pull=never --platform linux/amd64 --network none -e FLVX_TEST_DISPOSABLE_ALPINE_SHELL_FAILURE=1 -v /tmp/flvx-072-socket.test:/probe/socket.test:ro alpine:3.22 /probe/socket.test -test.run '^TestOpenRCCommandsCleanStageOnHelperExecFailure$' -test.v`.
Final host full socket race suite passed in 8.844s. Final follow-up changed only Linux test coverage; Mac socket/vet and diff checks passed again.
Independent Spec re-review confirmed the launch-failure gap is closed, with no remaining actionable spec gaps.