fix(relay): authenticate endpoints and fail closed on missing routes

This commit is contained in:
sagitchu
2026-11-08 23:31:08 +08:00
parent 46bfb86c1b
commit 7f0177491b
50 changed files with 5405 additions and 277 deletions
+57
View File
@@ -0,0 +1,57 @@
# Two-axis review
Fixed base: 3549fe5ad8a6e9b863b3e4f7b9bf738facff87ee plus all task working-tree
changes, including new files. The owner approved this scope and subsequently
authorized PR submission, merge and the next release tag. Initial review was
performed as independent Standards and Spec passes.
## Standards
No new hard violations found against applicable AGENTS.md rules. Repository
access, explicit table names, migrations, response envelopes and agent extension
boundaries comply.
Two non-blocking judgement calls:
1. Possible duplicated code: required_relay_transport_test.go repeats the native
TCP/TLS/KCP fixture in relay_auth_matrix_investigation_test.go. The historical
investigation fixture is deliberately retained separately in this patch, to
preserve the original evidence; helper extraction is deferred.
2. Possible speculative generality: buildFederationServiceConfig accepted a
variadic credential argument but only used its first item. Fixed by accepting
one nullable credential pointer, updating legacy callers explicitly. Focused
generator and federation authentication tests pass.
Summary: zero hard violations; two low-priority smells, one corrected and one
explicitly deferred.
## Spec
Three partial requirements found:
1. Create did not take the lifecycle lock, committing a visible tunnel before
runtime application. Concurrent delete/edit/reconnect could interleave with
stale creation. This violated the create/update/redeploy/delete scope.
2. Federated create held an open database transaction across provider/node
acknowledgments, contrary to the no-network-wait-in-transaction requirement.
3. Offline/timeouts and forward synchronization failures could be swallowed as
success, contrary to the requirement to expose partial application.
These were inherited behaviors within the explicitly promised repair scope,
not regressions introduced by the patch. No additional confirmed auth or
strict-route defect, and no unjustified scope creep found. Explicit legacy,
transport, old-agent and distributed-rollback limits remain documented.
Summary: three lifecycle completion gaps, all assigned for correction before
publication. Follow-up verification will be appended after the fixes.
## Review-driven recovery additions
Follow-up inspection found two additional lifecycle issues during the transaction
and pending-state changes: local port reservations must be published before the
remote acknowledgment wait, and pending updates must still clean obsolete
runtime and reconcile forwards. Both have focused regressions.
A cleanup failure must not discard the removed runtime's ownership. Persistent
cleanup records are being added in the existing database so retries can identify
old endpoints even after desired topology changes. Records are removed only after
acknowledgment and are canceled if current desired topology needs the resource
again. This is targeted cleanup recovery, not distributed transaction rollback.
+48
View File
@@ -0,0 +1,48 @@
# Final local validation
Local darwin/arm64 checkout, baseline 3549fe5; 2026-11-08.
Go toolchain inside all modules: 1.26.10. No dependencies installed.
No production configuration, commit, push, deployment or release.
## Final commands
- go-backend: go test -race ./... -skip '^TestCaptchaVerifyLoginContract$'
PASS. Handler 132.684s; contract 103.615s; repository 30.830s.
- go-backend: go vet ./... and go build ./cmd/paneld
PASS; executable output was directed to /tmp, not the repository.
- go-gost/x: go test -race ./... and go vet ./...
PASS, including authenticated A–B–C–D transition/rejection/recovery.
- go-gost: go test ./..., go vet ./..., go build .
PASS; executable output was directed to /tmp.
- git diff --check: PASS.
## Test-environment adjustments
Two pre-existing federation cache-fallback tests formerly relied on external
unreachable.invalid and peer.example requests failing. They now use loopback
httptest providers returning HTTP 503, preserving the fallback assertions.
The existing TestCaptchaVerifyLoginContract makes a real Cloudflare verification
request; it was excluded, not reported as passing. No external access restriction
was bypassed.
The broad race run exposed a pre-existing test-fixture GORM Update callback
registration racing WebSocket teardown's node-status update. Test setup now
registers those callbacks before WebSocket startup, with atomic flags for failure
injection. Two matching fixture tests passed with race detection repeated three
times, followed by the passing full backend race run above. Production behavior
was not changed for this fixture repair.
FLVX_POSTGRES_TEST_DSN is absent; PostgreSQL execution was not verified.
Three original investigation test files remain. The old bug-expecting gated test
is skipped and its complete original source preserved in the evidence directory.
## Review status and operational limits
Formal Matt Pocock two-axis Standards/Spec review is pending owner confirmation
of the fixed review base (suggested 3549fe5 plus all local uncommitted changes).
The originating task and local plan are the proposed spec.
This local patch does not secure existing production listeners by itself.
Existing legacy tunnels remain unauthenticated until a separately initiated
migration is implemented and performed. Required is a policy, not runtime
attestation. Agents must support chain.strict, protected tunnels require full
database recovery rather than credential-free configuration restore, and older
panel rollback can reintroduce anonymous receivers. The patch does not provide
cross-node transactional rollback or fix transport confidentiality.
@@ -0,0 +1,58 @@
# Relay authentication validation
Local darwin/arm64 checkout, base 3549fe5. No deploy/push/release or production
credentials. Three pre-existing investigation tests were preserved.
## Confirmed seam
Panel HTTP plus node/federation commands, using temporary SQLite repositories,
loopback HTTP/WebSocket peers, and captured command data. Native data-plane auth
matrix is recorded separately in the required-route validation evidence.
## Vertical regression evidence
- New local tunnel: observed failure because generated relay connector/receiver
authentication was absent; implemented purpose-separated endpoint credentials.
- Required tunnel configuration export: observed missing authentication-policy
marker; exported marker added, credentials remain excluded. Configuration
restore of protected tunnels explicitly rejects missing private state.
- Federation reserve/apply: observed missing capability advertisement; added
versioned contract, receiver authentication, persistent state, and no-downgrade.
- Full dual-panel middle/exit tunnel: observed zero authenticated receivers;
threaded receiving and next-target credentials through consumer requests.
- Federation strict middle policy: observed absent chain.strict; local and
federation middle generators now set it while exits retain direct dialing.
- Insert a new middle before an existing middle: observed rejected update with
hop-index credentials; receiver identity changed to tunnel/node/role, preserving
the existing credential and assigning the new receiver a distinct credential.
- Direct-to-relay conversion: observed absent relay authentication; conversion now
requires credentials while existing legacy relay tunnels remain explicit.
- Provider private JSON loss: observed successful reapply recreating protected
state; separately persisted auth version now rejects missing private material.
Additional command-seam checks cover local update stability, reconnect followed
by explicit public batch-redeploy (avoids the 30-second production debounce),
missing consumer private state rejection, old peer refusal before reservation,
legacy list visibility, matched per-hop credentials and secret-free list/export.
## Results
Eight authentication contract tests with race detection, repeated three times:
`go test -race ./tests/contract -run '^(TestNewRelayTunnelEmitsMatchingPrivateEndpointAuthentication|TestFederatedRelayRequiresVersionedAuthAndRejectsDowngrade|TestNewFederatedTunnelAuthenticatesEveryRelayHop|TestRelayAuthenticationSurvivesInsertingBeforeExistingMiddle|TestConvertingDirectTunnelRequiresRelayAuthentication|TestBackupImportCannotSilentlyDowngradeRequiredRelay|TestNewRelayTunnelRefusesLegacyFederationBeforeReservation|TestLegacyRelayStateRemainsExplicitInList)$' -count=3`
Earlier pass: 17.878 seconds. Final run after provider version guard and log
redaction: PASS, 19.584 seconds. Focused non-TLS generation
regression passes after narrowing its old overbroad no-metadata assertion.
`git diff --check` passes.
## Explicit limitations
- Authentication-only protection against anonymous relay clients, not credential
confidentiality over raw TCP/unverified TLS/public-default KCP or insecure
control-plane paths.
- Required mode is desired policy, not attestation of every participating agent.
Old agent strict-route support requires upgrading the agent.
- Existing legacy relay tunnels remain unprotected. No automatic migration,
credential rotation, or configuration-only credential recovery is implemented.
- SQLite/loopback verification only in this task; production topology, live
upgrades, and PostgreSQL runtime checks are not exercised here.
@@ -0,0 +1,41 @@
# Required-route repair validation
Local darwin/arm64 checkout, baseline 3549fe5; 2026-11-08 UTC.
## Red / green
- New parsed-service regression failed before implementation with:
"required chain is missing, but relay accepted a direct connection".
- Added opt-in handler metadata chain.strict and a router required-chain guard.
The same test passed with race detection repeated three times.
- New panel command contract failed before generation change:
"tunnel forward must require chain, got map[chain:chains_1 type:tcp]".
- Emitting strict metadata for type-2 forwards fixed it; type-1 direct forwards
remain unrestricted. Race detection repeated three times passed.
## Native data evidence
- Real loopback A–B–D, then A–B–C–D, refuses traffic while B's chain is missing;
C/D counters remain unchanged on the rejected request; restoring B's chain
restores the complete path.
- Parsed services reject missing chain names, empty candidate selection and an
absent chain configuration with strict policy; TCP and UDP requests covered.
- Authenticated native TCP/TLS/KCP relay transports carrying TCP/UDP targets
reject a required missing route and continue to support legitimate direct exit
services. TLS certificates are verified in the fixture.
- All TestRequiredRelay tests: race detection, count=3 passed, 1.836s.
- All go-gost/x tests: ordinary and race passed; go vet passed.
- Agent module tests (including lifecycle), build and go vet passed.
## Limits
The guard prevents direct destination dialing when the required selected route
is nil or empty. It does not promise zero DNS activity before rejection, nor
change generic GOST behavior without the strict option. FLVX's per-node chain
has one next-hop selection; arbitrary partially populated multi-hop GOST routes
are not claimed to be validated by this change. Existing established connections
are not rerouted by an edit. Older agents must be upgraded to enforce the new
metadata; ordinary command acknowledgment cannot prove unknown-binary support.
Additional composed regression: each A/B/C/D receiver has independent synthetic
credentials, and every upstream connector uses the next receiver's credential.
The authenticated A–B–D → A–B–C–D transition, missing-B-chain rejection and
recovery all pass on real TCP loopback sockets. All required-route regressions,
including this composition: race repeated three times, PASS, 1.912s.
@@ -0,0 +1,477 @@
package contract_test
import (
"bytes"
"encoding/json"
"fmt"
"go-backend/internal/auth"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
)
func TestTunnelInsertMiddleHopRecordedRuntimeContract(t *testing.T) {
secret := "contract-jwt-secret"
router, r := setupContractRouter(t, secret)
server := httptest.NewServer(router)
defer server.Close()
adminToken, err := auth.GenerateToken(1, "admin_user", 0, secret)
if err != nil {
t.Fatalf("generate admin token: %v", err)
}
now := time.Now().UnixMilli()
if err := r.DB().Exec(`
INSERT INTO user(id, user, pwd, role_id, exp_time, flow, in_flow, out_flow, flow_reset_time, num, created_time, updated_time, status)
VALUES(2, 'issue281_transition_user', 'pwd', 1, 2727251700000, 99999, 0, 0, 1, 99999, ?, ?, 1)
`, now, now).Error; err != nil {
t.Fatalf("insert user: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO tunnel(name, traffic_ratio, type, protocol, flow, created_time, updated_time, status, in_ip, inx)
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`, "issue281-transition-tunnel", 1.0, 2, "tls", 99999, now, now, 1, nil, 0).Error; err != nil {
t.Fatalf("insert tunnel: %v", err)
}
tunnelID := mustLastInsertID(t, r, "issue281-transition-tunnel")
insertNode := func(name, secretValue, ip, portRange string, inx int) int64 {
if err := r.DB().Exec(`
INSERT INTO node(name, secret, server_ip, server_ip_v4, server_ip_v6, port, interface_name, version, http, tls, socks, created_time, updated_time, status, tcp_listen_addr, udp_listen_addr, inx)
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`, name, secretValue, ip, ip, "", portRange, "", "v1", 1, 1, 1, now, now, 1, "[::]", "[::]", inx).Error; err != nil {
t.Fatalf("insert node %s: %v", name, err)
}
return mustLastInsertID(t, r, name)
}
entryA := insertNode("issue281-transition-entry-a", "issue281-transition-entry-a-secret", "10.52.0.1", "54000-54010", 0)
entryB := insertNode("issue281-transition-entry-b", "issue281-transition-entry-b-secret", "10.52.0.2", "54000-54010", 1)
entryC := insertNode("issue281-transition-entry-c", "issue281-transition-entry-c-secret", "10.52.0.3", "54000-54010", 2)
exitNodeID := insertNode("issue281-transition-exit", "issue281-transition-exit-secret", "10.52.0.4", "57000-57010", 3)
if err := r.DB().Exec(`
INSERT INTO chain_tunnel(tunnel_id, chain_type, node_id, port, strategy, inx, protocol)
VALUES(?, 1, ?, 54001, 'round', 1, 'tls')
`, tunnelID, entryA).Error; err != nil {
t.Fatalf("insert initial entry chain_tunnel: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO chain_tunnel(tunnel_id, chain_type, node_id, port, strategy, inx, protocol)
VALUES(?, 3, ?, 57001, 'round', 1, 'tls')
`, tunnelID, exitNodeID).Error; err != nil {
t.Fatalf("insert exit chain_tunnel: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO user_tunnel(id, user_id, tunnel_id, speed_id, num, flow, in_flow, out_flow, flow_reset_time, exp_time, status)
VALUES(282, 2, ?, NULL, 999, 99999, 0, 0, 1, 2727251700000, 1)
`, tunnelID).Error; err != nil {
t.Fatalf("insert user_tunnel: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO forward(user_id, user_name, name, tunnel_id, remote_addr, strategy, in_flow, out_flow, created_time, updated_time, status, inx)
VALUES(2, 'issue281_transition_user', 'issue281-transition-forward', ?, '1.1.1.1:443', 'fifo', 0, 0, ?, ?, 1, 0)
`, tunnelID, now, now).Error; err != nil {
t.Fatalf("insert forward: %v", err)
}
forwardID := mustLastInsertID(t, r, "issue281-transition-forward")
if err := r.DB().Exec(`INSERT INTO forward_port(forward_id, node_id, port) VALUES(?, ?, ?)`, forwardID, entryA, 54001).Error; err != nil {
t.Fatalf("insert forward_port: %v", err)
}
// This mock records control-plane commands only; it does not forward traffic.
type command struct {
node, kind string
data json.RawMessage
}
var mu sync.Mutex
var commands []command
for _, n := range []struct {
name, secret string
id int64
}{
{"A", "issue281-transition-entry-a-secret", entryA},
{"B", "issue281-transition-entry-b-secret", entryB},
{"C", "issue281-transition-entry-c-secret", entryC},
{"D", "issue281-transition-exit-secret", exitNodeID},
} {
name := n.name
stop := startMockNodeSessionWithCommandRecorder(t, server.URL, n.secret, func(kind string, data json.RawMessage) (bool, string) {
mu.Lock()
defer mu.Unlock()
commands = append(commands, command{name, kind, append(json.RawMessage(nil), data...)})
return false, ""
})
defer stop()
waitNodeStatus(t, r, n.id, 1)
}
// Let initial mock-node reconnect reconciliation finish before the topology edit.
time.Sleep(time.Second)
node := func(id int64) map[string]interface{} {
return map[string]interface{}{"nodeId": id, "protocol": "tls", "strategy": "round"}
}
update := func(hops [][]map[string]interface{}) {
mu.Lock()
commands = nil
mu.Unlock()
payload := map[string]interface{}{"id": tunnelID, "name": "issue281-transition-tunnel", "type": 2, "flow": 99999, "trafficRatio": 1.0, "status": 1,
"inNodeId": []map[string]interface{}{node(entryA)}, "chainNodes": hops,
"outNodeId": []map[string]interface{}{{"nodeId": exitNodeID, "protocol": "tls", "strategy": "round", "port": 57001}}}
raw, _ := json.Marshal(payload)
req := httptest.NewRequest(http.MethodPost, "/api/v1/tunnel/update", bytes.NewReader(raw))
req.Header.Set("Authorization", adminToken)
req.Header.Set("Content-Type", "application/json")
res := httptest.NewRecorder()
router.ServeHTTP(res, req)
assertCode(t, res, 0)
}
check := func(edges map[string]int64) {
mu.Lock()
defer mu.Unlock()
expectedChain := fmt.Sprintf("chains_%d", tunnelID)
got := map[string]bool{}
services := map[string]bool{}
for _, c := range commands {
if c.kind == "DeleteChains" {
t.Fatalf("unexpected chain deletion on %s: %s", c.node, c.data)
}
if c.kind == "UpdateChains" || c.kind == "AddChains" {
var raw map[string]interface{}
if err := json.Unmarshal(c.data, &raw); err != nil {
t.Fatal(err)
}
if nested, ok := raw["data"].(map[string]interface{}); ok {
raw = nested
}
target, ok := edges[c.node]
if !ok {
t.Fatalf("unexpected chain owner %s", c.node)
}
port := mustQueryInt(t, r, `SELECT port FROM chain_tunnel WHERE tunnel_id = ? AND node_id = ? LIMIT 1`, tunnelID, target)
ip := map[int64]string{entryB: "10.52.0.2", entryC: "10.52.0.3", exitNodeID: "10.52.0.4"}[target]
hops := raw["hops"].([]interface{})
nodes := hops[0].(map[string]interface{})["nodes"].([]interface{})
addr := nodes[0].(map[string]interface{})["addr"]
if len(hops) != 1 || len(nodes) != 1 || addr != fmt.Sprintf("%s:%d", ip, port) {
t.Fatalf("bad edge %s: %s", c.node, c.data)
}
got[c.node] = true
t.Logf("%s -> %s via %s", c.node, ip, addr)
}
if strings.EqualFold(c.kind, "UpdateService") || strings.EqualFold(c.kind, "AddService") {
var ss []map[string]interface{}
if err := json.Unmarshal(c.data, &ss); err != nil {
t.Fatal(err)
}
for _, s := range ss {
if s["name"] != fmt.Sprintf("tunnel_%d", tunnelID) {
continue
}
h := s["handler"].(map[string]interface{})
if c.node == "B" || c.node == "C" {
if h["chain"] != expectedChain {
t.Fatalf("missing chain on %s: %s", c.node, c.data)
}
}
if c.node == "D" && h["chain"] != nil {
t.Fatalf("exit has unexpected chain")
}
services[c.node] = true
}
}
}
for owner := range edges {
if !got[owner] {
t.Errorf("no chain update for %s", owner)
}
if owner != "A" && !services[owner] {
t.Errorf("no relay service for %s", owner)
}
}
if !services["D"] {
t.Error("no exit relay service")
}
}
update([][]map[string]interface{}{{node(entryB)}})
check(map[string]int64{"A": entryB, "B": exitNodeID})
update([][]map[string]interface{}{{node(entryB)}, {node(entryC)}})
check(map[string]int64{"A": entryB, "B": entryC, "C": exitNodeID})
// A remains the entry and the user's forward remains bound there.
after, _ := mustQueryInt64Int(t, r, `SELECT node_id,port FROM forward_port WHERE forward_id = ? LIMIT 1`, forwardID)
if after != entryA {
t.Fatalf("forward entry changed: %d", after)
}
}
func TestTunnelRedeployEditMissingChainWindowInvestigation(t *testing.T) {
secret := "contract-jwt-secret"
router, r := setupContractRouter(t, secret)
server := httptest.NewServer(router)
defer server.Close()
adminToken, err := auth.GenerateToken(1, "admin_user", 0, secret)
if err != nil {
t.Fatalf("generate admin token: %v", err)
}
now := time.Now().UnixMilli()
if err := r.DB().Exec(`
INSERT INTO user(id, user, pwd, role_id, exp_time, flow, in_flow, out_flow, flow_reset_time, num, created_time, updated_time, status)
VALUES(2, 'issue281_transition_user', 'pwd', 1, 2727251700000, 99999, 0, 0, 1, 99999, ?, ?, 1)
`, now, now).Error; err != nil {
t.Fatalf("insert user: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO tunnel(name, traffic_ratio, type, protocol, flow, created_time, updated_time, status, in_ip, inx)
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`, "issue281-transition-tunnel", 1.0, 2, "tls", 99999, now, now, 1, nil, 0).Error; err != nil {
t.Fatalf("insert tunnel: %v", err)
}
tunnelID := mustLastInsertID(t, r, "issue281-transition-tunnel")
insertNode := func(name, secretValue, ip, portRange string, inx int) int64 {
if err := r.DB().Exec(`
INSERT INTO node(name, secret, server_ip, server_ip_v4, server_ip_v6, port, interface_name, version, http, tls, socks, created_time, updated_time, status, tcp_listen_addr, udp_listen_addr, inx)
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`, name, secretValue, ip, ip, "", portRange, "", "v1", 1, 1, 1, now, now, 1, "[::]", "[::]", inx).Error; err != nil {
t.Fatalf("insert node %s: %v", name, err)
}
return mustLastInsertID(t, r, name)
}
entryA := insertNode("issue281-transition-entry-a", "issue281-transition-entry-a-secret", "10.52.0.1", "54000-54010", 0)
entryB := insertNode("issue281-transition-entry-b", "issue281-transition-entry-b-secret", "10.52.0.2", "54000-54010", 1)
entryC := insertNode("issue281-transition-entry-c", "issue281-transition-entry-c-secret", "10.52.0.3", "54000-54010", 2)
exitNodeID := insertNode("issue281-transition-exit", "issue281-transition-exit-secret", "10.52.0.4", "57000-57010", 3)
if err := r.DB().Exec(`
INSERT INTO chain_tunnel(tunnel_id, chain_type, node_id, port, strategy, inx, protocol)
VALUES(?, 1, ?, 54001, 'round', 1, 'tls')
`, tunnelID, entryA).Error; err != nil {
t.Fatalf("insert initial entry chain_tunnel: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO chain_tunnel(tunnel_id, chain_type, node_id, port, strategy, inx, protocol)
VALUES(?, 3, ?, 57001, 'round', 1, 'tls')
`, tunnelID, exitNodeID).Error; err != nil {
t.Fatalf("insert exit chain_tunnel: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO user_tunnel(id, user_id, tunnel_id, speed_id, num, flow, in_flow, out_flow, flow_reset_time, exp_time, status)
VALUES(282, 2, ?, NULL, 999, 99999, 0, 0, 1, 2727251700000, 1)
`, tunnelID).Error; err != nil {
t.Fatalf("insert user_tunnel: %v", err)
}
if err := r.DB().Exec(`
INSERT INTO forward(user_id, user_name, name, tunnel_id, remote_addr, strategy, in_flow, out_flow, created_time, updated_time, status, inx)
VALUES(2, 'issue281_transition_user', 'issue281-transition-forward', ?, '1.1.1.1:443', 'fifo', 0, 0, ?, ?, 1, 0)
`, tunnelID, now, now).Error; err != nil {
t.Fatalf("insert forward: %v", err)
}
forwardID := mustLastInsertID(t, r, "issue281-transition-forward")
if err := r.DB().Exec(`INSERT INTO forward_port(forward_id, node_id, port) VALUES(?, ?, ?)`, forwardID, entryA, 54001).Error; err != nil {
t.Fatalf("insert forward_port: %v", err)
}
// This mock records control-plane commands only; it does not forward traffic.
type command struct {
node, kind string
data json.RawMessage
}
var mu sync.Mutex
var commands []command
armed := false
chainPresent, servicePresent := map[string]bool{}, map[string]bool{}
atDeleteService, releaseDeleteService, editReachedA := make(chan struct{}), make(chan struct{}), make(chan struct{}, 1)
var gateOnce, releaseOnce sync.Once
release := func() { releaseOnce.Do(func() { close(releaseDeleteService) }) }
defer release()
for _, n := range []struct {
name, secret string
id int64
}{
{"A", "issue281-transition-entry-a-secret", entryA},
{"B", "issue281-transition-entry-b-secret", entryB},
{"C", "issue281-transition-entry-c-secret", entryC},
{"D", "issue281-transition-exit-secret", exitNodeID},
} {
name := n.name
stop := startMockNodeSessionWithCommandRecorder(t, server.URL, n.secret, func(kind string, data json.RawMessage) (bool, string) {
mu.Lock()
active := armed
shouldGate := active && name == "B" && kind == "DeleteService"
if active && name == "A" && kind == "UpdateChains" {
select {
case editReachedA <- struct{}{}:
default:
}
}
mu.Unlock()
if shouldGate {
gateOnce.Do(func() { close(atDeleteService); <-releaseDeleteService })
}
mu.Lock()
defer mu.Unlock()
if kind == "DeleteChains" {
chainPresent[name] = false
}
if kind == "AddChains" || kind == "UpdateChains" {
chainPresent[name] = true
}
if kind == "DeleteService" {
servicePresent[name] = false
}
if kind == "AddService" || kind == "UpdateService" {
servicePresent[name] = true
}
commands = append(commands, command{name, kind, append(json.RawMessage(nil), data...)})
return false, ""
})
defer stop()
waitNodeStatus(t, r, n.id, 1)
}
// Let initial mock-node reconnect reconciliation finish before the topology edit.
time.Sleep(time.Second)
node := func(id int64) map[string]interface{} {
return map[string]interface{}{"nodeId": id, "protocol": "tls", "strategy": "round"}
}
update := func(hops [][]map[string]interface{}) {
mu.Lock()
commands = nil
mu.Unlock()
payload := map[string]interface{}{"id": tunnelID, "name": "issue281-transition-tunnel", "type": 2, "flow": 99999, "trafficRatio": 1.0, "status": 1,
"inNodeId": []map[string]interface{}{node(entryA)}, "chainNodes": hops,
"outNodeId": []map[string]interface{}{{"nodeId": exitNodeID, "protocol": "tls", "strategy": "round", "port": 57001}}}
raw, _ := json.Marshal(payload)
req := httptest.NewRequest(http.MethodPost, "/api/v1/tunnel/update", bytes.NewReader(raw))
req.Header.Set("Authorization", adminToken)
req.Header.Set("Content-Type", "application/json")
res := httptest.NewRecorder()
router.ServeHTTP(res, req)
assertCode(t, res, 0)
}
check := func(edges map[string]int64) {
mu.Lock()
defer mu.Unlock()
expectedChain := fmt.Sprintf("chains_%d", tunnelID)
got := map[string]bool{}
services := map[string]bool{}
for _, c := range commands {
if c.kind == "DeleteChains" {
t.Fatalf("unexpected chain deletion on %s: %s", c.node, c.data)
}
if c.kind == "UpdateChains" || c.kind == "AddChains" {
var raw map[string]interface{}
if err := json.Unmarshal(c.data, &raw); err != nil {
t.Fatal(err)
}
if nested, ok := raw["data"].(map[string]interface{}); ok {
raw = nested
}
target, ok := edges[c.node]
if !ok {
t.Fatalf("unexpected chain owner %s", c.node)
}
port := mustQueryInt(t, r, `SELECT port FROM chain_tunnel WHERE tunnel_id = ? AND node_id = ? LIMIT 1`, tunnelID, target)
ip := map[int64]string{entryB: "10.52.0.2", entryC: "10.52.0.3", exitNodeID: "10.52.0.4"}[target]
hops := raw["hops"].([]interface{})
nodes := hops[0].(map[string]interface{})["nodes"].([]interface{})
addr := nodes[0].(map[string]interface{})["addr"]
if len(hops) != 1 || len(nodes) != 1 || addr != fmt.Sprintf("%s:%d", ip, port) {
t.Fatalf("bad edge %s: %s", c.node, c.data)
}
got[c.node] = true
t.Logf("%s -> %s via %s", c.node, ip, addr)
}
if strings.EqualFold(c.kind, "UpdateService") || strings.EqualFold(c.kind, "AddService") {
var ss []map[string]interface{}
if err := json.Unmarshal(c.data, &ss); err != nil {
t.Fatal(err)
}
for _, s := range ss {
if s["name"] != fmt.Sprintf("tunnel_%d", tunnelID) {
continue
}
h := s["handler"].(map[string]interface{})
if c.node == "B" || c.node == "C" {
if h["chain"] != expectedChain {
t.Fatalf("missing chain on %s: %s", c.node, c.data)
}
}
if c.node == "D" && h["chain"] != nil {
t.Fatalf("exit has unexpected chain")
}
services[c.node] = true
}
}
}
for owner := range edges {
if !got[owner] {
t.Errorf("no chain update for %s", owner)
}
if owner != "A" && !services[owner] {
t.Errorf("no relay service for %s", owner)
}
}
if !services["D"] {
t.Error("no exit relay service")
}
}
update([][]map[string]interface{}{{node(entryB)}})
check(map[string]int64{"A": entryB, "B": exitNodeID})
mu.Lock()
armed = true
mu.Unlock()
redeployDone := make(chan struct{})
go func() {
defer close(redeployDone)
raw, _ := json.Marshal(map[string]interface{}{"ids": []int64{tunnelID}})
req := httptest.NewRequest(http.MethodPost, "/api/v1/tunnel/batch-redeploy", bytes.NewReader(raw))
req.Header.Set("Authorization", adminToken)
req.Header.Set("Content-Type", "application/json")
res := httptest.NewRecorder()
router.ServeHTTP(res, req)
t.Logf("redeploy response: %s", res.Body.String())
}()
await := func(ch <-chan struct{}, label string) {
t.Helper()
select {
case <-ch:
case <-time.After(10 * time.Second):
release()
t.Fatalf("timeout waiting for %s", label)
}
}
await(atDeleteService, "B DeleteService gate")
editDone := make(chan struct{})
go func() { defer close(editDone); update([][]map[string]interface{}{{node(entryB)}, {node(entryC)}}) }()
await(editReachedA, "edit UpdateChains on A")
mu.Lock()
missing, live := !chainPresent["B"], servicePresent["B"]
mu.Unlock()
if !missing || !live {
release()
t.Fatalf("expected B live service with missing chain during overlap, missing=%v live=%v", missing, live)
}
t.Log("Deterministic overlap confirmed: edit reached A while redeploy had deleted B chain but had not deleted B relay service")
release()
await(editDone, "edit completion")
await(redeployDone, "redeploy completion")
mu.Lock()
defer mu.Unlock()
t.Logf("final mock runtime chain presence: A=%v B=%v C=%v; services B=%v C=%v D=%v", chainPresent["A"], chainPresent["B"], chainPresent["C"], servicePresent["B"], servicePresent["C"], servicePresent["D"])
}
@@ -0,0 +1,64 @@
# Tunnel lifecycle repair validation
Local darwin/arm64 checkout, baseline 3549fe5; 2026-11-08 UTC. No deployment, push, commit, dependency, or real node used.
## Agreed seams
Panel HTTP requests, temporary repository storage, and mock external agent/federation command interfaces. Tests observe runtime command payloads and API results. No private lifecycle helper tests.
## Red → green evidence
1. `go test ./tests/contract -run '^TestTunnelRedeployPreservesLiveRoutes$' -count=1`
- Red: `unexpected chain deletion on A: {"chain":"chains_1"}`.
- Green after replacing cleanup + create with existing chain/service upsert: 1.620s.
2. `go test ./tests/contract -run '^TestTunnelEditWaitsForRedeploy$' -count=1`
- Red: `edit changed runtime before redeploy finished` while the redeploy's exit service command was gated.
- Green after per-tunnel locking over snapshot, persistence and runtime application. Combined first two tests: 2.879s.
3. `go test ./tests/contract -run '^TestTunnelFailedRedeployPreservesLiveRoutes$' -count=1`
- Red: `failed redeploy removed existing runtime: DeleteService on B` on an injected bind failure.
- Green after removing destructive cleanup retry for upsert. Existing add/create retry behavior remains unchanged. Combined local tests: 3.922s.
4. `go test ./tests/contract -run '^TestFederationRedeployPreservesExistingBindings$' -count=1`
- Red: `redeploy destroyed existing federation runtime: [DeleteService DeleteService DeleteChains]`.
- Green: 0.653s after reusing stable resource keys and excluding existing bindings from failure cleanup.
Additional verification covers failed federation apply, concurrent delete waiting for redeploy, and final middle-hop target after concurrent edit. Full six-regression race invocation:
`go test -race ./tests/contract -run 'Test(Tunnel(RedeployPreservesLiveRoutes|FailedRedeployPreservesLiveRoutes|EditWaitsForRedeploy|DeleteWaitsForRedeploy)|Federation(Failed)?RedeployPreservesExistingBindings)' -count=3`
Result: PASS, 32.626s. The gated concurrency observation lasts 150ms; successful completion and the final B→C target are also asserted.
## Scope and limitations
- One process's Handler owns the locks. Different tunnels remain independent; migration locks source and target in sorted order. Update, redeploy/reconnect (through the shared redeploy path), ordinary/batch delete, and delete-with-forward migration participate. Best-exit probes remain unlocked while measuring, then revalidate owner/exit topology under the lifecycle lock before applying.
- Existing PR #550 agent mutation queue and service rollback were not modified. Panel upserts defer failure rollback to the agent rather than deleting a preserved listener.
- This is not a distributed multi-node transaction. Existing tunnel edits still commit desired state before node application; partial apply can leave some nodes ahead of others, and full atomic rollback was not introduced. Offline/deferred application behavior is inherited and is not proof of complete deployment.
- Forward CRUD and provider-side federation ownership are not newly serialized by this per-consumer-tunnel lock.
- Original bug-characterization source is preserved verbatim at `070-tunnel-insert-middle-hop-investigation.go.txt`; only its old-bug gated test is skipped in the active test file because waiting for the removed insecure event is no longer a passing regression.
## Release review follow-up (2026-11-08 UTC)
Review findings were repaired before publication:
- `TestTunnelDeleteWaitsForCreateRuntime`: red `delete removed runtime while create was still applying`; green after claiming the new ID's lifecycle lock before its first commit. Claim is nonblocking so it never waits for a lifecycle owner while holding a database transaction.
- `TestFederationCreateDoesNotHoldDatabaseDuringAcknowledgment`: red `federation create held database while waiting for acknowledgment`; green after splitting local desired-state persistence and final provider-binding persistence into short transactions around network work. Initial topology and allocated local ports are persisted with credentials before waiting for the provider; unallocated remote ports are NULL.
- `TestConcurrentCreatesReserveLocalPortsBeforeFederationAcknowledgment`: mixed local/remote topology with one available local receiver port. The second tunnel is rejected while the first awaits provider acknowledgment. Mutation proof disabling only initial prepared-chain persistence produced red `second create did not reject the reserved local port before provider acknowledgment`; restoring persistence produced green.
- `TestOfflineTunnelCreateReportsPendingRuntime`: red response was only `code:0,msg:操作成功`. Create/update now retain code-0 accepted-configuration compatibility but include explicit pending message and `runtimeStatus:pending`; batch redeploy reports pending in its failure count. Desired configuration/authentication and previously applied runtime are retained.
- `TestPendingTunnelUpdateStillRemovesObsoleteRelay`: red `pending update left removed relay live: map[]`; pending results now continue obsolete cleanup and required forward reconciliation before responding.
- `TestFailedObsoleteRelayRemovalPreservesItsChain`: red deleted both listener and chain despite rejected listener deletion. Cleanup now removes service before chain, preserves the chain on unacknowledged listener deletion, and reports cleanup errors in the pending outcome.
The original statement that offline application is silently inherited no longer applies to create/update/redeploy outcomes. This remains an asynchronous, multi-node workflow rather than a distributed transaction. Failed obsolete cleanup is reported; full durable cross-node rollback was not introduced.
After these changes, handler and contract suites passed in 11.951s and 16.189s respectively (prior to the final isolated cleanup-error regression/hardening, which also passed focused validation).
### Durable obsolete-runtime cleanup
A later review identified that a pending message alone loses obsolete endpoint ownership once topology rows are replaced. The repair adds `tunnel_runtime_cleanup` in the existing database (no external component). Each row contains tunnel ID, node ID, runtime kind and old receiver port. Topology updates queue ownership in the same transaction that replaces topology; tunnel deletion queues current runtime before deleting its desired rows. Cleanup records are not cascaded with the tunnel or authentication rows.
Reconciliation uses existing maintenance, reconnect, and redeploy paths under the tunnel lifecycle lock. Service deletion must be acknowledged before its chain is removed. Successful removal clears only acknowledged records. Re-desired roles cancel obsolete deletion safely; changed receiver ports remain reserved until an acknowledged service replacement. Multiple unresolved ports can coexist for the same named receiver. Automatic allocation, explicit relay/forward port validation, and federation port discovery include these reservations.
Public-seam regressions:
- Extended failed-cleanup test first failed `redeploy lost obsolete cleanup ownership after failure: map[]`; now a subsequent successful redeploy removes both obsolete resources and a third pass sends no repeated deletion.
- `TestRedesiredRelayKeepsOldPortReservedUntilReplacementAcknowledged` re-adds a removed node on a different receiver port with a simulated timeout, rejects another tunnel claiming the unresolved old port, then permits that port only after a successful replacement ACK. No delayed deletion reaches the re-desired listener.
- `TestDeletedTunnelCleanupSurvivesHandlerReplacement` deletes desired state despite a rejected listener deletion, verifies old-port reservation, then replaces the Handler and reconnects node sessions. Persisted ownership permits cleanup/retry and releases the port only after ACK. Subsequent retry does not remove the replacement tunnel.
The full repository, handler and contract suites passed (2.965s, 12.463s, 16.974s) before the final reconnect-specific test and forward-port reservation check. Focused race validation results are reported with the integration checkpoint.