mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-09-28 05:46:36 +08:00
[优化] 更新 openresty_observability_port 描述,增强健康检查逻辑,使用 stub_status 代替 openresty -t
This commit is contained in:
@@ -238,8 +238,9 @@ Agent 必须满足:
|
||||
* 发现新版本时先备份旧文件。
|
||||
* 写入主配置、路由配置与必要证书文件。
|
||||
* 写入新配置后执行 `openresty -t -c <main_config_path>`,再 reload;reload 发现运行时未启动时允许直接启动 OpenResty。
|
||||
* 周期性运行时健康检查不得调用 `openresty -t`,避免健康探针触发 upstream 域名同步解析;应优先请求本地 `openresty_observability_port` 上的 `/openflare/stub_status`,以 HTTP `200 OK` 作为 OpenResty 主进程和 worker 正在提供服务的判断依据。
|
||||
* 新配置激活失败时必须先尝试用目标配置恢复运行,再回滚到旧配置并重新拉起 OpenResty。
|
||||
* 回滚后 OpenResty 恢复正常时上报警告;如果本地没有历史主配置可恢复,必须允许写入内置安全兜底配置并拉起只监听 `80` 端口、统一返回 `503` 的 OpenResty 运行态。
|
||||
* 回滚后 OpenResty 恢复正常时上报警告;如果本地没有历史主配置可恢复,必须允许写入内置安全兜底配置并拉起对外只监听 `80` 端口、统一返回 `503` 的 OpenResty 运行态;兜底配置仍需保留本地 `stub_status` 健康检查入口。
|
||||
* 兜底运行态不得清除失败目标的阻断状态;应用记录必须能体现目标版本失败但 fallback runtime 已启动。存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
||||
* 某个目标 `version + checksum` 一旦应用失败并回退,Agent 必须在本地状态中阻断该目标的重复应用。
|
||||
|
||||
|
||||
@@ -58,7 +58,7 @@ Agent 发现新版本后会:
|
||||
5. reload;如果运行时未启动,则尝试用当前配置启动 OpenResty。
|
||||
6. 上报成功、警告或失败。
|
||||
|
||||
如果新配置激活失败,Agent 必须尝试恢复运行;回滚成功时上报警告。若本地没有历史主配置可回滚,Agent 会写入内置安全兜底配置并尝试拉起 OpenResty:该配置只监听 `80` 端口,不包含任何用户路由,统一返回 `503 Service Unavailable` 与 `OpenFlare: No Valid Configuration`。兜底启动成功时仍阻断失败目标版本并上报警告;存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
||||
如果新配置激活失败,Agent 必须尝试恢复运行;回滚成功时上报警告。若本地没有历史主配置可回滚,Agent 会写入内置安全兜底配置并尝试拉起 OpenResty:该配置对外只监听 `80` 端口,不包含任何用户路由,统一返回 `503 Service Unavailable` 与 `OpenFlare: No Valid Configuration`,同时保留本地 `stub_status` 健康检查入口。兜底启动成功时仍阻断失败目标版本并上报警告;存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
||||
|
||||
某个目标 `version + checksum` 一旦应用失败并回退,Agent 会在本地状态中阻断该目标重复应用。只有远端激活版本或 checksum 发生变化,才允许再次尝试。
|
||||
|
||||
|
||||
@@ -167,6 +167,8 @@ OpenResty runtime:
|
||||
ps aux | grep openresty
|
||||
```
|
||||
|
||||
Agent periodic health checks use local `http://127.0.0.1:<openresty_observability_port>/openflare/stub_status` instead of repeatedly running `openresty -t`. If a node is unhealthy, first confirm that the local observability port is listening. If `host not found in upstream` only appears during apply, the failure comes from config validation or reload, not the periodic health probe.
|
||||
|
||||
Use the actual `openresty_path` and `main_config_path` from `agent.json`.
|
||||
|
||||
## HTTPS Does Not Work
|
||||
|
||||
@@ -63,7 +63,7 @@ Agent supports the `-config` CLI flag, an `agent.json` file, and the `LOG_LEVEL`
|
||||
| `openresty_path` | OpenResty binary path | no | `openresty` |
|
||||
| `openresty_container_name` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
||||
| `openresty_docker_image` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
||||
| `openresty_observability_port` | Local observability port | no | `18081` |
|
||||
| `openresty_observability_port` | Local observability and OpenResty health-check port | no | `18081` |
|
||||
| `docker_binary` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
||||
| `data_dir` | Agent data directory | no | `data` under config directory |
|
||||
| `access_log_path` | OpenResty access log path | no | `data_dir/var/log/openflare/access.log` |
|
||||
|
||||
@@ -143,7 +143,7 @@ journalctl -u openflare-agent -f
|
||||
|
||||
注意:某个目标 `version + checksum` 一旦应用失败并回退,Agent 会在本地状态中阻断该目标重复应用。修正配置后需要重新发布生成新的 checksum,或激活旧版本回滚。
|
||||
|
||||
如果这是 Agent 首次应用配置,且本地没有历史 `nginx.conf` 可回滚,失败目标仍会被阻断,但 Agent 会尝试进入安全兜底运行态。此时应用记录和 Agent 日志会包含 `fallback runtime started`,OpenResty 只监听 `80` 端口并统一返回 `503` 与 `OpenFlare: No Valid Configuration`。修正配置并重新发布新版本后,Agent 会覆盖兜底配置并恢复正常代理。
|
||||
如果这是 Agent 首次应用配置,且本地没有历史 `nginx.conf` 可回滚,失败目标仍会被阻断,但 Agent 会尝试进入安全兜底运行态。此时应用记录和 Agent 日志会包含 `fallback runtime started`,OpenResty 对外只监听 `80` 端口并统一返回 `503` 与 `OpenFlare: No Valid Configuration`,同时保留本地 `stub_status` 健康检查入口。修正配置并重新发布新版本后,Agent 会覆盖兜底配置并恢复正常代理。
|
||||
|
||||
## OpenResty 应用失败
|
||||
|
||||
@@ -169,6 +169,8 @@ OpenResty 运行状态:
|
||||
ps aux | grep openresty
|
||||
```
|
||||
|
||||
Agent 周期性健康检查通过本地 `http://127.0.0.1:<openresty_observability_port>/openflare/stub_status` 判断 OpenResty 是否存活,不会反复执行 `openresty -t`。如果节点被标记为 unhealthy,优先确认该本地观测端口是否正在监听;如果只在应用配置时出现 `host not found in upstream`,说明失败来自配置校验或 reload,而不是周期性健康探针。
|
||||
|
||||
实际二进制路径和主配置路径以 `agent.json` 中的 `openresty_path` 与 `main_config_path` 为准。
|
||||
|
||||
## HTTPS 不生效
|
||||
|
||||
@@ -166,7 +166,7 @@ OpenResty 性能参数与缓存参数继续统一保存在 `Option` 表。当前
|
||||
| `openresty_path` | OpenResty 二进制路径 | 否 | `openresty` |
|
||||
| `openresty_container_name` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
||||
| `openresty_docker_image` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
||||
| `openresty_observability_port` | 本地观测端口 | 否 | `18081` |
|
||||
| `openresty_observability_port` | 本地观测与 OpenResty 健康检查端口 | 否 | `18081` |
|
||||
| `docker_binary` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
||||
| `data_dir` | Agent 数据目录 | 否 | 配置文件所在目录下的 `data` |
|
||||
| `main_config_path` | OpenResty 主配置写入路径 | 否 | `data_dir/etc/nginx/nginx.conf` |
|
||||
@@ -189,6 +189,7 @@ OpenResty 性能参数与缓存参数继续统一保存在 `Option` 表。当前
|
||||
* `heartbeat_interval` 与 `request_timeout` 支持毫秒整数或 Go duration 字符串。
|
||||
* Server 运行时配置 `AgentWebsocketUpgradeEnabled` 开启时,Agent 会在 HTTP 心跳成功后尝试升级为 WebSocket;连接失败或断开后自动退回 HTTP 心跳。
|
||||
* 未配置 `openresty_path` 时默认调用 `openresty`。
|
||||
* Agent 周期性健康检查会请求 `http://127.0.0.1:<openresty_observability_port>/openflare/stub_status`,不再通过高频 `openresty -t` 判断运行时健康;配置应用、启动恢复和 reload 前校验仍会执行 `openresty -t -c <main_config_path>`。
|
||||
* 如果 `agent.json` 不存在,但 `OPENFLARE_SERVER_URL` 与 Token 等环境变量足够,Agent 可以直接启动;两者同时存在时环境变量优先。
|
||||
* Agent 自动探测到私网 `node_ip` 时,Server 会在注册/心跳阶段优先保留 Agent 直连来源的公网地址,避免 NAT/多网卡场景误登记内网网卡地址。
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -16,6 +17,7 @@ import (
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"openflare-agent/internal/protocol"
|
||||
)
|
||||
@@ -171,9 +173,22 @@ http {
|
||||
server_name _;
|
||||
return 503 "OpenFlare: No Valid Configuration\n";
|
||||
}
|
||||
%s
|
||||
}
|
||||
`
|
||||
|
||||
const safeDefaultFallbackObservabilityServerBlock = `
|
||||
server {
|
||||
listen %s;
|
||||
server_name openflare-observability;
|
||||
access_log off;
|
||||
|
||||
location = /openflare/stub_status {
|
||||
stub_status;
|
||||
}
|
||||
}
|
||||
`
|
||||
|
||||
type ApplyOutcome struct {
|
||||
Status ApplyStatus
|
||||
Message string
|
||||
@@ -334,7 +349,10 @@ func (m *Manager) CheckHealth(ctx context.Context) error {
|
||||
return errors.New("openresty config not exists: waiting for initial sync")
|
||||
}
|
||||
}
|
||||
return m.Executor.CheckHealth(ctx)
|
||||
if m.OpenrestyObservabilityPort <= 0 {
|
||||
return m.Executor.CheckHealth(ctx)
|
||||
}
|
||||
return m.checkStubStatus(ctx)
|
||||
}
|
||||
|
||||
func (m *Manager) Restart(ctx context.Context) error {
|
||||
@@ -740,12 +758,39 @@ func (m *Manager) writeSafeDefaultFallbackFiles() error {
|
||||
if err := os.WriteFile(m.RouteConfigPath, nil, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(m.MainConfigPath, []byte(safeDefaultFallbackMainConfig), 0o644); err != nil {
|
||||
if err := os.WriteFile(m.MainConfigPath, []byte(m.safeDefaultFallbackMainConfig()), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) safeDefaultFallbackMainConfig() string {
|
||||
observabilityBlock := ""
|
||||
if listen := strings.TrimSpace(m.OpenrestyObservabilityListen); listen != "" {
|
||||
observabilityBlock = fmt.Sprintf(safeDefaultFallbackObservabilityServerBlock, listen)
|
||||
}
|
||||
return fmt.Sprintf(safeDefaultFallbackMainConfig, observabilityBlock)
|
||||
}
|
||||
|
||||
func (m *Manager) checkStubStatus(ctx context.Context) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 1500*time.Millisecond)
|
||||
defer cancel()
|
||||
openrestyStubUrl := fmt.Sprintf("http://127.0.0.1:%d/openflare/stub_status", m.OpenrestyObservabilityPort)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, openrestyStubUrl, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := (&http.Client{}).Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("openresty health endpoint unreachable: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("openresty health endpoint returned %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func removeLegacyPowConfig(path string) error {
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return nil
|
||||
|
||||
@@ -3,6 +3,8 @@ package nginx
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
@@ -328,6 +330,67 @@ func TestManagerApplyWritesSupportFilesAndReplacesPlaceholder(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagerCheckHealthUsesStubStatusInsteadOfConfigTest(t *testing.T) {
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("Listen failed: %v", err)
|
||||
}
|
||||
port := listener.Addr().(*net.TCPAddr).Port
|
||||
server := &http.Server{
|
||||
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/openflare/stub_status" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte("Active connections: 1\n"))
|
||||
}),
|
||||
}
|
||||
go func() {
|
||||
_ = server.Serve(listener)
|
||||
}()
|
||||
defer server.Shutdown(context.Background())
|
||||
|
||||
mainPath := filepath.Join(t.TempDir(), "nginx.conf")
|
||||
if err := os.WriteFile(mainPath, []byte("main"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile failed: %v", err)
|
||||
}
|
||||
manager := &Manager{
|
||||
MainConfigPath: mainPath,
|
||||
OpenrestyObservabilityPort: port,
|
||||
Executor: &fakeExecutor{
|
||||
testErr: errors.New("openresty -t should not be called"),
|
||||
},
|
||||
}
|
||||
if err := manager.CheckHealth(context.Background()); err != nil {
|
||||
t.Fatalf("CheckHealth failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagerCheckHealthFailsWhenStubStatusUnavailable(t *testing.T) {
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("Listen failed: %v", err)
|
||||
}
|
||||
port := listener.Addr().(*net.TCPAddr).Port
|
||||
if err := listener.Close(); err != nil {
|
||||
t.Fatalf("listener close failed: %v", err)
|
||||
}
|
||||
|
||||
mainPath := filepath.Join(t.TempDir(), "nginx.conf")
|
||||
if err := os.WriteFile(mainPath, []byte("main"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile failed: %v", err)
|
||||
}
|
||||
manager := &Manager{
|
||||
MainConfigPath: mainPath,
|
||||
OpenrestyObservabilityPort: port,
|
||||
Executor: &fakeExecutor{},
|
||||
}
|
||||
if err := manager.CheckHealth(context.Background()); err == nil {
|
||||
t.Fatal("expected CheckHealth to fail when stub_status is unavailable")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverDirectiveUsesExplicitResolvers(t *testing.T) {
|
||||
got := ResolverDirective("", []string{"10.0.0.2", "1.1.1.1"})
|
||||
if !strings.Contains(got, "resolver 10.0.0.2 1.1.1.1") {
|
||||
@@ -748,9 +811,10 @@ func TestManagerApplyStartsSafeFallbackWhenNoRollbackConfigExists(t *testing.T)
|
||||
testErrors: []error{errors.New("target config failed"), errors.New("rollback config missing"), nil},
|
||||
}
|
||||
manager := &Manager{
|
||||
MainConfigPath: mainPath,
|
||||
RouteConfigPath: routePath,
|
||||
Executor: executor,
|
||||
MainConfigPath: mainPath,
|
||||
RouteConfigPath: routePath,
|
||||
OpenrestyObservabilityListen: "127.0.0.1:18081",
|
||||
Executor: executor,
|
||||
}
|
||||
|
||||
outcome := manager.Apply(context.Background(), "bad-main", "bad-route", nil)
|
||||
@@ -773,6 +837,12 @@ func TestManagerApplyStartsSafeFallbackWhenNoRollbackConfigExists(t *testing.T)
|
||||
if !strings.Contains(string(mainData), "listen 80 default_server") {
|
||||
t.Fatalf("expected fallback to listen on port 80, got %s", string(mainData))
|
||||
}
|
||||
if !strings.Contains(string(mainData), "listen 127.0.0.1:18081") {
|
||||
t.Fatalf("expected fallback to expose local stub_status port, got %s", string(mainData))
|
||||
}
|
||||
if !strings.Contains(string(mainData), "stub_status;") {
|
||||
t.Fatalf("expected fallback to expose stub_status, got %s", string(mainData))
|
||||
}
|
||||
routeData, err := os.ReadFile(routePath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read route config: %v", err)
|
||||
|
||||
Reference in New Issue
Block a user