mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-10-04 15:06:37 +08:00
[优化] 更新 openresty_observability_port 描述,增强健康检查逻辑,使用 stub_status 代替 openresty -t
This commit is contained in:
@@ -238,8 +238,9 @@ Agent 必须满足:
|
|||||||
* 发现新版本时先备份旧文件。
|
* 发现新版本时先备份旧文件。
|
||||||
* 写入主配置、路由配置与必要证书文件。
|
* 写入主配置、路由配置与必要证书文件。
|
||||||
* 写入新配置后执行 `openresty -t -c <main_config_path>`,再 reload;reload 发现运行时未启动时允许直接启动 OpenResty。
|
* 写入新配置后执行 `openresty -t -c <main_config_path>`,再 reload;reload 发现运行时未启动时允许直接启动 OpenResty。
|
||||||
|
* 周期性运行时健康检查不得调用 `openresty -t`,避免健康探针触发 upstream 域名同步解析;应优先请求本地 `openresty_observability_port` 上的 `/openflare/stub_status`,以 HTTP `200 OK` 作为 OpenResty 主进程和 worker 正在提供服务的判断依据。
|
||||||
* 新配置激活失败时必须先尝试用目标配置恢复运行,再回滚到旧配置并重新拉起 OpenResty。
|
* 新配置激活失败时必须先尝试用目标配置恢复运行,再回滚到旧配置并重新拉起 OpenResty。
|
||||||
* 回滚后 OpenResty 恢复正常时上报警告;如果本地没有历史主配置可恢复,必须允许写入内置安全兜底配置并拉起只监听 `80` 端口、统一返回 `503` 的 OpenResty 运行态。
|
* 回滚后 OpenResty 恢复正常时上报警告;如果本地没有历史主配置可恢复,必须允许写入内置安全兜底配置并拉起对外只监听 `80` 端口、统一返回 `503` 的 OpenResty 运行态;兜底配置仍需保留本地 `stub_status` 健康检查入口。
|
||||||
* 兜底运行态不得清除失败目标的阻断状态;应用记录必须能体现目标版本失败但 fallback runtime 已启动。存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
* 兜底运行态不得清除失败目标的阻断状态;应用记录必须能体现目标版本失败但 fallback runtime 已启动。存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
||||||
* 某个目标 `version + checksum` 一旦应用失败并回退,Agent 必须在本地状态中阻断该目标的重复应用。
|
* 某个目标 `version + checksum` 一旦应用失败并回退,Agent 必须在本地状态中阻断该目标的重复应用。
|
||||||
|
|
||||||
|
|||||||
@@ -58,7 +58,7 @@ Agent 发现新版本后会:
|
|||||||
5. reload;如果运行时未启动,则尝试用当前配置启动 OpenResty。
|
5. reload;如果运行时未启动,则尝试用当前配置启动 OpenResty。
|
||||||
6. 上报成功、警告或失败。
|
6. 上报成功、警告或失败。
|
||||||
|
|
||||||
如果新配置激活失败,Agent 必须尝试恢复运行;回滚成功时上报警告。若本地没有历史主配置可回滚,Agent 会写入内置安全兜底配置并尝试拉起 OpenResty:该配置只监听 `80` 端口,不包含任何用户路由,统一返回 `503 Service Unavailable` 与 `OpenFlare: No Valid Configuration`。兜底启动成功时仍阻断失败目标版本并上报警告;存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
如果新配置激活失败,Agent 必须尝试恢复运行;回滚成功时上报警告。若本地没有历史主配置可回滚,Agent 会写入内置安全兜底配置并尝试拉起 OpenResty:该配置对外只监听 `80` 端口,不包含任何用户路由,统一返回 `503 Service Unavailable` 与 `OpenFlare: No Valid Configuration`,同时保留本地 `stub_status` 健康检查入口。兜底启动成功时仍阻断失败目标版本并上报警告;存在历史主配置但回滚后仍无法恢复运行时上报失败。
|
||||||
|
|
||||||
某个目标 `version + checksum` 一旦应用失败并回退,Agent 会在本地状态中阻断该目标重复应用。只有远端激活版本或 checksum 发生变化,才允许再次尝试。
|
某个目标 `version + checksum` 一旦应用失败并回退,Agent 会在本地状态中阻断该目标重复应用。只有远端激活版本或 checksum 发生变化,才允许再次尝试。
|
||||||
|
|
||||||
|
|||||||
@@ -167,6 +167,8 @@ OpenResty runtime:
|
|||||||
ps aux | grep openresty
|
ps aux | grep openresty
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Agent periodic health checks use local `http://127.0.0.1:<openresty_observability_port>/openflare/stub_status` instead of repeatedly running `openresty -t`. If a node is unhealthy, first confirm that the local observability port is listening. If `host not found in upstream` only appears during apply, the failure comes from config validation or reload, not the periodic health probe.
|
||||||
|
|
||||||
Use the actual `openresty_path` and `main_config_path` from `agent.json`.
|
Use the actual `openresty_path` and `main_config_path` from `agent.json`.
|
||||||
|
|
||||||
## HTTPS Does Not Work
|
## HTTPS Does Not Work
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ Agent supports the `-config` CLI flag, an `agent.json` file, and the `LOG_LEVEL`
|
|||||||
| `openresty_path` | OpenResty binary path | no | `openresty` |
|
| `openresty_path` | OpenResty binary path | no | `openresty` |
|
||||||
| `openresty_container_name` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
| `openresty_container_name` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
||||||
| `openresty_docker_image` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
| `openresty_docker_image` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
||||||
| `openresty_observability_port` | Local observability port | no | `18081` |
|
| `openresty_observability_port` | Local observability and OpenResty health-check port | no | `18081` |
|
||||||
| `docker_binary` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
| `docker_binary` | Deprecated Docker-control field, read for compatibility only | no | empty |
|
||||||
| `data_dir` | Agent data directory | no | `data` under config directory |
|
| `data_dir` | Agent data directory | no | `data` under config directory |
|
||||||
| `access_log_path` | OpenResty access log path | no | `data_dir/var/log/openflare/access.log` |
|
| `access_log_path` | OpenResty access log path | no | `data_dir/var/log/openflare/access.log` |
|
||||||
|
|||||||
@@ -143,7 +143,7 @@ journalctl -u openflare-agent -f
|
|||||||
|
|
||||||
注意:某个目标 `version + checksum` 一旦应用失败并回退,Agent 会在本地状态中阻断该目标重复应用。修正配置后需要重新发布生成新的 checksum,或激活旧版本回滚。
|
注意:某个目标 `version + checksum` 一旦应用失败并回退,Agent 会在本地状态中阻断该目标重复应用。修正配置后需要重新发布生成新的 checksum,或激活旧版本回滚。
|
||||||
|
|
||||||
如果这是 Agent 首次应用配置,且本地没有历史 `nginx.conf` 可回滚,失败目标仍会被阻断,但 Agent 会尝试进入安全兜底运行态。此时应用记录和 Agent 日志会包含 `fallback runtime started`,OpenResty 只监听 `80` 端口并统一返回 `503` 与 `OpenFlare: No Valid Configuration`。修正配置并重新发布新版本后,Agent 会覆盖兜底配置并恢复正常代理。
|
如果这是 Agent 首次应用配置,且本地没有历史 `nginx.conf` 可回滚,失败目标仍会被阻断,但 Agent 会尝试进入安全兜底运行态。此时应用记录和 Agent 日志会包含 `fallback runtime started`,OpenResty 对外只监听 `80` 端口并统一返回 `503` 与 `OpenFlare: No Valid Configuration`,同时保留本地 `stub_status` 健康检查入口。修正配置并重新发布新版本后,Agent 会覆盖兜底配置并恢复正常代理。
|
||||||
|
|
||||||
## OpenResty 应用失败
|
## OpenResty 应用失败
|
||||||
|
|
||||||
@@ -169,6 +169,8 @@ OpenResty 运行状态:
|
|||||||
ps aux | grep openresty
|
ps aux | grep openresty
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Agent 周期性健康检查通过本地 `http://127.0.0.1:<openresty_observability_port>/openflare/stub_status` 判断 OpenResty 是否存活,不会反复执行 `openresty -t`。如果节点被标记为 unhealthy,优先确认该本地观测端口是否正在监听;如果只在应用配置时出现 `host not found in upstream`,说明失败来自配置校验或 reload,而不是周期性健康探针。
|
||||||
|
|
||||||
实际二进制路径和主配置路径以 `agent.json` 中的 `openresty_path` 与 `main_config_path` 为准。
|
实际二进制路径和主配置路径以 `agent.json` 中的 `openresty_path` 与 `main_config_path` 为准。
|
||||||
|
|
||||||
## HTTPS 不生效
|
## HTTPS 不生效
|
||||||
|
|||||||
@@ -166,7 +166,7 @@ OpenResty 性能参数与缓存参数继续统一保存在 `Option` 表。当前
|
|||||||
| `openresty_path` | OpenResty 二进制路径 | 否 | `openresty` |
|
| `openresty_path` | OpenResty 二进制路径 | 否 | `openresty` |
|
||||||
| `openresty_container_name` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
| `openresty_container_name` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
||||||
| `openresty_docker_image` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
| `openresty_docker_image` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
||||||
| `openresty_observability_port` | 本地观测端口 | 否 | `18081` |
|
| `openresty_observability_port` | 本地观测与 OpenResty 健康检查端口 | 否 | `18081` |
|
||||||
| `docker_binary` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
| `docker_binary` | 旧 Docker 控制字段,仅兼容读取 | 否 | 空 |
|
||||||
| `data_dir` | Agent 数据目录 | 否 | 配置文件所在目录下的 `data` |
|
| `data_dir` | Agent 数据目录 | 否 | 配置文件所在目录下的 `data` |
|
||||||
| `main_config_path` | OpenResty 主配置写入路径 | 否 | `data_dir/etc/nginx/nginx.conf` |
|
| `main_config_path` | OpenResty 主配置写入路径 | 否 | `data_dir/etc/nginx/nginx.conf` |
|
||||||
@@ -189,6 +189,7 @@ OpenResty 性能参数与缓存参数继续统一保存在 `Option` 表。当前
|
|||||||
* `heartbeat_interval` 与 `request_timeout` 支持毫秒整数或 Go duration 字符串。
|
* `heartbeat_interval` 与 `request_timeout` 支持毫秒整数或 Go duration 字符串。
|
||||||
* Server 运行时配置 `AgentWebsocketUpgradeEnabled` 开启时,Agent 会在 HTTP 心跳成功后尝试升级为 WebSocket;连接失败或断开后自动退回 HTTP 心跳。
|
* Server 运行时配置 `AgentWebsocketUpgradeEnabled` 开启时,Agent 会在 HTTP 心跳成功后尝试升级为 WebSocket;连接失败或断开后自动退回 HTTP 心跳。
|
||||||
* 未配置 `openresty_path` 时默认调用 `openresty`。
|
* 未配置 `openresty_path` 时默认调用 `openresty`。
|
||||||
|
* Agent 周期性健康检查会请求 `http://127.0.0.1:<openresty_observability_port>/openflare/stub_status`,不再通过高频 `openresty -t` 判断运行时健康;配置应用、启动恢复和 reload 前校验仍会执行 `openresty -t -c <main_config_path>`。
|
||||||
* 如果 `agent.json` 不存在,但 `OPENFLARE_SERVER_URL` 与 Token 等环境变量足够,Agent 可以直接启动;两者同时存在时环境变量优先。
|
* 如果 `agent.json` 不存在,但 `OPENFLARE_SERVER_URL` 与 Token 等环境变量足够,Agent 可以直接启动;两者同时存在时环境变量优先。
|
||||||
* Agent 自动探测到私网 `node_ip` 时,Server 会在注册/心跳阶段优先保留 Agent 直连来源的公网地址,避免 NAT/多网卡场景误登记内网网卡地址。
|
* Agent 自动探测到私网 `node_ip` 时,Server 会在注册/心跳阶段优先保留 Agent 直连来源的公网地址,避免 NAT/多网卡场景误登记内网网卡地址。
|
||||||
|
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"io/fs"
|
"io/fs"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
@@ -16,6 +17,7 @@ import (
|
|||||||
"regexp"
|
"regexp"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"openflare-agent/internal/protocol"
|
"openflare-agent/internal/protocol"
|
||||||
)
|
)
|
||||||
@@ -171,9 +173,22 @@ http {
|
|||||||
server_name _;
|
server_name _;
|
||||||
return 503 "OpenFlare: No Valid Configuration\n";
|
return 503 "OpenFlare: No Valid Configuration\n";
|
||||||
}
|
}
|
||||||
|
%s
|
||||||
}
|
}
|
||||||
`
|
`
|
||||||
|
|
||||||
|
const safeDefaultFallbackObservabilityServerBlock = `
|
||||||
|
server {
|
||||||
|
listen %s;
|
||||||
|
server_name openflare-observability;
|
||||||
|
access_log off;
|
||||||
|
|
||||||
|
location = /openflare/stub_status {
|
||||||
|
stub_status;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
`
|
||||||
|
|
||||||
type ApplyOutcome struct {
|
type ApplyOutcome struct {
|
||||||
Status ApplyStatus
|
Status ApplyStatus
|
||||||
Message string
|
Message string
|
||||||
@@ -334,7 +349,10 @@ func (m *Manager) CheckHealth(ctx context.Context) error {
|
|||||||
return errors.New("openresty config not exists: waiting for initial sync")
|
return errors.New("openresty config not exists: waiting for initial sync")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return m.Executor.CheckHealth(ctx)
|
if m.OpenrestyObservabilityPort <= 0 {
|
||||||
|
return m.Executor.CheckHealth(ctx)
|
||||||
|
}
|
||||||
|
return m.checkStubStatus(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m *Manager) Restart(ctx context.Context) error {
|
func (m *Manager) Restart(ctx context.Context) error {
|
||||||
@@ -740,12 +758,39 @@ func (m *Manager) writeSafeDefaultFallbackFiles() error {
|
|||||||
if err := os.WriteFile(m.RouteConfigPath, nil, 0o644); err != nil {
|
if err := os.WriteFile(m.RouteConfigPath, nil, 0o644); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(m.MainConfigPath, []byte(safeDefaultFallbackMainConfig), 0o644); err != nil {
|
if err := os.WriteFile(m.MainConfigPath, []byte(m.safeDefaultFallbackMainConfig()), 0o644); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (m *Manager) safeDefaultFallbackMainConfig() string {
|
||||||
|
observabilityBlock := ""
|
||||||
|
if listen := strings.TrimSpace(m.OpenrestyObservabilityListen); listen != "" {
|
||||||
|
observabilityBlock = fmt.Sprintf(safeDefaultFallbackObservabilityServerBlock, listen)
|
||||||
|
}
|
||||||
|
return fmt.Sprintf(safeDefaultFallbackMainConfig, observabilityBlock)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Manager) checkStubStatus(ctx context.Context) error {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 1500*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
openrestyStubUrl := fmt.Sprintf("http://127.0.0.1:%d/openflare/stub_status", m.OpenrestyObservabilityPort)
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, openrestyStubUrl, nil)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
resp, err := (&http.Client{}).Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("openresty health endpoint unreachable: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return fmt.Errorf("openresty health endpoint returned %s", resp.Status)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func removeLegacyPowConfig(path string) error {
|
func removeLegacyPowConfig(path string) error {
|
||||||
if strings.TrimSpace(path) == "" {
|
if strings.TrimSpace(path) == "" {
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -3,6 +3,8 @@ package nginx
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
@@ -328,6 +330,67 @@ func TestManagerApplyWritesSupportFilesAndReplacesPlaceholder(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestManagerCheckHealthUsesStubStatusInsteadOfConfigTest(t *testing.T) {
|
||||||
|
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Listen failed: %v", err)
|
||||||
|
}
|
||||||
|
port := listener.Addr().(*net.TCPAddr).Port
|
||||||
|
server := &http.Server{
|
||||||
|
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path != "/openflare/stub_status" {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = w.Write([]byte("Active connections: 1\n"))
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
_ = server.Serve(listener)
|
||||||
|
}()
|
||||||
|
defer server.Shutdown(context.Background())
|
||||||
|
|
||||||
|
mainPath := filepath.Join(t.TempDir(), "nginx.conf")
|
||||||
|
if err := os.WriteFile(mainPath, []byte("main"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile failed: %v", err)
|
||||||
|
}
|
||||||
|
manager := &Manager{
|
||||||
|
MainConfigPath: mainPath,
|
||||||
|
OpenrestyObservabilityPort: port,
|
||||||
|
Executor: &fakeExecutor{
|
||||||
|
testErr: errors.New("openresty -t should not be called"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if err := manager.CheckHealth(context.Background()); err != nil {
|
||||||
|
t.Fatalf("CheckHealth failed: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagerCheckHealthFailsWhenStubStatusUnavailable(t *testing.T) {
|
||||||
|
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Listen failed: %v", err)
|
||||||
|
}
|
||||||
|
port := listener.Addr().(*net.TCPAddr).Port
|
||||||
|
if err := listener.Close(); err != nil {
|
||||||
|
t.Fatalf("listener close failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
mainPath := filepath.Join(t.TempDir(), "nginx.conf")
|
||||||
|
if err := os.WriteFile(mainPath, []byte("main"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile failed: %v", err)
|
||||||
|
}
|
||||||
|
manager := &Manager{
|
||||||
|
MainConfigPath: mainPath,
|
||||||
|
OpenrestyObservabilityPort: port,
|
||||||
|
Executor: &fakeExecutor{},
|
||||||
|
}
|
||||||
|
if err := manager.CheckHealth(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected CheckHealth to fail when stub_status is unavailable")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolverDirectiveUsesExplicitResolvers(t *testing.T) {
|
func TestResolverDirectiveUsesExplicitResolvers(t *testing.T) {
|
||||||
got := ResolverDirective("", []string{"10.0.0.2", "1.1.1.1"})
|
got := ResolverDirective("", []string{"10.0.0.2", "1.1.1.1"})
|
||||||
if !strings.Contains(got, "resolver 10.0.0.2 1.1.1.1") {
|
if !strings.Contains(got, "resolver 10.0.0.2 1.1.1.1") {
|
||||||
@@ -748,9 +811,10 @@ func TestManagerApplyStartsSafeFallbackWhenNoRollbackConfigExists(t *testing.T)
|
|||||||
testErrors: []error{errors.New("target config failed"), errors.New("rollback config missing"), nil},
|
testErrors: []error{errors.New("target config failed"), errors.New("rollback config missing"), nil},
|
||||||
}
|
}
|
||||||
manager := &Manager{
|
manager := &Manager{
|
||||||
MainConfigPath: mainPath,
|
MainConfigPath: mainPath,
|
||||||
RouteConfigPath: routePath,
|
RouteConfigPath: routePath,
|
||||||
Executor: executor,
|
OpenrestyObservabilityListen: "127.0.0.1:18081",
|
||||||
|
Executor: executor,
|
||||||
}
|
}
|
||||||
|
|
||||||
outcome := manager.Apply(context.Background(), "bad-main", "bad-route", nil)
|
outcome := manager.Apply(context.Background(), "bad-main", "bad-route", nil)
|
||||||
@@ -773,6 +837,12 @@ func TestManagerApplyStartsSafeFallbackWhenNoRollbackConfigExists(t *testing.T)
|
|||||||
if !strings.Contains(string(mainData), "listen 80 default_server") {
|
if !strings.Contains(string(mainData), "listen 80 default_server") {
|
||||||
t.Fatalf("expected fallback to listen on port 80, got %s", string(mainData))
|
t.Fatalf("expected fallback to listen on port 80, got %s", string(mainData))
|
||||||
}
|
}
|
||||||
|
if !strings.Contains(string(mainData), "listen 127.0.0.1:18081") {
|
||||||
|
t.Fatalf("expected fallback to expose local stub_status port, got %s", string(mainData))
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(mainData), "stub_status;") {
|
||||||
|
t.Fatalf("expected fallback to expose stub_status, got %s", string(mainData))
|
||||||
|
}
|
||||||
routeData, err := os.ReadFile(routePath)
|
routeData, err := os.ReadFile(routePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to read route config: %v", err)
|
t.Fatalf("failed to read route config: %v", err)
|
||||||
|
|||||||
Reference in New Issue
Block a user