tcjlj/OpenFlare
1
0
Fork 0
mirror of https://github.com/Rain-kl/OpenFlare.git synced 2026-10-02 06:56:36 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: add observability buffer for heartbeat data

Browse Source 
- Introduced `ObservabilityBufferStore` to manage buffered observability records.
- Implemented methods for upserting, replaying, and acknowledging records in the buffer.
- Enhanced `AgentNodePayload` and `NodePayload` to include buffered observability data.
- Updated tests to cover new functionality for observability buffering.
- Modified existing services to persist and handle buffered observability data during heartbeats.
- Added configuration options for observability buffer path and replay minutes.
- Updated documentation to reflect new observability features and configurations.
This commit is contained in:
ryan
2026-03-14 17:57:50 +08:00
parent 7628397785
commit e25b41fd75
18 changed files with 798 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/app-config.md
+11 -3
View File
@@ -239,6 +239,7 @@ go run ./cmd/agent -config ./agent.json
"openresty_container_name": "atsflare-openresty",
"openresty_docker_image": "openresty/openresty:alpine",
"openresty_observability_port": 18081,
"observability_replay_minutes": 15,
"heartbeat_interval": 10000,
"request_timeout": 10000
}
@@ -259,6 +260,8 @@ go run ./cmd/agent -config ./agent.json
"support_dir": "/usr/local/openresty/nginx/conf/support",
"openresty_support_dir": "/usr/local/openresty/nginx/conf/support",
"openresty_observability_port": 18081,
"observability_buffer_path": "./data/observability-buffer.json",
"observability_replay_minutes": 15,
"state_path": "./data/agent-state.json",
"heartbeat_interval": 10000,
"request_timeout": 10000
@@ -284,7 +287,9 @@ go run ./cmd/agent -config ./agent.json
| `route_config_path` | 路由配置文件写入路径 | 否 | 默认为 `data_dir` 下托管路径 | `/etc/nginx/conf.d/atsflare_routes.conf` |
| `support_dir` | Agent 在本机写入受管附属文件的目录，当前包含证书与 Lua 观测脚本 | 否 | 默认为 `data_dir` 下托管 support 目录 | `./data/etc/nginx/support` |
| `openresty_support_dir` | OpenResty 实际读取受管附属文件的目录 | 否 | 本机模式默认等于 `support_dir`；Docker 模式默认 `/etc/nginx/atsflare-support` | `/usr/local/openresty/nginx/conf/support` |
| `state_path` | Agent 本地状态文件路径 | 否 | 默认为 `data_dir` 下托管状态文件 | `./data/agent-state.json` |
| `observability_buffer_path` | Agent 本地观测补报缓冲文件路径；用于在 server 短暂离线时按时间窗口落盘待补传数据 | 否 | 默认为 `data_dir` 下托管观测缓冲文件 | `./data/var/lib/atsflare/observability-buffer.json` |
| `observability_replay_minutes` | Agent 恢复心跳后允许批量补传的最近观测窗口时长（分钟） | 否 | `15` | `30` |
| `state_path` | Agent 本地状态文件路径 | 否 | 默认为 `data_dir` 下托管状态文件 | `./data/agent-state.json` |
| `heartbeat_interval` | 心跳间隔 | 否 | `10000` 毫秒 | `10000` |
| `request_timeout` | HTTP 请求超时时间 | 否 | `10000` 毫秒 | `10000` |
@@ -297,6 +302,7 @@ go run ./cmd/agent -config ./agent.json
* `node_name` 与 `node_ip` 未填写时会自动探测；若自动探测失败，配置校验会报错
* 未配置 `openresty_path` 时，默认为 Docker OpenResty 模式
* `openresty_observability_port` 默认仅绑定本地回环地址；若节点本机已有端口冲突，可改为其他未占用端口
* `observability_replay_minutes` 只控制“允许补传最近多少分钟的窗口”；超出该窗口的历史观测会在本地自动裁剪
* 为兼容旧节点，Agent 仍可读取历史字段 `cert_dir` / `openresty_cert_dir`，但保存配置时会统一写回 `support_dir` / `openresty_support_dir`
* 配置保存时，`agent_version`、`nginx_version` 由程序运行时维护，不需要写入 JSON
* 第五版主配置接管完成后，本机模式下应优先通过 `main_config_path` 由 Agent 写入受管主配置，而不是依赖节点手工维护 include 规则
@@ -308,9 +314,10 @@ go run ./cmd/agent -config ./agent.json
| 字段 | 默认值 |
| --- | --- |
| `main_config_path` | 第五版 Docker 模式默认可落在 `data_dir/etc/nginx/nginx.conf`；本机模式建议显式配置 |
| `route_config_path` | `data_dir/etc/nginx/conf.d/atsflare_routes.conf` |
| `route_config_path` | `data_dir/etc/nginx/conf.d/atsflare_routes.conf` |
| `support_dir` | `data_dir/etc/nginx/support` |
| `state_path` | `data_dir/var/lib/atsflare/agent-state.json` |
| `observability_buffer_path` | `data_dir/var/lib/atsflare/observability-buffer.json` |
| `state_path` | `data_dir/var/lib/atsflare/agent-state.json` |
Docker OpenResty 模式下：
@@ -321,6 +328,7 @@ Docker OpenResty 模式下：
补充说明：
* Agent 当前会随受管配置一并向 OpenResty 注入 Lua 观测脚本，并在每次 heartbeat 前通过 `http://127.0.0.1:<openresty_observability_port>/atsflare/observability` 读取最近窗口请求指标
* 若 server 短暂离线，Agent 会把最近窗口观测先写入 `observability_buffer_path`，待 heartbeat 恢复后按时间窗口批量补传最近 `observability_replay_minutes` 分钟的数据
* 同一端口还会暴露仅本机可访问的 `stub_status`，用于采集 OpenResty 活动连接数
### 2.5 Agent 启动示例
docs/deployment.md
+4
View File
@@ -155,6 +155,7 @@ swag init -g main.go -o docs
"openresty_container_name": "atsflare-openresty",
"openresty_docker_image": "openresty/openresty:alpine",
"openresty_observability_port": 18081,
"observability_replay_minutes": 15,
"heartbeat_interval": 10000,
"request_timeout": 10000
}
@@ -170,6 +171,7 @@ swag init -g main.go -o docs
"openresty_container_name": "atsflare-openresty",
"openresty_docker_image": "openresty/openresty:alpine",
"openresty_observability_port": 18081,
"observability_replay_minutes": 15,
"heartbeat_interval": 10000,
"request_timeout": 10000
}
@@ -185,6 +187,7 @@ swag init -g main.go -o docs
* `node_name` 与 `node_ip` 可省略，未填写时自动探测
* 未配置 `openresty_path` 时，默认使用 Docker OpenResty 容器
* Agent 会在受管 OpenResty 中注入 Lua 观测脚本，并通过 `openresty_observability_port` 对本机暴露最近窗口指标与 `stub_status`
* Agent 还会把最近 `observability_replay_minutes` 分钟内未成功上报的观测窗口落盘到本地缓冲文件，待 server 恢复后自动批量补传
### 3.3 第五版新增部署约束
@@ -193,6 +196,7 @@ swag init -g main.go -o docs
* 本机 OpenResty 模式需要为 Agent 显式提供主配置文件写入路径
* Docker OpenResty 模式需要保证主配置、路由配置和证书目录位于同一套受管挂载路径中
* Docker OpenResty 模式会额外挂载一个仅本机可访问的 `127.0.0.1:<openresty_observability_port>` 观测端口，用于 Agent 在 heartbeat 前抓取 Lua 窗口指标
* Docker / 本机两种模式都会在 `data_dir/var/lib/atsflare/observability-buffer.json` 保留最近待补传窗口；若节点磁盘是临时盘，重启后该缓冲也会丢失
* 节点现存手工维护的主配置如继续保留，必须先迁移为 Server 渲染模板的等价配置，再切换到受管模式
* 主配置切换前必须预留回滚副本，并通过一次 `openresty -t` 失败演练验证回滚