Commit Graph

46 Commits

Author SHA1 Message Date
ryan 160e63558f fix(clickhouse): harden R/W path P0–P3 (cleanup, durability, rollups)
Honest TTL cleanup semantics; enqueue-safe dedup with flush retry and writer
metrics; model insert hooks; latest-per-node and hourly metric/openresty
rollups; small-host pool/async defaults, traffic hourly TTL, and UV labeling.
2026-07-10 10:34:04 +08:00
ryan 9b3555c569 fix(clickhouse): cut idle CPU from tiny parts and oversized merge pools
Observability writers flushed every few seconds with MinBatchSize unset,
creating constant small parts and merge load. Enable MinBatchSize with
MaxFlushWait, batch access logs more aggressively, and shrink ClickHouse
background pools for 3c hosts.
2026-07-10 10:08:32 +08:00
ryan b928928958 fix(observability): restore 24h capacity/network/disk trends via CH hourly agg
Node and dashboard 24h capacity, network, and disk IO charts only used the
latest limited raw snapshots (120/500 rows), so historical hour buckets stayed
empty. Prefer ClickHouse hourly aggregates with counter deltas, and fall back
to raw snapshots when aggregation is unavailable.
2026-07-10 09:48:45 +08:00
ryan 58624db397 perf(clickhouse): Phase 2 legacy governance — TTL cleanup, unified pool, MV, ops API
- Replace retention ALTER DELETE with MATERIALIZE TTL; use TRUNCATE for delete-all
- Remove GORM ClickHouse pool; migrate user access log reads to ChConn
- Drop query-side trim(remote_addr); enable wait_for_async_insert=1
- Add of_node_traffic_hourly MV and dashboard traffic trend fallback
- Add GET /admin/status/clickhouse operational metrics endpoint
2026-07-02 15:47:38 +08:00
ryan 38946d1af5 fix(clickhouse): resolve lint issues from optimization stack 2026-07-02 15:28:49 +08:00
ryan caf2ffcff4 perf(clickhouse): P1 TTL migrations, ORDER BY tune, remote_addr normalization 2026-07-02 15:25:03 +08:00
ryan 6525bef15d perf(clickhouse): P0/P1 access log and WAF query aggregation and SQL pagination 2026-07-02 15:23:17 +08:00
ryan 3e910f1961 perf(clickhouse): P0 dashboard/observability query limits, cache, slower polling 2026-07-02 15:23:17 +08:00
ryan ae618905a3 perf(clickhouse): P0 write path — remove heartbeat DELETE, batchwriter MinBatchSize, tune chwriter 2026-07-02 15:23:17 +08:00
ryan 7b1fea8194 refactor(db): use version string as primary key for ConfigVersion
-transition `of_config_versions` primary key from `id` to `version` string.
-add database migration files `202606270001_make_version_primary_key.sql` for PostgreSQL and SQLite.
-introduce AfterFind/AfterCreate GORM hooks to preserve JSON backward compatibility.
-refactor API controllers, logics, and front-end typescript definitions to receive `string` parameter.
2026-06-27 13:46:55 +08:00
ryan ac7b776378 refactor(db): use version string as primary key for ConfigVersion
-transition `of_config_versions` primary key from `id` to `version` string.
-add database migration files `202606270001_make_version_primary_key.sql` for PostgreSQL and SQLite.
-introduce AfterFind/AfterCreate GORM hooks to preserve JSON backward compatibility.
-refactor API controllers, logics, and front-end typescript definitions to receive `string` parameter.
2026-06-27 13:41:50 +08:00
ryan 13d6966cb5 refactor(db): use version string as primary key for ConfigVersion
-transition `of_config_versions` primary key from `id` to `version` string.
-add database migration files `202606270001_make_version_primary_key.sql` for PostgreSQL and SQLite.
-introduce AfterFind/AfterCreate GORM hooks to preserve JSON backward compatibility.
-refactor API controllers, logics, and front-end typescript definitions to receive `string` parameter.
2026-06-27 13:23:30 +08:00
ryan b89dc9ec7e fix(waf): correct whitelist logic to bypass and add config/IP-group edit broadcasts
- Transition WAF whitelist filter from strict block-on-miss to bypass-on-hit logic

- Hook up broadcastIPGroupToAgents to CreateIPGroup and UpdateIPGroup WAF logics

- Hook up BroadcastActiveConfig to PublishConfigVersion and ActivateConfigVersion version logics

- Update WAF Lua tests in manager_test.go
2026-06-26 20:47:30 +08:00
ryan 49eae80c78 修复目录权限问题 2026-06-22 23:21:04 +08:00
ryan 8ed91dbf97 修复证书问题 2026-06-22 22:56:10 +08:00
ryan 92ceecc6ce 迁移配置表 2026-06-22 22:23:49 +08:00
ryan 1e5f35b9a3 fix(openflare): refresh WS read deadline on JSON pong; capture frpc stderr
- read_pump: 收到客户端 JSON {"type":"pong"} 时调用 conn.SetReadDeadline 刷新
  服务端读超时。修复前,服务端仅在 WebSocket 协议层 Pong 帧时刷新 deadline,
  而客户端使用 JSON 应用层 pong 回复,导致服务端 90s 后超时关闭连接,
  客户端收到 EOF 并触发无限重连循环。在 Cloudflare 代理场景下,
  100s 空闲超时进一步加剧了此问题。

- frpc/manager: 捕获 frpc 子进程 stderr 并在进程异常退出时
  将其内容记录到结构化日志 stderr 字段,便于诊断 exit status 1 的
  具体原因(如配置格式错误、Auth Token 失败、relay 服务端不可达等)。
2026-06-22 14:22:20 +08:00
ryan 346024f346 feat(relay): support configurable frps webui port and fix node detail integration
- Implement configurable FRPS WebUI switch and custom port setting (relay_frps_web_ui_port) in system configs.
- Integrate settings into Relay Node detail manage page instead of global settings.
- Dynamically query server version to select matching Docker image tag for Relay installation.
- Clean up legacy code and fix backend linter/test warnings.
2026-06-22 12:48:54 +08:00
ryan ffe98f6307 移除Notice 2026-06-22 12:00:43 +08:00
ryan 0a3cd250d1 fix relay 2026-06-22 11:56:31 +08:00
ryan 16b02fd3f1 修复 Agent 升级版本比对逻辑 2026-06-22 11:26:55 +08:00
ryan 3a2878d070 feat(api): integrate TLS certificate renewal into async task framework
Replace native goroutines in RenewCertificate logic with Asynq task dispatching to support queue execution, retry capability, and detailed task execution logs.
2026-06-22 10:54:15 +08:00
ryan 6ed3c0c81f 收敛 Pages 部署包读取路径 2026-06-21 12:04:54 +08:00
ryan 13a375e042 修复 Agent 部署 Pages 问题 2026-06-21 11:53:54 +08:00
ryan 0f904b4b6d 修复代理路由详情认证配置 Tab:移除 PoW 配置(PoW 仅在 WAF 规则组中设置);保留 Basic Auth 保存能力;移除页头重复的「保存当前分区」按钮。 2026-06-21 11:26:37 +08:00
ryan e479ae75e6 修复 Pages 路由发布失败并报 pages module is not available:配置快照发布流程补齐 Pages 项目激活部署解析与 pages_deployment 写入。 2026-06-21 11:12:37 +08:00
ryan ce2b931a78 修复 Pages 上传或节点同步时报 pages file size out of bounds:允许 ZIP 包内的 0 字节文件,并兼容未声明解压大小的 ZIP 条目。 2026-06-21 10:55:48 +08:00
ryan 6e86901a58 修复节点详情 OpenResty 连接数与吞吐显示为「—」:节点可观测 API 将 OpenResty 观测数据合并进 metric_snapshots;指标文案改为「请求/分钟」(近 60 秒窗口),连接数为 0 时正常显示 0。 2026-06-21 10:52:06 +08:00
ryan b12a9b0185 fix: 修复应用日志异常膨胀 2026-06-21 10:21:18 +08:00
ryan 117d473c27 fix: 修复 WAF 规则组保存/绑定网站时报 of_waf_rule_group_bindings_pkey 冲突 2026-06-20 22:05:30 +08:00
ryan b04a358e5e fix: 配置版本列表按 created_at 倒序展示 2026-06-20 21:32:18 +08:00
ryan 889e79c8b8 fix: 收敛子代理站点标识双轨逻辑 2026-06-20 21:03:13 +08:00
ryan 9bf7e3cd1b fix: 修复 WAF 规则组 PoW 策略发布后边缘不生效 2026-06-20 20:47:38 +08:00
ryan 8751c0dee3 fix(openflare): mmdb 国家名节点在世界地图使用正确质心
- 从 world-geo 生成国家质心表,Server 在仅有 ISO/国家名时补全 geo 坐标
- 全球态势板在缺少经纬度时按 geo_name 解析质心,避免 fallback 到美国
2026-06-20 19:46:33 +08:00
ryan 498a9ed3ff fix(openflare): Agent 上报 IP 后由 Server 自动解析节点地理位置
- 启动时按 of_options.GeoIPProvider 初始化 pkg/geoip(bootstrap + runtime)
- mmdb 模式从内置 GeoLite2 种子到 data/;保存归属方式后热刷新 Provider
- Agent/Relay 心跳在服务端根据 IP 写入 geo 字段,尊重 geo_manual_override
- ipinfo 归属名称改为 City, Region, Country 可读格式
2026-06-20 19:36:17 +08:00
ryan 6c46f5d24f fix(openflare): Pages 部署包经 upload 存储下载
Agent 下载 Pages 包时统一通过 upload_id 走文件存储 API;legacy
artifact_path 仅用于一次性回填 upload 并清空路径。部署视图暴露
upload_id,并补充回归测试与 changelog。
2026-06-20 19:21:12 +08:00
ryan 8c872f9b32 refactor(bootstrap): 引入解耦的全局生命周期管理器以隔离业务停机钩子
- 新建 internal/lifecycle 包,提供全局线程安全的 Shutdown 钩子注册与调度能力。
- 在 chwriter 与 risk_control 初始化阶段通过 OnShutdown 将其 Stop 函数注册到管理器中。
- bootstrap.Stop() 函数仅委托调用 lifecycle.Stop(),不再硬编码引入业务包,防止后续框架同步产生合并冲突。
2026-06-20 09:58:59 +08:00
ryan 2a0ebd16fa fix(backend): 修复优雅停机失效、系统设置并发读写冲突、文件服务API信封绕过以及测试uploads目录污染
- 修复 ClickHouse Batch Writer 与 RiskControl LogWriter 优雅停机,确保退出前队列数据正确刷盘。
- 修复 OpenFlare 系统配置参数全局变量并发读写的 Data Race 冲突,在读取配置时引入读锁保护。
- 修复 upload 模块的 API 错误响应格式,使用 response.Abort* 代替原始的 c.AbortWithStatus 与 c.JSON,保证全局信封格式统一。
- 修复 pkg/utils/network 的 isPrivateIPv4 以使用标准库 net.IP.IsPrivate() 校验,修复 format 的 Bytes2Size 边界,修改大小单位因子变量为只读常量。
- 重构 upload 文件服务与路由器测试,使用 t.TempDir() 代替硬编码的 uploads 相对路径写入和删除,解决测试目录文件污染问题。
2026-06-20 09:33:42 +08:00
ryan b3a55d4ab5 refactor(core): optimize performance, fix concurrency and clean up AGENTS.md design violations
- Concurrency: Added lock protection to WebSocket writes, fixed timer leaks, and prevented config cache listener context leaks.
- Performance: Added memory cache in ObservabilityBufferStore, periodic cleaning in CH Deduplicator, and buffered ZIP batch download writes.
- Design: Introduced Redis caching for OAuth session/tokens, sanitized raw DB error messages, segregated handlers and logics, and standard CAP response envelopes.
2026-06-20 09:09:02 +08:00
ryan b6c4181c70 fix(openflare): serialize access log snowflake IDs as strings
- Return AccessLogView.id as string to avoid JS Number precision loss
- Store OpenFlareAccessLog IDs as uint64 with json id,string
- Update frontend AccessLogItem.id type to string
2026-06-19 20:10:09 +08:00
ryan 3187934f72 fix(openflare): ClickHouse count scan and dashboard traffic metrics
- Scan ClickHouse count()/countIf() aggregates as uint64 before int64 conversion
- Fix access log list count, region stats, and delete pre-count queries
- Aggregate dashboard 24h traffic totals from hourly trend buckets
- Show current hour value in traffic trend chart summary
- Rename docker-compose service to openflare
- Remove redundant config preview section from performance page
2026-06-19 17:34:19 +08:00
ryan 9491b2a744 feat(clickhouse): wire batchwriter into business ingestion paths
Migrate risk_control audit logs to internal/db/batchwriter and add
openflare/chwriter with per-table async flush for observability
timeseries and node access logs.

Replace single-row ClickHouse inserts and pre-insert SELECT count()
dedup with repository BatchInsert* APIs, in-process TTL dedup for
metric/report snapshots, and bootstrap initialization on API startup.
2026-06-19 17:20:39 +08:00
ryan 578110f615 observation clickhouse 2026-06-19 17:20:39 +08:00
ryan 32861c5db9 fix lint 2026-06-19 17:20:39 +08:00
ryan 0b34792709 refactor(edge): Server 协议统一与 wsclient 收敛(Phase 3 Batch 3)
- openflare/agent/relay/flared 协议类型改为 pkg/protocol 别名,删除重复 struct
- 新增 edge/wsclient Preset 表,三组件 wsclient 改为薄包装
- 更新设计文档、计划与 changelog
2026-06-19 15:00:46 +08:00
ryan 63cd906cfc refactor(repo): consolidate openflare-server to root and move subprojects to internal/apps
- Merge all files inside openflare-server to the repository root directory.
- Relocate agent, relay, and flared subprojects from internal/ to internal/apps/.
- Combine docker-compose files and update build context paths to root.
- Update GitHub workflows and Dockerfiles to refer to new directories and package names.
- Rewrite Go package imports across all files.
- Resolve database renew test race condition and clean up docs.
2026-06-19 14:23:29 +08:00