Commit Graph

1561 Commits

Author SHA1 Message Date
ryan 99f6f3231a fix: 修复 WAF 规则组保存/绑定网站时报 of_waf_rule_group_bindings_pkey 冲突 2026-06-20 21:33:46 +08:00
ryan b04a358e5e fix: 配置版本列表按 created_at 倒序展示 2026-06-20 21:32:18 +08:00
ryan 6fc39d9e80 fix: 修复 WAF 规则组 PoW 策略发布后边缘不生效 2026-06-20 21:05:25 +08:00
ryan 889e79c8b8 fix: 收敛子代理站点标识双轨逻辑 2026-06-20 21:03:13 +08:00
ryan 9bf7e3cd1b fix: 修复 WAF 规则组 PoW 策略发布后边缘不生效 2026-06-20 20:47:38 +08:00
ryan 8751c0dee3 fix(openflare): mmdb 国家名节点在世界地图使用正确质心
- 从 world-geo 生成国家质心表,Server 在仅有 ISO/国家名时补全 geo 坐标
- 全球态势板在缺少经纬度时按 geo_name 解析质心,避免 fallback 到美国
2026-06-20 19:46:33 +08:00
ryan 498a9ed3ff fix(openflare): Agent 上报 IP 后由 Server 自动解析节点地理位置
- 启动时按 of_options.GeoIPProvider 初始化 pkg/geoip(bootstrap + runtime)
- mmdb 模式从内置 GeoLite2 种子到 data/;保存归属方式后热刷新 Provider
- Agent/Relay 心跳在服务端根据 IP 写入 geo 字段,尊重 geo_manual_override
- ipinfo 归属名称改为 City, Region, Country 可读格式
2026-06-20 19:36:17 +08:00
ryan ec53629971 fix(frontend): 修复配置版本快照侧栏无法滚动
将快照与发布预览 Sheet 内容区改为 flex-1 min-h-0 overflow-y-auto,
并固定侧栏高度为 h-svh,与项目内其他可滚动 Sheet 一致。
2026-06-20 19:22:23 +08:00
ryan 6c46f5d24f fix(openflare): Pages 部署包经 upload 存储下载
Agent 下载 Pages 包时统一通过 upload_id 走文件存储 API;legacy
artifact_path 仅用于一次性回填 upload 并清空路径。部署视图暴露
upload_id,并补充回归测试与 changelog。
2026-06-20 19:21:12 +08:00
ryan e077b12328 fix(frontend): cap envelope mismatch 2026-06-20 19:04:55 +08:00
ryan 570b639e07 fix(agent): commit GeoLite2 mmdb as build fallback
Vendor GeoLite2-Country.mmdb in the repository so agent builds still
work when the remote download is unavailable. Update the fetch script
and agent Dockerfile to prefer a fresh download and fall back to the
committed database file.
2026-06-20 14:04:42 +08:00
ryan 3a368119e5 fix(ci): fetch GeoLite2 mmdb before agent build
Agent embeds GeoLite2-Country.mmdb but the file is not checked into the
repository. Download it in CI, Docker, and Makefile build paths via
scripts/fetch-agent-geoip-mmdb.sh, and correct the gitignore exception
path for the geoipdata package.
2026-06-20 14:03:43 +08:00
ryan 6e6bce6a03 Optimize CI 2026-06-20 13:53:54 +08:00
ryan 6975a6c290 sync ci 2026-06-20 13:44:24 +08:00
ryan ef5d5b46af chore(release): bump version to v1.3.1
### 🛠 修复
- 修复了风控中间件测试在 ClickHouse 批写架构迁移后无法正确初始化的问题。
- 修复了 OpenTelemetry trace provider 初始化时 semconv Schema URL 版本冲突导致进程无法启动的问题。

### ⚡️ 优化与改进
- 新增 ClickHouse 独立 OLAP 管线,以 goose 迁移作为唯一 schema 来源,并抽取 analytics repository 统一访问日志读写。
- 新增通用 ClickHouse batchwriter 批量写入框架,支持各业务域独立缓冲 flush 管道与默认批处理调优。
- 风控访问日志与管理端日志查询改为经 repository 层批写与查询,移除内联 SQL 与手动 DDL 维护路径。
- OAuth Access Token 与会话校验引入 RAM+Redis 缓存,降低高频鉴权路径的数据库读取压力。
- 上传元数据、Auth Source 与系统配置批量读取接入三层缓存(RAM→Redis→DB),并通过 pub/sub 支持多节点失效同步。
- 上传统计增量更新收敛为单事务写入,减少 ingest/remove 路径的锁竞争与统计偏差风险。
- ClickHouse 迁移与连接初始化增加进程隔离,避免与主库迁移互相阻塞。
- 引入 lifecycle 优雅停机钩子,确保进程退出前 flush 批写缓冲与释放资源。

### 💄 其他/体验
- 新增 cache-framework 与 clickhouse-batchwriter 开发技能,并更新 AGENTS.md Skill 关联索引。
- 登录与 Token 校验路径增加缓存预热,缩短冷启动后首次鉴权延迟。
2026-06-20 11:20:11 +08:00
ryan a1f6459c09 fix(trace): 使用 NewSchemaless 避免 semconv schema 版本冲突
业务 Resource 改为 NewSchemaless 合并,继承 resource.Default() 的 SDK 内置
schema URL,不再硬编码 semconv 版本路径。
2026-06-20 11:20:01 +08:00
ryan 280bb63cbd chore(release): bump version to v1.3.1
### 🛠 修复
- 修复了风控中间件测试在 ClickHouse 批写架构迁移后无法正确初始化的问题。

### ⚡️ 优化与改进
- 新增 ClickHouse 独立 OLAP 管线,以 goose 迁移作为唯一 schema 来源,并抽取 analytics repository 统一访问日志读写。
- 新增通用 ClickHouse batchwriter 批量写入框架,支持各业务域独立缓冲 flush 管道与默认批处理调优。
- 风控访问日志与管理端日志查询改为经 repository 层批写与查询,移除内联 SQL 与手动 DDL 维护路径。
- OAuth Access Token 与会话校验引入 RAM+Redis 缓存,降低高频鉴权路径的数据库读取压力。
- 上传元数据、Auth Source 与系统配置批量读取接入三层缓存(RAM→Redis→DB),并通过 pub/sub 支持多节点失效同步。
- 上传统计增量更新收敛为单事务写入,减少 ingest/remove 路径的锁竞争与统计偏差风险。
- ClickHouse 迁移与连接初始化增加进程隔离,避免与主库迁移互相阻塞。
- 引入 lifecycle 优雅停机钩子,确保进程退出前 flush 批写缓冲与释放资源。

### 💄 其他/体验
- 新增 cache-framework 与 clickhouse-batchwriter 开发技能,并更新 AGENTS.md Skill 关联索引。
- 登录与 Token 校验路径增加缓存预热,缩短冷启动后首次鉴权延迟。
2026-06-20 10:51:27 +08:00
ryan f52c8db21a perf(cache): 三层缓存框架补强
- 新增 cache-framework skill,规范 RAM→Redis→DB 读路径、失效与 pub/sub
- 上传元数据 Otter+Redis 缓存与多节点失效;Auth Source 缓存与 pub/sub
- ListSystemConfigsByKeys 补 Redis 层;上传统计单事务;登录/Token 缓存预热
- cleanup 任务补 upload meta 失效钩子
2026-06-20 10:20:35 +08:00
ryan cdac1f8a45 perf(cache): 三层缓存框架补强
- 新增 cache-framework skill,规范 RAM→Redis→DB 读路径、失效与 pub/sub
- 上传元数据 Otter+Redis 缓存与多节点失效;Auth Source 缓存与 pub/sub
- ListSystemConfigsByKeys 补 Redis 层;上传统计单事务;登录/Token 缓存预热
- cleanup 任务补 upload meta 失效钩子
2026-06-20 10:20:23 +08:00
ryan 8c872f9b32 refactor(bootstrap): 引入解耦的全局生命周期管理器以隔离业务停机钩子
- 新建 internal/lifecycle 包,提供全局线程安全的 Shutdown 钩子注册与调度能力。
- 在 chwriter 与 risk_control 初始化阶段通过 OnShutdown 将其 Stop 函数注册到管理器中。
- bootstrap.Stop() 函数仅委托调用 lifecycle.Stop(),不再硬编码引入业务包,防止后续框架同步产生合并冲突。
2026-06-20 09:58:59 +08:00
ryan d490030b75 fix: test 2026-06-20 09:58:37 +08:00
ryan 200525a1ab perf: refactor 2026-06-20 09:54:48 +08:00
ryan cac6e88bc0 perf: refactor 2026-06-20 09:46:29 +08:00
ryan 080be1e03a perf: access token cache 2026-06-20 09:46:29 +08:00
ryan 2a0ebd16fa fix(backend): 修复优雅停机失效、系统设置并发读写冲突、文件服务API信封绕过以及测试uploads目录污染
- 修复 ClickHouse Batch Writer 与 RiskControl LogWriter 优雅停机,确保退出前队列数据正确刷盘。
- 修复 OpenFlare 系统配置参数全局变量并发读写的 Data Race 冲突,在读取配置时引入读锁保护。
- 修复 upload 模块的 API 错误响应格式,使用 response.Abort* 代替原始的 c.AbortWithStatus 与 c.JSON,保证全局信封格式统一。
- 修复 pkg/utils/network 的 isPrivateIPv4 以使用标准库 net.IP.IsPrivate() 校验,修复 format 的 Bytes2Size 边界,修改大小单位因子变量为只读常量。
- 重构 upload 文件服务与路由器测试,使用 t.TempDir() 代替硬编码的 uploads 相对路径写入和删除,解决测试目录文件污染问题。
2026-06-20 09:33:42 +08:00
ryan b3a55d4ab5 refactor(core): optimize performance, fix concurrency and clean up AGENTS.md design violations
- Concurrency: Added lock protection to WebSocket writes, fixed timer leaks, and prevented config cache listener context leaks.
- Performance: Added memory cache in ObservabilityBufferStore, periodic cleaning in CH Deduplicator, and buffered ZIP batch download writes.
- Design: Introduced Redis caching for OAuth session/tokens, sanitized raw DB error messages, segregated handlers and logics, and standard CAP response envelopes.
2026-06-20 09:09:02 +08:00
ryan 5bed2bae9f migrate database 2026-06-19 22:18:52 +08:00
ryan d0a9958711 perf: clickhouse isolation 2026-06-19 21:31:03 +08:00
ryan a98d266278 feat(db): add ClickHouse batchwriter framework and skill
Introduce internal/db/batchwriter as a reusable generic buffered writer
for per-domain ClickHouse flush pipelines, with unit tests and default
batch tuning aligned with audit log ingestion.

Add clickhouse-batchwriter agent skill and cross-references in AGENTS.md
and database-migration. Business layers are not wired yet.
2026-06-19 20:59:07 +08:00
ryan b6c4181c70 fix(openflare): serialize access log snowflake IDs as strings
- Return AccessLogView.id as string to avoid JS Number precision loss
- Store OpenFlareAccessLog IDs as uint64 with json id,string
- Update frontend AccessLogItem.id type to string
2026-06-19 20:10:09 +08:00
ryan 3187934f72 fix(openflare): ClickHouse count scan and dashboard traffic metrics
- Scan ClickHouse count()/countIf() aggregates as uint64 before int64 conversion
- Fix access log list count, region stats, and delete pre-count queries
- Aggregate dashboard 24h traffic totals from hourly trend buckets
- Show current hour value in traffic trend chart summary
- Rename docker-compose service to openflare
- Remove redundant config preview section from performance page
2026-06-19 17:34:19 +08:00
ryan 9491b2a744 feat(clickhouse): wire batchwriter into business ingestion paths
Migrate risk_control audit logs to internal/db/batchwriter and add
openflare/chwriter with per-table async flush for observability
timeseries and node access logs.

Replace single-row ClickHouse inserts and pre-insert SELECT count()
dedup with repository BatchInsert* APIs, in-process TTL dedup for
metric/report snapshots, and bootstrap initialization on API startup.
2026-06-19 17:20:39 +08:00
ryan ca6c20ebd9 feat(db): add ClickHouse batchwriter framework and skill
Introduce internal/db/batchwriter as a reusable generic buffered writer
for per-domain ClickHouse flush pipelines, with unit tests and default
batch tuning aligned with audit log ingestion.

Add clickhouse-batchwriter agent skill and cross-references in AGENTS.md
and database-migration. Business layers are not wired yet.
2026-06-19 17:20:39 +08:00
ryan 578110f615 observation clickhouse 2026-06-19 17:20:39 +08:00
ryan 32861c5db9 fix lint 2026-06-19 17:20:39 +08:00
ryan 0b34792709 refactor(edge): Server 协议统一与 wsclient 收敛(Phase 3 Batch 3)
- openflare/agent/relay/flared 协议类型改为 pkg/protocol 别名,删除重复 struct
- 新增 edge/wsclient Preset 表,三组件 wsclient 改为薄包装
- 更新设计文档、计划与 changelog
2026-06-19 15:00:46 +08:00
ryan db9a9f98fd refactor(edge): 抽取边缘运行时共享包并完成 Phase 3 重构
- 新增 internal/apps/edge/,三组件改为薄包装,删除 3000+ 行重复代码
- Agent 心跳周期下沉至 heartbeat/cycle.go
- 协议类型迁入 pkg/protocol/agent.go
- 补充设计文档与 changelog
2026-06-19 14:56:00 +08:00
ryan cc5e53c51e 文档更新 2026-06-19 14:45:17 +08:00
ryan 9eeeb09d2f feat(repo): rename output server binary from wavelet to openflare-server 2026-06-19 14:36:46 +08:00
ryan 84303d25b2 feat(repo): add build targets for agent, relay, and flared to Makefile 2026-06-19 14:33:32 +08:00
ryan 63cd906cfc refactor(repo): consolidate openflare-server to root and move subprojects to internal/apps
- Merge all files inside openflare-server to the repository root directory.
- Relocate agent, relay, and flared subprojects from internal/ to internal/apps/.
- Combine docker-compose files and update build context paths to root.
- Update GitHub workflows and Dockerfiles to refer to new directories and package names.
- Rewrite Go package imports across all files.
- Resolve database renew test race condition and clean up docs.
2026-06-19 14:23:29 +08:00
ryan 19d476ed7f refactor 2026-06-19 14:10:21 +08:00
ryan 8506a03f1c refactor(clickhouse): route of_node_access_logs through analytics repository
Move all ClickHouse DML for OpenFlare node access logs into
internal/repository/analytics; model layer keeps domain aggregation and
in-memory test store via a thin adapter. Aligns with goose-managed DDL
and openflare database startup dependency.
2026-06-19 12:13:18 +08:00
ryan 8a00f53b16 feat(clickhouse): integrate goose migrations and analytics repository
Merge upstream Wavelet patch to manage ClickHouse DDL via goose
(goose_clickhouse_version), add GORM ChDB access, and centralize
w_user_access_logs reads/writes in internal/repository/analytics.
OpenFlare of_node_access_logs DDL moves to goose/clickhouse; remove
manual clickhouse_schema startup and support-files SQL duplicates.
2026-06-19 12:11:41 +08:00
ryan 00593742c3 feat(clickhouse): integrate goose migrations and analytics repository
Add a separate ClickHouse OLAP pipeline with goose/clickhouse DDL as the
sole schema source, model/analytics for ORM mapping, and repository/analytics
for reads (ChDB/GORM) and batch writes (ChConn). Refactor risk_control and
admin/logs to use the repository layer instead of inline SQL. Remove the
manual support-files DDL and document the workflow in database-migration skill.
2026-06-19 12:04:22 +08:00
ryan fff26aa343 merge: migrate of_node_access_logs to ClickHouse
Integrate clickhouse worktree: node access logs now stored in ClickHouse
(openflare database) with mandatory startup dependency.
2026-06-19 11:45:51 +08:00
ryan 425ca89765 feat(openflare): migrate of_node_access_logs to ClickHouse
Move node access log storage from PostgreSQL/SQLite to ClickHouse
(database: openflare). System startup now requires a healthy ClickHouse
connection and auto-initializes the schema. Agent heartbeat writes access
logs via batch insert; goose migration drops the legacy relational table.
2026-06-19 11:45:48 +08:00
ryan 7eb943f02f update doc 2026-06-19 11:45:22 +08:00
ryan d78449cbc9 refactor(repo): replace legacy openflare-server with Wavelet rename
Delete the old monolithic openflare-server implementation and rename
Wavelet/ to openflare-server/ to complete the migration consolidation.
Update CI workflows, agent Dockerfiles, and deployment docs for the new
layout (frontend/, docker/Dockerfile).
2026-06-19 11:37:34 +08:00
ryan aa11c0f52a merge: replace legacy openflare-server with Wavelet rename 2026-06-19 11:30:05 +08:00