Commit Graph

5 Commits

Author SHA1 Message Date
ryan a4dd5ca9e1 feat(dashboard): 首页请求趋势拆分状态码并合并容量到业务流量
- 24 小时请求趋势拆分展示请求总量与 200/400/500 状态码请求量,独占一行;
  时间桶聚合新增 status_200/400/500_count(CH countIf、PG FILTER),
  请求趋势改为基于原始桶聚合(小时 rollup 无状态码口径)
- 首页移除宿主机磁盘指标,容量趋势(CPU/内存)并入业务流量卡片展示
- 压缩协议 traffic_24h 扩展为 7 元组,前端归一化同步更新
2026-08-13 11:10:37 +08:00
ryan 75d1fcf345 feat(access-logs): 日志明细支持按状态码筛选并折叠次要搜索项,修复首页来源分布无数据
- 修复 PostgreSQL/SQLite 日志库下首页「来源分布」卡片无数据:RegionCounts 对空
  节点 ID 误拼 node_id = '' 恒空条件,改为空节点 ID 表示全节点聚合(对齐 CH 语义),
  并过滤空白归属地
- /access-logs?tab=list 新增状态码筛选:状态码下拉含常用 2xx/3xx/4xx/5xx 选项,
  校验 100-599,非法值返回 400;ClickHouse 与 PostgreSQL/SQLite 日志库均支持
- 搜索框折叠:默认仅显示来源 IP 与状态码,节点 ID/访问域名/请求路径折叠进
  「更多筛选」
2026-08-13 09:59:32 +08:00
ryan 3d4d99081e fix(log): PG 日志库批量写入为零 ID 行生成雪花 ID
PostgreSQL 日志表 id 为 NOT NULL 且无默认值,而 GORM 将零值 uint64
主键视为自增并省略 id 列,导致 node access log / 可观测指标等批量
落库持续报 "null value in column id violates not-null constraint"。
在 BatchInsert* 落库前为零 ID 行生成雪花 ID(与 ClickHouse 写入路径
一致),并新增单元回归与 PG 集成回归测试覆盖六张日志表。
2026-08-09 13:47:13 +08:00
ryan f530cd4025 perf(log): optimize PG log store queries and expired partition cleanup
- Count/节点访问日志统计改为单次扫描聚合,WAF 按 IP 聚合由三次扫描合并为两次
- IPSummaries 归属地改为取过滤窗口内最新记录(对齐 ClickHouse argMax 口径),
  子查询带窗口条件,可分区裁剪并命中索引
- 新增 goose 迁移:of_node_access_logs (logged_at DESC, id DESC) 前导索引与
  lower(trim(host)) 表达式索引,加速列表排序与主机过滤
- 过期日志清理先按数据校验直接 DROP 完全过期整月分区,再对边界月逐行删除;
  启动时兜底预建当月及未来 2 个月分区,跨月停机重启后首次写入不再报
  "no partition of relation found"
2026-08-09 10:20:58 +08:00
ryan 7d71f1e4e1 feat(log): decouple log storage from ClickHouse with switchable logstore
- New internal/repository/logstore abstraction: exported domain interfaces
  (AccessLogStore/ObservabilityStore/UserAccessLogStore/StatusStore),
  config-driven provider (Active/Build/Migrating/SetConfigReader), GORM
  implementation for PostgreSQL/SQLite (incl. hourly rollups computed in
  real time, migration listers, PG partition maintenance), and a ClickHouse
  wrapper preserving the native batch path; repository facade delegates to
  logstore; import-lint test enforces apps never import analyticsrepo.
- ClickHouse is now optional: the log DB is either the main DB (postgres
  when database.enabled, else sqlite) or clickhouse; boot validation +
  first-run seed; log_database / log_db_migration are protected keys.
- New user task 切换日志数据库 (of_log_db_switch): freeze log writes,
  drain batch writers, copy all 6 raw log tables by id (preserving IDs)
  with target-partition pre-creation for PG, flip log_database on success,
  clear the freeze flag on failure.
- Per-store retention (log_retention_days_*) with expiry cleanup folded
  into the daily system_cleanup task; legacy database_auto_cleanup_* and
  of_database_auto_cleanup decommissioned.
- goose migrations: 6 log tables in PG (2 monthly-partitioned) + SQLite,
  retention config seeds, schedule cleanup; GET
  /api/v1/admin/status/log-database endpoint; frontend retention settings,
  switch-task UI and status badge; changelog and docs updated.

docs(plan): log database decoupling implementation plan

docs(design): log database decoupling design (ClickHouse optional)
2026-08-08 19:43:01 +08:00