feat(log): decouple log storage from ClickHouse with switchable logstore

- New internal/repository/logstore abstraction: exported domain interfaces
  (AccessLogStore/ObservabilityStore/UserAccessLogStore/StatusStore),
  config-driven provider (Active/Build/Migrating/SetConfigReader), GORM
  implementation for PostgreSQL/SQLite (incl. hourly rollups computed in
  real time, migration listers, PG partition maintenance), and a ClickHouse
  wrapper preserving the native batch path; repository facade delegates to
  logstore; import-lint test enforces apps never import analyticsrepo.
- ClickHouse is now optional: the log DB is either the main DB (postgres
  when database.enabled, else sqlite) or clickhouse; boot validation +
  first-run seed; log_database / log_db_migration are protected keys.
- New user task 切换日志数据库 (of_log_db_switch): freeze log writes,
  drain batch writers, copy all 6 raw log tables by id (preserving IDs)
  with target-partition pre-creation for PG, flip log_database on success,
  clear the freeze flag on failure.
- Per-store retention (log_retention_days_*) with expiry cleanup folded
  into the daily system_cleanup task; legacy database_auto_cleanup_* and
  of_database_auto_cleanup decommissioned.
- goose migrations: 6 log tables in PG (2 monthly-partitioned) + SQLite,
  retention config seeds, schedule cleanup; GET
  /api/v1/admin/status/log-database endpoint; frontend retention settings,
  switch-task UI and status badge; changelog and docs updated.

docs(plan): log database decoupling implementation plan

docs(design): log database decoupling design (ClickHouse optional)
This commit is contained in:
ryan
2026-08-08 11:36:56 +08:00
parent 734fe45baa
commit 7d71f1e4e1
95 changed files with 10569 additions and 3728 deletions
+14 -2
View File
@@ -197,8 +197,6 @@ Server 的所有核心基础配置定义在 `config.yaml` 中,且均支持环
| `node_offline_threshold` | `int` | 在管理后台中判定节点失去心跳并标注为离线状态的无响应阈值(毫秒) | `60000` (60s) |
| `agent_update_repo` | `string` | Agent 节点更新下载自身二进制的 Release 仓库源 | `Rain-kl/OpenFlare` |
| `geoip_provider` | `string` | GeoIP 提供商,支持 `maxmind` 等,用于 WAF 防护时地域分析 | `ipinfo` |
| `database_auto_cleanup_enabled` | `bool` | 是否在每天凌晨 3:00 自动清理过期观测历史日志(降低数据库空间) | `true` |
| `database_auto_cleanup_retention_days` | `int` | 自动清理观测数据(访问日志、度量曲线、审计等)的默认保留天数 | `30` |
### 5. Uptime Kuma 监控联动同步
| 配置键 (Key) | 数据类型 | 作用说明 | 默认值 |
@@ -272,6 +270,20 @@ Server 的所有核心基础配置定义在 `config.yaml` 中,且均支持环
---
### 8. 日志存储(Log Database)
日志存储解耦后的运行时配置:日志主库由「切换日志数据库」任务管理(内部/受保护 key,禁止管理员手动修改),保留天数按存储库分别在业务配置中设置。
| 配置键 (Key) | 数据类型 | 作用说明 | 默认值 |
| --- | --- | --- | --- |
| `log_database` | `string` | 当前日志主库(`postgres` / `sqlite` / `clickhouse`)。**内部受保护 key**:仅「切换日志数据库」迁移任务写入,管理员不可手动创建/修改 | 随主库(PostgreSQL 启用时为 `postgres`,否则 `sqlite`;ClickHouse 启用时优先 `clickhouse`) |
| `log_db_migration` | `string` | 日志迁移冻结标记(`migrating` 或空)。**内部受保护 key**:仅迁移任务写入,置位期间日志写入返回 503「日志数据库迁移中,暂不可写」 | 空 |
| `log_retention_days_postgres` | `int` | PostgreSQL 日志库的过期清理保留天数(过期日志由系统垃圾清理每日任务删除) | `90` |
| `log_retention_days_sqlite` | `int` | SQLite 日志库的过期清理保留天数 | `90` |
| `log_retention_days_clickhouse` | `int` | ClickHouse 日志库的过期清理保留天数 | `90` |
---
## 前端构建环境变量
| 环境变量 | 作用 | 默认值 |