mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-09-30 14:26:36 +08:00
ryan
454542c1d0
- README 默认英文：README.en.md → README.md（英文为默认），中文移至 README.zh-CN.md，语言切换链接同步 - 恢复被删除的 docs/en/ 英文文档（git 历史 cc5e53c5^），删除 4 篇已废弃文件 - 英文导航 config.ts 对齐中文结构（新增 Deployment/Changelog 侧栏，同步 Guide/Design 条目） - 翻译 15 篇中文新增文档：guide 5 篇（certificates/pages-usage/proxy-config/uptime-kuma/zone-domain-migration）+ design 10 篇（zone-design/cloudflare-pointing/waf-orchestration/origin-error-page/edge-cache-design/pages-design/logstore/kuma-design/login-captcha/observability 三篇） - en 首页更新（新增 Pages 特性、tagline 同步）；changelog 英文入口指向中文版 - vitepress 构建验证：43 个英文页面全部渲染 注意：29 篇旧英文文档为恢复版，部分内容（如 deployment/server、reference/configuration）可能落后于中文，需后续逐篇同步
87 lines
5.0 KiB
Markdown
87 lines
5.0 KiB
Markdown
# Log Store Decoupling
|
|
|
|
You will learn: which tables are log-purpose, why they must not be pinned to ClickHouse, and which code path a new log table must follow.
|
|
|
|
Observability fields and the reporting protocol are still governed by [Observability Protocol & Tables](./observability-data-model.md); this document only defines **where data is stored and how to switch databases**.
|
|
|
|
---
|
|
|
|
## 1. Goals
|
|
|
|
* **ClickHouse optional**: when not enabled, PostgreSQL (or SQLite when the primary DB is off) fully takes over writes, queries, aggregation, and cleanup.
|
|
* **Upper layers don't touch the underlying DB**: apps only face `internal/repository/logstore` (or the `repository` facade). `repository/analytics` and `db.ChConn` / `db.ChDB` are only used by logstore's ClickHouse implementation.
|
|
* **Switchable**: 「Switch Log Database」in Task Management copies data between PostgreSQL/SQLite and ClickHouse and flips the primary; writes are frozen during migration, the switch only happens on success, and source data is not deleted.
|
|
|
|
---
|
|
|
|
## 2. What Counts as a Log Table
|
|
|
|
A table enters logstore only if it meets all of:
|
|
|
|
* Append-only writes, almost no row updates
|
|
* Query or aggregate by time, deletable by retention days
|
|
* Must still support writes and queries when ClickHouse is off
|
|
* Does not participate in transactional consistency for websites / nodes / certificates, etc.
|
|
|
|
**Don't** make these log tables: Zones, nodes, config versions, task executions, upload metadata. These go through the business primary DB `repository`.
|
|
|
|
Current log domains:
|
|
|
|
| Domain | Interface | Tables |
|
|
| --- | --- | --- |
|
|
| Node access logs | `AccessLogStore` | `of_node_access_logs` |
|
|
| Observability time series | `ObservabilityStore` | `of_node_metric_snapshots` / `of_node_edge_health` / `of_node_obs_frps` / `of_node_obs_frpc` |
|
|
| User access audit | `UserAccessLogStore` | `w_user_access_logs` |
|
|
|
|
Hourly materialized views on ClickHouse (e.g. `of_access_log_hourly`) only serve CH query acceleration. PostgreSQL / SQLite **do not** build isomorphic aggregation tables; queries aggregate in real time from raw logs.
|
|
|
|
---
|
|
|
|
## 3. Layering
|
|
|
|
| Layer | Path | Responsibility |
|
|
| --- | --- | --- |
|
|
| Abstraction | `internal/repository/logstore` | Interfaces + `Active` / `BuildForMigration`; selects implementation by `log_database` |
|
|
| CH implementation | `logstore/clickhouse_store.go` | Delegates to `repository/analytics` (native batch + existing aggregation SQL) |
|
|
| Primary DB implementation | `logstore/postgres_store.go` | PostgreSQL (high-frequency tables partitioned monthly) and SQLite (plain tables) share GORM |
|
|
| Model | `internal/model/analytics` | Entities and batch SQL, no IO |
|
|
| Enqueue | `chwriter` / `risk_control` + `batchwriter` | `FlushFunc` calls `logstore.Active`; node logs / observability enqueue via hooks |
|
|
| Constraint | `logstore/imports_test.go` | apps are forbidden from importing `repository/analytics` |
|
|
|
|
`log_database` has only two legal states: **follow the business primary DB** (`postgres` or `sqlite`) or **`clickhouse`**. "Primary PostgreSQL + log SQLite" does not exist. `log_database` / `log_db_migration` are protected and cannot be changed from the admin panel.
|
|
|
|
At startup: `log_database=clickhouse` but ClickHouse not enabled → startup is refused; you must re-enable ClickHouse, switch back to the primary DB, and only then turn it off.
|
|
|
|
---
|
|
|
|
## 4. Switch Protocol
|
|
|
|
Task type `of_log_db_switch` (admin name 「Switch Log Database」), parameter `target`.
|
|
|
|
1. Validate the target is legal and not the current DB.
|
|
2. Write `log_db_migration=migrating`, drain in-flight batchwriter (`Drain`, not `Stop` writer). Writes return a clear error afterward (HTTP 503), not queued backlog.
|
|
3. Clear the target log tables, then copy by id in pages; call `EnsurePartitions` on the PostgreSQL target before copying.
|
|
4. Only on full success write `log_database=target` and clear the migration marker; on failure clear the marker and writes continue on the source DB.
|
|
5. Source data is not deleted; re-clear the target before retry to guarantee idempotency.
|
|
|
|
Don't invent another switch protocol, and don't connect `analyticsrepo` directly inside tasks.
|
|
|
|
---
|
|
|
|
## 5. Adding a New Log Table
|
|
|
|
Column names must be identical across the three goose migrations (ClickHouse / PostgreSQL / SQLite). Key points:
|
|
|
|
* High-frequency tables: CH uses `MergeTree` + `toYYYYMM`; PG uses `PARTITION BY RANGE(time column)` with the partition key in the primary key; SQLite uses a plain table + indexes.
|
|
* IDs use snowflake `uint64`, preserved as-is during migration.
|
|
* Writes go through a dedicated `batchwriter`; flush calls `logstore.Active`, not `analyticsrepo.BatchInsert`.
|
|
* The switch task's `copy*` must cover the new table; cleanup uses existing `log_retention_days_*` or `metric_retention_days`, don't use the wrong TTL.
|
|
|
|
Runtime config: [Configuration Reference · Log Storage](../reference/configuration.md#8-日志存储log-database).
|
|
|
|
---
|
|
|
|
## 6. Related Docs
|
|
|
|
* Observability fields and reporting protocol: [Observability Protocol & Tables](./observability-data-model.md)
|