Commit Graph

25 Commits

Author SHA1 Message Date
ryan 50678756d4 feat(zone): add normalized zone domain schema 2026-07-12 14:23:25 +08:00
ryan da1dd92404 fix(observability): merge rollup+raw hourly trends and backfill history
Prefer capacity/openresty rollups only when they cover the 24h window;
otherwise merge per hour so raw fills pre-MV gaps and rollup wins on
overlap. Add a one-time ANTI JOIN backfill migration for the last 30 days.
2026-07-10 11:27:51 +08:00
ryan 160e63558f fix(clickhouse): harden R/W path P0–P3 (cleanup, durability, rollups)
Honest TTL cleanup semantics; enqueue-safe dedup with flush retry and writer
metrics; model insert hooks; latest-per-node and hourly metric/openresty
rollups; small-host pool/async defaults, traffic hourly TTL, and UV labeling.
2026-07-10 10:34:04 +08:00
ryan 9b3555c569 fix(clickhouse): cut idle CPU from tiny parts and oversized merge pools
Observability writers flushed every few seconds with MinBatchSize unset,
creating constant small parts and merge load. Enable MinBatchSize with
MaxFlushWait, batch access logs more aggressively, and shrink ClickHouse
background pools for 3c hosts.
2026-07-10 10:08:32 +08:00
ryan b312460ddf chore(release): v3.1.1
### ⚡️ 优化与改进
- 将 cap_login_enabled 默认值由 true 变更为 false,默认关闭登录界面 PoW 人机验证。
2026-07-06 12:29:53 +08:00
ryan 44bba0f19a fix(clickhouse): remove unsupported MODIFY ORDER BY from migration
ClickHouse keeps the implicit PRIMARY KEY when shortening ORDER BY,
which fails with "Primary key must be a prefix of the sorting key".
TTL-only changes are safe and unblock goose startup; narrowing ORDER BY
would require table recreation.
2026-07-02 16:45:48 +08:00
ryan f0eca028f9 fix(clickhouse): cast DateTime64 to DateTime in TTL migration 2026-07-02 16:21:30 +08:00
ryan 58624db397 perf(clickhouse): Phase 2 legacy governance — TTL cleanup, unified pool, MV, ops API
- Replace retention ALTER DELETE with MATERIALIZE TTL; use TRUNCATE for delete-all
- Remove GORM ClickHouse pool; migrate user access log reads to ChConn
- Drop query-side trim(remote_addr); enable wait_for_async_insert=1
- Add of_node_traffic_hourly MV and dashboard traffic trend fallback
- Add GET /admin/status/clickhouse operational metrics endpoint
2026-07-02 15:47:38 +08:00
ryan caf2ffcff4 perf(clickhouse): P1 TTL migrations, ORDER BY tune, remote_addr normalization 2026-07-02 15:25:03 +08:00
ryan 28c14eb054 perf(clickhouse): enable async_insert and tune connection/buffer defaults 2026-07-02 15:23:17 +08:00
ryan ae618905a3 perf(clickhouse): P0 write path — remove heartbeat DELETE, batchwriter MinBatchSize, tune chwriter 2026-07-02 15:23:17 +08:00
ryan 23be2f9296 fix(db): 新增 PostgreSQL 数据库自增序列全局同步迁移脚本
为了解决因历史数据以显式 ID 方式迁移导致 PostgreSQL 自增序列计数器不同步,产生主键冲突唯一性约束报错(如 WAF 规则组和 IP 组保存失败)的问题,在 PostgreSQL 迁移中加入了对所有相关表 pg_get_serial_sequence 重置的代码。同时,在 SQLite 中补齐了对应的同名迁移文件。
2026-06-30 16:13:37 +08:00
ryan 8d8814b416 refactor(oauth): replace legacy oauth cache with standard ram cache and add pubsub synchronization
- Replaced custom map-based cache in apps/oauth/cache.go with standard pkg/cache/ram framework.
- Implemented Redis Pub/Sub invalidation channels for distributed token and user cache synchronization.
- Created apps/oauth/cache_test.go to verify local cache operations and pub/sub broadcasts.

refactor(cache): generic RAM cache with CoW and unified preheating

Replaced L2 Redis cache and old cache package with process-local generic pkg/cache/ram. Implemented Copy-on-Write for reads, fine-grained locks per type for writes, and unified preheating in bootstrap. Changed cache invalidation to lazy-loading to resolve SQLite deadlocks during transactions.
2026-06-27 14:32:27 +08:00
ryan 02ebb81929 refactor(oauth): replace legacy oauth cache with standard ram cache and add pubsub synchronization
- Replaced custom map-based cache in apps/oauth/cache.go with standard pkg/cache/ram framework.
- Implemented Redis Pub/Sub invalidation channels for distributed token and user cache synchronization.
- Created apps/oauth/cache_test.go to verify local cache operations and pub/sub broadcasts.

refactor(cache): generic RAM cache with CoW and unified preheating

Replaced L2 Redis cache and old cache package with process-local generic pkg/cache/ram. Implemented Copy-on-Write for reads, fine-grained locks per type for writes, and unified preheating in bootstrap. Changed cache invalidation to lazy-loading to resolve SQLite deadlocks during transactions.
2026-06-27 14:26:13 +08:00
ryan 13d6966cb5 refactor(db): use version string as primary key for ConfigVersion
-transition `of_config_versions` primary key from `id` to `version` string.
-add database migration files `202606270001_make_version_primary_key.sql` for PostgreSQL and SQLite.
-introduce AfterFind/AfterCreate GORM hooks to preserve JSON backward compatibility.
-refactor API controllers, logics, and front-end typescript definitions to receive `string` parameter.
2026-06-27 13:23:30 +08:00
ryan 92ceecc6ce 迁移配置表 2026-06-22 22:23:49 +08:00
ryan 346024f346 feat(relay): support configurable frps webui port and fix node detail integration
- Implement configurable FRPS WebUI switch and custom port setting (relay_frps_web_ui_port) in system configs.
- Integrate settings into Relay Node detail manage page instead of global settings.
- Dynamically query server version to select matching Docker image tag for Relay installation.
- Clean up legacy code and fix backend linter/test warnings.
2026-06-22 12:48:54 +08:00
ryan ffe98f6307 移除Notice 2026-06-22 12:00:43 +08:00
ryan 0a3cd250d1 fix relay 2026-06-22 11:56:31 +08:00
ryan 117d473c27 fix: 修复 WAF 规则组保存/绑定网站时报 of_waf_rule_group_bindings_pkey 冲突 2026-06-20 22:05:30 +08:00
ryan 99f6f3231a fix: 修复 WAF 规则组保存/绑定网站时报 of_waf_rule_group_bindings_pkey 冲突 2026-06-20 21:33:46 +08:00
ryan 5bed2bae9f migrate database 2026-06-19 22:18:52 +08:00
ryan ca6c20ebd9 feat(db): add ClickHouse batchwriter framework and skill
Introduce internal/db/batchwriter as a reusable generic buffered writer
for per-domain ClickHouse flush pipelines, with unit tests and default
batch tuning aligned with audit log ingestion.

Add clickhouse-batchwriter agent skill and cross-references in AGENTS.md
and database-migration. Business layers are not wired yet.
2026-06-19 17:20:39 +08:00
ryan 578110f615 observation clickhouse 2026-06-19 17:20:39 +08:00
ryan 63cd906cfc refactor(repo): consolidate openflare-server to root and move subprojects to internal/apps
- Merge all files inside openflare-server to the repository root directory.
- Relocate agent, relay, and flared subprojects from internal/ to internal/apps/.
- Combine docker-compose files and update build context paths to root.
- Update GitHub workflows and Dockerfiles to refer to new directories and package names.
- Rewrite Go package imports across all files.
- Resolve database renew test race condition and clean up docs.
2026-06-19 14:23:29 +08:00