tcjlj/OpenFlare
1
0
Fork 0
mirror of https://github.com/Rain-kl/OpenFlare.git synced 2026-09-29 05:56:38 +08:00
Code Issues Packages Projects Releases Wiki Activity
952 Commits 3 Branches 61 Tags
3d4d99081e1458d2bc2b67c5e82bd762e4881ec6
Commit Graph

187 Commits

Author SHA1 Message Date
ryan 3d4d99081e fix(log): PG 日志库批量写入为零 ID 行生成雪花 ID 
PostgreSQL 日志表 id 为 NOT NULL 且无默认值，而 GORM 将零值 uint64
主键视为自增并省略 id 列，导致 node access log / 可观测指标等批量
落库持续报 "null value in column id violates not-null constraint"。
在 BatchInsert* 落库前为零 ID 行生成雪花 ID（与 ClickHouse 写入路径
一致），并新增单元回归与 PG 集成回归测试覆盖六张日志表。
 2026-08-09 13:47:13 +08:00
ryan 1d0f2d6342 fix(log): hard-set log retention days default to 30, drop legacy inheritance 
log_retention_days_* 迁移不再继承旧键 database_auto_cleanup_retention_days
的值，统一默认 30 天。此前若旧键残留异常小值（如 2 天）会被静默带入，
导致升级后首次垃圾清理把大部分日志直接删掉。PG/SQLite 双方言同步修改，
文档默认值 90 -> 30。
 2026-08-09 10:48:45 +08:00
ryan e3f603f72a fix(security): stop logging UptimeKuma socket payload content 2026-08-09 10:42:21 +08:00
ryan 3b010bb15e feat(log): disable user access log recording 
- 移除全局用户访问日志采集中间件与批写入 writer（risk_control 包整包删除），
  不再写入 w_user_access_logs；存量数据与管理端访问日志统计页面保留
- 日志库迁移任务不再排空用户访问日志队列，状态接口不再展示其缓冲队列统计
- 迁移测试的系统配置 seed 计数断言更新为当前实际值（86 → 95），
  注释改为提示新增配置 seed 时同步更新
 2026-08-09 10:35:39 +08:00
ryan f530cd4025 perf(log): optimize PG log store queries and expired partition cleanup 
- Count/节点访问日志统计改为单次扫描聚合，WAF 按 IP 聚合由三次扫描合并为两次
- IPSummaries 归属地改为取过滤窗口内最新记录（对齐 ClickHouse argMax 口径），
  子查询带窗口条件，可分区裁剪并命中索引
- 新增 goose 迁移：of_node_access_logs (logged_at DESC, id DESC) 前导索引与
  lower(trim(host)) 表达式索引，加速列表排序与主机过滤
- 过期日志清理先按数据校验直接 DROP 完全过期整月分区，再对边界月逐行删除；
  启动时兜底预建当月及未来 2 个月分区，跨月停机重启后首次写入不再报
  "no partition of relation found"
 2026-08-09 10:20:58 +08:00
ryan 08d28c2c8e feat(log): drop empty old-month PG partitions during cleanup 
系统垃圾清理任务删除过期日志后，顺带清理旧月份空分区表：
PostgreSQL 按月分区的访问日志表（节点/用户）在数据删除后若该月
分区已无数据，则自动删除对应分区表，避免历史分区表无限累积。

- 仅删除「当前月之前」且为空的月份分区，当月/未来月及仍有数据的分区保留
- ClickHouse/SQLite 为 no-op（CH 分区随数据删除自动消失）
- 修复既有集成测试 pg_inherits 查询（inhrelid → inhparent）
- 新增单元测试与 PG 集成测试
 2026-08-09 09:33:59 +08:00
ryan 34a0896ff8 chore(task): run system garbage cleanup once daily 
系统定期垃圾清理 cron 由每 2 小时（0 */2 * * *）改为每日凌晨 3 点
（0 3 * * *，Asia/Shanghai），降低非必要高频扫描。新增 PG/SQLite
双方言 goose 迁移（含 Down 回滚）与迁移测试。
 2026-08-09 09:25:30 +08:00
ryan 0c22e76f4b fix(frontend): optimization 2026-08-09 09:14:54 +08:00
ryan bd2183c8bb feat(log): add independent short retention for performance metrics 
性能指标（CPU/内存/磁盘/网络）不再跟随 log_retention_days_*，新增三库共用
的 metric_retention_days 配置（默认 3 天），系统垃圾清理按独立短留存清理
指标快照；访问日志保留时长不变。新增 PG/SQLite 双方言 goose 迁移 seed。
 2026-08-09 09:04:33 +08:00
ryan 9df2437e47 fix(config): set ClickHouse to disabled by default and update related documentation 2026-08-09 08:54:26 +08:00
ryan e8c414aa12 fix: ch migrate 2026-08-09 08:47:56 +08:00
ryan 7e8aa5fa0f Merge branch 'codex/log-database-decoupling' 
# Conflicts:
#	docs/changelog/index.md
#	frontend/app/(main)/error-pages/page.tsx
#	internal/infra/persistence/migrator/migrator_test.go
 2026-08-09 08:37:37 +08:00
ryan 6487ce666d fix(security): harden PoW XSS, email header injection and UptimeKuma log redaction 2026-08-08 21:52:46 +08:00
ryan 9797fcdb2f fix(openflare): improve SWOfflineDomains validation and snapshot diff logic 2026-08-08 20:52:07 +08:00
Ryan 93ec3096f3 Potential fix for pull request finding 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-08-08 20:48:54 +08:00
Ryan 0e34301c92 Potential fix for pull request finding 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-08-08 20:48:37 +08:00
Ryan 12b5271f92 Potential fix for pull request finding 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-08-08 20:47:33 +08:00
Ryan 8ee966434d Potential fix for pull request finding 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-08-08 20:46:54 +08:00
ryan 7d93d3d2a1 fix(log): address remaining CodeRabbit suggestions for log database switch and migrations 2026-08-08 20:36:04 +08:00
Ryan 074edf17a1 Potential fix for pull request finding 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-08-08 20:30:36 +08:00
Ryan 6faf525af0 Potential fix for pull request finding 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-08-08 20:30:17 +08:00
ryan a6fc2b7737 fix(log): address CodeRabbit review findings for log database decoupling 2026-08-08 20:15:36 +08:00
ryan ca21ff3a5b feat(option): add sw offline 
fix(openresty): scope sw injection per cert partition

fix(lint): satisfy revive and biome format for sw offline feature

docs: sw offline scope changelog

fix(frontend): use scoped query key for sw scope zones

fix(frontend): hide preview link in sw contact page editor

feat(frontend): add sw scope domain picker and contact page fields

refactor(frontend): generalize html editor workspace for reuse

feat(openresty): scope sw offline injection by route domains

feat(openresty): add sw offline domains snapshot field

feat(option): add sw offline domains scope option

docs: fill html editor workspace generalization detail

docs: sw offline scope implementation plan

docs: sw offline scope design

test(openresty): assert single merged access block in sw enabled servers

fix(openresty): restrict sw intercept to https server blocks

fix(openresty): version sw offline cache by html content

fix(agent): escape redir in sw challenge page to prevent xss

fix(agent): return sw.runtime module table and add lua spec

docs: sw offline fallback changelog

fix(frontend): memoize option map to preserve unsaved contact page edits

feat(frontend): add response pages module with contact page tab

feat(agent): ship sw offline lua assets and placeholder substitution

feat(config): wire sw offline options into config snapshot

feat(openresty): render sw offline assets and challenge intercept

feat(openresty): add sw offline ConfigSnapshot fields and placeholder

feat(db): seed sw offline options

feat(option): add sw offline config keys and validation

docs: add service worker offline fallback implementation plan

docs: adopt global-option pattern for SW offline fallback (matches origin error page)

docs: unify offline contact page with error pages as response pages

docs: service worker offline fallback design (issue #23)
 2026-08-08 20:14:28 +08:00
ryan 7d71f1e4e1 feat(log): decouple log storage from ClickHouse with switchable logstore 
- New internal/repository/logstore abstraction: exported domain interfaces
  (AccessLogStore/ObservabilityStore/UserAccessLogStore/StatusStore),
  config-driven provider (Active/Build/Migrating/SetConfigReader), GORM
  implementation for PostgreSQL/SQLite (incl. hourly rollups computed in
  real time, migration listers, PG partition maintenance), and a ClickHouse
  wrapper preserving the native batch path; repository facade delegates to
  logstore; import-lint test enforces apps never import analyticsrepo.
- ClickHouse is now optional: the log DB is either the main DB (postgres
  when database.enabled, else sqlite) or clickhouse; boot validation +
  first-run seed; log_database / log_db_migration are protected keys.
- New user task 切换日志数据库 (of_log_db_switch): freeze log writes,
  drain batch writers, copy all 6 raw log tables by id (preserving IDs)
  with target-partition pre-creation for PG, flip log_database on success,
  clear the freeze flag on failure.
- Per-store retention (log_retention_days_*) with expiry cleanup folded
  into the daily system_cleanup task; legacy database_auto_cleanup_* and
  of_database_auto_cleanup decommissioned.
- goose migrations: 6 log tables in PG (2 monthly-partitioned) + SQLite,
  retention config seeds, schedule cleanup; GET
  /api/v1/admin/status/log-database endpoint; frontend retention settings,
  switch-task UI and status badge; changelog and docs updated.

docs(plan): log database decoupling implementation plan

docs(design): log database decoupling design (ClickHouse optional)
 2026-08-08 19:43:01 +08:00
ryan 6738abdec1 feat(openresty): add origin error page GET-only option 
Allow restricting custom origin error HTML to GET requests so other
methods pass through origin responses. Adds option seed, snapshot field,
edge limit_except/Lua handling, and admin UI switch.
 2026-08-06 20:22:44 +08:00
ryan d17d8457f3 chore: upgrade dependence 2026-08-06 17:41:06 +08:00
ryan 3328d3d121 refactor(agent): stop embedding GeoIP MMDB in agent binary 
Agent ships without City/Country MMDB in the binary; Docker images COPY
databases into data_dir, bare installs seed via download on first start.
Server keeps Country-only embed for optional MaxMind control-plane use.
Also harden fetch script nonempty check and reject non-file MMDB paths.
 2026-08-06 16:24:57 +08:00
ryan 076bf8b95c Merge branch 'feat/origin-error-page' 2026-08-06 14:16:28 +08:00
ryan 7d47db1f34 feat(option): seed and validate origin error page options 2026-08-06 13:59:54 +08:00
ryan 68d8f786cc feat(openresty): render origin error page directives 
Wire origin error page into OpenResty proxy route rendering: ConfigSnapshot
fields, default HTML template SupportFile, proxy_intercept_errors + error_page
with status-preserving internal Lua location, and Agent placeholder substitution
for __OPENFLARE_ERROR_PAGE_TMPL__. Pages routes are excluded.
 2026-08-06 13:52:40 +08:00
ryan 1a4a03a20d fix(agent): clear sticky LastError on successful sync paths 
Pages reconcile could succeed while agent state retained a previous
network error, so health events stayed active indefinitely. Clear
LastError whenever sync completes successfully without re-applying
config, and cover the paths with regression tests.
 2026-08-06 13:47:48 +08:00
ryan 4eced2b721 feat(cloudflare): add DNS pointing integration 2026-08-04 13:30:40 +08:00
ryan ea7658815a fix(migration): quote reserved authorization column 2026-08-04 12:49:58 +08:00
ryan 3edcdb9e9f feat(cloudflare): add DNS pointing integration 
Implement Cloudflare connection management, pointing groups and members, asynchronous A-record reconciliation, node IP triggers, admin APIs, management pages, migrations, tests, and documentation.
 2026-08-04 12:32:37 +08:00
ryan 943818f7d4 refactor(repository): 收敛 model/repository 分层为唯一持久化入口 
将 OpenFlare 与平台业务的数据访问从 model 与 apps 直连迁入 repository，
model 仅保留实体与无 IO 规则；补充 code-check 架构守卫与开发规范。
 2026-07-24 17:00:17 +08:00
ryan 23a5488203 refactor(http): remove dead internal/util HTTP client wrapper 
Drop internal/util (unused httppool wrapper and dead StringArray) and
rely on pkg/httppool plus oauth context injection for HTTP clients.
 2026-07-24 15:49:52 +08:00
ryan d99c5b7c43 refactor(pkg): merge pkg/utils into pkg/util 
Consolidate pure helper packages under pkg/util and update imports.
 2026-07-24 15:45:10 +08:00
ryan 33a1c32cf8 refactor(structure): group platform, infra, and shared packages 
Reorganize internal packages into platform/infra/shared layers and update
imports, docs, and seed-count tests to match current system configs.
 2026-07-24 15:41:59 +08:00
ryan f28aa6520e fix(lint): 消除 linter 告警 2026-07-20 15:53:53 +08:00
ryan d58b4b6b0e feat(waf): 自动 IP 组 lookback 支持 60m/1h 时长写法 
将 lookback_minutes 替换为 lookback，移除最小 5 分钟回看限制，并兼容旧字段。
 2026-07-20 15:48:16 +08:00
ryan 866f1df5e3 fix(waf): IP 组同步间隔下限改为 1 分钟 
移除同步周期 5 分钟限制；回看窗口仍保持最小 5 分钟。
 2026-07-20 15:41:31 +08:00
ryan 351e8ce78c feat(waf): StatusRatio/StatusCount 支持 2xx/4xx/5xx 类写法 
自动 IP 组表达式可按状态码类汇总占比与计数，兼容原有精确状态码。
 2026-07-20 15:39:14 +08:00
ryan a261c01a9c fix(db): correct table name to of_proxy_routes in migration 2026-07-20 14:06:07 +08:00
ryan ae5345c03e feat(rate-limit): add default request rate limit configuration 
- Support openresty_default_limit_req_per_ip in system_configs.
- Add limit_req and limit_req_status 429 directive generation in openresty renderer.
- Implement route-level limit_req_per_ip override and explicit disable.
- Add frontend UI inputs and validation in rate limits tab config.
- Update swagger API docs and changelog for v3.4.3-beta.3.
 2026-07-20 10:58:13 +08:00
ryan a963b8bf54 chore(prettier): format 并清理无用 import 
make prettier 统一格式化，goimports 与 eslint unused-imports 自动移除未使用导入。
 2026-07-19 20:51:12 +08:00
ryan 4481677ef3 refactor(pages): 公开部署源任务并默认每日扫描 
移除 Pages 部署源任务的 InternalOnly 限制，任务管理可查看与调度；
将 scanner cron 与 GitHub latest 默认检查间隔调整为每天一次，
并优化部署历史列表展示。
 2026-07-19 20:41:09 +08:00
ryan 20249d917c feat(rate-limits): use 3-minute trend buckets 
Rate-limit analysis requests overview with bucket_minutes=3; RPS uses
count/180. Overview still defaults to 60-minute buckets.
 2026-07-19 20:30:20 +08:00
ryan abe8fb8268 refactor(pages): 精简部署源模型并重构详情页交互 
将 Remote 网络策略收敛为 allow_insecure，去掉脱敏与无用字段；
Pages 详情拆为部署/设置 Tab，统一卡片样式与来源信息展示。
 2026-07-19 20:23:31 +08:00
ryan f0b51a99b3 fix(db): 重编号 Pages 部署源迁移避免版本冲突 
main 已占用 202607190001（OpenResty 默认限流），
将 Pages source runtime / scanner seed 顺延为
202607190002、202607190003，并同步迁移测试期望配置数。
 2026-07-19 19:41:30 +08:00
ryan c92f986978 merge: 合并 PR #22 Pages 部署源 V2 到 feat/pages-source-sync-v2 
基于最新 main 合并 deqiying/feat/pages-source-sync-v2，
解决 docs/changelog/index.md 与限流相关条目的冲突。
 2026-07-19 19:36:48 +08:00