From f38667446467473c373edf962a7fc29b1939255b Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 16:14:09 +0800 Subject: [PATCH 1/8] =?UTF-8?q?docs(pages):=20=E5=AE=8C=E5=96=84=E9=83=A8?= =?UTF-8?q?=E7=BD=B2=E6=BA=90=20V2=20=E5=AE=9E=E7=8E=B0=E6=96=B9=E6=A1=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/architecture.md | 4 +- docs/design/pages-design.md | 12 +- docs/plan/20260719-pages-source-sync-v2.md | 1437 ++++++++++++++++++++ docs/plan/index.md | 1 + 4 files changed, 1451 insertions(+), 3 deletions(-) create mode 100644 docs/plan/20260719-pages-source-sync-v2.md diff --git a/docs/design/architecture.md b/docs/design/architecture.md index 0f63aa51..f58a915c 100644 --- a/docs/design/architecture.md +++ b/docs/design/architecture.md @@ -80,7 +80,7 @@ OpenResty (Agent, TLS/WAF) * 提供管理端 REST API（`/api/v1/d/*`），通过 **Session Cookie** 鉴权，可选 `X-Access-Token` 访问令牌。 * 边缘节点协议走 `/api/v1/agent|relay|tunnel/*`，分别使用 `X-Agent-Token` / `X-Tunnel-Token` 鉴权。 * 包含配置编译器（Compiler），将数据库中的规则、证书与全局参数统一编译为不可变的配置快照及 OpenResty 物理配置文件文本。 -* 存储 Pages 部署 ZIP 包于本地 Artifacts 目录，并向 Agent 提供受控的下载接口。 +* 统一接收 Pages 本地上传、Remote URL 与公开 GitHub Release 预构建产物，完成来源检查、受限下载、归档校验、不可变 deployment 与原子激活，并向 Agent 提供受控的 latest 下载接口。未来仓库源码构建由独立 Server build executor 扩展，Agent 不执行第三方拉取或构建命令。 * 后台集成 Uptime Kuma 监控同步服务，自动为可用站点维护 HTTP 探测任务。 * 启动入口为根目录 `main.go` + `internal/cmd/`（`api` / `worker` / `scheduler` / `all`）；OpenFlare 业务在 `internal/apps/openflare/`，边缘协议处理在 `internal/apps/openflare/{agent,relay,flared}/`。 * *详细设计请参阅：[Agent 与发布模型设计](./agent-design.md) 以及 [Uptime Kuma 监控同步设计](./kuma-design.md)* @@ -160,7 +160,7 @@ OpenResty 健康与连接数 --> 边缘健康（瞬时，不作 24h 业务总量 当前系统核心实体包括： * **反代与配置**：`zones` (根域管理边界), `zone_domains` (明确域名与证书/路由关联), `proxy_routes` (路由策略), `origins` (源站), `config_versions` (配置版本), `tls_certificates` (证书). 详见 [Zone 与域名资源设计](./zone-design.md)。 -* **Pages 静态托管**：`pages_projects` (Pages项目), `pages_deployments` (不可变部署), `pages_deployment_files` (部署文件清单). +* **Pages 静态托管**：`pages_projects` (Pages项目), `pages_project_sources` / `pages_project_source_runtime` (可变来源配置与运行态), `pages_deployments` (不可变部署), `pages_deployment_files` (部署文件清单). * **节点与穿透**：`nodes` (节点), `tunnels` (隧道客户端), `node_system_profiles` (系统概况), `apply_logs` (应用日志). * **WAF 与安全**：`waf_rule_groups` (WAF规则组), `waf_ip_groups` (WAF IP组), `waf_rule_group_bindings` (网站WAF绑定). * **系统与账号**：`acme_accounts` (ACME账户), `dns_accounts` (DNS账户), `geoip_update_configs` (GeoIP更新配置). diff --git a/docs/design/pages-design.md b/docs/design/pages-design.md index 9f7fded0..07343276 100644 --- a/docs/design/pages-design.md +++ b/docs/design/pages-design.md @@ -19,13 +19,23 @@ ## 核心功能 Pages 静态托管子系统包含以下核心能力： -* **Direct Upload 部署模式**：支持直接上传预构建的静态资源压缩包（`zip`、`tar.gz`、`tar.xz`、`tar.bz2`、`tar`、`7z`），或填写 URL 由控制面代为下载导入（允许内网地址与自签证书），省去复杂的 Git 集成和构建环境依赖。 +* **预构建产物部署**：支持直接上传静态资源压缩包，也可为项目保存一个 Remote URL 或公开 GitHub Release asset 来源。外部来源只由 Server 访问，成功同步后统一创建或复用不可变 deployment 并原子激活。 * **不可变部署快照**：每次上传产生一个带唯一 ID 和 SHA-256 Checksum 的不可变部署记录。支持按系统配置保留最近 N 个历史部署，并可随时激活和回滚。 * **SPA Fallback 支持**：支持对单页应用（SPA）进行 Fallback 路由配置，请求找不到静态文件时自动重定向到入口文件。 * **内置 API 反代服务**：支持在 Pages 规则内一键启用 API 代理，消除跨域问题，将请求转发给指定的后端服务。 * **安全包校验与解压缩**：内置路径逃逸防御、防软链接劫持、文件大小/数量上限与可配置上传包体积控制，保障节点物理安全。 * **可配置限额**：管理员可在运维设置中调整「部署包大小上限」与「历史部署保留数」。 +### 部署源与未来构建边界 + +项目当前支持 manual、Remote URL、GitHub Release 三种来源视图。无 source 记录即 manual；切换或删除 source 不删除历史 deployment，也不改变当前 active deployment。Remote URL 只允许手动“同步并发布”；GitHub Release 支持 latest/tag 手动检查与同步，只有 latest 可选择定时检查和自动更新。 + +source 是可变配置，deployment 是不可变事实。source 配置与运行态游标、状态、租约分别存储；deployment 只保存创建时的安全 provenance 快照。所有来源最终都进入同一条“下载或接收产物 → 真实字节与入口校验 → `upload.Ingest` → deployment → 原子激活”管线，Agent 不感知来源类型。 + +后续从 Git 仓库拉取源码并自动构建时，将新增独立 `git_repository` provider 与隔离的 build executor。它输出受限的预构建产物后继续复用上述导入管线；不得把 clone、依赖安装或任意构建命令下发给 Agent，也不得把 branch/build/env 字段塞入现有 `github_release` source。当前 V2 不增加这些未来字段或空任务，只稳定 provider 输出、source discriminated view 与 deployment provenance 三个扩展边界。 + +管理端信息架构参考 Cloudflare Pages 当前把 [Git integration](https://developers.cloudflare.com/pages/configuration/git-integration/) 与 [Direct Upload](https://developers.cloudflare.com/pages/get-started/direct-upload/) 分离、并统一展示生产状态与历史部署的方式：OpenFlare 项目详情按“当前生产部署 → 部署源 → 部署历史”组织。OpenFlare 仍允许切换来源并保留历史部署，不采用 Cloudflare 项目创建后来源不可切换的限制。 + --- ## Pages 静态托管架构 diff --git a/docs/plan/20260719-pages-source-sync-v2.md b/docs/plan/20260719-pages-source-sync-v2.md new file mode 100644 index 00000000..47859802 --- /dev/null +++ b/docs/plan/20260719-pages-source-sync-v2.md @@ -0,0 +1,1437 @@ +# Pages 项目部署源与 GitHub Releases 自动更新 V2 实现方案 + +日期：2026-07-19 +状态：实施中 +方案版本：V2（设计修订版，不代表新增 `/api/v2`） + +关联材料： + +* 原方案：[`20260718-pages-source-sync.md`](./20260718-pages-source-sync.md) +* 设计审核：[`20260719-pages-source-sync-design-review.md`](./20260719-pages-source-sync-design-review.md) +* 表结构审核：[`20260719-pages-source-sync-schema-revision.md`](./20260719-pages-source-sync-schema-revision.md) + +> 本文是完整、独立且可直接实施的技术方案，取代原方案成为该功能唯一实现基线。原方案与两份审核文档仅用于追溯设计演进；开发时不需要再将它们与本文拼接，也不得沿用其中与本文冲突的宽表、11 态状态机、`activate=false` 或四重 fence 设计。 + +--- + +## 0. 结论摘要 + +V2 保留原方案正确的主链路：外部来源只由 Server 控制面访问，所有包都进入统一的不可变 deployment 管线，Agent 仍只从 Server 拉取当前 active package。审核意见中的高优先级问题按以下规则一次性收敛： + +1. source 配置与运行态拆为 `of_pages_project_sources`、`of_pages_project_source_runtime` 两张表；runtime 不再冗余 `project_id`。 +2. 来源同步固定为“解析/下载 → 校验 → 创建或复用 deployment → 原子激活”，API 不提供 `activate` 开关。 +3. `sync_status` 只保留 `idle | checking | update_available | syncing | failed | attention` 六态；排队状态使用现有 `TaskExecution`，不在 source runtime 重复保存。 +4. 互斥只由 runtime lease 负责；过期结果只在最终事务校验 source `config_version`、project `content_config_version`、`lease_token` 与 lease 未过期时间，不再传递通用 `expected_revision`。 +5. 人工激活不同 deployment 时，只要项目存在 source，就在同一事务中 fence 在途任务；若自动更新已开启，同时强制关闭，避免人工回滚被下一轮 latest 静默覆盖。 +6. `history_count=1` 时，手动上传允许临时保留 active 与最新 candidate 两条；激活后恢复严格上限。source sync 因创建与激活同事务完成，不产生未激活候选。 +7. Remote URL 只支持手动“同步并发布”；只有 GitHub `latest` 支持定时检查和可选自动更新，GitHub `tag` 只支持手动检查/同步。 +8. Remote URL 使用显式 `remote_url_set` 控制“保留或替换”密文 URL；API、日志、任务 payload 和 deployment provenance 均不得泄漏 query token。 +9. 阶段 0 先修复 `RootDir`、归档真实展开限制、Agent 全量内存下载和历史裁剪问题，再接入远端来源。 + +--- + +## 1. 目标与背景 (Goal & Context) + +### 1.1 当前实现与问题 + +当前 Pages 已支持： + +* 管理员本地上传压缩包； +* 同步调用 `POST /api/v1/d/pages/:id/deployments/upload-from-url` 完成一次性 URL 导入； +* 使用 `RootDir + EntryFile` 检查归档并通过 `upload.Ingest` 保存； +* 创建不可变 deployment、手动激活、保留历史版本； +* Agent 通过 latest hash/package 接口拉取 active deployment，并原子切换本地 `current`。 + +现状的主要问题不是缺少下载函数，而是缺少项目级、可持续管理的来源模型：URL 每次都要重新输入，GitHub Release 无版本游标和自动检查，任务与 deployment 也没有可审计的来源快照。同时，现有代码还存在必须在自动化前修复的边界： + +* `RootDir` 已用于 Server 校验，但 OpenResty `LocalRoot` 未稳定追加该目录； +* Agent 将整个 package 读入 `[]byte`，解压时关闭实际限制； +* tar family 的部分检查/解压路径会按声明大小物化成员内容； +* `history_count=1` 时，刚上传的未激活 candidate 会被旧 active 挤掉； +* `PolicyDedupNewRecord` 当前复制既有 record metadata，且新 record 持久化失败时可能误删复用的共享 object； +* 一次性 URL client 允许私网与不安全 TLS，不能作为新持久来源的默认网络策略； +* 前端上传 payload、deployment 类型、固定入口文件提示和请求超时与后端契约存在漂移。 + +### 1.2 功能目标 + +本方案交付以下能力： + +* Pages 项目可保持手动模式，或配置一个持久 Remote URL / GitHub Release 来源； +* Remote URL 可重复手动同步，每次按下载内容 SHA-256 幂等创建或复用 deployment 并激活； +* GitHub 支持 `latest`（默认）与固定 `tag`，asset 名称默认精确匹配 `dist.zip`； +* GitHub `latest` 可按 5～1440 分钟间隔定时检查，默认 60 分钟；自动更新默认关闭； +* 检查发现新版本但自动更新关闭时，只展示更新，不下载； +* 同一 Release 下 asset 被替换时进入 `attention`，必须由管理员确认指定 revision 后才允许同步； +* `RootDir` / `EntryFile` 继续作为项目级内容配置统一作用于本地、Remote 与 GitHub 包，不在 source 中复制一套入口字段； +* 所有来源统一使用现有 Pages 归档检查、上传、deployment、激活、历史裁剪和 Agent 分发链路； +* 自动化执行可互斥、可 fence、可恢复，且失败不会改变旧 active deployment。 + +### 1.3 来源能力矩阵 + +| 模式 | source 行 | 触发方式 | 检查更新 | 自动更新 | revision | 成功结果 | +| --- | --- | --- | --- | --- | --- | --- | +| 手动本地上传 | 无 | 管理员上传 | 无 | 无 | 不用于幂等 | 创建 candidate，管理员再激活 | +| 一次性 URL（兼容） | 无 | 旧同步 API | 无 | 无 | 不用于幂等 | 每次创建 candidate，管理员再激活 | +| 持久 Remote URL | 有 | 管理员“同步并发布” | 不提供 | 不提供 | 下载内容 SHA-256 | 创建或复用并强制激活 | +| GitHub Release tag | 有 | 管理员检查/同步 | 仅手动 | 不提供 | Release/asset 元数据哈希 | 创建或复用并强制激活 | +| GitHub Release latest | 有 | 手动或 scanner | 定时 | 可选，默认关闭 | Release/asset 元数据哈希 | 创建或复用并强制激活 | + +无 source 行即手动模式。切换或删除 source 不删除 deployment，也不改变当前 active deployment。 + +### 1.4 默认值 + +| 配置 | 默认值 | 边界 | +| --- | --- | --- | +| GitHub selector | `latest` | `latest` / `tag` | +| Release asset | `dist.zip` | basename，精确且区分大小写 | +| 自动更新 | `false` | 仅 GitHub latest 可开启 | +| 检查间隔 | 60 分钟 | 5～1440 分钟 | +| scanner cron | `*/5 * * * *` | 固定，无新增系统设置 | +| scanner 单批 | 20 个 source | 按 `next_check_at, source_id` 排序 | +| check lease | 2 分钟 | 到期可恢复 | +| sync lease | 15 分钟 | 长下载期间按需续租 | +| Remote 网络策略 | `public` | `public` / `trusted_internal` | + +### 1.5 非目标 + +本次不实现： + +* GitHub 私有仓库、GitHub Token、GitHub App 或其它代码托管平台； +* source archive、`zipball_url` / `tarball_url` 回退； +* asset glob、正则、优先级列表或 semver 自行排序； +* Remote URL 的定时轮询或自动更新； +* 多 source、分支构建、Webhook、CI 构建、预览环境； +* `remote_url` 数据库加密列；V2 先保证最小暴露和全链路脱敏； +* 额外同步历史表、租约表、Provider 分表或全局 GitHub 响应缓存； +* Agent 直接访问 GitHub 或 Remote URL。 + +上述“仓库构建”属于明确的后续能力，不在 V2 偷跑实现；但 V2 的 Provider、部署来源视图和导入管线必须保留可扩展边界，避免未来只能把 Git clone/build 逻辑塞入 `github_release` 分支或重写 deployment 主链路。 + +--- + +## 2. 设计与决策 (Design & Decisions) + +### 2.1 核心原则 + +1. **Server 单一信任边界**：第三方网络访问、digest 校验与归档检查都在 Server 完成。 +2. **source 可变，deployment 不可变**：source 表示当前配置；deployment 保存创建时的最小来源快照，不随 source 编辑。 +3. **检查不等于部署**：GitHub check 只更新远端游标；只有 sync 才下载、创建并激活。 +4. **成功才切换**：网络和归档工作在事务外；active pointer、deployment 与 applied cursor 在最终事务原子提交。 +5. **状态面最小化**：source runtime 只保存控制面稳定状态，队列细节和阶段日志复用现有 TaskExecution。 +6. **人工操作优先**：人工激活或回滚必须 fence 自动任务，且不能被自动更新静默覆盖。 +7. **平台能力复用**：文件摄取继续通过 `upload.Ingest`；普通文件删除使用 `upload.Remove` / `RemoveOwned`，Pages 保留类型在复检业务引用后使用同包的 `RemoveLockedTx`；任务继续使用现有 task/Asynq 框架。 + +### 2.2 总体架构 + +```mermaid +flowchart LR + Admin["管理员 / Pages 详情页"] --> SourceAPI["Pages Source API"] + SourceAPI --> ConfigDB[("Source Config")] + SourceAPI --> RuntimeDB[("Source Runtime")] + SourceAPI -->|"手动 check / sync"| ActionTask["Pages Source Action Task"] + + Scheduler["Scheduler"] --> ScanTask["Pages Source Scan Task"] + ScanTask -->|"串行检查到期 latest"| GitHubAPI["GitHub Releases API"] + ScanTask --> RuntimeDB + ScanTask -->|"限量 orphan record 补偿"| Upload + ScanTask -->|"发现更新且 auto=true"| ActionTask + + ActionTask --> Remote["Remote URL / GitHub Asset"] + ActionTask --> Pipeline["统一导入管线"] + Pipeline --> Inspect["真实展开与入口校验"] + Inspect --> Upload["upload.Ingest"] + Upload --> DeploymentDB[("Pages Deployments")] + DeploymentDB --> Activate["原子激活 + applied cursor"] + Activate --> RuntimeDB + + Agent["Agent latest hash/package 对账"] --> DeploymentDB + Agent --> Current["projects/{id}/current"] + Current --> OpenResty["OpenResty 静态服务"] +``` + +scanner 本身是一个正式 TaskHandler，并非绕过任务框架。它在单次执行中先限量恢复 lease/orphan record，再串行检查到期的 GitHub latest source，避免一次 cron 批量投递 20 个并行 GitHub 请求。手动操作和自动下载使用统一 action task；scanner 不执行长时间 package 下载。 + +### 2.3 领域对象与不变量 + +| 对象 | 生命周期 | 不变量 | +| --- | --- | --- | +| PagesProject | 可变 | `RootDir` / `EntryFile` 的实质变化递增 `content_config_version` | +| PagesProjectSource | 可变配置 | 每项目最多一条；不保存状态、游标或 lease | +| PagesProjectSourceRuntime | 可变运行态 | 与 source 1:1；状态、游标、lease 只写本表 | +| PagesDeployment | 不可变事实 | 持久来源 revision 幂等；provenance 创建后不回写 | +| PagesDeploymentFile | 不可变清单 | 只属于一个 deployment | + +核心不变量： + +* source 与 runtime 必须同事务创建、同事务删除；无 source 就无 runtime。 +* deployment 的 `source_identity` 与 `source_revision` 必须同时为非空值或同时为 SQL `NULL`。 +* 持久来源 sync 成功时，deployment、files、active pointer 和 runtime applied cursor 在同一事务提交。 +* source/project 配置变化或人工激活可以使任务过期；过期任务不得改变 active、runtime 或其它 deployment。 +* source API 永不返回完整 Remote URL，任务 payload 永不携带 URL。 + +### 2.4 数据模型 + +#### 2.4.1 关系总览 + +```text +of_pages_projects + └── 0..1 of_pages_project_sources + └── 1..1 of_pages_project_source_runtime + +of_pages_projects + └── 0..N of_pages_deployments + └── 0..N of_pages_deployment_files +``` + +不建立物理外键；删除顺序由 Pages service 事务显式保证。 + +#### 2.4.2 `of_pages_project_sources`：纯配置 + +| 字段 | 类型 / DB 默认 | 说明 | +| --- | --- | --- | +| `id` | PK | source ID | +| `project_id` | bigint/integer | 项目 ID，唯一索引 | +| `source_type` | varchar(32), `''` | 服务层写 `remote_url` / `github_release` | +| `remote_url` | text, `''` | 仅 Remote；可含 query secret，禁止回显 | +| `remote_network_policy` | varchar(32), `''` | 服务层写 `public` / `trusted_internal` | +| `github_repository` | varchar(255), `''` | 规范化为 `{owner}/{repo}` | +| `release_selector` | varchar(16), `''` | `latest` / `tag` | +| `release_tag` | varchar(255), `''` | tag 模式必填，latest 必须空 | +| `asset_name` | varchar(255), `''` | 服务层默认写 `dist.zip` | +| `auto_update_enabled` | bool, `false` | 仅 GitHub latest 可为 true | +| `check_interval_minutes` | int, `0` | GitHub latest 服务层默认写 60 | +| `config_version` | int, `0` | 创建显式写 1；实质配置变化或人工 fence 时递增 | +| `source_identity` | char(64), `''` | 无凭据的稳定身份 SHA-256 | +| `created_at` / `updated_at` | datetime | 审计时间 | + +配置表禁止加入 `sync_status`、`etag`、`last_seen_*`、`last_applied_*` 或 `lease_*`。 + +#### 2.4.3 `source_identity` + +GitHub： + +```text +SHA-256("github|" + owner_repo + "|" + selector + "|" + tag + "|" + asset_name) +``` + +Remote： + +```text +SHA-256("remote_url|" + canonical_scheme_host_port_path) +``` + +Remote canonical identity 使用小写 scheme/host、移除默认端口并保留规范化 path；明确排除 query、fragment 和 userinfo。下载 URL 仍保存管理员输入的完整值，但 URL userinfo 和 fragment 本身不允许保存。 + +identity 变化时，同事务重置 runtime 的 ETag、seen/applied cursor、detail、错误、检查时间和 lease；当前 active deployment 不变。仅 query token、自动更新、检查间隔或网络策略变化时 identity 不变，保留 cursor，但仍递增 `config_version`、清 lease 并按现有 cursor 重算稳定状态。 + +#### 2.4.4 `of_pages_project_source_runtime`：纯运行态 + +| 字段 | 类型 / DB 默认 | 说明 | +| --- | --- | --- | +| `source_id` | PK | 与 source 1:1 的逻辑关联 | +| `etag` | varchar(512), `''` | GitHub 条件请求 | +| `last_seen_revision` | char(64), `''` | 最近解析到的 revision | +| `last_seen_detail` | text, `''` | 已校验的安全 JSON 对象字符串 | +| `last_applied_revision` | char(64), `''` | 当前 source 视角下已激活 revision | +| `last_applied_detail` | text, `''` | 与 applied revision 配套的安全 JSON | +| `sync_status` | varchar(32), `''` | 创建时服务层显式写 `idle` | +| `last_error` | text, `''` | 脱敏后的最近错误 | +| `last_checked_at` | nullable datetime | GitHub 最近完成检查时间 | +| `last_synced_at` | nullable datetime | 最近成功同步并激活时间 | +| `next_check_at` | nullable datetime | 仅 GitHub latest 非空；普通索引 | +| `lease_expires_at` | nullable datetime | 当前租约截止时间 | +| `lease_token` | varchar(64), `''` | 每次获取租约生成的新 token | +| `updated_at` | datetime | 运行态更新时间 | + +runtime 刻意不保存 `project_id`：scanner 本来就必须 join source 读取 `source_type`、selector 和 config version；重复保存 project ID 只会引入漂移和额外索引。scanner 查询以 `next_check_at` 索引定位 runtime，再 join source。 + +detail 使用跨 PostgreSQL/SQLite 一致的 text，并由 Go typed struct 统一 marshal/unmarshal；比较和幂等只读取 revision 列，禁止解析 JSON 做 CAS。GitHub detail 最小形状为： + +```json +{ + "provider": "github", + "release_id": "123456", + "asset_id": "789", + "tag": "v1.2.3", + "asset_name": "dist.zip", + "asset_updated_at": "2026-07-18T12:00:00Z", + "digest": "sha256:..." +} +``` + +Remote detail 只保存无密钥显示信息，例如： + +```json +{ + "provider": "remote_url", + "display_name": "dist.zip" +} +``` + +#### 2.4.5 状态机 + +状态固定为： + +```text +idle | checking | update_available | syncing | failed | attention +``` + +```mermaid +stateDiagram-v2 + [*] --> idle + idle --> checking: GitHub check + update_available --> checking: 再次 check + failed --> checking: 重试 check + attention --> checking: 再次 check + checking --> idle: 无更新 + checking --> update_available: 有更新且不自动同步 + checking --> attention: 同 Release asset 被替换 + checking --> failed: 检查失败 + idle --> syncing: 手动 sync + update_available --> syncing: 手动或自动 sync + failed --> syncing: 手动重试 + attention --> syncing: 确认指定 revision + syncing --> idle: 同步并激活成功 + syncing --> failed: 下载/校验/提交失败 + syncing --> attention: 替换风险未确认 +``` + +约定： + +* `syncing` 覆盖下载、校验、Ingest、创建和激活；详细阶段只写 task 日志。 +* `failed` 可以与“已有待更新 revision”同时存在；API 的 `update_available` 始终由 revision 派生，而非由状态字符串判断。 +* `attention` 是 GitHub 供应链确认状态，不等同于普通失败。 +* `queued` / `succeeded` 属于 `w_task_executions`，不进入 runtime。 + +派生规则： + +```text +update_available = + last_seen_revision != '' AND + last_seen_revision != last_applied_revision +``` + +#### 2.4.6 `of_pages_projects` 增量 + +新增： + +| 字段 | 类型 / 默认 | 说明 | +| --- | --- | --- | +| `content_config_version` | int, `0` | 仅 `RootDir` / `EntryFile` 实质变化时 +1 | + +SPA Fallback、API Proxy、名称、描述、启停等变化不影响归档内容校验，不递增该版本。 + +#### 2.4.7 `of_pages_deployments` 精简 provenance + +| 字段 | 类型 / DB 默认 | 说明 | +| --- | --- | --- | +| `source_type` | varchar(32), `''` | `manual_upload` / `manual_url` / `remote_url` / `github_release` | +| `source_identity` | nullable char(64) | 持久 source 快照；手动/一次性 URL 必须为 SQL `NULL` | +| `source_revision` | nullable char(64) | 持久 source 幂等键；手动/一次性 URL 必须为 SQL `NULL` | +| `source_label` | varchar(255), `''` | tag 或安全文件名，不含 query | +| `source_meta` | text, `''` | 安全 JSON 审计快照，不含 URL/token | +| `trigger_type` | varchar(32), `''` | `manual_upload` / `manual_url` / `manual_sync` / `scheduled_auto_update` | + +不再增加独立的 release/asset/digest 宽列；这些只在 `source_meta` 保留审计快照。deployment 列表 API 只返回安全的 `source_type`、`source_label`、`trigger_type`，不直接输出原始 meta JSON。 + +revision 生成： + +```text +github_raw = github:::: +github_revision = SHA-256(github_raw) + +remote_revision = SHA-256(downloaded_package_bytes) +``` + +GitHub 未提供 digest 时，`declared_digest` 为空；同步仍必须计算 package SHA-256 作为 deployment checksum。若 GitHub 提供 `sha256:` digest，则下载后必须严格校验。 + +#### 2.4.8 索引与迁移 + +索引： + +```text +UNIQUE of_pages_project_sources(project_id) +INDEX of_pages_project_source_runtime(next_check_at) +UNIQUE of_pages_deployments(project_id, deployment_number) +UNIQUE of_pages_deployments(project_id, source_identity, source_revision) + WHERE source_identity IS NOT NULL AND source_revision IS NOT NULL +``` + +PostgreSQL 与 SQLite 均创建同语义的部分唯一索引。禁止用空字符串代替 deployment 的 NULL provenance，否则手动重复上传会被误判为同一来源版本。 + +新增双方言 migration： + +1. `202607190001_add_pages_source_runtime.sql`：两张 source 表、project content version、deployment provenance、索引和存量回填。 +2. `202607190002_seed_pages_source_scan.sql`：幂等插入 `of_pages_source_scan` 的 5 分钟 schedule。 + +存量 deployment 只能可靠回填为 `source_type=manual_upload`、`trigger_type=manual_upload`，identity/revision 保持 NULL；现有记录无法反推出是否来自旧一次性 URL。schedule seed 不写死 ID，使用 `WHERE NOT EXISTS (task_type = 'of_pages_source_scan')`，Down 仅按该 task type 删除。 + +#### 2.4.9 写入矩阵 + +| 操作 | source config | runtime | deployment | +| --- | --- | --- | --- | +| 创建 source | 新建 | 同事务新建 idle | 不变 | +| 编辑 source | 实质变化时 version +1 | identity 变则 reset，否则保留 cursor、清 lease | 不变 | +| 删除 source | 删除 | 同事务删除 | 全部保留 | +| GitHub check / 304 | 不变 | seen、时间、状态、下次检查 | 不变 | +| source sync | 不变 | syncing → applied/idle | 创建或复用并激活 | +| 人工激活/回滚 | 必要时关闭 auto、version +1 | 清 lease，按目标 provenance 更新 applied | 切 active | +| project 删除 | 删除 | 先删除 | 按现有流程删除 | + +### 2.5 任务、租约与并发 + +#### 2.5.1 任务类型 + +只新增两个任务： + +| Meta Type | Asynq Type | 职责 | +| --- | --- | --- | +| `of_pages_source_scan` | `openflare:pages_source_scan` | 恢复过期 lease/orphan record；串行检查一批到期 GitHub latest source；必要时投递 sync action | +| `of_pages_source_action` | `openflare:pages_source_action` | 执行管理员 check/sync 或 scanner 触发的 sync | + +两者都在 `internal/task/handlers/register.go` 显式注册 Handler 与 TaskMeta。现有 `bootstrap.RegisterTasks()` 已覆盖 API、worker、scheduler 和 all 入口，不新增 `init()`，也不修改 `internal/router/router.go`、`internal/bootstrap/bootstrap.go` 或 `internal/cmd` 的装配职责。 + +两类任务都标记为 `TaskMeta.InternalOnly=true`。通用 Admin Task 类型列表、手工 dispatch 与 schedule 创建/更新必须隐藏或拒绝 internal-only meta；scheduler 与 Pages 内部 dispatch 仍使用完整 registry。这样客户端不能绕过 Pages Handler 自行伪造 `source_id`、`config_version` 或 `actor`。 + +action payload 只包含： + +```json +{ + "source_id": 42, + "config_version": 3, + "action": "check", + "actor": "user:1234567890", + "target_revision": "", + "confirmed_revision": "" +} +``` + +规则： + +* `action` 仅为 `check` / `sync`； +* `target_revision` 只由 scanner 在自动 sync 时写入，用于锁定本次 check 发现的 revision；手动 sync 为空； +* `confirmed_revision` 只在确认 `attention` 时携带 UI 当前看到的精确 revision；不用单纯 boolean 确认未知的未来版本； +* payload 不携带 Remote URL、GitHub 下载 URL、ETag、`content_config_version` 或通用 `expected_revision`；`target_revision` 是自动检查结果约束，不参与配置 fencing； +* `actor` 手动操作为 `user:`，自动任务为 `system:pages-source-sync`，禁止空字符串表示系统。 + +调用 `task.DispatchTask` 时，框架级 `triggeredBy` 继续使用 `manual` / `system`；具体操作者只放在已校验且无密钥的 action payload 中，供 deployment `created_by` 与审计日志使用。 + +该 payload 是 Server 内部契约：HTTP Handler 只接受 action 所需业务字段，再从路由项目、当前 source 和 OAuth context 组装 `source_id/config_version/actor`，禁止客户端直接指定或冒充这些值。 + +`content_config_version` 在 sync Worker 获取 lease 后读取并形成执行快照，最终事务再次检查。这样既能阻止旧入口配置被激活，又不把每次项目变更传播进队列 payload。 + +#### 2.5.2 lease 规则 + +lease 只解决“同一 source 同时只能有一个执行者”： + +* check 获取 2 分钟短 lease，并将状态切为 `checking`； +* sync 获取 15 分钟长 lease，并将状态切为 `syncing`； +* 获取使用 `source_id + config_version + lease 已过期` 的 CAS； +* 续租、状态写入、终态和释放必须同时满足 `lease_token` 匹配且 `lease_expires_at > now`；续租不再 join project/source 版本； +* 最终事务前强制续租一次；最终提交仍必须再次检查 token 与未过期时间，不能让“尚未被新 Worker 改写 token 的过期 lease”通过； +* source 配置变化、RootDir/EntryFile 变化或人工激活统一调用 `fenceAndNormalizeRuntime`：清 token/expiry；若当前 seen/applied 仍构成同 Release 替换则为 `attention`，否则 seen≠applied 为 `update_available`，其余为 `idle`；source 删除则同事务直接删除 runtime/source，行不存在即 fence； +* 未拿到 lease 的重复任务写一条 no-op task 日志并成功结束，不制造 runtime 错误。 + +最终提交的锁顺序固定为： + +```text +project -> source（存在时） -> runtime（存在时） -> upload（所有相关 ID 升序） +``` + +提交前只校验： + +```text +source.config_version == captured_source_version +project.content_config_version == captured_content_version +runtime.lease_token == worker_token +runtime.lease_expires_at > transaction_now +target_upload.status == used +``` + +source/runtime 条件只适用于持久 source；本地上传和一次性 URL 仍必须先锁 project、最后锁目标 upload。create-or-load 选中的既有 deployment 与本次新建但最终未使用的 upload 不同时，两个 upload ID 在最后一层按升序加锁，避免多行反序。上述任一条件不满足，任务按“配置、执行权或上传记录已变化”结束，不覆盖新 runtime 状态；若本次创建了 upload record，则进入补偿。revision 幂等由 deployment 部分唯一索引负责，不再增加第四套通用 revision fence。 + +#### 2.5.3 scanner 流程 + +每 5 分钟执行： + +1. 扫描所有 runtime 中 lease 已过期且状态为 `checking/syncing` 的行；恢复 UPDATE 必须再次 CAS 原 token 且 `lease_expires_at <= now`，避免覆盖刚续租的 Worker。成功后清 lease、状态设为 `failed`，记录“上次任务租约已过期”，GitHub latest 的 `next_check_at` 调整为近期重试。 +2. 执行 2.7.2 的限量 orphan record reconciliation；单条失败只告警并保留候选，不中断 source 检查。 +3. join source 查询 `github_release + latest + next_check_at <= now`，按 `next_check_at, source_id` 排序，最多取 20 条。 +4. 对每条 source 尝试获取短 lease；失败说明另一个 scanner/action 已处理，直接跳过。 +5. 在当前 scanner TaskHandler 内串行调用 GitHub check，共享同一 check service；单个 source 失败只落该 runtime，不中断其它 source。 +6. `304` 仍更新 `last_checked_at/next_check_at`，并根据已保存的 seen/applied revision 重新判断是否待同步。 +7. 发现更新后无论 auto 开关，都先原子写 seen cursor、将状态落为 `update_available` 并释放短 lease；auto 开启时再投递带本次 `target_revision` 的 `action=sync`，sync Worker 获取长 lease 后才切为 `syncing`。 +8. sync 入队失败：保持 `update_available`，记录安全错误，并把 `next_check_at` 调整为短退避，后续 scanner 可再次尝试。 +9. 下次检查时间使用 interval 加 source-ID 派生的小幅 jitter，避免整点集中请求。 + +scanner 直接串行 check 而不是先批量投递 check action，目的是减少 GitHub 并发和一层“派发预占”状态。重叠的 scanner 实例仍通过每个 source 的 lease 互斥；不增加全局 scanner 锁。 + +#### 2.5.4 action 流程 + +手动 check： + +1. Handler 校验 source 为 GitHub；Remote 直接返回稳定 400，不入队。 +2. action Worker 校验 payload `config_version`，获取短 lease。 +3. 解析 Release/asset，更新 seen、ETag、检查时间与状态后释放 lease。 +4. 手动 check 永远不隐式下载；即使 auto 已开启，也只由 scanner 检查路径触发自动 sync，避免“点击检查”产生意外发布。 + +手动或自动 sync： + +1. 校验 source/config version，获取长 lease并读取 project content version。 +2. GitHub 在 lease 内重新解析目标 Release/asset；Remote 直接下载。这样刚保存 source 时无需等待一次 check 才能同步。 +3. scanner 自动 sync 若携带 `target_revision`，本次新解析 target 必须与其相等；不相等说明 latest 在 check 与执行间变化，任务将新 target 安全写为 seen，按本次 target 归一为 `attention/update_available`，释放 lease 并把 `next_check_at` 提前，禁止直接部署未经原 check 锁定的新 revision。 +4. 非空 `confirmed_revision` 必须先与本次 target 完全相等，否则要求刷新后重试；再以本次 target 与 applied detail 判断同 Release 替换，构成替换且未确认当前 target 时写 `attention` 并停止。 +5. 流式下载、digest/checksum 校验、归档检查与 Ingest。 +6. 进入最终事务完成 create-or-load、激活与 applied cursor；提交后严格裁剪历史。 + +永久业务错误（非法配置、asset 不存在、未确认 attention）通过 task 框架的 `PermanentError` 包装为 `asynq.SkipRetry`，不进行 Asynq 快速重试；瞬时网络/存储错误按 TaskMeta 的有限次数退避重试。包装后的 `Error()` 只暴露脱敏 domain message。重复任务、旧 config version 和丢失 lease 作为成功 no-op 结束，避免无意义重试。Provider/Action Handler 在把 error 返回 task executor 前必须转换为不含 URL/query/header/body 的安全 domain error；原始错误也只能经统一 URL 脱敏后写内部日志，防止 TaskExecution `error_message/log/result` 持久化密钥。 + +### 2.6 Provider 设计 + +#### 2.6.0 Provider 扩展边界与未来仓库构建 + +V2 Provider 只负责把某个外部来源解析为一个经过约束的不可变归档候选，不负责直接写 deployment、切 active 或操作 Agent。Pages service 继续统一承担归档检查、`upload.Ingest`、deployment create-or-load、激活、历史裁剪与补偿。当前 Remote URL 与 GitHub Release 都实现这一窄边界。 + +为后续“从仓库拉代码自动构建”预留以下设计约束，但本期不增加数据库列、API 或空实现： + +* 后续新增独立 `git_repository` source/provider，禁止复用或扩展 `github_release` 语义；Release asset 是预构建产物来源，repository source 是源码与构建来源，两者凭据、revision、失败阶段和 UI 配置完全不同。 +* repository provider 的输出仍必须是临时目录中的受限归档/构建产物描述，再进入现有统一导入管线；build checkout、依赖安装、命令执行和日志隔离属于未来独立 build executor，不进入 Agent，也不绕过 `upload.Ingest`。 +* source view 与前端表单继续使用 discriminated union；未来可以新增 repository variant，而无需给 Remote/GitHub Release 视图加入无关的 branch、build command、output directory 或 environment 字段。 +* deployment provenance 保留 `source_type/source_identity/source_revision/source_label/source_meta/trigger_type` 的通用事实边界；未来 repository revision 可使用 commit SHA，安全 `source_meta` 可保存 branch/build 输出摘要，但不得保存凭据或完整环境变量。 +* TaskExecution 继续承载阶段日志。未来构建可增加 resolve/checkout/build/package 阶段，但 source runtime 不因此扩展为构建步骤状态机。 + +该边界参考 Cloudflare Pages 当前将 [Git integration](https://developers.cloudflare.com/pages/configuration/git-integration/) 与 [Direct Upload](https://developers.cloudflare.com/pages/get-started/direct-upload/) 分成不同来源体验、同时把生产部署与历史部署统一呈现的产品结构；OpenFlare 保留自己的“来源可切换且历史部署不删除”决策，不照搬 Cloudflare 创建后不可切换来源的限制。 + +#### 2.6.1 持久 Remote URL + +Remote 来源只提供“同步并发布”，不提供 check、定时检查或自动更新。每次同步： + +1. 按 source 保存的 network policy 构建下载 client； +2. 流式写入 Server 临时文件，同时计算 SHA-256 和实际压缩包大小； +3. 以内容 SHA-256 生成 revision；若同 identity/revision deployment 已存在，跳过 Ingest，直接进入安全激活； +4. 新 revision 使用统一归档/上传/激活管线； +5. 成功后 seen 与 applied 同时更新为该 revision，状态回到 `idle`。 + +归档格式优先使用配置 URL path 的安全 basename；名称缺失或无可识别扩展名时，使用 `pagesarchive.DetectFormat` 对临时文件至少前 512 字节做 magic sniff，覆盖 tar 在偏移位置的签名，不能沿用当前仅 16 字节的探测。redirect 最终 URL 和 `Content-Disposition` 不进入 provenance，避免签名地址或不可信文件名泄漏。 + +Remote URL 的 query 可用于签名 token。API 返回： + +* `has_remote_url=true`； +* `display_url=https://example.com/dist.zip?***`； +* 永不返回原始 URL。 + +编辑时使用显式 `remote_url_set`： + +* 新建 Remote、从 GitHub 切换到 Remote：必须为 `true` 且 URL 非空； +* 编辑现有 Remote 但只改 network policy：必须为 `false`，同时省略 `remote_url`； +* 替换地址：为 `true` 并提交新 URL； +* `false` 却携带 URL，或 `true` 但 URL 为空，均返回 400； +* 前端绝不能把 `display_url` 当作可保存值。 + +#### 2.6.2 GitHub Releases + +仓库地址只接受： + +```text +https://github.com/{owner}/{repo} +https://github.com/{owner}/{repo}.git +``` + +保存时规范化为 `{owner}/{repo}`；拒绝非 `https`、非 `github.com`、userinfo、query、fragment、额外 path 及空 owner/repo。V2 只访问公开仓库。 + +Release 解析： + +* latest：`GET /repos/{owner}/{repo}/releases/latest`；采用 GitHub 的 latest 语义，不拉列表、不自行比较 semver； +* tag：`GET /repos/{owner}/{repo}/releases/tags/{url.PathEscape(tag)}`；固定 tag 不进入 scanner； +* asset：只接受 `state=uploaded` 且 `name == asset_name` 的精确、区分大小写匹配； +* asset 不存在时，安全错误最多列出该 Release 前 10 个 asset 名，单项与总错误长度均截断； +* 不回退到源码 archive。 + +API client 使用新的窄包 `internal/integration/githubrelease`，集中 Release/asset HTTP 契约、redirect、ETag 与限流解析；Pages 模块只负责 source 规则、revision 和状态映射。当前 node/edge/admin updater 的旧实现不在本功能中强制迁移，但后续新增调用方必须复用该包，避免继续增加 feature-local GitHub client。 + +* 发送 `Accept: application/vnd.github+json`、固定 `User-Agent`；实现基线固定 `X-GitHub-Api-Version: 2026-03-10`，收敛为一个常量； +* 保存 ETag 并发送 `If-None-Match`； +* 处理 `Retry-After`、`X-RateLimit-Remaining`、`X-RateLimit-Reset`，按服务端指示设置 `next_check_at`，禁止紧循环； +* asset 下载使用 `/repos/{owner}/{repo}/releases/assets/{asset_id}` 与 `Accept: application/octet-stream`，兼容 `200` 内容和 `302` 跳转； +* GitHub 始终使用严格 TLS；asset redirect 仅允许 HTTPS、最多 5 次，每跳解析并校验公网 IP；跨 host 删除 `Authorization`、`Cookie`、`Referer` 和条件请求 header； +* 元数据与下载错误只保留 status、request id、repo、tag、asset 等安全上下文。 + +参考官方文档： + +* [GitHub Releases REST API](https://docs.github.com/en/rest/releases/releases) +* [GitHub Release Assets REST API](https://docs.github.com/en/rest/releases/assets) +* [GitHub REST API 最佳实践](https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api) +* [GitHub REST API Rate Limits](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api) + +未认证公共请求存在严格额度，V2 通过 ETag、串行 scanner、jitter 与服务端退避降低消耗，不承诺大规模仓库轮询。多项目共享仓库缓存留到出现真实规模瓶颈后再设计。 + +#### 2.6.3 `attention` 与 digest 失败边界 + +* 每次 check/sync 都以本次新解析的 target 判断：`target.release_id == applied.release_id` 且 revision 变化时进入 `attention`；禁止用过期的 runtime seen 代替本次 target； +* 管理员同步时必须提交与当前 seen 完全相等的 `confirmed_revision`；状态变化后旧确认自动失效； +* declared digest 与实际 package checksum 不一致：`failed`，不能用 attention 确认绕过； +* 同一 revision 重复点击由部分唯一索引和 lease 双重保证只产生一条 deployment； +* 后续 latest 已推进到不同 release ID 时，不再满足同 Release 替换条件，应转为普通 `update_available` 并按 auto 策略继续；attention 不设计成永久 hold。 + +### 2.7 统一导入、激活与回滚 + +#### 2.7.1 source sync 原子提交 + +source sync 不创建长期 candidate，固定执行以下顺序： + +1. 获取 source lease，快照 source config version、project content version、`RootDir`、`EntryFile`。 +2. 事务外解析并流式下载到临时文件，计算 checksum；临时文件在所有退出路径删除。 +3. 使用快照的 `RootDir + EntryFile` 做真实展开限制、路径与入口校验，得到 manifest。 +4. 先查询相同 project/source identity/revision 的 deployment；存在则不调用 Ingest。 +5. 不存在时调用 `upload.Ingest`，使用现有 Pages upload type 与 `PolicyDedupNewRecord`；upload metadata 的 `Extra` 写入固定 marker 版本、十进制字符串形式的 `pages_project_id` 及可选 `pages_source_id`，供孤儿补偿判断，绝不写 URL 或 token。平台需先修正 dedup 新记录语义：新 record 采用本次请求的业务 metadata，仅从既有 object 继承存储归属 `Bucket`，不能继续复制既有 record 的业务 `Extra`；dedup record 写库失败时也绝不能删除并非本次 Ingest 创建的共享 object。 +6. 最终事务先按 `project -> source -> runtime` 加锁并校验双 version、lease token/expiry；project 锁同时串行化本项目所有 V2 deployment 创建、激活、裁剪与 orphan 判定。 +7. create-or-load 必须使用 GORM `clause.OnConflict{DoNothing: true}`（或等价 `INSERT ... ON CONFLICT DO NOTHING`），再按 `(project_id, source_identity, source_revision)` 查询 winner，禁止依赖普通唯一冲突后继续查询已 aborted 的 PostgreSQL 事务。若冲突仅来自 deployment number 且 revision winner 不存在，则在 project 锁内重新分配编号并有限重试。 +8. 确定目标 deployment 后，将目标 upload 与本次 Ingest upload（若不同）按 ID 升序锁定；目标 upload 必须仍为 `used`。唯一竞争产生的多余 upload 只记录为事务后的补偿目标，禁止在 Pages 事务内调用另起事务的 `upload.Remove`。 +9. 取消旧 active、激活目标 deployment、更新 project active pointer，并更新 runtime applied/seen/status/时间。 +10. 提交后立即补偿未被采用的 upload，再执行严格历史裁剪；事务回滚则补偿本次 Ingest upload。裁剪失败不回滚已成功激活，但必须告警并由下一次裁剪自愈。 + +任何最终事务前的失败都保持旧 active。`created_by` / `trigger_type` 约定： + +| 触发 | `created_by` | `trigger_type` | +| --- | --- | --- | +| 本地上传 | `user:` | `manual_upload` | +| 一次性 URL | `user:` | `manual_url` | +| 持久来源手动 sync | `user:` | `manual_sync` | +| scanner 自动更新 | `system:pages-source-sync` | `scheduled_auto_update` | + +#### 2.7.2 Ingest 补偿与延迟记录恢复 + +`upload.Remove` 当前只会软删除 upload record、调整统计并失效缓存，不会删除底层 object；因此实现与验收不得宣称 defer 调用后物理文件已回收。 + +新创建的 Pages upload record 统一使用以下无密钥 marker；`pages_source_id` 只在持久 source sync 时存在，手动上传与一次性 URL 省略该键： + +```json +{ + "pages_ingest_marker": "pages_deployment_v2", + "pages_project_id": "123", + "pages_source_id": "456" +} +``` + +ID 使用十进制字符串，cleanup 必须严格解析并校验关联归属；marker 不作为权限凭证，只作为“允许进入 Pages 孤儿判定”的一个条件。`project_slug`、归档格式等可由正式模型/Upload 列获得且当前无读取方，不再复制进新 record 的 `Extra`。 + +`openflare_pages_deployment` 由 upload 平台集中定义并导出为保留 type，Pages 与通用 Handler 复用同一常量：通用 `POST /api/v1/upload` 必须拒绝客户端提交该值，通用管理员/用户删除入口及 `upload.Remove` / `RemoveOwned` 也必须拒绝删除该类型；cleanup 候选还必须满足 `user_id == repository.GetSystemUser(ctx).ID`。marker、保留 type、system owner 三项缺一不可，避免普通用户伪造 metadata 后被后台任务误删。 + +V2 采用两层处理： + +1. 立即补偿：只要 Ingest 创建了新 upload record 而最终事务未引用它，就调用 Pages 内部 `removePagesUploadIfUnreferenced`；该函数锁 project（存在时）与 upload、再次确认没有任何 deployment 引用，再调用 `upload.RemoveLockedTx`。补偿错误必须写可告警日志，不能 `_ =` 静默忽略。 +2. 延迟记录补偿：Pages scanner 每轮最多选择 100 条超过 2 小时、状态仍为 `used`、system owner、type 为 `openflare_pages_deployment`、无 deployment 引用且带 V2 Pages marker 的 upload。这覆盖“立即补偿调用本身失败”的恢复路径；PostgreSQL 使用 JSONB 路径、SQLite 使用 `json_extract` 将 marker 纳入 SQL 候选条件，避免存量合法记录长期占满批次。任一条件不满足的记录一律跳过，禁止仅凭 type/时间推断孤儿。 + +`upload.Remove`、`RemoveOwned` 与 Pages 内部删除路径必须共用同一幂等删除原语：事务内锁定包含 deleted 状态的 record，再由 `RemoveLockedTx` 以 `id + status IN (pending, used)` 做 CAS；只有 `RowsAffected == 1` 才递减统计，已 deleted 视为成功 no-op。事务成功后无论本次是否发生状态迁移都失效该 record 的 metadata cache，以便顺带修复前次提交后 cache invalidation 中断；这样立即补偿、延迟补偿、历史裁剪和管理员删除并发时不会重复扣减统计。`Remove` / `RemoveOwned` 在锁内发现保留 type 时返回稳定 domain error，不得调用 `RemoveLockedTx`。 + +cleanup 最终 recheck 与软删除必须在同一数据库临界区完成： + +1. 事务外读取 candidate 快照，严格解析 system owner、marker、`pages_project_id/pages_source_id`；格式错误直接跳过并告警； +2. 事务内统一按 `project -> source（存在时） -> runtime（存在时） -> upload` 加锁；项目或 marker 指向的 source 已不存在属于合法 orphan 场景，应继续检查；只有 source ID 仍存在但其 `project_id` 与 marker 不同才跳过并告警。禁止先锁 upload 再反向读取 runtime； +3. source 存在时若 runtime 有未过期 lease，回滚并跳过；随后锁 upload，再次确认 ID、marker、归属、`used` 状态和 2 小时阈值； +4. 在持有 project/upload 锁的情况下确认不存在任何 deployment 引用该 upload；所有 V2 deployment 创建路径也必须遵循同一锁顺序，避免检查后又插入引用； +5. 通过 upload 平台提供的事务内幂等 `RemoveLockedTx` 完成软删除与统计更新，提交后统一失效 upload metadata cache；业务模块不得直接调用 repository 或改 `w_uploads`； +6. Pages 最终提交若后获得 upload 行锁，必须因 status 已 deleted 而终止；若 deployment 提交先完成，cleanup 在引用检查时跳过。两者竞争时只能有一方成功，绝不允许 deployment 指向 deleted upload。 + +V2 不物理删除 object，也不硬删除 upload record：`PolicyDedupNewRecord` 可能共享 `file_path`，当前平台没有能与并发 dedup 创建原子协调的引用锁/引用计数，先检查再删除仍有竞态。软删除后的 object 和记录保持可识别，待 upload 平台提供安全的统一 blob GC 后回收；Pages 业务包不得直接调用 storage backend。清理失败保留 active 候选供下次重试，并输出数量与错误上下文。 + +同一安全边界也适用于现有 `system:cleanup`：阶段 0 将 pending upload 清理收敛为 `RemoveLockedTx` 的记录级软删除、统计与 cache 失效，停止直接 `backend.Delete(file_path)`。仅增加“是否还有 active record”检查仍无法闭合“检查后并发 dedup 新 record”的竞态，不能作为物理删除依据；所有 upload blob 的物理回收统一留给未来具备引用协调能力的平台 GC。 + +#### 2.7.3 人工激活/回滚硬约束 + +通过现有 activation API 人工激活不同于当前 active ID 的 deployment 时： + +1. 按全局顺序锁 project、当前 source（如有）、runtime 与目标 deployment 的 upload；目标 upload 非 `used` 时拒绝激活； +2. 若存在 source，始终 `config_version + 1` 并清 lease，fence 已排队和正在执行的 source task； +3. 若 `auto_update_enabled=true`，同事务强制改为 false； +4. 目标 deployment identity 等于当前 source identity 时，将 runtime applied 更新为目标 revision/detail；否则清空 applied；若归一后的 seen/applied 仍构成同 Release 替换则保持 `attention`，否则状态为 `idle` 或 `update_available`； +5. 切换 active deployment 后提交； +6. 输出结构化审计日志：actor、project、旧/新 deployment、是否关闭 auto、目标 source type/identity（不含 URL）。 + +重复激活当前 active 视为 no-op，不关闭自动更新。该规则刻意比“只在 identity 不同才关闭”更严格：即使回滚到同一 GitHub source 的旧 revision，下一轮 latest 也可能覆盖人工选择。 + +前端确认框必须明确提示：“激活其它历史部署会终止当前来源任务；若已开启自动更新，将同时关闭自动更新。”成功后同时刷新 project、source 与 deployment queries。 + +#### 2.7.4 `history_count=1` + +手动上传仍保留“先上传 candidate、再人工激活”的现有交互，但裁剪增加 `preserveCandidateID`： + +* 上传完成后保留当前 active 与本次新 candidate；即使 history limit 为 1，也允许临时最多 2 条； +* 再次上传时只保护 active 与最新 candidate，旧 candidate 可被裁剪； +* candidate 激活后执行 strict prune，不再传 preserve ID，恢复总数 `<= history_count`； +* source sync 在同一事务内创建并激活，提交后直接 strict prune； +* prune 删除 deployment/files 后，artifact record 也统一交给 `removePagesUploadIfUnreferenced`；通用文件管理永远不直接删除 Pages 保留类型； +* `history_count<=0` 继续表示不限制。 + +手动 candidate 创建与 source 最终提交都先锁 project 再分配 deployment number，并由 `(project_id, deployment_number)` 唯一索引兜底。这是一项明确的产品例外，不新增 candidate 状态或额外保留配置。 + +preserve/strict prune 每次都在事务内先锁 project，再重新读取 active 与候选集合后决定删除项；禁止沿用事务外快照做删除判断。deployment/files 提交后，待删除 artifact 再交给无引用复检路径软删除。 + +### 2.8 安全与数据面前置修复 + +#### 2.8.1 `RootDir` / `EntryFile` + +统一使用一个严格的逻辑路径规范化函数： + +* `RootDir` 允许空字符串表示归档根目录；非空 RootDir 与 EntryFile 只接受 UTF-8 相对 POSIX 路径，空 EntryFile 由服务层归一为 `index.html`； +* 拒绝绝对路径、`.` / `..` segment、反斜线、Windows drive、NUL/控制字符、引号、分号及超长值； +* 逻辑归档路径用 `path` 处理，不用平台相关 `filepath`；落盘路径仍用 `filepath` 并再次执行目录逃逸检查； +* project 已有 active deployment 时，更新 RootDir/EntryFile 前用现有 deployment file manifest 验证新入口存在；失败保持原配置； +* 阶段 0 先完成严格校验、manifest 验证与 LocalRoot 一致性；阶段 1 随 source DDL 增加 `content_config_version` 后，实质变化再递增该版本并清当前 source lease； +* snapshot 与 rebind 构建 `LocalRoot` 时安全追加规范化 RootDir，确保 Server 检查路径与 OpenResty 实际服务路径一致。 + +归档继续保留现有 common-root 语义：若所有文件共享唯一首层目录，检查与解压都会先剥离该目录，随后再解释 `RootDir`。阶段 0 以测试固化该规则，未来仓库构建产物也必须输出符合相同 artifact contract 的目录结构。 + +#### 2.8.2 Remote SSRF 与 TLS + +`public` 策略： + +* 仅允许 `http` / `https`，最多 5 次 redirect，不使用环境代理； +* 每次连接前解析 host，拒绝 loopback、private、link-local、multicast、unspecified 及其它非公网地址； +* 自定义 `DialContext` 直接连接已校验 IP，不能在校验后再次按 host 解析，防止 DNS rebinding； +* 每次 redirect 重新执行 scheme、host 与 IP 检查； +* HTTPS 严格证书验证，禁止 `InsecureSkipVerify`； +* 设置连接、响应头、整体下载超时，并以实际流量强制 package size 上限。 + +专用 client 通过 `pkg/httppool` 新增的可配置 transport factory 复用连接池参数与 OTel instrumentation，同时显式注入 no-proxy、受控 DialContext 和 TLS policy；不能直接使用当前会读取环境代理的 `DefaultTransport()`。 + +`trusted_internal` 策略是管理员显式选择的信任边界：允许私网目标与自签 TLS，但仍执行 http(s)、redirect、超时、真实大小和归档限制；UI 必须展示醒目风险提示。新 source 默认永远是 `public`。 + +旧 `upload-from-url` 为兼容现有行为，内部映射到共享 downloader 的 trusted-internal 兼容策略，不再保留第二套 HTTP client；新 UI 不再暴露该入口。 + +#### 2.8.3 归档真实限制 + +Server 与 Agent 都必须按实际读取字节执行： + +* 压缩包字节数、单文件展开字节数、总展开字节数、文件数； +* Content-Length/asset size 只用于提前拒绝，不能代替流式上限； +* 拒绝 Zip-Slip、绝对路径、Windows drive、symlink、hardlink、device/特殊条目； +* tar/tar.gz/tar.xz/tar.bz2 检查与解压不得把所有成员 body 物化到内存； +* zip/7z 声明大小必须在实际复制时再次验证； +* Server manifest 中的 `file_count/total_size` 来自实际检查结果。 + +继续复用现有 Pages 设置：压缩包默认 100 MiB、硬上限 2048 MiB、文件数 1000、展开总量按现有规则计算；不新增一组 source 专用大小设置。 + +#### 2.8.4 Agent 流式下载与本地硬上限 + +latest hash 响应扩展为： + +```json +{ + "project_id": 1, + "deployment_id": 2, + "hash": "sha256-hex", + "package_size": 1048576, + "file_count": 128, + "total_size": 8388608 +} +``` + +Agent： + +1. 先读取 metadata，并拒绝超过 Agent 编译期绝对上限的值；绝对上限不得被 Server 响应放大。 +2. 将 package response 流式写入 release 临时文件，使用 `io.LimitedReader` 约束实际压缩字节，并在写入同时计算 SHA-256。 +3. 再次读取 latest metadata；hash/deployment 发生变化时删除临时文件并按现有有限次数重试。 +4. 使用 `pagesarchive.ExtractFile` 的流式实现解压到 `.tmp`，开启文件数、单文件和总量限制；Server metadata 只作为更小的预期上限，仍受本地绝对 cap 约束。 +5. 完整校验、写 marker 后才原子切换 current；失败保留旧 current。 + +编译期绝对上限固定为压缩包 2 GiB、文件数 1000、单文件 8 GiB、总展开 8 GiB，与 Server 当前硬边界一致；Server metadata 只能收紧这些值。`file_count>0 && total_size=0` 是全部零字节文件的合法情况，不能被 limits 的默认值逻辑放大。 + +Agent 继续只访问 Server，不解析 source provenance，也不访问第三方 URL。 + +### 2.9 API 与鉴权 + +沿用当前 Pages/admin action-style 路由；所有接口使用 `apiutil.AdminMiddlewares()`，成功 HTTP 200，错误通过 `response.Abort*` 交给全局 ErrorHandler。 + +| 方法 | 路由 | 语义 | +| --- | --- | --- | +| GET | `/api/v1/d/pages/:id/source` | 返回 discriminated source view；无 source 返回 manual | +| POST | `/api/v1/d/pages/:id/source/update` | 创建或更新 source | +| POST | `/api/v1/d/pages/:id/source/delete` | 幂等切回 manual；deployment/active 保留 | +| POST | `/api/v1/d/pages/:id/source/check` | GitHub 手动检查；Remote 返回稳定 400 | +| POST | `/api/v1/d/pages/:id/source/sync` | Remote/GitHub 同步并强制激活 | + +#### 2.9.1 Source update payload + +Remote： + +```json +{ + "source_type": "remote_url", + "remote_url_set": true, + "remote_url": "https://artifacts.example.com/dist.zip?token=secret", + "remote_network_policy": "public" +} +``` + +GitHub latest： + +```json +{ + "source_type": "github_release", + "repository_url": "https://github.com/owner/repo", + "release_selector": "latest", + "asset_name": "dist.zip", + "auto_update_enabled": false, + "check_interval_minutes": 60 +} +``` + +GitHub tag： + +```json +{ + "source_type": "github_release", + "repository_url": "https://github.com/owner/repo", + "release_selector": "tag", + "release_tag": "v1.2.3", + "asset_name": "dist.zip" +} +``` + +使用 discriminated validation：Remote 不接受 GitHub/auto 字段；tag 不接受开启 auto 或非零 interval；latest 必须没有 tag；模式外已知字段非零即 400。数据库产品默认由 service 归一并显式写入，不依赖 GORM/DB 默认推断。 + +source type 切换时必须在同一事务清空另一 Provider 的全部列：Remote → GitHub 清除完整 `remote_url/network_policy`，GitHub → Remote 清除 repository/selector/tag/asset/auto/interval。禁止只改 `source_type` 而让 query token 或失效配置继续滞留数据库。 + +GitHub source 新建或实质更新成功后，在数据库事务提交后异步投递首次 check；无实质变化不重复投递。队列入队不是数据库事务的一部分，因此入队失败时 source 仍保存成功：响应中的 `check_task=null`、`warning` 给出可重试提示，同时 runtime 标为 failed；用户可点击检查，latest scanner 也会在近期重试。 + +创建或更新 latest source 时先将 `next_check_at` 设为 `now + interval + jitter`；tag 始终为 NULL。首次 check 入队失败时将 latest 的 `next_check_at` 提前到下一轮 scanner，成功 check 则按 interval 重算。首次 check 本身只负责发现版本，不因保存动作隐式发布；若管理员同时开启 auto，后续 scanner 或显式 sync 再执行发布。 + +update 响应： + +```json +{ + "error_msg": "", + "data": { + "source": {}, + "check_task": { + "task_id": "manual_of_pages_source_action_...", + "execution_id": "1234567890", + "action": "check" + }, + "warning": "" + } +} +``` + +#### 2.9.2 Source view + +manual： + +```json +{ + "source_type": "manual" +} +``` + +Remote view 只返回 Remote 有效字段： + +```json +{ + "source_type": "remote_url", + "has_remote_url": true, + "display_url": "https://artifacts.example.com/dist.zip?***", + "remote_network_policy": "public", + "sync_status": "idle", + "last_applied": { + "revision": "sha256-hex", + "label": "dist.zip" + }, + "last_synced_at": "2026-07-19T10:00:00Z", + "last_error": "" +} +``` + +GitHub view： + +```json +{ + "source_type": "github_release", + "github_repository": "owner/repo", + "release_selector": "latest", + "release_tag": "", + "asset_name": "dist.zip", + "auto_update_enabled": false, + "check_interval_minutes": 60, + "sync_status": "update_available", + "update_available": true, + "last_seen": { + "revision": "revision-hex", + "label": "v1.2.3", + "asset_name": "dist.zip" + }, + "last_applied": { + "revision": "revision-hex", + "label": "v1.2.2", + "asset_name": "dist.zip" + }, + "last_checked_at": "2026-07-19T10:00:00Z", + "last_synced_at": "2026-07-18T10:00:00Z", + "next_check_at": "2026-07-19T11:00:00Z", + "last_error": "" +} +``` + +API 不返回 config/content version、lease、ETag、raw detail JSON、GitHub asset URL 或完整 Remote URL。detail 先反序列化为内部 typed struct，再映射为上述安全 view。 + +#### 2.9.3 Action request/receipt + +check 无请求体。普通 sync 的规范请求体为 `{}`；Handler 同时把空 body 的 `io.EOF` 视为默认空请求，避免 `BaseService.post(..., undefined)` 稳定返回 400。只在 attention 确认时提交： + +```json +{ + "confirmed_revision": "revision-hex-currently-shown" +} +``` + +action 成功入队返回： + +```json +{ + "task_id": "manual_of_pages_source_action_...", + "execution_id": "1234567890", + "action": "sync" +} +``` + +Handler 在 `task.DispatchTask` 返回后，按 task ID 读取已先创建的 TaskExecution，并返回 numeric execution ID 的字符串形式。前端复用现有 task execution detail API 轮询 `pending/running/succeeded/failed`，source runtime 不增加 queued 状态。 + +典型错误： + +| 条件 | HTTP | 文案语义 | +| --- | --- | --- | +| payload/模式字段非法 | 400 | 指出当前来源允许的配置 | +| Remote 调用 check | 400 | `远程地址来源不支持检查更新，请使用立即同步` | +| attention 未确认或确认已过期 | 400 | 要求刷新并确认当前 revision | +| 项目/source 不存在 | 404 | 安全的资源不存在提示 | +| source 有有效 lease | 409 | 来源任务正在执行 | +| 入队/数据库内部失败 | 500 | 通用安全提示，底层错误写日志 | + +上述 attention/lease 检查是 Handler 的 best-effort preflight；preflight 与 Worker 获取 lease 之间仍可能发生竞态。竞态中的权威结果由 Worker 的 target revision、lease 与最终事务校验决定，并通过脱敏的 TaskExecution 成功 no-op 或失败结果反馈，API 不承诺把所有异步竞态同步映射成 400/409。 + +#### 2.9.4 旧一次性 URL + +`POST /api/v1/d/pages/:id/deployments/upload-from-url` 在 V2 保留： + +* Swagger description 标记 Deprecated； +* 不创建 source，不写 identity/revision，每次仍创建新的 manual URL candidate； +* 内部复用新的流式 downloader、归档校验和 candidate 裁剪规则； +* 为保持兼容，映射到 trusted-internal 网络策略； +* 新前端移除入口，最早在下一个 major version 才考虑删除。 + +### 2.10 前端方案 + +#### 2.10.1 页面结构 + +当前 `detail/page.tsx` 仅转发 `page-client.tsx`，且 `page-client.tsx` 已接近复杂度阈值。V2 将路由骨架、标题、外层布局与 Suspense 直接移回物理入口 `page.tsx`，再拆出高状态密度组件；禁止继续保留纯转发页面： + +```text +detail/page.tsx + ├── pages-source-card.tsx + ├── pages-source-dialog.tsx + ├── deployment-history.tsx + └── deployment-files-panel.tsx +``` + +六个 source status 的 badge/文案映射直接放在 `pages-source-card.tsx`，不再创建薄的 `pages-source-status.tsx`。现有 `page-client.tsx` 的剩余 query/交互逻辑在拆分后移入对应业务组件，不再作为同名页面容器保留。 + +信息层级参考 Cloudflare Pages 当前项目页，但使用 OpenFlare 现有设计系统实现，不复制品牌视觉： + +1. 顶部项目摘要优先显示当前生产部署、入口路径与关键动作； +2. “部署源”卡片单独表达当前 source、远端游标与同步动作，来源设置不与 deployment 行内操作混杂； +3. “部署历史”展示不可变部署事实与来源快照，当前 active 置顶突出，历史回滚保持显式确认； +4. source dialog 以 manual / Remote URL / GitHub Release 的分步选择呈现；未来新增 repository source 时只增加新的 discriminated step，不改写现有三类表单字段。 + +#### 2.10.2 能力分离 + +Remote 卡片只显示： + +* 脱敏 URL、network policy、最近同步、已应用 revision、最近错误； +* “编辑来源”“同步并发布”“切换回手动”； +* 不显示检查、自动更新、检查间隔或 next check。 + +编辑 Remote 默认 `remote_url_set=false` 并展示只读 masked URL；用户点击“更换地址”后才出现空输入框。`trusted_internal` 需要二次风险提示。 + +GitHub latest 卡片显示检查、同步、自动更新、间隔、远端/已应用版本和 next check。GitHub tag 显示手动检查/同步，隐藏自动更新与周期字段。`attention` 使用 Alert + 确认弹窗，提交卡片当前 revision。 + +source 历史信息与 deployment 历史分工： + +* source 卡片显示当前远端状态； +* deployment 行只显示创建时快照，例如 `GitHub · v1.2.3 · 定时更新`； +* 历史区域明确标注“部署时来源快照”，不重复展示远端最新状态。 + +#### 2.10.3 上传与契约修复 + +* `DeploymentUploadDialog` 移除 URL tab，只保留本地上传； +* 显示项目实际 `root_dir + entry_file`，不再硬编码 `index.html`； +* multipart 只发送 `package`，删除后端未消费的 root/entry 字段； +* `PagesDeployment` 类型删除后端不返回的 `root_dir/entry_file`，增加安全 provenance 字段； +* 兼容 URL service 使用与后端 10 分钟相容的 timeout，直到 UI/API 最终移除； +* deployment query 的 `isError` 单独渲染错误组件，不能降级成“暂无部署”。 + +#### 2.10.4 轮询 + +* 用户 action 拿到 `execution_id` 后轮询现有 TaskExecution；pending/running 继续，succeeded/failed 停止； +* 终态统一 invalid project/source/deployments/files queries；失败展示 TaskExecution 安全文案并重新读取 source `last_error`； +* source status 为 checking/syncing 时，以约 2 秒频率刷新 source； +* GitHub latest 空闲时以低频刷新或在 `next_check_at` 附近刷新，确保 scanner 发现更新后页面无需手动刷新；Remote/tag 空闲时不持续轮询； +* 所有轮询设置前端最长等待时间，超时停止自动请求并提供手动刷新； +* 操作按钮在本地 mutation、TaskExecution pending/running 或 source lease busy 任一条件成立时禁用。 + +在实现任何 Next.js 改动前，先读取 `frontend/node_modules/next/dist/docs/` 中与 App Router、Client Component、数据获取相关的当前版本文档，并遵循项目 shadcn 与页面拆分规范。 + +### 2.11 日志、可观测性与敏感信息 + +source status 不承担详细执行日志。TaskExecution 日志使用稳定阶段前缀： + +```text +[check] [resolve] [download] [verify] [ingest] [activate] [cleanup] +``` + +日志可以记录 source/project ID、repo、tag、asset name、revision 前缀、HTTP status、GitHub request ID、字节数和耗时；不得记录 Remote 原始 URL/query、asset 临时下载 URL、Cookie/Authorization 或响应 body。 + +scanner TaskResult 和结构化日志至少记录： + +* 到期总数、选取数、成功/失败/跳过数； +* 检查 backlog； +* GitHub 403/429 与退避截止时间； +* 自动 sync 投递成功/失败数； +* lease 过期恢复数。 +* orphan 候选、已补偿、仍被引用、lease busy、非法 marker 与失败数。 + +当前仓库没有统一业务 metrics abstraction，V2 不为该功能单独引入一套指标框架；后续接入全局 OTel metrics 时再把上述计数提升为 metrics。 + +### 2.12 关键取舍 + +| 决策 | 采用方案 | 未采用方案与原因 | +| --- | --- | --- | +| runtime project ID | 不冗余，scanner join source | 冗余列需额外一致性维护，且无法消除读取 config 的 join | +| sync 语义 | 固定创建/复用并激活 | `activate=false` 与 history=1 冲突，并扩大 UI/状态机 | +| 状态 | 六态 | 11 态与 TaskExecution 重复，容易卡在中间态 | +| scanner | TaskHandler 内串行 check，自动更新再投 sync | 批量投递 check 会放大 GitHub 并发并需要派发预占状态 | +| fencing | lease + source/project 双 version 最终校验 | 通用 expected revision 是第四套重复 fence；仅 attention 使用精确确认 revision | +| 回滚 | 人工激活其它部署即 fence；auto 强制关闭 | 只靠 UI 提示无法阻止下一轮 latest 覆盖回滚 | +| Remote URL 编辑 | `remote_url_set` 显式保留/替换 | masked URL 回填、空串或省略语义容易误清密钥 | +| GitHub client | 新建窄 `internal/integration/githubrelease` 包，Pages 复用 | 仓库已有多套 feature-local Release 访问，再新增 Pages 私有 client 会继续扩大重复；本阶段不强制迁移旧调用方 | +| orphan upload | Pages 无引用复检软删除 + scanner 延迟记录补偿；物理 blob GC 后续统一建设 | Pages 直接删 storage 违反平台边界且可能误删 dedup 共享对象；通用 upload cleanup 反向依赖 Pages 状态也会破坏模块边界 | +| scanner 批量 | V2 固定 20，并记录 backlog | 现阶段新增系统设置只扩大配置面；出现真实容量瓶颈后再配置化或改延迟任务 | + +--- + +## 3. 具体修改文件清单 (Proposed Changes) + +以下为实施边界；同一阶段可在不改变职责的前提下合并测试文件，不应再拆出只有常量转发的薄文件。 + +### 3.1 后端 Server + +#### [NEW] `internal/model/openflare_pages_source.go` + +* `PagesProjectSource`、`PagesProjectSourceRuntime` 模型与表名。 + +#### [MODIFY] `internal/model/openflare_pages.go` + +* project content version；deployment nullable provenance。 + +#### [NEW] `internal/apps/openflare/pages/source.go` + +* discriminated input/view、默认值、identity、脱敏、source CRUD。 + +#### [NEW] `internal/apps/openflare/pages/source_provider.go` + +* Provider 内部接口、Remote public/trusted downloader、共享流式下载结果。 + +#### [NEW] `internal/integration/githubrelease/client.go` + +* 可复用的 GitHub latest/tag/asset client、ETag、rate limit、受控 redirect 与安全错误。 + +#### [MODIFY] `pkg/httppool/httppool.go` + +* 增加保留现有池参数/OTel 的可配置 transport factory，供 SSRF-safe DialContext、no-proxy 与 TLS policy 使用；默认 client 行为不变。 + +#### [NEW] `internal/apps/openflare/pages/source_sync.go` + +* runtime 状态、lease、check/sync service、原子 create-or-load/activate、补偿。 + +#### [NEW] `internal/apps/openflare/pages/source_tasks.go` + +* scanner/action TaskMeta、payload validation、Handler 与限量 orphan record reconciliation。 + +#### [MODIFY] `internal/apps/openflare/pages/logics.go` + +* 统一 deployment 创建/激活；created_by/provenance；人工回滚硬约束；candidate/strict prune；Pages artifact 无引用复检删除。 + +#### [MODIFY] `internal/apps/openflare/pages/helpers.go` + +* 严格 RootDir/EntryFile、真实归档限制、manifest 与 Agent metadata；移除对 Pages 保留 type 的通用 `upload.Remove` 调用。 + +#### [MODIFY] `internal/apps/openflare/pages/download_url.go` + +* 旧 URL 导入改用共享 downloader，删除独立不安全 client 分叉；无扩展名归档使用至少 512 字节 format sniff。 + +#### [MODIFY] `internal/apps/openflare/pages/routers.go` + +* 5 个 source Handler；从 OAuth context 获取真实 user ID；Swagger；旧 URL deprecated。 + +#### [MODIFY] `internal/apps/openflare/pages/errs.go` + +* source、Provider、lease 与 attention 的稳定安全错误文案。 + +#### [MODIFY] `internal/router/v1/openflare/register_pages.go` + +* 注册 5 条 source 路由；不在顶层 router 直接挂业务 Handler。 + +#### [MODIFY] `internal/task/handlers/register.go` + +* 显式注册 scanner/action Handler 与 TaskMeta。 + +#### [MODIFY] `internal/apps/upload/ingest/helpers.go` + +* `PolicyDedupNewRecord` 的新 record 使用本次请求业务 metadata，并只继承既有 object 的 `Bucket`，保证每条业务记录的归属信息独立。 +* 将“本次是否真实写入 object”作为持久化失败补偿的显式条件；dedup record 创建/统计失败不得删除复用的既有 `file_path`。 + +#### [MODIFY] `internal/apps/upload/ingest/remove.go`、`internal/apps/upload/exports.go` + +* 增加仅供已持有 upload 行锁的事务编排使用的 `RemoveLockedTx`，统一 CAS 软删除与统计更新；`Remove` / `RemoveOwned` 也改用该原语并将已删除视为 no-op，但对 Pages 保留 type 返回稳定拒绝错误。 +* 提供提交后调用的 cache invalidation 出口，禁止调用方直接依赖 upload cache 子包。 + +#### [MODIFY] `internal/repository/upload.go` + +* upload 软删除更新增加 active status 条件并返回 `RowsAffected`，保证只有一次真实状态迁移会触发统计扣减。 + +#### [MODIFY] `internal/apps/upload/task/cleanup.go` + +* pending upload 清理改用幂等软删除原语并停止直接删除可能被 dedup record 共享的 object；物理 blob GC 不在 Pages V2 内伪实现。 + +#### [MODIFY] `internal/apps/upload/handler/routers.go`、`file_management.go`、`logics.go` + +* 通用上传 API 拒绝创建 Pages 保留 type，通用管理员/用户文件删除拒绝移除该 type，并同步更新 Swagger 错误说明。 + +#### [MODIFY] `internal/apps/upload/shared/constants.go`、`errs.go` + +* 在 upload 平台集中定义保留 type `openflare_pages_deployment` 与安全错误，由 `exports.go` 导出并供 Pages/Handler 共用。 + +#### [NEW/MODIFY TEST] Pages、upload 与 model tests + +* `source_test.go`、`source_provider_test.go`、`source_sync_test.go`、`internal/integration/githubrelease/client_test.go` 与 `pkg/httppool/httppool_test.go`； +* `logics_test.go`、`routers_test.go`、`internal/apps/upload/ingest/helpers_test.go`、`internal/apps/upload/ingest/remove_test.go`、`internal/apps/upload/handler/routers_test.go`、`internal/apps/upload/task/tasks_test.go`； +* model/迁移测试覆盖 NULL 部分索引与双版本。 + +### 3.2 数据库迁移 + +#### [NEW] PostgreSQL + +* `internal/db/migrator/goose/postgres/202607190001_add_pages_source_runtime.sql` +* `internal/db/migrator/goose/postgres/202607190002_seed_pages_source_scan.sql` + +#### [NEW] SQLite + +* `internal/db/migrator/goose/sqlite/202607190001_add_pages_source_runtime.sql` +* `internal/db/migrator/goose/sqlite/202607190002_seed_pages_source_scan.sql` + +版本号若已被其它分支占用，实施时只顺延编号，不改变 DDL/DML 拆分。 + +SQLite `0001` 的 Down 必须通过重建受影响表完整移除新增列、约束与索引，不接受只删除 source/runtime 表却遗留 project/deployment 列的伪回滚；PostgreSQL 与 SQLite 都需要真实 Up/Down/Up 验证。 + +### 3.3 Agent、协议与归档库 + +#### [MODIFY] `pkg/pagesarchive/entry.go`、`path.go`、`inspect.go`、`list.go`、`extract.go`、`limits.go` + +* tar family 流式检查/解压、实际字节限制、特殊条目拒绝与 ExtractFile。 + +#### [MODIFY] `pkg/protocol/agent.go` + +* latest hash response 增加 package/file/total size。 + +#### [MODIFY] `internal/apps/openflare/agent/routers.go` + +* 返回 Agent 限额 metadata，保持 package 流式响应。 + +#### [MODIFY] `internal/apps/agent/httpclient/client.go` + +* package 下载从 `[]byte` 改为受限流式写入。 + +#### [MODIFY] `internal/apps/agent/sync/service.go`、`pages.go` + +* client interface、临时文件、hash race 复核、ExtractFile 与本地绝对 cap。 + +#### [MODIFY] `internal/apps/openflare/config_version/pages_snapshot.go`、`internal/apps/openflare/pages/rebind.go` + +* `LocalRoot` 安全追加规范化 RootDir。 + +### 3.4 前端 Web + +#### [MODIFY] `frontend/lib/services/openflare/types.ts` + +* source union、action receipt、safe provenance；清理 deployment/upload 漂移字段。 + +#### [MODIFY] `frontend/lib/services/openflare/pages.service.ts`、`index.ts` + +* 5 个 source API；本地 upload 只发 package；兼容 URL timeout。 + +#### [NEW] `frontend/app/(main)/pages/detail/components/pages-source-card.tsx` + +* source query、能力分离、状态与 TaskExecution 轮询。 + +#### [NEW] `frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx` + +* Remote/GitHub discriminated form、URL replacement 与 trusted warning。 + +#### [NEW] `frontend/app/(main)/pages/detail/components/deployment-history.tsx` + +* deployment query、激活/删除、历史 provenance 和回滚提示。 + +#### [NEW] `frontend/app/(main)/pages/detail/components/deployment-files-panel.tsx` + +* deployment files query 与错误态。 + +#### [MODIFY] `frontend/app/(main)/pages/detail/page.tsx` + +* 直接承载路由骨架、标题、布局与 Suspense，并组合上述组件。 + +#### [DELETE] `frontend/app/(main)/pages/detail/page-client.tsx` + +* 拆分完成后移除纯转发容器；业务逻辑归入 page 与就近子组件。 + +#### [MODIFY] `frontend/app/(main)/pages/components/deployment-upload-dialog.tsx`、`pages-utils.ts` + +* 本地上传单模式、真实入口显示与 source query key。 + +#### [MODIFY/NEW TEST] 前端测试 + +* `frontend/tests/openflare/pages-service.test.ts` +* `frontend/tests/openflare/pages-source-ui.test.tsx` + +### 3.5 文档与生成物（代码实施时） + +#### [MODIFY] + +* `docs/design/pages-design.md` +* `docs/design/index.md` +* `docs/design/architecture.md` +* `docs/guide/pages-usage.md` +* `README.md` +* `docs/changelog/index.md` 的 `[Unreleased]`（仅实际代码变更后） + +API 实现后运行 `make swagger` 更新 `docs/docs.go`、`docs/swagger.json`、`docs/swagger.yaml`，禁止手工编辑生成物。本计划文档不加入 `docs/config.ts` 的用户文档导航。 + +--- + +## 4. 验证计划 (Verification Plan) + +### 4.1 数据库与模型 + +* PostgreSQL/SQLite 空库 Up、现有库升级和 Down； +* source/runtime 同事务 1:1，无 source 项目保持 manual； +* config/runtime 无审核中已删除的宽表冗余列； +* identity 变化 reset runtime，query token/策略变化保留 cursor； +* 持久 source 同 revision 并发只产生一条 deployment； +* PostgreSQL/SQLite 的 create-or-load 使用 conflict-do-nothing 后可在同一事务读取 winner；deployment number 独立冲突能有限重试； +* 手动/旧 URL identity/revision 为 NULL，可重复导入相同包； +* `(project_id, deployment_number)` 并发唯一； +* schedule seed 无固定 ID、可重复 Up，Down 不影响其它 schedule。 + +建议： + +```bash +go test ./internal/db/migrator ./internal/model +``` + +### 4.2 source、Provider 与 API + +* Remote/GitHub discriminated validation、默认值与非法模式字段； +* Remote `remote_url_set` 新建/保留/替换全部分支； +* Remote/GitHub 双向切换会清空非当前 Provider 列，旧 query token 不残留； +* Remote 无扩展名 tar 与常见合法扩展名归档均能识别，redirect/Content-Disposition 不污染安全 label； +* URL/repository 规范化、identity 不包含凭据； +* source view、错误、task payload、deployment meta、日志均无 query token； +* GitHub latest/tag endpoint、exact asset、asset 缺失的有限候选列表； +* ETag/304、200/302 asset、403/404/429、Retry-After/reset； +* asset redirect 仅 HTTPS、最多 5 次、逐跳公网 IP 校验，并移除敏感 header/Referer； +* digest 正确、不匹配、缺失； +* Remote check 稳定 400；source delete 幂等且保留 active/history； +* 普通上传 API 提交保留 type `openflare_pages_deployment` 时稳定拒绝；管理员/用户通用删除 API 对该类型同样返回稳定冲突，已被 deployment 引用与暂时无引用两种情况都不能绕过； +* source save 后 check 入队成功与“配置已保存但入队失败”的部分成功语义； +* Handler 全部使用 Abort*，Swagger 声明实际 Failure 状态。 +* Provider/TaskResult/TaskExecution 的 `error_message/log/result` 不含 Remote query token、临时下载 URL 或敏感 header。 + +GitHub/Remote 使用 `httptest.Server` 或可注入 RoundTripper，不在普通单测访问真实外网。 + +### 4.3 并发、状态与回滚 + +* 六态转换，无 queued/succeeded runtime 残留； +* 重复 action 只有一个 lease owner，其余 no-op； +* lease 到期但 token 尚未被接管时，旧 Worker 仍无法续租/写终态/激活；scanner 随后恢复 failed； +* source 编辑/删除期间的旧任务不能提交； +* 下载期间修改 RootDir/EntryFile，旧 content version 不能激活； +* scanner 单个 source 失败不阻塞后续 source；304 且已有待更新 revision 时仍能自动投递； +* scanner 看到 revision A、sync 执行时 latest 已变为 B：`target_revision` 不匹配，A/B 均不被该任务激活，并提前下一次检查； +* auto=false 只更新 cursor，不下载；tag 不进入 scanner； +* 人工激活其它 deployment 时 fence 在途任务、关闭 auto，并正确更新/清空 applied； +* 同 identity 旧 revision 回滚同样关闭 auto；重复激活当前 active 不关闭； +* same-release replacement 进入 attention，错误 confirmed revision 不能绕过；digest mismatch 进入 failed； +* attention 后 latest 推进到不同 release ID 时恢复普通 update_available/auto 路径，不形成永久 hold； +* source sync 任何失败均保留旧 active。 +* 复用已有 deployment 或人工激活时，目标 upload 已 deleted 会被拒绝，不产生失效 active pointer。 + +### 4.4 历史与上传补偿 + +* history=1 手动上传后保留 active + 最新 candidate；再次上传替换旧 candidate； +* candidate 激活后严格恢复 1 条；source sync 激活后只保留新 active； +* Ingest 成功但最终事务失败时 upload record 被软删除并记录补偿结果； +* `PolicyDedupNewRecord` 复用 object 时，新 record 保留本次请求的 Pages marker/项目/source metadata，仅继承既有 object 的 `Bucket`； +* 故障注入 dedup record 创建或统计失败，既有 upload/object 仍可读取，只有本次真实新写 object 才允许在持久化失败时删除； +* 立即 `removePagesUploadIfUnreferenced` 失败时 record 保持 `used`；scanner 隔离期后只处理带 V2 marker 且无 deployment 引用的 Pages upload； +* 普通用户即使伪造保留 type、完整 marker 和真实项目/source ID，也因 HTTP 保留 type 校验与 system owner 双重条件不会进入 cleanup； +* 对普通 upload，`Remove`、`RemoveOwned` 并发删除同一 record 时只有一次 `RowsAffected=1`，上传统计只扣减一次；Pages 补偿/cleanup/历史裁剪共享同一断言； +* source lease 未过期时 cleanup 跳过；过期 Worker 不能续租，后续最终提交因 lease/upload 状态失败； +* cleanup 与最终部署事务按同一 `project -> source -> runtime -> upload` 顺序竞争，分别覆盖 cleanup 先提交、deployment 先提交两种结果，断言不存在指向 deleted upload 的 deployment； +* 管理员通用删除与人工激活并发时删除请求被保留 type 策略拒绝，激活只可能看到 `used` target； +* marker 缺失/损坏或仍存在的 source 归属不一致时只告警并跳过；项目/source 已删除时按合法 orphan 继续补偿； +* PostgreSQL JSONB 与 SQLite `json_extract` 候选查询只选 V2 marker，并以 100 条为批次上限； +* cleanup 不物理删除 object、不硬删记录，dedup 共享 file path 不受影响； +* `system:cleanup` 对过期 pending record 只做一次软删除/统计扣减，不再调用 storage backend 删除共享 object； +* record 补偿失败保留重试候选并产生可告警日志。 + +### 4.5 归档、网络与 Agent + +* chunked 实际 body 超限、伪造 Content-Length、单文件/总量/文件数超限； +* zip/tar/tar.gz/tar.xz/tar.bz2/7z 的合法包与压缩炸弹； +* Zip-Slip、绝对路径、Windows drive、symlink/hardlink/special entry； +* public 拒绝 loopback/private/link-local、redirect 到私网和 DNS rebinding； +* public 拒绝自签 TLS，trusted_internal 仅显式选择后允许； +* Agent 大包不进入 `[]byte`，实际下载超过 metadata/absolute cap 即失败； +* metadata 被伪造为超大值时本地 cap 仍生效； +* hash race 不切换 current；解压失败保留旧 current； +* RootDir + EntryFile 从上传检查到 OpenResty root/index 端到端一致； +* 自动激活后 Agent 通过现有周期 latest 对账拉取，无需发布新的主配置。 + +建议： + +```bash +go test ./pkg/pagesarchive \ + ./internal/apps/openflare/pages \ + ./internal/apps/openflare/agent \ + ./internal/apps/agent/httpclient \ + ./internal/apps/agent/sync \ + ./internal/apps/upload/ingest \ + ./internal/apps/upload/handler \ + ./internal/apps/upload/task +``` + +### 4.6 前端 + +* 三类 source view 与 latest/tag/Remote 能力分离； +* Remote masked URL 不会被保存回后端，replace 开关 payload 正确； +* attention revision 确认、trusted warning、回滚关闭 auto 提示； +* TaskExecution pending/running/terminal 轮询与超时； +* scanner 自动更新后的低频 source 刷新； +* deployment query 错误不显示为空列表； +* 本地上传只发 package，显示真实入口； +* 用户输入完整 URL 时只存在于受控输入框与本地 form state；保存后不重新渲染原值，也不进入 masked view、toast、console、日志或测试 snapshot。输入框使用 password/reveal 交互。 + +建议： + +```bash +cd frontend +pnpm exec vitest run +pnpm exec tsc --noEmit +pnpm lint +``` + +### 4.7 最终项目门禁与手工验收 + +代码完成后： + +```bash +make swagger +make prettier +make code-check +``` + +手工最小矩阵： + +1. 本地上传 → candidate → 激活 → Agent current 更新； +2. public Remote 同步相同/不同内容，验证复用与新 deployment； +3. trusted internal Remote 的私网/自签场景与风险提示； +4. GitHub tag 检查、同步； +5. GitHub latest 检查到更新，auto off 只提示；auto on 自动激活； +6. 自动更新后人工回滚，验证 auto 被关闭且下次 scanner 不打回 latest； +7. asset 同 Release 替换，验证 attention 与精确 revision 确认； +8. Server/Worker 在下载、Ingest、最终提交不同阶段中断，验证旧 active、lease 恢复与 orphan upload 记录补偿。 + +--- + +## 5. 分阶段实施 + +### 阶段 0：安全与一致性前置（独立合并/发布） + +* RootDir/EntryFile 严格路径与 LocalRoot 端到端一致； +* Server 真实归档限制和 tar 流式实现； +* Agent 流式下载、ExtractFile、metadata 与绝对 cap； +* history=1 candidate preserve/strict prune； +* created_by 真实 actor； +* upload dedup 新记录 metadata/对象所有权语义、幂等 CAS 软删除、HTTP 保留 type 创建/删除边界；现有 Pages prune/补偿切换到无引用复检删除并记录错误，`system:cleanup` 停止不安全的物理 object 删除；物理 blob GC 保持独立平台后续项。 + +验收：不引入 source 表/API 的情况下，现有本地上传与旧 URL 路径全部回归；大包内存与路径安全测试通过。 + +### 阶段 1：数据模型与 Remote 手动同步 + +* `0001` DDL migration、model、source CRUD/view； +* runtime 六态、lease、config/content fence； +* Remote public/trusted downloader； +* action sync、原子 create-or-load/activate； +* Remote source card/dialog；旧 URL UI 移除但 API 保留。 + +验收：Remote 只能手动同步并发布；相同内容幂等；失败保持旧 active；立即补偿失败可观测；URL 全链路脱敏。 + +本阶段 API/DTO 变化完成后立即运行 `make swagger` 并将生成物纳入阶段验证，不把 Swagger 漂移累积到阶段 4。 + +### 阶段 2：GitHub 手动检查与同步 + +* GitHub client、latest/tag、ETag、asset/digest、rate limit； +* action check、首次异步 check、update_available； +* attention + exact confirmed revision； +* latest 的 auto 开关在本阶段不出现在 UI，服务层拒绝 `auto_update_enabled=true`； +* GitHub source UI 和 deployment provenance。 + +验收：tag/latest 均可手动检查/同步；auto 仍保持 false；asset 替换不能未经确认激活。 + +本阶段 API/DTO 变化后再次运行 `make swagger`，保证阶段 2 可独立合并发布。 + +### 阶段 3：latest scanner 与自动更新 + +* scanner task、`0002` schedule seed、过期 lease 恢复； +* serial batch、jitter、退避、自动 sync dispatch； +* marker 白名单 orphan record 延迟补偿与统一锁顺序竞态测试； +* 自动更新开关与 interval； +* 人工回滚关闭 auto 的完整前后端交互； +* TaskExecution 与后台状态轮询。 + +验收：auto 默认 false；开启后只对 latest 生效；回滚不会被自动打回；多 source 失败隔离和 backlog 日志可用。 + +本阶段 API/DTO 变化后再次运行 `make swagger`，阶段 4 只做最终一致性复检。 + +### 阶段 4：文档、生成物与全门禁 + +* 同步 Pages design/architecture/guide/README 与中文 changelog； +* 生成 Swagger； +* 运行前后端测试、`make prettier`、`make code-check`； +* 按 §4.7 完成手工矩阵并记录未覆盖的真实外部场景。 + +每个阶段只提交本阶段明确路径并独立验证；阶段 0 不与后续 source 功能捆绑成一个大提交。 + +--- + +## 6. 完成定义 + +只有同时满足以下条件，V2 才视为实现完成： + +* 三类来源能力边界与 UI/API 完全一致； +* 数据模型为 config/runtime 分离的瘦表，状态不超过六态； +* source sync 无 candidate/activate 分支，revision 幂等且原子激活； +* 人工回滚可以硬性终止自动覆盖； +* Remote/GitHub/日志/task/deployment 均无密钥泄漏； +* Server 与 Agent 都执行真实字节上限，Agent 不再全量内存下载； +* history=1、本地 candidate、source sync 和清理补偿均有自动化覆盖； +* PostgreSQL、SQLite、后端、Agent、前端及项目门禁全部通过； +* 实际代码、Swagger、中文设计/使用文档和 changelog 同步。 diff --git a/docs/plan/index.md b/docs/plan/index.md index 974203a5..07ee80d4 100644 --- a/docs/plan/index.md +++ b/docs/plan/index.md @@ -19,6 +19,7 @@ * [访问日志 cache_status 明细可见](./20260718-access-log-cache-status.md)：上报 `$upstream_cache_status`，明细展示命中/回源/未缓存三态。 * [边缘缓存默认 static 策略](./20260718-edge-cache-static-default.md)：开启缓存默认仅静态扩展名；存量 url→all。 * [访问日志 IP 明细 Tab](./20260719-access-log-ip-tab.md)：第三 Tab 按 IP 聚合列表（时间窗/流量/2xx 比例）；IP 情报迁入独立详情；日志详情仅请求字段。 +* [Pages 项目部署源与 GitHub Releases 自动更新 V2](./20260719-pages-source-sync-v2.md)：统一 Remote URL / GitHub Release 来源、不可变部署、自动检查更新与安全回滚，并预留独立仓库构建 Provider 边界。 ## 使用建议 From 4e8ec232641c0210a28bfdbade97bc57383ef4d4 Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 16:42:45 +0800 Subject: [PATCH 2/8] =?UTF-8?q?fix(pages):=20=E6=94=B6=E7=B4=A7=E9=83=A8?= =?UTF-8?q?=E7=BD=B2=E5=8C=85=E4=B8=8E=20Agent=20=E5=90=8C=E6=AD=A5?= =?UTF-8?q?=E8=BE=B9=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 完成 V2 Phase 0 安全与一致性前置：统一真实归档限额、流式拉取、候选裁剪、保留上传删除语义及 Pages 路由引用锁。 --- docs/changelog/index.md | 4 + docs/docs.go | 45 +- docs/swagger.json | 45 +- docs/swagger.yaml | 28 + internal/apps/agent/httpclient/client.go | 131 +++- internal/apps/agent/httpclient/client_test.go | 109 +++ internal/apps/agent/sync/pages.go | 659 +++++++++++++++--- internal/apps/agent/sync/pages_stream_test.go | 388 +++++++++++ internal/apps/agent/sync/service.go | 8 +- internal/apps/agent/sync/service_test.go | 146 +++- internal/apps/openflare/agent/routers.go | 9 +- .../config_version/pages_snapshot.go | 30 +- .../config_version/pages_snapshot_test.go | 14 +- internal/apps/openflare/pages/download_url.go | 2 +- internal/apps/openflare/pages/errs.go | 2 + internal/apps/openflare/pages/helpers.go | 115 ++- internal/apps/openflare/pages/logics.go | 432 ++++++++---- internal/apps/openflare/pages/logics_test.go | 236 ++++++- .../apps/openflare/pages/package_metadata.go | 64 ++ .../openflare/pages/package_metadata_test.go | 81 +++ internal/apps/openflare/pages/rebind.go | 29 +- internal/apps/openflare/pages/rebind_test.go | 9 +- internal/apps/openflare/pages/routers.go | 24 +- internal/apps/openflare/pages/routers_test.go | 100 +++ internal/apps/openflare/proxy_route/logics.go | 61 ++ .../apps/openflare/proxy_route/logics_test.go | 55 +- internal/apps/upload/exports.go | 24 +- .../apps/upload/handler/file_management.go | 13 +- internal/apps/upload/handler/routers.go | 5 + internal/apps/upload/handler/routers_test.go | 92 +++ internal/apps/upload/ingest/errors.go | 3 + internal/apps/upload/ingest/helpers.go | 27 +- internal/apps/upload/ingest/ingest_test.go | 235 ++++++- internal/apps/upload/ingest/remove.go | 68 +- internal/apps/upload/shared/constants.go | 2 + internal/apps/upload/shared/errs.go | 1 + internal/apps/upload/task/cleanup.go | 34 +- internal/apps/upload/task/tasks_test.go | 30 +- internal/repository/upload.go | 17 +- pkg/pagesarchive/entry.go | 63 +- pkg/pagesarchive/extract.go | 209 ++++-- pkg/pagesarchive/inspect.go | 282 +++++--- pkg/pagesarchive/list.go | 221 ++---- pkg/pagesarchive/path.go | 92 ++- pkg/pagesarchive/security_test.go | 499 +++++++++++++ pkg/protocol/agent.go | 6 + pkg/protocol/agent_test.go | 15 + 47 files changed, 4005 insertions(+), 759 deletions(-) create mode 100644 internal/apps/agent/httpclient/client_test.go create mode 100644 internal/apps/agent/sync/pages_stream_test.go create mode 100644 internal/apps/openflare/pages/package_metadata.go create mode 100644 internal/apps/openflare/pages/package_metadata_test.go create mode 100644 internal/apps/openflare/pages/routers_test.go create mode 100644 pkg/pagesarchive/security_test.go diff --git a/docs/changelog/index.md b/docs/changelog/index.md index fec2320f..db529121 100644 --- a/docs/changelog/index.md +++ b/docs/changelog/index.md @@ -30,6 +30,10 @@ sidebar: false - WAF 规则编辑器支持为节点自定义显示名称，并从节点库拖放到画布指定位置添加节点。 - WAF 规则画布支持右键删除节点或连线，并屏蔽浏览器默认右键菜单。 +### 修复 + +- 修复 Pages 部署包路径校验、归档展开限额、历史版本裁剪、代理路由绑定与 Agent 下载过程中的安全和一致性问题；大包改为流式处理，部署入口、旧版目录切换、保留版本及上传记录在并发场景下更加可靠。 + ## [v3.4.0] - 2026-07-19 ### 新增 diff --git a/docs/docs.go b/docs/docs.go index 6e2473b2..df744b22 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -3738,6 +3738,12 @@ const docTemplate = `{ "schema": { "$ref": "#/definitions/response.Any" } + }, + "409": { + "description": "系统保留类型或存储只读", + "schema": { + "$ref": "#/definitions/response.Any" + } } } } @@ -12535,6 +12541,12 @@ const docTemplate = `{ "$ref": "#/definitions/response.Any" } }, + "409": { + "description": "系统保留类型或存储只读", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, "500": { "description": "内部错误", "schema": { @@ -12729,6 +12741,12 @@ const docTemplate = `{ "schema": { "$ref": "#/definitions/response.Any" } + }, + "409": { + "description": "系统保留类型或存储只读", + "schema": { + "$ref": "#/definitions/response.Any" + } } } } @@ -14574,11 +14592,20 @@ const docTemplate = `{ "deployment_id": { "type": "integer" }, + "file_count": { + "type": "integer" + }, "hash": { "type": "string" }, + "package_size": { + "type": "integer" + }, "project_id": { "type": "integer" + }, + "total_size": { + "type": "integer" } } }, @@ -16568,9 +16595,15 @@ const docTemplate = `{ "observability.AccessLogView": { "type": "object", "properties": { + "bytes_sent": { + "type": "integer" + }, "cache_status": { "type": "string" }, + "created_at": { + "type": "string" + }, "host": { "type": "string" }, @@ -16595,6 +16628,12 @@ const docTemplate = `{ "remote_addr": { "type": "string" }, + "request_length": { + "type": "integer" + }, + "request_time_ms": { + "type": "integer" + }, "status_code": { "type": "integer" }, @@ -19310,7 +19349,8 @@ const docTemplate = `{ "block", "ip_match", "geo_match", - "pow" + "pow", + "ua_check" ], "x-enum-varnames": [ "RuleNodeStart", @@ -19318,7 +19358,8 @@ const docTemplate = `{ "RuleNodeBlock", "RuleNodeIPMatch", "RuleNodeGeoMatch", - "RuleNodePoW" + "RuleNodePoW", + "RuleNodeUACheck" ] }, "waf.RulePosition": { diff --git a/docs/swagger.json b/docs/swagger.json index 7355be74..efdab2a3 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -3731,6 +3731,12 @@ "schema": { "$ref": "#/definitions/response.Any" } + }, + "409": { + "description": "系统保留类型或存储只读", + "schema": { + "$ref": "#/definitions/response.Any" + } } } } @@ -12528,6 +12534,12 @@ "$ref": "#/definitions/response.Any" } }, + "409": { + "description": "系统保留类型或存储只读", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, "500": { "description": "内部错误", "schema": { @@ -12722,6 +12734,12 @@ "schema": { "$ref": "#/definitions/response.Any" } + }, + "409": { + "description": "系统保留类型或存储只读", + "schema": { + "$ref": "#/definitions/response.Any" + } } } } @@ -14567,11 +14585,20 @@ "deployment_id": { "type": "integer" }, + "file_count": { + "type": "integer" + }, "hash": { "type": "string" }, + "package_size": { + "type": "integer" + }, "project_id": { "type": "integer" + }, + "total_size": { + "type": "integer" } } }, @@ -16561,9 +16588,15 @@ "observability.AccessLogView": { "type": "object", "properties": { + "bytes_sent": { + "type": "integer" + }, "cache_status": { "type": "string" }, + "created_at": { + "type": "string" + }, "host": { "type": "string" }, @@ -16588,6 +16621,12 @@ "remote_addr": { "type": "string" }, + "request_length": { + "type": "integer" + }, + "request_time_ms": { + "type": "integer" + }, "status_code": { "type": "integer" }, @@ -19303,7 +19342,8 @@ "block", "ip_match", "geo_match", - "pow" + "pow", + "ua_check" ], "x-enum-varnames": [ "RuleNodeStart", @@ -19311,7 +19351,8 @@ "RuleNodeBlock", "RuleNodeIPMatch", "RuleNodeGeoMatch", - "RuleNodePoW" + "RuleNodePoW", + "RuleNodeUACheck" ] }, "waf.RulePosition": { diff --git a/docs/swagger.yaml b/docs/swagger.yaml index 6bbf8815..a52e0600 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -760,10 +760,16 @@ definitions: properties: deployment_id: type: integer + file_count: + type: integer hash: type: string + package_size: + type: integer project_id: type: integer + total_size: + type: integer type: object github_com_Rain-kl_Wavelet_pkg_protocol.WAFIPGroup: properties: @@ -2085,8 +2091,12 @@ definitions: type: object observability.AccessLogView: properties: + bytes_sent: + type: integer cache_status: type: string + created_at: + type: string host: type: string id: @@ -2103,6 +2113,10 @@ definitions: type: string remote_addr: type: string + request_length: + type: integer + request_time_ms: + type: integer status_code: type: integer user_agent: @@ -3908,6 +3922,7 @@ definitions: - ip_match - geo_match - pow + - ua_check type: string x-enum-varnames: - RuleNodeStart @@ -3916,6 +3931,7 @@ definitions: - RuleNodeIPMatch - RuleNodeGeoMatch - RuleNodePoW + - RuleNodeUACheck waf.RulePosition: properties: x: @@ -6157,6 +6173,10 @@ paths: description: 文件不存在 schema: $ref: '#/definitions/response.Any' + "409": + description: 系统保留类型或存储只读 + schema: + $ref: '#/definitions/response.Any' security: - SessionCookie: [] summary: 删除文件 @@ -11589,6 +11609,10 @@ paths: description: 未登录 schema: $ref: '#/definitions/response.Any' + "409": + description: 系统保留类型或存储只读 + schema: + $ref: '#/definitions/response.Any' "500": description: 内部错误 schema: @@ -11622,6 +11646,10 @@ paths: description: 文件不存在 schema: $ref: '#/definitions/response.Any' + "409": + description: 系统保留类型或存储只读 + schema: + $ref: '#/definitions/response.Any' security: - SessionCookie: [] summary: 删除我的文件 diff --git a/internal/apps/agent/httpclient/client.go b/internal/apps/agent/httpclient/client.go index 34e1398a..4e4e2efb 100644 --- a/internal/apps/agent/httpclient/client.go +++ b/internal/apps/agent/httpclient/client.go @@ -1,9 +1,13 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + // Package httpclient provides an authenticated HTTP client for the agent. package httpclient import ( "context" "encoding/json" + "errors" "fmt" "io" "net/http" @@ -13,6 +17,8 @@ import ( edgehttp "github.com/Rain-kl/Wavelet/internal/apps/edge/httpclient" ) +const pagesControlResponseMaxBytes = int64(64 * 1024) + // Client is a HTTP client used by the agent to communicate with the control plane server. type Client struct { base *edgehttp.Client @@ -98,23 +104,43 @@ func (c *Client) GetPagesDeploymentHash(ctx context.Context, deploymentID uint) return resp.Data.Hash, nil } -// DownloadPagesDeploymentPackage downloads the deployment package for the given Pages deployment ID. -func (c *Client) DownloadPagesDeploymentPackage(ctx context.Context, deploymentID uint) ([]byte, error) { - res, err := c.base.DoRaw(ctx, http.MethodGet, fmt.Sprintf("/api/v1/agent/pages/deployments/%d/package", deploymentID), nil) - if err != nil { - return nil, err - } - defer func() { _ = res.Body.Close() }() - if res.StatusCode != http.StatusOK { - return nil, edgehttp.ReadHTTPError(res) - } - return io.ReadAll(res.Body) +// DownloadPagesDeploymentPackage streams the deployment package into dst while +// enforcing maxBytes against both advertised and actual response sizes. +func (c *Client) DownloadPagesDeploymentPackage( + ctx context.Context, + deploymentID uint, + dst io.Writer, + maxBytes int64, +) (int64, error) { + return c.downloadPagesPackage( + ctx, + fmt.Sprintf("/api/v1/agent/pages/deployments/%d/package", deploymentID), + dst, + maxBytes, + ) } // GetPagesProjectLatestHash returns the active deployment package hash for a Pages project. func (c *Client) GetPagesProjectLatestHash(ctx context.Context, projectID uint) (*protocol.PagesProjectLatestHashResponse, error) { + res, err := c.base.DoRaw( + ctx, + http.MethodGet, + fmt.Sprintf("/api/v1/agent/pages/projects/%d/latest/hash", projectID), + nil, + ) + if err != nil { + return nil, err + } + defer func() { _ = res.Body.Close() }() + body, err := readPagesControlResponse(res, pagesControlResponseMaxBytes) + if err != nil { + return nil, err + } + if res.StatusCode != http.StatusOK { + return nil, edgehttp.ReadBodyError(body, res.Status) + } resp := protocol.APIResponse[protocol.PagesProjectLatestHashResponse]{} - if err := c.base.GetJSON(ctx, fmt.Sprintf("/api/v1/agent/pages/projects/%d/latest/hash", projectID), &resp); err != nil { + if err := json.Unmarshal(body, &resp); err != nil { return nil, err } if err := edgehttp.APIError(resp.ErrorMsg); err != nil { @@ -123,17 +149,86 @@ func (c *Client) GetPagesProjectLatestHash(ctx context.Context, projectID uint) return &resp.Data, nil } -// DownloadPagesProjectLatestPackage downloads the active deployment package for a Pages project. -func (c *Client) DownloadPagesProjectLatestPackage(ctx context.Context, projectID uint) ([]byte, error) { - res, err := c.base.DoRaw(ctx, http.MethodGet, fmt.Sprintf("/api/v1/agent/pages/projects/%d/latest/package", projectID), nil) +// DownloadPagesProjectLatestPackage streams the active deployment package into +// dst while enforcing maxBytes against both advertised and actual sizes. +func (c *Client) DownloadPagesProjectLatestPackage( + ctx context.Context, + projectID uint, + dst io.Writer, + maxBytes int64, +) (int64, error) { + return c.downloadPagesPackage( + ctx, + fmt.Sprintf("/api/v1/agent/pages/projects/%d/latest/package", projectID), + dst, + maxBytes, + ) +} + +func (c *Client) downloadPagesPackage( + ctx context.Context, + path string, + dst io.Writer, + maxBytes int64, +) (int64, error) { + if dst == nil { + return 0, errors.New("pages package destination is required") + } + if maxBytes <= 0 { + return 0, errors.New("pages package byte limit must be positive") + } + res, err := c.base.DoRaw(ctx, http.MethodGet, path, nil) if err != nil { - return nil, err + return 0, err } defer func() { _ = res.Body.Close() }() if res.StatusCode != http.StatusOK { - return nil, edgehttp.ReadHTTPError(res) + body, readErr := readPagesControlResponse(res, pagesControlResponseMaxBytes) + if readErr != nil { + return 0, readErr + } + return 0, edgehttp.ReadBodyError(body, res.Status) } - return io.ReadAll(res.Body) + return copyPagesPackageResponse(dst, res, maxBytes) +} + +func readPagesControlResponse(res *http.Response, maxBytes int64) ([]byte, error) { + if res.ContentLength > maxBytes { + return nil, fmt.Errorf( + "pages control response Content-Length %d exceeds limit %d", + res.ContentLength, + maxBytes, + ) + } + limited := &io.LimitedReader{R: res.Body, N: maxBytes + 1} + body, err := io.ReadAll(limited) + if err != nil { + return nil, fmt.Errorf("read pages control response: %w", err) + } + if int64(len(body)) > maxBytes { + return nil, fmt.Errorf("pages control response body exceeds limit %d", maxBytes) + } + return body, nil +} + +func copyPagesPackageResponse(dst io.Writer, res *http.Response, maxBytes int64) (int64, error) { + if res.ContentLength > maxBytes { + return 0, fmt.Errorf( + "pages package Content-Length %d exceeds limit %d", + res.ContentLength, + maxBytes, + ) + } + + limited := &io.LimitedReader{R: res.Body, N: maxBytes + 1} + written, err := io.Copy(dst, limited) + if err != nil { + return written, fmt.Errorf("stream pages package: %w", err) + } + if written > maxBytes { + return written, fmt.Errorf("pages package body exceeds limit %d", maxBytes) + } + return written, nil } // SetToken updates the authentication token used for API requests. diff --git a/internal/apps/agent/httpclient/client_test.go b/internal/apps/agent/httpclient/client_test.go new file mode 100644 index 00000000..613c2dda --- /dev/null +++ b/internal/apps/agent/httpclient/client_test.go @@ -0,0 +1,109 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package httpclient + +import ( + "bytes" + "context" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestDownloadPagesProjectLatestPackageRejectsChunkedBodyOverLimit(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + if flusher, ok := w.(http.Flusher); ok { + flusher.Flush() + } + _, _ = io.WriteString(w, "123456") + })) + defer server.Close() + + client := New(server.URL, "test-token", time.Second) + var dst bytes.Buffer + written, err := client.DownloadPagesProjectLatestPackage( + context.Background(), + 7, + &dst, + 4, + ) + if err == nil || !strings.Contains(err.Error(), "body exceeds limit") { + t.Fatalf("DownloadPagesProjectLatestPackage(chunked, limit=4) error = %v, want body limit error", err) + } + if written != 5 { + t.Errorf("DownloadPagesProjectLatestPackage(chunked, limit=4) written = %d, want 5", written) + } +} + +func TestCopyPagesPackageResponseRejectsAdvertisedContentLengthBeforeWrite(t *testing.T) { + response := &http.Response{ + Body: io.NopCloser(strings.NewReader("123456")), + ContentLength: 6, + } + var dst bytes.Buffer + written, err := copyPagesPackageResponse(&dst, response, 4) + if err == nil || !strings.Contains(err.Error(), "Content-Length") { + t.Fatalf("copyPagesPackageResponse(Content-Length=6, limit=4) error = %v, want Content-Length limit error", err) + } + if written != 0 || dst.Len() != 0 { + t.Errorf("copyPagesPackageResponse(Content-Length=6, limit=4) wrote (%d, %d buffered), want no writes", written, dst.Len()) + } +} + +func TestCopyPagesPackageResponseRejectsForgedSmallContentLength(t *testing.T) { + response := &http.Response{ + Body: io.NopCloser(strings.NewReader("123456")), + ContentLength: 2, + } + var dst bytes.Buffer + written, err := copyPagesPackageResponse(&dst, response, 4) + if err == nil || !strings.Contains(err.Error(), "body exceeds limit") { + t.Fatalf("copyPagesPackageResponse(forged Content-Length=2, limit=4) error = %v, want body limit error", err) + } + if written != 5 { + t.Errorf("copyPagesPackageResponse(forged Content-Length=2, limit=4) written = %d, want 5", written) + } +} + +func TestDownloadPagesProjectLatestPackageBoundsChunkedErrorResponse(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusBadRequest) + if flusher, ok := w.(http.Flusher); ok { + flusher.Flush() + } + _, _ = io.WriteString(w, strings.Repeat("x", int(pagesControlResponseMaxBytes+1))) + })) + defer server.Close() + + client := New(server.URL, "test-token", time.Second) + var dst bytes.Buffer + _, err := client.DownloadPagesProjectLatestPackage(context.Background(), 7, &dst, 1024) + if err == nil || !strings.Contains(err.Error(), "control response body exceeds limit") { + t.Fatalf("DownloadPagesProjectLatestPackage(large chunked 400) error = %v, want bounded response error", err) + } + if dst.Len() != 0 { + t.Errorf("DownloadPagesProjectLatestPackage(large chunked 400) wrote %d package bytes, want 0", dst.Len()) + } +} + +func TestGetPagesProjectLatestHashBoundsChunkedMetadataResponse(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + if flusher, ok := w.(http.Flusher); ok { + flusher.Flush() + } + _, _ = io.WriteString(w, strings.Repeat("x", int(pagesControlResponseMaxBytes+1))) + })) + defer server.Close() + + client := New(server.URL, "test-token", time.Second) + _, err := client.GetPagesProjectLatestHash(context.Background(), 7) + if err == nil || !strings.Contains(err.Error(), "control response body exceeds limit") { + t.Fatalf("GetPagesProjectLatestHash(large chunked metadata) error = %v, want bounded response error", err) + } +} diff --git a/internal/apps/agent/sync/pages.go b/internal/apps/agent/sync/pages.go index 67ab219a..5364ec4d 100644 --- a/internal/apps/agent/sync/pages.go +++ b/internal/apps/agent/sync/pages.go @@ -1,3 +1,6 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + // Package sync applies control-plane configuration to the local agent runtime. package sync @@ -20,9 +23,13 @@ import ( ) const ( - pagesDirPerm = 0o755 - pagesFilePerm = 0o644 - pagesManifestFilePerm = 0o644 + pagesDirPerm = 0o755 + pagesFilePerm = 0o644 + pagesManifestFilePerm = 0o644 + agentPagesMaxPackageBytes = int64(2 * 1024 * 1024 * 1024) + agentPagesMaxFiles = 1000 + agentPagesMaxFileBytes = int64(8 * 1024 * 1024 * 1024) + agentPagesMaxTotalBytes = int64(8 * 1024 * 1024 * 1024) // pagesLatestPullAttempts covers a race where the active deployment changes // between the hash probe and the package download. pagesLatestPullAttempts = 2 @@ -45,6 +52,11 @@ type pagesProjectRef struct { Checksum string } +type pagesPackageLimits struct { + PackageBytes int64 + Extraction pagesarchive.Limits +} + type pagesDeploymentMarker struct { ProjectID uint `json:"project_id"` DeploymentID uint `json:"deployment_id,omitempty"` @@ -191,10 +203,11 @@ func (s *Service) ensurePagesProject(ctx context.Context, snapshot *state.Snapsh if err != nil { return fmt.Errorf("fetch Pages project %d latest hash: %w", projectID, err) } - hash := strings.TrimSpace(latest.Hash) - if hash == "" { - return fmt.Errorf("pages project %d latest hash is empty", projectID) + limits, err := validatePagesPackageMetadata(projectID, latest) + if err != nil { + return err } + hash := strings.TrimSpace(latest.Hash) effective := pagesProjectRef{ ProjectID: projectID, DeploymentID: latest.DeploymentID, @@ -211,44 +224,65 @@ func (s *Service) ensurePagesProject(ctx context.Context, snapshot *state.Snapsh return nil } - packageBytes, err := s.client.DownloadPagesProjectLatestPackage(ctx, projectID) + packagePath, got, err := s.downloadPagesProjectPackage(ctx, projectID, latest, limits.PackageBytes) if err != nil { return fmt.Errorf("download Pages project %d latest package: %w", projectID, err) } - got := checksumBytes(packageBytes) // Re-probe latest after download to detect activation races. - // Accept the package only when its content hash still matches latest. + // A deployment-id-only change is still a latest-pointer race even when + // deduplication makes both deployments share the same package hash. verify, err := s.client.GetPagesProjectLatestHash(ctx, projectID) if err != nil { + _ = os.Remove(packagePath) return fmt.Errorf("re-fetch Pages project %d latest hash: %w", projectID, err) } - verifyHash := strings.TrimSpace(verify.Hash) - if verifyHash == "" { - return fmt.Errorf("pages project %d latest hash is empty", projectID) + if _, err := validatePagesPackageMetadata(projectID, verify); err != nil { + _ = os.Remove(packagePath) + return err } - if got != verifyHash { + if !samePagesPackageMetadata(latest, verify) { + _ = os.Remove(packagePath) lastErr = fmt.Errorf( - "pages project %d package/hash race: downloaded %s, latest now %s (attempt %d/%d)", - projectID, got, verifyHash, attempt+1, pagesLatestPullAttempts, + "pages project %d latest metadata changed during download: deployment %d/%s -> %d/%s (attempt %d/%d)", + projectID, + latest.DeploymentID, + strings.TrimSpace(latest.Hash), + verify.DeploymentID, + strings.TrimSpace(verify.Hash), + attempt+1, + pagesLatestPullAttempts, ) - slog.Warn("pages latest package race, retrying", + slog.Warn("pages latest metadata race, retrying", + "project_id", projectID, + "before_deployment_id", latest.DeploymentID, + "before_hash", strings.TrimSpace(latest.Hash), + "after_deployment_id", verify.DeploymentID, + "after_hash", strings.TrimSpace(verify.Hash), + "attempt", attempt+1, + ) + continue + } + if got != hash { + _ = os.Remove(packagePath) + lastErr = fmt.Errorf( + "pages project %d package hash mismatch: downloaded %s, expected %s (attempt %d/%d)", + projectID, got, hash, attempt+1, pagesLatestPullAttempts, + ) + slog.Warn("pages latest package hash mismatch, retrying", "project_id", projectID, "downloaded_hash", got, - "latest_hash", verifyHash, + "expected_hash", hash, "attempt", attempt+1, ) continue } - effective = pagesProjectRef{ - ProjectID: projectID, - DeploymentID: verify.DeploymentID, - Checksum: got, - } releaseDir = pagesProjectReleaseDir(s.pagesDir, projectID, got) - if err := extractPagesPackage(packageBytes, releaseDir, effective); err != nil { - return err + extractErr := extractPagesPackageFile(packagePath, releaseDir, effective, limits.Extraction, latest) + _ = os.Remove(packagePath) + if extractErr != nil { + return extractErr } if err := switchPagesProjectCurrentDir(s.pagesDir, projectID, releaseDir); err != nil { return err @@ -264,6 +298,145 @@ func (s *Service) ensurePagesProject(ctx context.Context, snapshot *state.Snapsh return fmt.Errorf("pages project %d latest pull failed", projectID) } +func validatePagesPackageMetadata( + projectID uint, + metadata *protocol.PagesProjectLatestHashResponse, +) (pagesPackageLimits, error) { + if metadata == nil { + return pagesPackageLimits{}, fmt.Errorf("pages project %d latest metadata is missing", projectID) + } + if metadata.ProjectID != projectID { + return pagesPackageLimits{}, fmt.Errorf( + "pages project %d latest metadata has project id %d", + projectID, + metadata.ProjectID, + ) + } + if metadata.DeploymentID == 0 { + return pagesPackageLimits{}, fmt.Errorf("pages project %d latest deployment id is missing", projectID) + } + if strings.TrimSpace(metadata.Hash) == "" { + return pagesPackageLimits{}, fmt.Errorf("pages project %d latest hash is empty", projectID) + } + if metadata.PackageSize <= 0 { + return pagesPackageLimits{}, fmt.Errorf("pages project %d package size must be positive", projectID) + } + if metadata.PackageSize > agentPagesMaxPackageBytes { + return pagesPackageLimits{}, fmt.Errorf( + "pages project %d package size %d exceeds agent limit %d", + projectID, + metadata.PackageSize, + agentPagesMaxPackageBytes, + ) + } + if metadata.FileCount <= 0 { + return pagesPackageLimits{}, fmt.Errorf("pages project %d file count must be positive", projectID) + } + if metadata.FileCount > agentPagesMaxFiles { + return pagesPackageLimits{}, fmt.Errorf( + "pages project %d file count %d exceeds agent limit %d", + projectID, + metadata.FileCount, + agentPagesMaxFiles, + ) + } + if metadata.TotalSize < 0 { + return pagesPackageLimits{}, fmt.Errorf("pages project %d total size cannot be negative", projectID) + } + if metadata.TotalSize > agentPagesMaxTotalBytes { + return pagesPackageLimits{}, fmt.Errorf( + "pages project %d total size %d exceeds agent limit %d", + projectID, + metadata.TotalSize, + agentPagesMaxTotalBytes, + ) + } + + // pagesarchive treats zero limits as defaults. A one-byte extraction guard + // plus the exact post-extraction manifest check below preserves the valid + // case of one or more zero-byte files while still enforcing total_size=0. + extractedBytes := metadata.TotalSize + if extractedBytes == 0 { + extractedBytes = 1 + } + maxFileBytes := extractedBytes + if maxFileBytes > agentPagesMaxFileBytes { + maxFileBytes = agentPagesMaxFileBytes + } + + return pagesPackageLimits{ + PackageBytes: metadata.PackageSize, + Extraction: pagesarchive.Limits{ + MaxFiles: metadata.FileCount, + MaxFileBytes: maxFileBytes, + MaxTotalBytes: extractedBytes, + }, + }, nil +} + +func samePagesPackageMetadata( + before *protocol.PagesProjectLatestHashResponse, + after *protocol.PagesProjectLatestHashResponse, +) bool { + if before == nil || after == nil { + return false + } + return before.ProjectID == after.ProjectID && + before.DeploymentID == after.DeploymentID && + strings.TrimSpace(before.Hash) == strings.TrimSpace(after.Hash) && + before.PackageSize == after.PackageSize && + before.FileCount == after.FileCount && + before.TotalSize == after.TotalSize +} + +func (s *Service) downloadPagesProjectPackage( + ctx context.Context, + projectID uint, + metadata *protocol.PagesProjectLatestHashResponse, + maxBytes int64, +) (packagePath string, hash string, err error) { + releasesRoot := filepath.Join(s.pagesDir, "projects", fmt.Sprintf("%d", projectID), "releases") + if err := os.MkdirAll(releasesRoot, pagesDirPerm); err != nil { + return "", "", err + } + packageFile, err := os.CreateTemp(releasesRoot, ".package-*.tmp") + if err != nil { + return "", "", err + } + packagePath = packageFile.Name() + keep := false + defer func() { + if closeErr := packageFile.Close(); err == nil && closeErr != nil { + err = closeErr + } + if !keep || err != nil { + _ = os.Remove(packagePath) + packagePath = "" + } + }() + + hasher := sha256.New() + written, err := s.client.DownloadPagesProjectLatestPackage( + ctx, + projectID, + io.MultiWriter(packageFile, hasher), + maxBytes, + ) + if err != nil { + return "", "", err + } + if written != metadata.PackageSize { + return "", "", fmt.Errorf( + "pages project %d package size %d does not match metadata %d", + projectID, + written, + metadata.PackageSize, + ) + } + keep = true + return packagePath, hex.EncodeToString(hasher.Sum(nil)), nil +} + // cleanupPagesProjectStaleReleases keeps only keepHash under projects/{id}/releases. // Must be called only after the keepHash release is ready and current points at it. func cleanupPagesProjectStaleReleases(baseDir string, projectID uint, keepHash string) error { @@ -372,40 +545,258 @@ type pagesDeploymentSource struct { Checksum string `json:"checksum"` } -func extractPagesPackage(packageBytes []byte, releaseDir string, project pagesProjectRef) error { - tmpDir := releaseDir + ".tmp" - _ = os.RemoveAll(tmpDir) - if err := os.MkdirAll(tmpDir, pagesDirPerm); err != nil { +func extractPagesPackageFile( + packagePath string, + releaseDir string, + project pagesProjectRef, + limits pagesarchive.Limits, + expected *protocol.PagesProjectLatestHashResponse, +) error { + if err := os.MkdirAll(filepath.Dir(releaseDir), pagesDirPerm); err != nil { return err } - format, err := pagesarchive.DetectFormat("", packageBytes) + stagingDir, err := os.MkdirTemp( + filepath.Dir(releaseDir), + "."+filepath.Base(releaseDir)+"-*.tmp", + ) if err != nil { - _ = os.RemoveAll(tmpDir) - return fmt.Errorf("detect Pages package format: %w", err) + return err } - // Control plane already inspected and accepted this package. - if err := pagesarchive.ExtractBytes(packageBytes, format, tmpDir, pagesarchive.ExtractOptions{ + cleanupStaging := true + defer func() { + if cleanupStaging { + removePagesStagingUnlessCurrent(stagingDir, pagesCurrentDirFromRelease(releaseDir)) + } + }() + + if err := pagesarchive.ExtractFile(packagePath, "", stagingDir, pagesarchive.ExtractOptions{ StripCommonRoot: true, - EnforceLimits: false, + EnforceLimits: true, + Limits: limits, }); err != nil { - _ = os.RemoveAll(tmpDir) return fmt.Errorf("extract Pages package: %w", err) } - if err := writePagesMarker(tmpDir, project); err != nil { - _ = os.RemoveAll(tmpDir) + if err := validateExtractedPagesMetadata(stagingDir, expected); err != nil { return err } - _ = os.RemoveAll(releaseDir) - return os.Rename(tmpDir, releaseDir) + if err := writePagesMarker(stagingDir, project); err != nil { + return err + } + if err := promotePagesRelease(stagingDir, releaseDir, project); err != nil { + return err + } + cleanupStaging = false + return nil } -func switchPagesProjectCurrentDir(baseDir string, projectID uint, releaseDir string) error { - currentDir := pagesProjectCurrentDir(baseDir, projectID) - previousDir := currentDir + ".previous" - _ = os.RemoveAll(previousDir) +func validateExtractedPagesMetadata( + dir string, + expected *protocol.PagesProjectLatestHashResponse, +) error { + if expected == nil { + return nil + } + fileCount := 0 + totalSize := int64(0) + err := filepath.WalkDir(dir, func(path string, entry os.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() { + return nil + } + info, err := entry.Info() + if err != nil { + return err + } + if !info.Mode().IsRegular() { + return fmt.Errorf("pages extracted entry is not a regular file: %s", path) + } + fileCount++ + if fileCount > agentPagesMaxFiles { + return fmt.Errorf("pages extracted file count exceeds agent limit %d", agentPagesMaxFiles) + } + if info.Size() < 0 || info.Size() > agentPagesMaxTotalBytes-totalSize { + return fmt.Errorf("pages extracted size exceeds agent limit %d", agentPagesMaxTotalBytes) + } + totalSize += info.Size() + return nil + }) + if err != nil { + return fmt.Errorf("validate extracted Pages package: %w", err) + } + if fileCount != expected.FileCount || totalSize != expected.TotalSize { + return fmt.Errorf( + "pages extracted metadata mismatch: got %d files/%d bytes, expected %d files/%d bytes", + fileCount, + totalSize, + expected.FileCount, + expected.TotalSize, + ) + } + return nil +} + +func promotePagesRelease(stagingDir string, releaseDir string, project pagesProjectRef) error { + return promotePagesReleaseWithCopy(stagingDir, releaseDir, project, copyPagesDir) +} + +func promotePagesReleaseWithCopy( + stagingDir string, + releaseDir string, + project pagesProjectRef, + copyDir func(string, string) error, +) error { + currentDir := pagesCurrentDirFromRelease(releaseDir) + defer removePagesStagingUnlessCurrent(stagingDir, currentDir) + currentUsesRelease, err := pagesCurrentTargetsRelease(currentDir, releaseDir) + if err != nil { + return err + } + if !currentUsesRelease { + if err := os.RemoveAll(releaseDir); err != nil { + return err + } + return os.Rename(stagingDir, releaseDir) + } + + // A same-hash repair cannot remove releaseDir while current still resolves + // through it. Keep traffic on the fully validated staging tree, rebuild the + // canonical release, then atomically point current back to the canonical path. + if err := switchPagesCurrentDir(currentDir, stagingDir, os.Rename); err != nil { + return fmt.Errorf("switch Pages current to repair staging: %w", err) + } + backupDir := stagingDir + ".previous" + if err := os.Rename(releaseDir, backupDir); err != nil { + restoreErr := switchPagesCurrentDir(currentDir, releaseDir, os.Rename) + return errors.Join( + fmt.Errorf("move previous Pages release aside: %w", err), + restoreErr, + ) + } + + rollback := func(cause error) error { + var rollbackErrors []error + rollbackErrors = append(rollbackErrors, cause) + if err := os.RemoveAll(releaseDir); err != nil { + rollbackErrors = append(rollbackErrors, fmt.Errorf("remove failed Pages release repair: %w", err)) + } + if err := os.Rename(backupDir, releaseDir); err != nil { + rollbackErrors = append(rollbackErrors, fmt.Errorf("restore previous Pages release: %w", err)) + return errors.Join(rollbackErrors...) + } + if err := switchPagesCurrentDir(currentDir, releaseDir, os.Rename); err != nil { + rollbackErrors = append(rollbackErrors, fmt.Errorf("restore previous Pages current target: %w", err)) + } + return errors.Join(rollbackErrors...) + } + + if err := copyDir(stagingDir, releaseDir); err != nil { + return rollback(fmt.Errorf("copy repaired Pages release: %w", err)) + } + if !pagesProjectReleaseReady(releaseDir, project) { + return rollback(errors.New("repaired Pages release is not ready")) + } + if err := switchPagesCurrentDir(currentDir, releaseDir, os.Rename); err != nil { + return rollback(fmt.Errorf("switch Pages current to repaired release: %w", err)) + } + if err := os.RemoveAll(backupDir); err != nil { + slog.Warn("failed to remove previous Pages release", "path", backupDir, "error", err) + } + if err := os.RemoveAll(stagingDir); err != nil { + slog.Warn("failed to remove Pages repair staging", "path", stagingDir, "error", err) + } + return nil +} + +func pagesCurrentDirFromRelease(releaseDir string) string { + return filepath.Join(filepath.Dir(filepath.Dir(releaseDir)), "current") +} + +func pagesCurrentTargetsRelease(currentDir string, releaseDir string) (bool, error) { + if _, err := os.Lstat(currentDir); err != nil { + if os.IsNotExist(err) { + return false, nil + } + return false, err + } + currentInfo, err := os.Stat(currentDir) + if err != nil { + if os.IsNotExist(err) { + return false, nil + } + return false, fmt.Errorf("stat Pages current target: %w", err) + } + releaseInfo, err := os.Stat(releaseDir) + if err != nil { + if os.IsNotExist(err) { + return false, nil + } + return false, err + } + return os.SameFile(currentInfo, releaseInfo), nil +} + +func removePagesStagingUnlessCurrent(stagingDir string, currentDir string) { + currentUsesStaging, err := pagesCurrentTargetsRelease(currentDir, stagingDir) + if err == nil && currentUsesStaging { + slog.Error("preserving Pages staging because current still references it", "path", stagingDir) + return + } + if removeErr := os.RemoveAll(stagingDir); removeErr != nil { + slog.Warn("failed to remove Pages staging", "path", stagingDir, "error", removeErr) + } +} + +func verifyPagesCurrentTarget(currentDir string, releaseDir string) error { + currentInfo, err := os.Stat(currentDir) + if err != nil { + return fmt.Errorf("stat Pages current target: %w", err) + } + releaseInfo, err := os.Stat(releaseDir) + if err != nil { + return fmt.Errorf("stat Pages release target: %w", err) + } + if !os.SameFile(currentInfo, releaseInfo) { + return fmt.Errorf("pages current target does not resolve to release %s", releaseDir) + } + return nil +} + +func switchPagesCurrentDir( + currentDir string, + releaseDir string, + rename func(string, string) error, +) error { + return switchPagesCurrentDirWithOps(currentDir, releaseDir, rename, os.Symlink) +} + +func switchPagesCurrentDirWithOps( + currentDir string, + releaseDir string, + rename func(string, string) error, + symlink func(string, string) error, +) error { if err := os.MkdirAll(filepath.Dir(currentDir), pagesDirPerm); err != nil { return err } + currentInfo, currentErr := os.Lstat(currentDir) + if currentErr != nil && !os.IsNotExist(currentErr) { + return currentErr + } + if currentErr == nil && currentInfo.Mode()&os.ModeSymlink == 0 { + return fallbackCopyPagesCurrentDir(currentDir, releaseDir, rename) + } + + previousTarget := "" + hadPrevious := currentErr == nil + if hadPrevious { + var err error + previousTarget, err = os.Readlink(currentDir) + if err != nil { + return err + } + } relTarget, err := filepath.Rel(filepath.Dir(currentDir), releaseDir) if err != nil { @@ -413,44 +804,123 @@ func switchPagesProjectCurrentDir(baseDir string, projectID uint, releaseDir str } tmpSymlink := currentDir + ".tmp" - _ = os.Remove(tmpSymlink) - - symlinkErr := os.Symlink(relTarget, tmpSymlink) - if symlinkErr != nil { - return fallbackCopyPagesCurrentDir(currentDir, previousDir, releaseDir) - } - _ = os.Remove(tmpSymlink) - - if _, err := os.Lstat(currentDir); err == nil { - if err := os.Rename(currentDir, previousDir); err != nil { - return err - } - } - - if err := os.Symlink(relTarget, currentDir); err != nil { - if _, restoreErr := os.Lstat(previousDir); restoreErr == nil { - _ = os.Rename(previousDir, currentDir) - } + if err := os.Remove(tmpSymlink); err != nil && !os.IsNotExist(err) { return err } - _ = os.RemoveAll(previousDir) + if err := symlink(relTarget, tmpSymlink); err != nil { + _ = os.Remove(tmpSymlink) + return fallbackCopyPagesCurrentDir(currentDir, releaseDir, rename) + } + defer func() { _ = os.Remove(tmpSymlink) }() + if err := verifyPagesCurrentTarget(tmpSymlink, releaseDir); err != nil { + return err + } + if err := rename(tmpSymlink, currentDir); err != nil { + return err + } + if err := verifyPagesCurrentTarget(currentDir, releaseDir); err != nil { + rollbackErr := rollbackPagesCurrentSymlink( + currentDir, + previousTarget, + hadPrevious, + rename, + ) + return errors.Join(err, rollbackErr) + } return nil } -func fallbackCopyPagesCurrentDir(currentDir, previousDir, releaseDir string) error { - if _, err := os.Lstat(currentDir); err == nil { - if err := os.Rename(currentDir, previousDir); err != nil { - return err - } - } - if err := copyPagesDir(releaseDir, currentDir); err != nil { - _ = os.RemoveAll(currentDir) - if _, restoreErr := os.Lstat(previousDir); restoreErr == nil { - _ = os.Rename(previousDir, currentDir) - } +func fallbackCopyPagesCurrentDir( + currentDir string, + releaseDir string, + rename func(string, string) error, +) error { + stagingDir := currentDir + ".copy.tmp" + previousDir := currentDir + ".previous" + if err := os.RemoveAll(stagingDir); err != nil { return err } - _ = os.RemoveAll(previousDir) + if err := copyPagesDir(releaseDir, stagingDir); err != nil { + _ = os.RemoveAll(stagingDir) + return err + } + if err := os.RemoveAll(previousDir); err != nil { + _ = os.RemoveAll(stagingDir) + return err + } + + hadPrevious := false + if _, err := os.Lstat(currentDir); err == nil { + if err := rename(currentDir, previousDir); err != nil { + _ = os.RemoveAll(stagingDir) + return err + } + hadPrevious = true + } else if !os.IsNotExist(err) { + _ = os.RemoveAll(stagingDir) + return err + } + if err := rename(stagingDir, currentDir); err != nil { + var restoreErr error + if hadPrevious { + restoreErr = rename(previousDir, currentDir) + } + _ = os.RemoveAll(stagingDir) + return errors.Join(err, restoreErr) + } + if err := os.RemoveAll(previousDir); err != nil { + slog.Warn("failed to remove previous Pages current directory", "path", previousDir, "error", err) + } + return nil +} + +func switchPagesProjectCurrentDir(baseDir string, projectID uint, releaseDir string) error { + return switchPagesProjectCurrentDirWithRename(baseDir, projectID, releaseDir, os.Rename) +} + +func switchPagesProjectCurrentDirWithRename( + baseDir string, + projectID uint, + releaseDir string, + rename func(string, string) error, +) error { + return switchPagesCurrentDir(pagesProjectCurrentDir(baseDir, projectID), releaseDir, rename) +} + +func rollbackPagesCurrentSymlink( + currentDir string, + previousTarget string, + hadPrevious bool, + rename func(string, string) error, +) error { + if !hadPrevious { + if err := os.Remove(currentDir); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove unverified Pages current symlink: %w", err) + } + return nil + } + rollbackSymlink := currentDir + ".rollback.tmp" + if err := os.Remove(rollbackSymlink); err != nil && !os.IsNotExist(err) { + return err + } + if err := os.Symlink(previousTarget, rollbackSymlink); err != nil { + return err + } + defer func() { _ = os.Remove(rollbackSymlink) }() + if err := rename(rollbackSymlink, currentDir); err != nil { + return fmt.Errorf("restore previous Pages current symlink: %w", err) + } + gotTarget, err := os.Readlink(currentDir) + if err != nil { + return fmt.Errorf("verify restored Pages current symlink: %w", err) + } + if gotTarget != previousTarget { + return fmt.Errorf( + "restored Pages current symlink target %q does not match %q", + gotTarget, + previousTarget, + ) + } return nil } @@ -467,24 +937,30 @@ func copyPagesDir(sourceDir string, targetDir string) error { if entry.IsDir() { return os.MkdirAll(targetPath, pagesDirPerm) } - input, err := os.Open(sourcePath) //nolint:gosec // sourcePath is under managed PagesDir walk root - if err != nil { - return err - } - defer func() { _ = input.Close() }() - if err := os.MkdirAll(filepath.Dir(targetPath), pagesDirPerm); err != nil { - return err - } - output, err := os.OpenFile(targetPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, pagesFilePerm) //nolint:gosec // targetPath is under managed PagesDir walk root - if err != nil { - return err - } - defer func() { _ = output.Close() }() - _, err = io.Copy(output, input) - return err + return copyPagesFile(sourcePath, targetPath) }) } +func copyPagesFile(sourcePath string, targetPath string) error { + input, err := os.Open(sourcePath) //nolint:gosec // sourcePath is under managed PagesDir walk root + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(targetPath), pagesDirPerm); err != nil { + _ = input.Close() + return err + } + output, err := os.OpenFile(targetPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, pagesFilePerm) //nolint:gosec // targetPath is under managed PagesDir walk root + if err != nil { + _ = input.Close() + return err + } + _, copyErr := io.Copy(output, input) + outputCloseErr := output.Close() + inputCloseErr := input.Close() + return errors.Join(copyErr, outputCloseErr, inputCloseErr) +} + func markerMatches(dir string, project pagesProjectRef) bool { data, err := os.ReadFile(filepath.Join(dir, ".openflare-pages.json")) //nolint:gosec // dir is managed PagesDir if err != nil { @@ -515,8 +991,3 @@ func pagesProjectCurrentDir(baseDir string, projectID uint) string { func pagesProjectReleaseDir(baseDir string, projectID uint, checksum string) string { return filepath.Join(baseDir, "projects", fmt.Sprintf("%d", projectID), "releases", checksum) } - -func checksumBytes(data []byte) string { - sum := sha256.Sum256(data) - return hex.EncodeToString(sum[:]) -} diff --git a/internal/apps/agent/sync/pages_stream_test.go b/internal/apps/agent/sync/pages_stream_test.go new file mode 100644 index 00000000..8c16e50d --- /dev/null +++ b/internal/apps/agent/sync/pages_stream_test.go @@ -0,0 +1,388 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package sync + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/Rain-kl/Wavelet/internal/apps/agent/protocol" + "github.com/Rain-kl/Wavelet/internal/apps/agent/state" +) + +func TestEnsurePagesProjectRejectsMetadataBeyondAgentCapsBeforeDownload(t *testing.T) { + packageBytes := testPagesPackage(t, map[string]string{"index.html": "x"}) + base := protocol.PagesProjectLatestHashResponse{ + ProjectID: 1, + DeploymentID: 1, + Hash: testBytesChecksum(packageBytes), + PackageSize: int64(len(packageBytes)), + FileCount: 1, + TotalSize: 1, + } + tests := []struct { + name string + mutate func(*protocol.PagesProjectLatestHashResponse) + }{ + { + name: "package size", + mutate: func(metadata *protocol.PagesProjectLatestHashResponse) { + metadata.PackageSize = agentPagesMaxPackageBytes + 1 + }, + }, + { + name: "file count", + mutate: func(metadata *protocol.PagesProjectLatestHashResponse) { + metadata.FileCount = agentPagesMaxFiles + 1 + }, + }, + { + name: "total size", + mutate: func(metadata *protocol.PagesProjectLatestHashResponse) { + metadata.TotalSize = agentPagesMaxTotalBytes + 1 + }, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + metadata := base + test.mutate(&metadata) + client := &fakeClient{ + pagesPackages: map[uint][]byte{1: packageBytes}, + pagesMetadata: map[uint]protocol.PagesProjectLatestHashResponse{1: metadata}, + } + service := New(client, &fakeManager{}, nil) + service.SetPagesDir(t.TempDir()) + err := service.ensurePagesProject(context.Background(), &state.Snapshot{}, 1) + if err == nil || !strings.Contains(err.Error(), "agent limit") { + t.Fatalf("ensurePagesProject(%s metadata) error = %v, want agent limit error", test.name, err) + } + if client.pagesPackageDownloads != 0 { + t.Errorf("ensurePagesProject(%s metadata) downloads = %d, want 0", test.name, client.pagesPackageDownloads) + } + }) + } +} + +func TestEnsurePagesProjectRetriesSameHashDifferentDeployment(t *testing.T) { + packageBytes := testPagesPackage(t, map[string]string{"index.html": "same"}) + hash := testBytesChecksum(packageBytes) + client := &racingLatestClient{ + pkgA: packageBytes, + pkgB: packageBytes, + hashA: hash, + hashB: hash, + } + service := New(client, &fakeManager{}, nil) + pagesDir := t.TempDir() + service.SetPagesDir(pagesDir) + snapshot := &state.Snapshot{PagesDeployments: []state.PagesDeployment{{ProjectID: 42}}} + + if err := service.ensurePagesProject(context.Background(), snapshot, 42); err != nil { + t.Fatalf("ensurePagesProject(same hash deployment race) error = %v", err) + } + if client.downloadCalls != 2 { + t.Errorf("ensurePagesProject(same hash deployment race) downloads = %d, want 2", client.downloadCalls) + } + if snapshot.PagesDeployments[0].DeploymentID != 2 || snapshot.PagesDeployments[0].Hash != hash { + t.Errorf("snapshot Pages deployment = %+v, want deployment 2/hash %s", snapshot.PagesDeployments[0], hash) + } +} + +func TestEnsurePagesProjectExtractionFailureCleansTempAndPreservesCurrent(t *testing.T) { + projectID := uint(9) + oldPackage := testPagesPackage(t, map[string]string{"index.html": "old"}) + oldHash := testBytesChecksum(oldPackage) + newPackage := testPagesPackage(t, map[string]string{"index.html": "new"}) + newHash := testBytesChecksum(newPackage) + pagesDir := t.TempDir() + oldRelease := pagesProjectReleaseDir(pagesDir, projectID, oldHash) + if err := extractTestPagesPackage(t, oldPackage, oldRelease, pagesProjectRef{ + ProjectID: projectID, + DeploymentID: 1, + Checksum: oldHash, + }); err != nil { + t.Fatalf("extractTestPagesPackage(old) error = %v", err) + } + if err := switchPagesProjectCurrentDir(pagesDir, projectID, oldRelease); err != nil { + t.Fatalf("switchPagesProjectCurrentDir(old) error = %v", err) + } + + client := &fakeClient{ + pagesPackages: map[uint][]byte{projectID: newPackage}, + pagesMetadata: map[uint]protocol.PagesProjectLatestHashResponse{ + projectID: { + ProjectID: projectID, + DeploymentID: 2, + Hash: newHash, + PackageSize: int64(len(newPackage)), + FileCount: 1, + TotalSize: 2, // Smaller than the actual three-byte file. + }, + }, + } + service := New(client, &fakeManager{}, nil) + service.SetPagesDir(pagesDir) + err := service.ensurePagesProject(context.Background(), &state.Snapshot{}, projectID) + if err == nil { + t.Fatal("ensurePagesProject(metadata-tightened extraction) error = nil, want error") + } + current, readErr := os.ReadFile(pagesProjectCurrentDir(pagesDir, projectID) + "/index.html") + if readErr != nil { + t.Fatalf("read old current after failed extraction error = %v", readErr) + } + if string(current) != "old" { + t.Errorf("current content after failed extraction = %q, want %q", current, "old") + } + entries, readErr := os.ReadDir(filepath.Join(pagesDir, "projects", "9", "releases")) + if readErr != nil { + t.Fatalf("read releases after failed extraction error = %v", readErr) + } + if len(entries) != 1 || entries[0].Name() != oldHash { + t.Errorf("releases after failed extraction = %v, want only %s", entries, oldHash) + } +} + +func TestEnsurePagesProjectAcceptsAllZeroByteFiles(t *testing.T) { + packageBytes := testPagesPackage(t, map[string]string{ + "index.html": "", + ".gitkeep": "", + }) + client := &fakeClient{pagesPackages: map[uint][]byte{5: packageBytes}} + service := New(client, &fakeManager{}, nil) + pagesDir := t.TempDir() + service.SetPagesDir(pagesDir) + + if err := service.ensurePagesProject(context.Background(), &state.Snapshot{}, 5); err != nil { + t.Fatalf("ensurePagesProject(all-zero files) error = %v", err) + } + for _, name := range []string{"index.html", ".gitkeep"} { + info, err := os.Stat(filepath.Join(pagesProjectCurrentDir(pagesDir, 5), name)) + if err != nil { + t.Errorf("stat all-zero file %q error = %v", name, err) + continue + } + if info.Size() != 0 { + t.Errorf("all-zero file %q size = %d, want 0", name, info.Size()) + } + } +} + +func TestSwitchPagesProjectCurrentDirRenameFailureKeepsPreviousCurrent(t *testing.T) { + pagesDir := t.TempDir() + projectID := uint(21) + oldRelease := pagesProjectReleaseDir(pagesDir, projectID, "old") + newRelease := pagesProjectReleaseDir(pagesDir, projectID, "new") + for path, content := range map[string]string{ + oldRelease: "old", + newRelease: "new", + } { + if err := os.MkdirAll(path, pagesDirPerm); err != nil { + t.Fatalf("mkdir release %q error = %v", path, err) + } + if err := os.WriteFile(filepath.Join(path, "index.html"), []byte(content), pagesFilePerm); err != nil { + t.Fatalf("write release %q error = %v", path, err) + } + } + if err := switchPagesProjectCurrentDir(pagesDir, projectID, oldRelease); err != nil { + t.Fatalf("seed previous current error = %v", err) + } + currentDir := pagesProjectCurrentDir(pagesDir, projectID) + renameErr := errors.New("injected current rename failure") + err := switchPagesProjectCurrentDirWithRename( + pagesDir, + projectID, + newRelease, + func(oldPath string, newPath string) error { + if oldPath == currentDir+".tmp" && newPath == currentDir { + return renameErr + } + return os.Rename(oldPath, newPath) + }, + ) + if !errors.Is(err, renameErr) { + t.Fatalf("switchPagesProjectCurrentDirWithRename() error = %v, want injected rename error", err) + } + current, err := os.ReadFile(filepath.Join(currentDir, "index.html")) + if err != nil { + t.Fatalf("read previous current after rename failure error = %v", err) + } + if string(current) != "old" { + t.Errorf("current after rename failure = %q, want %q", current, "old") + } + if _, err := os.Lstat(currentDir + ".tmp"); !os.IsNotExist(err) { + t.Errorf("temporary current symlink remains after rename failure: %v", err) + } +} + +func TestPromoteSameHashReleaseFailureRestoresPreviousCurrent(t *testing.T) { + pagesDir := t.TempDir() + projectID := uint(22) + hash := "same-hash" + project := pagesProjectRef{ProjectID: projectID, DeploymentID: 2, Checksum: hash} + releaseDir := pagesProjectReleaseDir(pagesDir, projectID, hash) + if err := os.MkdirAll(releaseDir, pagesDirPerm); err != nil { + t.Fatalf("mkdir previous same-hash release error = %v", err) + } + if err := os.WriteFile(filepath.Join(releaseDir, "index.html"), []byte("old"), pagesFilePerm); err != nil { + t.Fatalf("write previous same-hash release error = %v", err) + } + if err := writePagesMarker(releaseDir, project); err != nil { + t.Fatalf("write previous same-hash marker error = %v", err) + } + if err := switchPagesProjectCurrentDir(pagesDir, projectID, releaseDir); err != nil { + t.Fatalf("seed same-hash current error = %v", err) + } + stagingDir, err := os.MkdirTemp(filepath.Dir(releaseDir), ".same-hash-*.tmp") + if err != nil { + t.Fatalf("create same-hash staging error = %v", err) + } + if err := os.WriteFile(filepath.Join(stagingDir, "index.html"), []byte("new"), pagesFilePerm); err != nil { + t.Fatalf("write repaired same-hash release error = %v", err) + } + if err := writePagesMarker(stagingDir, project); err != nil { + t.Fatalf("write repaired same-hash marker error = %v", err) + } + copyErr := errors.New("injected same-hash copy failure") + err = promotePagesReleaseWithCopy( + stagingDir, + releaseDir, + project, + func(_ string, targetDir string) error { + if err := os.MkdirAll(targetDir, pagesDirPerm); err != nil { + return err + } + if err := os.WriteFile(filepath.Join(targetDir, "index.html"), []byte("partial"), pagesFilePerm); err != nil { + return err + } + return copyErr + }, + ) + if !errors.Is(err, copyErr) { + t.Fatalf("promotePagesReleaseWithCopy() error = %v, want injected copy error", err) + } + for name, path := range map[string]string{ + "current": filepath.Join(pagesProjectCurrentDir(pagesDir, projectID), "index.html"), + "release": filepath.Join(releaseDir, "index.html"), + } { + content, readErr := os.ReadFile(path) + if readErr != nil { + t.Fatalf("read restored %s after same-hash repair failure error = %v", name, readErr) + } + if string(content) != "old" { + t.Errorf("restored %s after same-hash repair failure = %q, want %q", name, content, "old") + } + } + if _, err := os.Stat(stagingDir); !os.IsNotExist(err) { + t.Errorf("same-hash staging remains after successful rollback: %v", err) + } +} + +func TestPromotePagesReleaseRepairsDanglingCurrent(t *testing.T) { + pagesDir := t.TempDir() + projectID := uint(23) + releaseDir := pagesProjectReleaseDir(pagesDir, projectID, "new-hash") + currentDir := pagesProjectCurrentDir(pagesDir, projectID) + requireTestMkdirAll(t, filepath.Dir(currentDir)) + relTarget, err := filepath.Rel(filepath.Dir(currentDir), releaseDir) + if err != nil { + t.Fatalf("relative release target error = %v", err) + } + if err := os.Symlink(relTarget, currentDir); err != nil { + t.Skipf("symlink unsupported: %v", err) + } + + requireTestMkdirAll(t, filepath.Dir(releaseDir)) + stagingDir, err := os.MkdirTemp(filepath.Dir(releaseDir), ".dangling-*.tmp") + if err != nil { + t.Fatalf("create dangling repair staging error = %v", err) + } + if err := os.WriteFile(filepath.Join(stagingDir, "index.html"), []byte("repaired"), pagesFilePerm); err != nil { + t.Fatalf("write dangling repair staging error = %v", err) + } + project := pagesProjectRef{ProjectID: projectID, DeploymentID: 1, Checksum: "new-hash"} + if err := writePagesMarker(stagingDir, project); err != nil { + t.Fatalf("write dangling repair marker error = %v", err) + } + + if err := promotePagesRelease(stagingDir, releaseDir, project); err != nil { + t.Fatalf("promotePagesRelease(dangling current) error = %v", err) + } + content, err := os.ReadFile(filepath.Join(currentDir, "index.html")) + if err != nil { + t.Fatalf("read repaired dangling current error = %v", err) + } + if string(content) != "repaired" { + t.Errorf("repaired dangling current = %q, want %q", content, "repaired") + } +} + +func TestSwitchPagesCurrentDirCopiesOverLegacyDirectory(t *testing.T) { + pagesDir := t.TempDir() + currentDir := filepath.Join(pagesDir, "current") + releaseDir := filepath.Join(pagesDir, "releases", "new") + requireTestMkdirAll(t, currentDir) + requireTestMkdirAll(t, releaseDir) + if err := os.WriteFile(filepath.Join(currentDir, "index.html"), []byte("old"), pagesFilePerm); err != nil { + t.Fatalf("write legacy current error = %v", err) + } + if err := os.WriteFile(filepath.Join(releaseDir, "index.html"), []byte("new"), pagesFilePerm); err != nil { + t.Fatalf("write new release error = %v", err) + } + + if err := switchPagesCurrentDir(currentDir, releaseDir, os.Rename); err != nil { + t.Fatalf("switchPagesCurrentDir(legacy directory) error = %v", err) + } + content, err := os.ReadFile(filepath.Join(currentDir, "index.html")) + if err != nil { + t.Fatalf("read copied legacy current error = %v", err) + } + if string(content) != "new" { + t.Errorf("copied legacy current = %q, want %q", content, "new") + } +} + +func TestSwitchPagesCurrentDirFallsBackWhenSymlinkUnavailable(t *testing.T) { + pagesDir := t.TempDir() + currentDir := filepath.Join(pagesDir, "current") + releaseDir := filepath.Join(pagesDir, "releases", "new") + requireTestMkdirAll(t, releaseDir) + if err := os.WriteFile(filepath.Join(releaseDir, "index.html"), []byte("new"), pagesFilePerm); err != nil { + t.Fatalf("write fallback release error = %v", err) + } + symlinkErr := errors.New("injected symlink unavailable") + if err := switchPagesCurrentDirWithOps( + currentDir, + releaseDir, + os.Rename, + func(string, string) error { return symlinkErr }, + ); err != nil { + t.Fatalf("switchPagesCurrentDirWithOps(symlink unavailable) error = %v", err) + } + info, err := os.Lstat(currentDir) + if err != nil { + t.Fatalf("lstat copied current error = %v", err) + } + if !info.IsDir() { + t.Errorf("copied current mode = %v, want directory", info.Mode()) + } + content, err := os.ReadFile(filepath.Join(currentDir, "index.html")) + if err != nil { + t.Fatalf("read fallback current error = %v", err) + } + if string(content) != "new" { + t.Errorf("fallback current = %q, want %q", content, "new") + } +} + +func requireTestMkdirAll(t *testing.T, dir string) { + t.Helper() + if err := os.MkdirAll(dir, pagesDirPerm); err != nil { + t.Fatalf("mkdir %q error = %v", dir, err) + } +} diff --git a/internal/apps/agent/sync/service.go b/internal/apps/agent/sync/service.go index 30562c91..6c684b02 100644 --- a/internal/apps/agent/sync/service.go +++ b/internal/apps/agent/sync/service.go @@ -1,3 +1,6 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + package sync import ( @@ -7,6 +10,7 @@ import ( "encoding/json" "errors" "fmt" + "io" "log/slog" "sort" "strconv" @@ -31,9 +35,9 @@ const ( type ConfigClient interface { GetActiveConfig(ctx context.Context) (*protocol.ActiveConfigResponse, error) GetPagesDeploymentHash(ctx context.Context, deploymentID uint) (string, error) - DownloadPagesDeploymentPackage(ctx context.Context, deploymentID uint) ([]byte, error) + DownloadPagesDeploymentPackage(ctx context.Context, deploymentID uint, dst io.Writer, maxBytes int64) (int64, error) GetPagesProjectLatestHash(ctx context.Context, projectID uint) (*protocol.PagesProjectLatestHashResponse, error) - DownloadPagesProjectLatestPackage(ctx context.Context, projectID uint) ([]byte, error) + DownloadPagesProjectLatestPackage(ctx context.Context, projectID uint, dst io.Writer, maxBytes int64) (int64, error) ReportApplyLog(ctx context.Context, payload protocol.ApplyLogPayload) error SyncWAFIPGroups(ctx context.Context, payload protocol.WAFIPGroupSyncRequest) (*protocol.WAFIPGroupSyncResponse, error) } diff --git a/internal/apps/agent/sync/service_test.go b/internal/apps/agent/sync/service_test.go index ecec6338..01b0a2fd 100644 --- a/internal/apps/agent/sync/service_test.go +++ b/internal/apps/agent/sync/service_test.go @@ -1,3 +1,6 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + package sync import ( @@ -7,6 +10,7 @@ import ( "crypto/sha256" "encoding/hex" "fmt" + "io" "os" "path/filepath" "strings" @@ -16,6 +20,7 @@ import ( "github.com/Rain-kl/Wavelet/internal/apps/agent/nginx" "github.com/Rain-kl/Wavelet/internal/apps/agent/protocol" "github.com/Rain-kl/Wavelet/internal/apps/agent/state" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" ) type fakeExecutor struct { @@ -28,15 +33,17 @@ func testPagesSourceConfigJSON(projectID, deploymentID uint, checksum string) st } type fakeClient struct { - config protocol.ActiveConfigResponse - reports []protocol.ApplyLogPayload - wafSyncCalls []protocol.WAFIPGroupSyncRequest - pagesPackages map[uint][]byte // key: project_id (latest package) - pagesHashes map[uint]string // key: project_id - pagesLatestDeployIDs map[uint]uint // key: project_id → deployment_id - wafSyncResult protocol.WAFIPGroupSyncResponse - fetchCalls int - hashCalls int + config protocol.ActiveConfigResponse + reports []protocol.ApplyLogPayload + wafSyncCalls []protocol.WAFIPGroupSyncRequest + pagesPackages map[uint][]byte // key: project_id (latest package) + pagesHashes map[uint]string // key: project_id + pagesLatestDeployIDs map[uint]uint // key: project_id → deployment_id + pagesMetadata map[uint]protocol.PagesProjectLatestHashResponse + pagesPackageDownloads int + wafSyncResult protocol.WAFIPGroupSyncResponse + fetchCalls int + hashCalls int } type fakeManager struct { @@ -98,17 +105,28 @@ func (f *fakeClient) GetPagesDeploymentHash(ctx context.Context, deploymentID ui return f.projectHash(deploymentID) } -func (f *fakeClient) DownloadPagesDeploymentPackage(ctx context.Context, deploymentID uint) ([]byte, error) { +func (f *fakeClient) DownloadPagesDeploymentPackage( + ctx context.Context, + deploymentID uint, + dst io.Writer, + maxBytes int64, +) (int64, error) { for projectID, depID := range f.pagesLatestDeployIDs { if depID == deploymentID { - return f.projectPackage(projectID) + return f.writeProjectPackage(projectID, dst, maxBytes) } } - return f.projectPackage(deploymentID) + return f.writeProjectPackage(deploymentID, dst, maxBytes) } func (f *fakeClient) GetPagesProjectLatestHash(ctx context.Context, projectID uint) (*protocol.PagesProjectLatestHashResponse, error) { f.hashCalls++ + if f.pagesMetadata != nil { + if metadata, ok := f.pagesMetadata[projectID]; ok { + result := metadata + return &result, nil + } + } hash, err := f.projectHash(projectID) if err != nil { return nil, err @@ -119,15 +137,32 @@ func (f *fakeClient) GetPagesProjectLatestHash(ctx context.Context, projectID ui deploymentID = id } } + packageBytes, err := f.projectPackage(projectID) + if err != nil { + return nil, err + } + fileCount, totalSize, err := testPagesPackageStats(packageBytes) + if err != nil { + return nil, err + } return &protocol.PagesProjectLatestHashResponse{ ProjectID: projectID, DeploymentID: deploymentID, Hash: hash, + PackageSize: int64(len(packageBytes)), + FileCount: fileCount, + TotalSize: totalSize, }, nil } -func (f *fakeClient) DownloadPagesProjectLatestPackage(ctx context.Context, projectID uint) ([]byte, error) { - return f.projectPackage(projectID) +func (f *fakeClient) DownloadPagesProjectLatestPackage( + ctx context.Context, + projectID uint, + dst io.Writer, + maxBytes int64, +) (int64, error) { + f.pagesPackageDownloads++ + return f.writeProjectPackage(projectID, dst, maxBytes) } func (f *fakeClient) projectHash(projectID uint) (string, error) { @@ -155,6 +190,22 @@ func (f *fakeClient) projectPackage(projectID uint) ([]byte, error) { return packageBytes, nil } +func (f *fakeClient) writeProjectPackage(projectID uint, dst io.Writer, maxBytes int64) (int64, error) { + packageBytes, err := f.projectPackage(projectID) + if err != nil { + return 0, err + } + limited := &io.LimitedReader{R: bytes.NewReader(packageBytes), N: maxBytes + 1} + written, err := io.Copy(dst, limited) + if err != nil { + return written, err + } + if written > maxBytes { + return written, fmt.Errorf("test pages package exceeds limit %d", maxBytes) + } + return written, nil +} + func (f *fakeClient) ReportApplyLog(ctx context.Context, payload protocol.ApplyLogPayload) error { f.reports = append(f.reports, payload) return nil @@ -572,7 +623,9 @@ func TestSyncOnceRejectsPagesZipSlipBeforeApply(t *testing.T) { service.SetPagesDir(t.TempDir()) err := service.SyncOnce(context.Background(), &protocol.ActiveConfigMeta{Version: "20260309-102", Checksum: "pages-config-checksum"}) - if err == nil || (!strings.Contains(err.Error(), "escapes deployment root") && !strings.Contains(err.Error(), "escapes directory")) { + if err == nil || (!strings.Contains(err.Error(), "escapes deployment root") && + !strings.Contains(err.Error(), "escapes directory") && + !strings.Contains(err.Error(), "dot segment")) { t.Fatalf("expected zip-slip rejection, got %v", err) } if len(manager.applyRouteContents) != 0 { @@ -1316,7 +1369,7 @@ func TestSyncOnceRedownloadsPagesDeploymentWhenServerHashChanges(t *testing.T) { } pagesDir := t.TempDir() releaseDir := pagesProjectReleaseDir(pagesDir, projectID, initialHash) - if err = extractPagesPackage(initialPackage, releaseDir, pagesProjectRef{ + if err = extractTestPagesPackage(t, initialPackage, releaseDir, pagesProjectRef{ ProjectID: projectID, Checksum: initialHash, }); err != nil { @@ -1385,20 +1438,42 @@ func (r *racingLatestClient) GetPagesProjectLatestHash(ctx context.Context, proj // 2: verify after downloading B → B (race) // 3+: stable on B for retry hash, dep := r.hashA, uint(1) + packageBytes := r.pkgA if r.hashCall >= 2 { hash, dep = r.hashB, 2 + packageBytes = r.pkgB + } + fileCount, totalSize, err := testPagesPackageStats(packageBytes) + if err != nil { + return nil, err } return &protocol.PagesProjectLatestHashResponse{ ProjectID: projectID, DeploymentID: dep, Hash: hash, + PackageSize: int64(len(packageBytes)), + FileCount: fileCount, + TotalSize: totalSize, }, nil } -func (r *racingLatestClient) DownloadPagesProjectLatestPackage(ctx context.Context, projectID uint) ([]byte, error) { +func (r *racingLatestClient) DownloadPagesProjectLatestPackage( + ctx context.Context, + projectID uint, + dst io.Writer, + maxBytes int64, +) (int64, error) { r.downloadCalls++ // Always return package B (what "latest download" would stream mid-race / after). - return r.pkgB, nil + limited := &io.LimitedReader{R: bytes.NewReader(r.pkgB), N: maxBytes + 1} + written, err := io.Copy(dst, limited) + if err != nil { + return written, err + } + if written > maxBytes { + return written, fmt.Errorf("test pages package exceeds limit %d", maxBytes) + } + return written, nil } func TestEnsurePagesProjectSurvivesHashPackageRace(t *testing.T) { @@ -1641,6 +1716,41 @@ func testPagesPackage(t *testing.T, files map[string]string) []byte { return buffer.Bytes() } +func extractTestPagesPackage( + t *testing.T, + packageBytes []byte, + releaseDir string, + project pagesProjectRef, +) error { + t.Helper() + packagePath := filepath.Join(t.TempDir(), "pages-package.zip") + if err := os.WriteFile(packagePath, packageBytes, pagesFilePerm); err != nil { + t.Fatalf("write test Pages package error = %v", err) + } + return extractPagesPackageFile(packagePath, releaseDir, project, pagesarchive.Limits{ + MaxFiles: agentPagesMaxFiles, + MaxFileBytes: agentPagesMaxFileBytes, + MaxTotalBytes: agentPagesMaxTotalBytes, + }, nil) +} + +func testPagesPackageStats(packageBytes []byte) (int, int64, error) { + reader, err := zip.NewReader(bytes.NewReader(packageBytes), int64(len(packageBytes))) + if err != nil { + return 0, 0, err + } + fileCount := 0 + totalSize := int64(0) + for _, file := range reader.File { + if file.FileInfo().IsDir() { + continue + } + fileCount++ + totalSize += int64(file.UncompressedSize64) //nolint:gosec // test packages are memory-bounded + } + return fileCount, totalSize, nil +} + func testBytesChecksum(data []byte) string { sum := sha256.Sum256(data) return hex.EncodeToString(sum[:]) diff --git a/internal/apps/openflare/agent/routers.go b/internal/apps/openflare/agent/routers.go index e39b27be..dd840a63 100644 --- a/internal/apps/openflare/agent/routers.go +++ b/internal/apps/openflare/agent/routers.go @@ -224,14 +224,17 @@ func GetPagesProjectLatestHashHandler(c *gin.Context) { if !ok { return } - deploymentID, hash, err := pages.GetProjectLatestPackageHash(c.Request.Context(), projectID) + metadata, err := pages.GetProjectLatestPackageMetadata(c.Request.Context(), projectID) if apiutil.AbortBadRequestOnError(c, err) { return } c.JSON(http.StatusOK, response.OK(protocol.PagesProjectLatestHashResponse{ ProjectID: projectID, - DeploymentID: deploymentID, - Hash: hash, + DeploymentID: metadata.DeploymentID, + Hash: metadata.Hash, + PackageSize: metadata.PackageSize, + FileCount: metadata.FileCount, + TotalSize: metadata.TotalSize, })) } diff --git a/internal/apps/openflare/config_version/pages_snapshot.go b/internal/apps/openflare/config_version/pages_snapshot.go index d7012829..b7486f67 100644 --- a/internal/apps/openflare/config_version/pages_snapshot.go +++ b/internal/apps/openflare/config_version/pages_snapshot.go @@ -7,9 +7,11 @@ import ( "context" "errors" "fmt" + "path" "strings" "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" openrestyrender "github.com/Rain-kl/Wavelet/pkg/render/openresty" "gorm.io/gorm" ) @@ -58,23 +60,41 @@ func buildPagesRouteSnapshot( } pagesProjectID = route.PagesProjectID - deployment = buildSnapshotPagesDeployment(project, activeDeployment) + deployment, err = buildSnapshotPagesDeployment(project, activeDeployment) + if err != nil { + return "", nil, nil, nil, fmt.Errorf("路由 %s Pages 配置无效: %w", route.SiteName, err) + } originURL = fmt.Sprintf("openflare-pages://project/%d", project.ID) return originURL, []string{originURL}, pagesProjectID, deployment, nil } -func buildSnapshotPagesDeployment(project *model.PagesProject, activeDeployment *model.PagesDeployment) *openrestyrender.PagesDeployment { +func buildSnapshotPagesDeployment( + project *model.PagesProject, + activeDeployment *model.PagesDeployment, +) (*openrestyrender.PagesDeployment, error) { if project == nil || activeDeployment == nil { - return nil + return nil, errors.New("pages 项目或部署为空") + } + rootDir, err := pagesarchive.NormalizeLogicalPath(strings.TrimSpace(project.RootDir), true) + if err != nil { + return nil, fmt.Errorf("pages 根目录不合法: %w", err) } entryFile := strings.TrimSpace(project.EntryFile) if entryFile == "" { entryFile = defaultPagesSnapshotEntryFile } + entryFile, err = pagesarchive.NormalizeLogicalPath(entryFile, false) + if err != nil { + return nil, fmt.Errorf("pages 入口文件不合法: %w", err) + } fallbackPath := strings.TrimSpace(project.SPAFallbackPath) if fallbackPath == "" { fallbackPath = defaultPagesSnapshotFallbackPath } + localRoot := openrestyrender.PagesProjectLocalRoot(project.ID) + if rootDir != "" { + localRoot = path.Join(localRoot, rootDir) + } return &openrestyrender.PagesDeployment{ ProjectID: project.ID, ProjectSlug: strings.TrimSpace(project.Slug), @@ -90,6 +110,6 @@ func buildSnapshotPagesDeployment(project *model.PagesProject, activeDeployment APIProxyRewrite: strings.TrimSpace(project.APIProxyRewrite), // Root is project-scoped so Agents can swap active packages without // re-publishing main config (nginx root stays stable). - LocalRoot: openrestyrender.PagesProjectLocalRoot(project.ID), - } + LocalRoot: localRoot, + }, nil } diff --git a/internal/apps/openflare/config_version/pages_snapshot_test.go b/internal/apps/openflare/config_version/pages_snapshot_test.go index 9b6fb1a9..476a9ed5 100644 --- a/internal/apps/openflare/config_version/pages_snapshot_test.go +++ b/internal/apps/openflare/config_version/pages_snapshot_test.go @@ -28,6 +28,7 @@ func TestBuildSnapshotRoutesPages(t *testing.T) { Enabled: true, SPAFallbackEnabled: true, SPAFallbackPath: "/index.html", + RootDir: "public/site", EntryFile: "index.html", } require.NoError(t, conn.Create(project).Error) @@ -64,7 +65,7 @@ func TestBuildSnapshotRoutesPages(t *testing.T) { require.NotNil(t, snapshotRoute.PagesDeployment) assert.Equal(t, deployment.ID, snapshotRoute.PagesDeployment.DeploymentID) assert.Equal(t, deployment.Checksum, snapshotRoute.PagesDeployment.Checksum) - assert.Equal(t, "__OPENFLARE_PAGES_DIR__/projects/1/current", snapshotRoute.PagesDeployment.LocalRoot) + assert.Equal(t, "__OPENFLARE_PAGES_DIR__/projects/1/current/public/site", snapshotRoute.PagesDeployment.LocalRoot) _, err = renderSnapshotConfig(bundle.SnapshotJSON, nil) require.NoError(t, err) @@ -78,6 +79,17 @@ func TestBuildSnapshotRoutesPages(t *testing.T) { require.NotNil(t, decoded.Routes[0].PagesDeployment) } +func TestBuildSnapshotPagesDeploymentRejectsUnsafeStoredPaths(t *testing.T) { + deployment := &model.PagesDeployment{ID: 1, ProjectID: 1, Checksum: "checksum"} + for _, project := range []*model.PagesProject{ + {ID: 1, RootDir: "../escape", EntryFile: "index.html"}, + {ID: 1, RootDir: "public", EntryFile: "/index.html"}, + } { + _, err := buildSnapshotPagesDeployment(project, deployment) + require.Error(t, err) + } +} + func requireDB(t *testing.T, ctx context.Context) *gorm.DB { t.Helper() conn := db.DB(ctx) diff --git a/internal/apps/openflare/pages/download_url.go b/internal/apps/openflare/pages/download_url.go index 9de2938e..5959d7e7 100644 --- a/internal/apps/openflare/pages/download_url.go +++ b/internal/apps/openflare/pages/download_url.go @@ -27,7 +27,7 @@ import ( const ( pagesURLDownloadTimeout = 10 * time.Minute pagesURLMaxRedirects = 5 - pagesMagicSniffBytes = 16 + pagesMagicSniffBytes = 512 pagesURLDialTimeout = 30 * time.Second pagesURLTLSHandshake = 15 * time.Second pagesBrowserUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" diff --git a/internal/apps/openflare/pages/errs.go b/internal/apps/openflare/pages/errs.go index b4591c78..602825a1 100644 --- a/internal/apps/openflare/pages/errs.go +++ b/internal/apps/openflare/pages/errs.go @@ -34,4 +34,6 @@ const ( errPagesPackageNotInActiveConfig = "pages 部署尚未进入激活配置" errPagesDeploymentHashMissing = "pages 部署包哈希缺失" errPagesInvalidSnapshotFormat = "配置快照格式无效" + errPagesActorMissing = "无法识别当前用户" + errPagesEntryFileMissing = "当前激活部署中不存在指定入口文件" ) diff --git a/internal/apps/openflare/pages/helpers.go b/internal/apps/openflare/pages/helpers.go index b1f2c959..90db9996 100644 --- a/internal/apps/openflare/pages/helpers.go +++ b/internal/apps/openflare/pages/helpers.go @@ -15,13 +15,17 @@ import ( "path" "path/filepath" "regexp" + "strconv" "strings" "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" "github.com/Rain-kl/Wavelet/internal/repository" "github.com/Rain-kl/Wavelet/pkg/logger" "github.com/Rain-kl/Wavelet/pkg/pagesarchive" + "gorm.io/gorm" + "gorm.io/gorm/clause" ) const ( @@ -31,11 +35,14 @@ const ( defaultPagesMaxHistoryCount = 20 defaultPagesEntryFile = "index.html" defaultPagesFallbackPath = "/index.html" - pagesDeploymentUploadType = "openflare_pages_deployment" + pagesIngestMarkerKey = "pages_ingest_marker" + pagesIngestMarkerV2 = "pages_deployment_v2" + pagesProjectIDMetadataKey = "pages_project_id" pagesMaxPathLength = 512 bytesPerMiB = 1024 * 1024 pagesExtractedSizeMultiplier = 4 pagesMinExtractedSizeBytes = 100 * bytesPerMiB + pagesRowLockStrength = "UPDATE" ) var pagesSlugPattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,126}[a-z0-9]$|^[a-z0-9]$`) @@ -115,30 +122,14 @@ func normalizePagesSlug(raw string) string { func validateAndNormalizePagesRootDir(raw string) (string, error) { value := strings.TrimSpace(raw) - if value == "" { - return "", nil - } if len(value) > pagesMaxPathLength { return "", errors.New("pages 根目录长度不能超过 512") } - if strings.Contains(value, "\\") || strings.ContainsAny(value, "\"';") { - return "", errors.New("pages 根目录包含不支持的字符") + normalized, err := pagesarchive.NormalizeLogicalPath(value, true) + if err != nil { + return "", fmt.Errorf("pages 根目录不合法: %w", err) } - for _, r := range value { - if r <= 0x20 || r == 0x7f { - return "", errors.New("pages 根目录不能包含空白或控制字符") - } - } - cleaned := path.Clean(filepath.ToSlash(value)) - if cleaned == "." || cleaned == "/" { - return "", nil - } - for _, segment := range strings.Split(cleaned, "/") { - if segment == "." || segment == ".." { - return "", errors.New("pages 根目录不能包含 . 或 .. 路径段") - } - } - return strings.TrimPrefix(cleaned, "/"), nil + return normalized, nil } func normalizePagesFallbackPath(raw string) (string, error) { @@ -186,12 +177,19 @@ func normalizeStoredPagesFallbackPath(value string) string { return normalized } -func normalizePagesEntryFile(raw string) string { - value := path.Clean(strings.TrimSpace(filepath.ToSlash(raw))) - if value == "." || value == "/" { - return defaultPagesEntryFile +func validateAndNormalizePagesEntryFile(raw string) (string, error) { + value := strings.TrimSpace(raw) + if value == "" { + value = defaultPagesEntryFile } - return strings.TrimPrefix(value, "/") + if len(value) > pagesMaxPathLength { + return "", errors.New("pages 入口文件长度不能超过 512") + } + normalized, err := pagesarchive.NormalizeLogicalPath(value, false) + if err != nil { + return "", fmt.Errorf("pages 入口文件不合法: %w", err) + } + return normalized, nil } func persistPagesUploadTemp(fileHeader *multipart.FileHeader, maxPackageBytes int64) (string, string, int64, pagesarchive.Format, error) { @@ -262,7 +260,7 @@ func ingestPagesDeploymentPackage( ctx context.Context, localPath string, checksum string, - projectSlug string, + projectID uint, fileName string, format pagesarchive.Format, ) (upload.IngestResult, error) { @@ -275,28 +273,27 @@ func ingestPagesDeploymentPackage( MimeType: pagesarchive.MIMEType(format), Extension: extension, Hash: checksum, - Type: pagesDeploymentUploadType, + Type: upload.ReservedPagesDeploymentType, AccessMode: &accessMode, SkipExtensionCheck: true, Policy: upload.PolicyDedupNewRecord, Metadata: model.UploadMetadata{ Extra: map[string]any{ - "project_slug": projectSlug, - "format": string(format), + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: strconv.FormatUint(uint64(projectID), 10), }, }, }) } -func removeDeploymentArtifact(ctx context.Context, deployment *model.PagesDeployment) { +func removeDeploymentArtifact(ctx context.Context, projectID uint, deployment *model.PagesDeployment) { if deployment == nil { return } if deployment.UploadID == 0 { return } - if _, err := upload.Remove(ctx, deployment.UploadID); err != nil { - // Soft-delete / storage cleanup failure must not undo DB prune; log for ops. + if err := removePagesUploadIfUnreferenced(ctx, projectID, deployment.UploadID); err != nil { logger.WarnF(ctx, "[Pages] remove deployment artifact failed: deployment_id=%d upload_id=%d error=%v", deployment.ID, deployment.UploadID, err, @@ -304,10 +301,58 @@ func removeDeploymentArtifact(ctx context.Context, deployment *model.PagesDeploy } } +// removePagesUploadIfUnreferenced soft-deletes a reserved Pages upload only +// after locking its project (when present), locking the upload, and rechecking +// deployment references in the same transaction. +func removePagesUploadIfUnreferenced(ctx context.Context, projectID uint, uploadID uint64) error { + if uploadID == 0 { + return nil + } + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + if projectID != 0 { + var project model.PagesProject + projectErr := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error + if projectErr != nil && !errors.Is(projectErr, gorm.ErrRecordNotFound) { + return projectErr + } + } + + var uploadRecord model.Upload + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ?", uploadID). + First(&uploadRecord).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil + } + return err + } + if uploadRecord.Type != upload.ReservedPagesDeploymentType { + return fmt.Errorf("pages 部署包上传类型不匹配: %s", uploadRecord.Type) + } + + var references int64 + if err := tx.Model(&model.PagesDeployment{}). + Where("upload_id = ?", uploadID). + Count(&references).Error; err != nil { + return err + } + if references > 0 { + return nil + } + _, err := upload.RemoveLockedTx(tx, &uploadRecord) + return err + }) + // Always invalidate after transaction completion, including idempotent no-op, + // so a prior post-commit cache interruption can heal on retry. + upload.InvalidateUploadMetaCache(ctx, uploadID) + return err +} + func inspectPagesPackage(packagePath string, format pagesarchive.Format, rootDir string, entryFile string, limits pagesLimits) (*deploymentManifest, error) { archiveManifest, err := pagesarchive.InspectFile(packagePath, format, pagesarchive.InspectOptions{ - RootDir: rootDir, - EntryFile: entryFile, + RootDir: rootDir, + EntryFile: entryFile, + VerifySizes: true, Limits: pagesarchive.Limits{ MaxFiles: limits.MaxFiles, MaxFileBytes: limits.ExtractedBytes, diff --git a/internal/apps/openflare/pages/logics.go b/internal/apps/openflare/pages/logics.go index f588c02d..622564bc 100644 --- a/internal/apps/openflare/pages/logics.go +++ b/internal/apps/openflare/pages/logics.go @@ -12,6 +12,7 @@ import ( "mime/multipart" "net/url" "os" + "path" "strings" "time" @@ -21,6 +22,7 @@ import ( "github.com/Rain-kl/Wavelet/pkg/logger" "github.com/Rain-kl/Wavelet/pkg/pagesarchive" "gorm.io/gorm" + "gorm.io/gorm/clause" ) // DeploymentPackage is a streamable Pages deployment artifact for agent download. @@ -137,28 +139,40 @@ func CreateProject(ctx context.Context, input Input) (*View, error) { // UpdateProject 更新 Pages 项目。 func UpdateProject(ctx context.Context, id uint, input Input) (*View, error) { - project, err := model.GetPagesProjectByID(ctx, id) + var project *model.PagesProject + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var existing model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&existing, id).Error; err != nil { + return err + } + updated := existing + var err error + project, err = buildProject(&updated, input) + if err != nil { + return err + } + if (existing.RootDir != project.RootDir || existing.EntryFile != project.EntryFile) && + existing.ActiveDeploymentID != nil && *existing.ActiveDeploymentID != 0 { + if err := ensureDeploymentEntry(tx, *existing.ActiveDeploymentID, project.RootDir, project.EntryFile); err != nil { + return err + } + } + return tx.Model(&existing).Updates(map[string]any{ + "name": project.Name, + "slug": project.Slug, + "description": project.Description, + "enabled": project.Enabled, + "spa_fallback_enabled": project.SPAFallbackEnabled, + "spa_fallback_path": project.SPAFallbackPath, + "api_proxy_enabled": project.APIProxyEnabled, + "api_proxy_path": project.APIProxyPath, + "api_proxy_pass": project.APIProxyPass, + "api_proxy_rewrite": project.APIProxyRewrite, + "root_dir": project.RootDir, + "entry_file": project.EntryFile, + }).Error + }) if err != nil { - return nil, err - } - project, err = buildProject(project, input) - if err != nil { - return nil, err - } - if err = db.DB(ctx).Model(project).Updates(map[string]any{ - "name": project.Name, - "slug": project.Slug, - "description": project.Description, - "enabled": project.Enabled, - "spa_fallback_enabled": project.SPAFallbackEnabled, - "spa_fallback_path": project.SPAFallbackPath, - "api_proxy_enabled": project.APIProxyEnabled, - "api_proxy_path": project.APIProxyPath, - "api_proxy_pass": project.APIProxyPass, - "api_proxy_rewrite": project.APIProxyRewrite, - "root_dir": project.RootDir, - "entry_file": project.EntryFile, - }).Error; err != nil { if isUniqueConstraintError(err) { return nil, errors.New(errPagesSlugExists) } @@ -167,24 +181,49 @@ func UpdateProject(ctx context.Context, id uint, input Input) (*View, error) { return buildProjectView(ctx, project) } +func ensureDeploymentEntry(conn *gorm.DB, deploymentID uint, rootDir, entryFile string) error { + targetPath := entryFile + if rootDir != "" { + targetPath = path.Join(rootDir, entryFile) + } + var count int64 + if err := conn.Model(&model.PagesDeploymentFile{}). + Where("deployment_id = ? AND path = ?", deploymentID, targetPath). + Count(&count).Error; err != nil { + return err + } + if count == 0 { + return fmt.Errorf("%s: %s", errPagesEntryFileMissing, targetPath) + } + return nil +} + // DeleteProject 删除 Pages 项目。 func DeleteProject(ctx context.Context, id uint) error { project, err := model.GetPagesProjectByID(ctx, id) if err != nil { return err } - routeCount, err := model.CountProxyRoutesByPagesProjectID(ctx, project.ID) - if err != nil { - return err - } - if routeCount > 0 { - return errors.New(errPagesDeleteReferenced) - } - deployments, err := model.ListPagesDeployments(ctx, project.ID) - if err != nil { - return err - } - return db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var deployments []model.PagesDeployment + err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var lockedProject model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&lockedProject, project.ID).Error; err != nil { + return err + } + if tx.Migrator().HasTable(&model.ProxyRoute{}) { + var routeCount int64 + if err := tx.Model(&model.ProxyRoute{}). + Where("pages_project_id = ?", project.ID). + Count(&routeCount).Error; err != nil { + return err + } + if routeCount > 0 { + return errors.New(errPagesDeleteReferenced) + } + } + if err := tx.Where("project_id = ?", project.ID).Find(&deployments).Error; err != nil { + return err + } if err := tx.Where( "deployment_id IN (?)", tx.Model(&model.PagesDeployment{}).Select("id").Where("project_id = ?", project.ID), @@ -197,11 +236,15 @@ func DeleteProject(ctx context.Context, id uint) error { if err := tx.Delete(project).Error; err != nil { return err } - for index := range deployments { - removeDeploymentArtifact(ctx, &deployments[index]) - } return nil }) + if err != nil { + return err + } + for index := range deployments { + removeDeploymentArtifact(ctx, project.ID, &deployments[index]) + } + return nil } // ListProjectDeployments 列出项目的全部部署。 @@ -298,7 +341,10 @@ func createDeploymentFromTempPackage( if err != nil { return nil, err } - entryFile := normalizePagesEntryFile(project.EntryFile) + entryFile, err := validateAndNormalizePagesEntryFile(project.EntryFile) + if err != nil { + return nil, err + } manifest, err := inspectPagesPackage(tempPath, format, rootDir, entryFile, limits) if err != nil { return nil, err @@ -307,7 +353,7 @@ func createDeploymentFromTempPackage( ctx, tempPath, checksum, - project.Slug, + project.ID, fileName, format, ) @@ -317,11 +363,20 @@ func createDeploymentFromTempPackage( ingestCommitted := false defer func() { if !ingestCommitted && ingestResult.Created { - _, _ = upload.Remove(ctx, ingestResult.Upload.ID) + if removeErr := removePagesUploadIfUnreferenced(ctx, project.ID, ingestResult.Upload.ID); removeErr != nil { + logger.ErrorF(ctx, + "[Pages] compensate deployment upload failed: project_id=%d upload_id=%d error=%v", + project.ID, ingestResult.Upload.ID, removeErr, + ) + } } }() deployment := &model.PagesDeployment{} err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var lockedProject model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&lockedProject, project.ID).Error; err != nil { + return err + } var maxNumber int if err := tx.Model(&model.PagesDeployment{}). Where("project_id = ?", project.ID). @@ -357,7 +412,7 @@ func createDeploymentFromTempPackage( } ingestCommitted = true - if pruneErr := pruneProjectDeploymentHistory(ctx, project.ID, limits.HistoryCount); pruneErr != nil { + if pruneErr := pruneProjectDeploymentHistory(ctx, project.ID, limits.HistoryCount, deployment.ID); pruneErr != nil { logger.ErrorF(ctx, "[Pages] prune deployment history failed: project_id=%d keep=%d error=%v", project.ID, limits.HistoryCount, pruneErr, @@ -381,7 +436,7 @@ func createDeploymentFromTempPackage( // Concurrency: DB row deletes run in a single transaction after a consistent read // of project + deployments. Concurrent uploads may briefly exceed keepCount; the // next successful prune brings the project back within the limit (eventual). -func pruneProjectDeploymentHistory(ctx context.Context, projectID uint, keepCount int) error { +func pruneProjectDeploymentHistory(ctx context.Context, projectID uint, keepCount int, preserveCandidateID uint) error { if keepCount <= 0 { return nil } @@ -390,7 +445,7 @@ func pruneProjectDeploymentHistory(ctx context.Context, projectID uint, keepCoun // that inserted another deployment between our list and delete. var lastErr error for pass := 0; pass < 2; pass++ { - deleted, err := pruneProjectDeploymentHistoryOnce(ctx, projectID, keepCount) + deleted, err := pruneProjectDeploymentHistoryOnce(ctx, projectID, keepCount, preserveCandidateID) if err != nil { lastErr = err break @@ -404,73 +459,79 @@ func pruneProjectDeploymentHistory(ctx context.Context, projectID uint, keepCoun // pruneProjectDeploymentHistoryOnce performs one list → select → delete cycle. // Returns the number of deployments deleted from the database. -func pruneProjectDeploymentHistoryOnce(ctx context.Context, projectID uint, keepCount int) (int, error) { - project, err := model.GetPagesProjectByID(ctx, projectID) - if err != nil { - return 0, fmt.Errorf("load pages project: %w", err) - } - deployments, err := model.ListPagesDeployments(ctx, projectID) - if err != nil { - return 0, fmt.Errorf("list pages deployments: %w", err) - } - if len(deployments) <= keepCount { - return 0, nil - } - - var activeID uint - if project.ActiveDeploymentID != nil { - activeID = *project.ActiveDeploymentID - } - toDelete := selectDeploymentsToPrune(deployments, activeID, keepCount) - if len(toDelete) == 0 { - return 0, nil - } - - // Delete metadata in one transaction so partial prune does not leave - // orphan file-list rows without a parent deployment. - if err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { +func pruneProjectDeploymentHistoryOnce(ctx context.Context, projectID uint, keepCount int, preserveCandidateID uint) (int, error) { + var deletedDeployments []model.PagesDeployment + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { + return fmt.Errorf("load pages project: %w", err) + } + var deployments []model.PagesDeployment + if err := tx.Where("project_id = ?", projectID).Order("id desc").Find(&deployments).Error; err != nil { + return fmt.Errorf("list pages deployments: %w", err) + } + var activeID uint + if project.ActiveDeploymentID != nil { + activeID = *project.ActiveDeploymentID + } + // Preserve mode is signaled by a non-zero candidate ID, but the lock-time + // newest non-active deployment wins. This prevents concurrent upload A/B + // prune passes from deleting each other's newer candidate. + resolvedCandidateID := resolveLatestCandidateID(deployments, activeID, preserveCandidateID != 0) + toDelete := selectDeploymentsToPrune(deployments, activeID, resolvedCandidateID, keepCount) for index := range toDelete { deployment := toDelete[index] - // Never delete the active deployment even if project pointer raced. - if activeID != 0 && deployment.ID == activeID { - continue - } - if project.ActiveDeploymentID != nil && deployment.ID == *project.ActiveDeploymentID { - continue - } if err := tx.Where("deployment_id = ?", deployment.ID).Delete(&model.PagesDeploymentFile{}).Error; err != nil { return fmt.Errorf("delete deployment files id=%d: %w", deployment.ID, err) } - if err := tx.Where("id = ? AND project_id = ?", deployment.ID, projectID). - Delete(&model.PagesDeployment{}).Error; err != nil { - return fmt.Errorf("delete deployment id=%d: %w", deployment.ID, err) + result := tx.Where("id = ? AND project_id = ?", deployment.ID, projectID). + Delete(&model.PagesDeployment{}) + if result.Error != nil { + return fmt.Errorf("delete deployment id=%d: %w", deployment.ID, result.Error) + } + if result.RowsAffected == 1 { + deletedDeployments = append(deletedDeployments, deployment) } } return nil - }); err != nil { + }) + if err != nil { return 0, err } - // Artifacts are best-effort outside the transaction (object storage I/O). - for index := range toDelete { - deployment := toDelete[index] - if activeID != 0 && deployment.ID == activeID { - continue - } - removeDeploymentArtifact(ctx, &deployment) + for index := range deletedDeployments { + deployment := deletedDeployments[index] + removeDeploymentArtifact(ctx, projectID, &deployment) } logger.InfoF(ctx, "[Pages] pruned deployment history: project_id=%d keep=%d deleted=%d", - projectID, keepCount, len(toDelete), + projectID, keepCount, len(deletedDeployments), ) - return len(toDelete), nil + return len(deletedDeployments), nil +} + +func resolveLatestCandidateID(deployments []model.PagesDeployment, activeID uint, preserve bool) uint { + if !preserve { + return 0 + } + for _, deployment := range deployments { + if deployment.ID != activeID { + return deployment.ID + } + } + return 0 } // selectDeploymentsToPrune returns deployments that should be removed under the // "at most keepCount, always keep active, fill with newest" policy. // deployments must be ordered newest-first (id desc). -func selectDeploymentsToPrune(deployments []model.PagesDeployment, activeID uint, keepCount int) []model.PagesDeployment { +func selectDeploymentsToPrune( + deployments []model.PagesDeployment, + activeID uint, + preserveCandidateID uint, + keepCount int, +) []model.PagesDeployment { if keepCount <= 0 || len(deployments) <= keepCount { return nil } @@ -486,6 +547,17 @@ func selectDeploymentsToPrune(deployments []model.PagesDeployment, activeID uint } } } + // A freshly uploaded manual candidate is temporarily protected in addition + // to the active deployment. This intentionally permits two rows when the + // configured history limit is one. + if preserveCandidateID != 0 { + for _, deployment := range deployments { + if deployment.ID == preserveCandidateID { + keepIDs[preserveCandidateID] = struct{}{} + break + } + } + } // 2) Fill remaining slots from newest to oldest. for _, deployment := range deployments { if len(keepIDs) >= keepCount { @@ -521,26 +593,69 @@ func ActivateDeployment(ctx context.Context, projectID uint, deploymentID uint) if deployment.ProjectID != project.ID { return nil, errors.New(errPagesDeploymentMismatch) } + if deployment.UploadID == 0 { + if err = ensureDeploymentUploadRecord(ctx, deployment); err != nil { + return nil, err + } + } now := time.Now() if err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { + return err + } + var deployment model.PagesDeployment + if err := tx.First(&deployment, deploymentID).Error; err != nil { + return err + } + if deployment.ProjectID != project.ID { + return errors.New(errPagesDeploymentMismatch) + } + rootDir, err := validateAndNormalizePagesRootDir(project.RootDir) + if err != nil { + return err + } + entryFile, err := validateAndNormalizePagesEntryFile(project.EntryFile) + if err != nil { + return err + } + if err := ensureDeploymentEntry(tx, deployment.ID, rootDir, entryFile); err != nil { + return err + } + var uploadRecord model.Upload + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ?", deployment.UploadID). + First(&uploadRecord).Error; err != nil { + return errors.New(errPagesPackageUploadMissing) + } + if uploadRecord.Status != model.UploadStatusUsed || uploadRecord.Type != upload.ReservedPagesDeploymentType { + return errors.New(errPagesPackageUploadMissing) + } if err := tx.Model(&model.PagesDeployment{}). Where("project_id = ?", project.ID). Update("status", model.PagesDeploymentStatusUploaded).Error; err != nil { return err } - if err := tx.Model(deployment).Updates(map[string]any{ + if err := tx.Model(&deployment).Updates(map[string]any{ "status": model.PagesDeploymentStatusActive, "activated_at": &now, }).Error; err != nil { return err } - return tx.Model(project).Updates(map[string]any{ + return tx.Model(&project).Updates(map[string]any{ "active_deployment_id": deployment.ID, }).Error }); err != nil { return nil, err } - return GetProject(ctx, project.ID) + limits := resolvePagesLimits(ctx) + if pruneErr := pruneProjectDeploymentHistory(ctx, projectID, limits.HistoryCount, 0); pruneErr != nil { + logger.ErrorF(ctx, + "[Pages] strict prune after activation failed: project_id=%d keep=%d error=%v", + projectID, limits.HistoryCount, pruneErr, + ) + } + return GetProject(ctx, projectID) } // GetDeploymentPackageHash returns the upload SHA-256 hash of the deployment package. @@ -728,22 +843,97 @@ func hydrateLegacyDeploymentUpload( ctx, artifactPath, deployment.Checksum, - project.Slug, + project.ID, fmt.Sprintf("pages-deployment-%d.zip", deployment.ID), pagesarchive.FormatZip, ) if err != nil { return nil, err } - if err := db.DB(ctx).Model(deployment).Updates(map[string]any{ - "upload_id": ingestResult.Upload.ID, - "artifact_path": "", - }).Error; err != nil { + winnerUploadID, err := attachLegacyDeploymentUpload( + ctx, + project.ID, + deployment.ID, + ingestResult.Upload.ID, + ) + if ingestResult.Created && (err != nil || winnerUploadID != ingestResult.Upload.ID) { + if removeErr := removePagesUploadIfUnreferenced(ctx, project.ID, ingestResult.Upload.ID); removeErr != nil { + logger.ErrorF(ctx, + "[Pages] compensate legacy deployment upload failed: project_id=%d upload_id=%d error=%v", + project.ID, ingestResult.Upload.ID, removeErr, + ) + } + } + if err != nil { return nil, err } - deployment.UploadID = ingestResult.Upload.ID + winner, err := upload.GetActiveUpload(ctx, winnerUploadID) + if err != nil { + return nil, err + } + deployment.UploadID = winnerUploadID deployment.ArtifactPath = "" - return &ingestResult.Upload, nil + return &winner, nil +} + +func attachLegacyDeploymentUpload( + ctx context.Context, + projectID uint, + deploymentID uint, + uploadID uint64, +) (uint64, error) { + winnerUploadID := uint64(0) + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var err error + winnerUploadID, err = attachLegacyDeploymentUploadTx(tx, projectID, deploymentID, uploadID) + return err + }) + return winnerUploadID, err +} + +func attachLegacyDeploymentUploadTx( + tx *gorm.DB, + projectID uint, + deploymentID uint, + uploadID uint64, +) (uint64, error) { + var lockedProject model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&lockedProject, projectID).Error; err != nil { + return 0, err + } + var lockedDeployment model.PagesDeployment + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&lockedDeployment, deploymentID).Error; err != nil { + return 0, err + } + if lockedDeployment.ProjectID != lockedProject.ID { + return 0, errors.New(errPagesDeploymentMismatch) + } + if lockedDeployment.UploadID != 0 { + return lockedDeployment.UploadID, nil + } + + var uploadRecord model.Upload + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ?", uploadID). + First(&uploadRecord).Error; err != nil { + return 0, err + } + if uploadRecord.Status != model.UploadStatusUsed || uploadRecord.Type != upload.ReservedPagesDeploymentType { + return 0, errors.New(errPagesPackageUploadMissing) + } + result := tx.Model(&model.PagesDeployment{}). + Where("id = ? AND project_id = ? AND upload_id = 0", lockedDeployment.ID, lockedProject.ID). + Updates(map[string]any{ + "upload_id": uploadRecord.ID, + "artifact_path": "", + }) + if result.Error != nil { + return 0, result.Error + } + if result.RowsAffected != 1 { + return 0, errors.New(errPagesPackageUploadMissing) + } + return uploadRecord.ID, nil } // ensureDeploymentInActiveSnapshot allows download of a specific deployment when @@ -842,30 +1032,34 @@ func parseSnapshotRoutes(snapshotJSON string) ([]snapshotRouteRef, error) { // DeleteDeployment 删除 Pages 部署。 func DeleteDeployment(ctx context.Context, projectID uint, deploymentID uint) error { - project, err := model.GetPagesProjectByID(ctx, projectID) - if err != nil { - return err - } - deployment, err := model.GetPagesDeploymentByID(ctx, deploymentID) - if err != nil { - return err - } - if deployment.ProjectID != project.ID { - return errors.New(errPagesDeploymentMismatch) - } - if project.ActiveDeploymentID != nil && *project.ActiveDeploymentID == deployment.ID { - return errors.New(errPagesDeleteActiveDeploy) - } - return db.DB(ctx).Transaction(func(tx *gorm.DB) error { - if err := tx.Where("deployment_id = ?", deployment.ID).Delete(&model.PagesDeploymentFile{}).Error; err != nil { + var removed model.PagesDeployment + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { return err } - if err := tx.Delete(deployment).Error; err != nil { + if err := tx.First(&removed, deploymentID).Error; err != nil { + return err + } + if removed.ProjectID != project.ID { + return errors.New(errPagesDeploymentMismatch) + } + if project.ActiveDeploymentID != nil && *project.ActiveDeploymentID == removed.ID { + return errors.New(errPagesDeleteActiveDeploy) + } + if err := tx.Where("deployment_id = ?", removed.ID).Delete(&model.PagesDeploymentFile{}).Error; err != nil { + return err + } + if err := tx.Delete(&removed).Error; err != nil { return err } - removeDeploymentArtifact(ctx, deployment) return nil }) + if err != nil { + return err + } + removeDeploymentArtifact(ctx, projectID, &removed) + return nil } func buildProject(existing *model.PagesProject, input Input) (*model.PagesProject, error) { @@ -923,7 +1117,11 @@ func buildProject(existing *model.PagesProject, input Input) (*model.PagesProjec return nil, err } existing.RootDir = rootDir - existing.EntryFile = normalizePagesEntryFile(input.EntryFile) + entryFile, err := validateAndNormalizePagesEntryFile(input.EntryFile) + if err != nil { + return nil, err + } + existing.EntryFile = entryFile return existing, nil } diff --git a/internal/apps/openflare/pages/logics_test.go b/internal/apps/openflare/pages/logics_test.go index ba3d52ff..230f3352 100644 --- a/internal/apps/openflare/pages/logics_test.go +++ b/internal/apps/openflare/pages/logics_test.go @@ -17,6 +17,7 @@ import ( "path/filepath" "testing" + "github.com/Rain-kl/Wavelet/internal/apps/upload" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" "github.com/Rain-kl/Wavelet/internal/repository" @@ -163,6 +164,89 @@ func TestCreateProjectRejectsUnsafeFallbackPath(t *testing.T) { assert.Contains(t, err.Error(), "回退路径") } +func TestCreateProjectRejectsUnsafeContentPaths(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + + rootDirs := []string{"/public", "public/../dist", "C:/public", `public\\dist`, "./public", "public\x00dist"} + for index, rootDir := range rootDirs { + _, err := CreateProject(ctx, Input{ + Name: fmt.Sprintf("Unsafe Root %d", index), + Slug: fmt.Sprintf("unsafe-root-%d", index), + RootDir: rootDir, + EntryFile: "index.html", + }) + require.Error(t, err, rootDir) + } + + entryFiles := []string{"/index.html", "../index.html", "C:/index.html", `public\\index.html`, "./index.html", "index.html;bad"} + for index, entryFile := range entryFiles { + _, err := CreateProject(ctx, Input{ + Name: fmt.Sprintf("Unsafe Entry %d", index), + Slug: fmt.Sprintf("unsafe-entry-%d", index), + EntryFile: entryFile, + }) + require.Error(t, err, entryFile) + } +} + +func TestUpdateProjectValidatesActiveDeploymentEntry(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + ctx := context.Background() + + project, err := CreateProject(ctx, Input{ + Name: "Content Root", + Slug: "content-root", + Enabled: true, + EntryFile: "index.html", + }) + require.NoError(t, err) + deployment, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "site.zip", testPagesZip(t, map[string]string{ + "index.html": "root", + "dist/index.html": "dist", + })), "user:1") + require.NoError(t, err) + staleCandidate, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "stale.zip", testPagesZip(t, map[string]string{ + "index.html": "stale", + })), "user:1") + require.NoError(t, err) + _, err = ActivateDeployment(ctx, project.ID, deployment.ID) + require.NoError(t, err) + + updated, err := UpdateProject(ctx, project.ID, Input{ + Name: project.Name, + Slug: project.Slug, + Enabled: true, + RootDir: "dist", + EntryFile: "index.html", + }) + require.NoError(t, err) + assert.Equal(t, "dist", updated.RootDir) + + _, err = ActivateDeployment(ctx, project.ID, staleCandidate.ID) + require.Error(t, err) + assert.Contains(t, err.Error(), errPagesEntryFileMissing) + + _, err = UpdateProject(ctx, project.ID, Input{ + Name: project.Name, + Slug: project.Slug, + Enabled: true, + RootDir: "missing", + EntryFile: "index.html", + }) + require.Error(t, err) + assert.Contains(t, err.Error(), errPagesEntryFileMissing) + + stored, err := model.GetPagesProjectByID(ctx, project.ID) + require.NoError(t, err) + assert.Equal(t, "dist", stored.RootDir) + assert.Equal(t, "index.html", stored.EntryFile) +} + func TestUploadDeploymentAcceptsZeroByteFiles(t *testing.T) { cleanup := setupPagesTestDB(t) defer cleanup() @@ -213,6 +297,13 @@ func TestUploadDeploymentStoresPackageInUploadFramework(t *testing.T) { var uploadCount int64 require.NoError(t, db.DB(ctx).Model(&model.Upload{}).Count(&uploadCount).Error) assert.Equal(t, int64(1), uploadCount) + var uploadRecord model.Upload + require.NoError(t, db.DB(ctx).First(&uploadRecord, storedDeployment.UploadID).Error) + assert.Equal(t, upload.ReservedPagesDeploymentType, uploadRecord.Type) + assert.Equal(t, pagesIngestMarkerV2, uploadRecord.Metadata.Extra[pagesIngestMarkerKey]) + assert.Equal(t, fmt.Sprint(project.ID), uploadRecord.Metadata.Extra[pagesProjectIDMetadataKey]) + assert.NotContains(t, uploadRecord.Metadata.Extra, "project_slug") + assert.NotContains(t, uploadRecord.Metadata.Extra, "format") } func TestOpenDeploymentPackageHydratesLegacyArtifactPath(t *testing.T) { @@ -448,34 +539,42 @@ func TestSelectDeploymentsToPruneKeepsActiveAndNewest(t *testing.T) { {ID: 2, ProjectID: 1}, {ID: 1, ProjectID: 1}, } - toDelete := selectDeploymentsToPrune(deployments, 1, 2) + toDelete := selectDeploymentsToPrune(deployments, 1, 0, 2) require.Len(t, toDelete, 2) assert.Equal(t, uint(3), toDelete[0].ID) assert.Equal(t, uint(2), toDelete[1].ID) // active is newest; keep=2 → keep {4,3}, prune {2,1} - toDelete = selectDeploymentsToPrune(deployments, 4, 2) + toDelete = selectDeploymentsToPrune(deployments, 4, 0, 2) require.Len(t, toDelete, 2) assert.Equal(t, uint(2), toDelete[0].ID) assert.Equal(t, uint(1), toDelete[1].ID) // no active; keep=2 → keep {4,3} - toDelete = selectDeploymentsToPrune(deployments, 0, 2) + toDelete = selectDeploymentsToPrune(deployments, 0, 0, 2) require.Len(t, toDelete, 2) assert.Equal(t, uint(2), toDelete[0].ID) assert.Equal(t, uint(1), toDelete[1].ID) // keep=1 with active → only active, prune the rest - toDelete = selectDeploymentsToPrune(deployments, 2, 1) + toDelete = selectDeploymentsToPrune(deployments, 2, 0, 1) require.Len(t, toDelete, 3) for _, item := range toDelete { assert.NotEqual(t, uint(2), item.ID) } // already within limit - assert.Nil(t, selectDeploymentsToPrune(deployments[:2], 4, 2)) + assert.Nil(t, selectDeploymentsToPrune(deployments[:2], 4, 0, 2)) // unlimited - assert.Nil(t, selectDeploymentsToPrune(deployments, 1, 0)) + assert.Nil(t, selectDeploymentsToPrune(deployments, 1, 0, 0)) + + // history=1 temporarily preserves active plus the freshly uploaded candidate. + toDelete = selectDeploymentsToPrune(deployments, 2, 4, 1) + require.Len(t, toDelete, 2) + assert.Equal(t, uint(3), toDelete[0].ID) + assert.Equal(t, uint(1), toDelete[1].ID) + assert.Equal(t, uint(4), resolveLatestCandidateID(deployments, 2, true)) + assert.Zero(t, resolveLatestCandidateID(deployments, 2, false)) } func TestPruneProjectDeploymentHistory(t *testing.T) { @@ -540,6 +639,131 @@ func TestPruneProjectDeploymentHistory(t *testing.T) { assert.True(t, hasLatest, "newest deployment must fill remaining slot") } +func TestHistoryCountOnePreservesFreshCandidateUntilActivation(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + ctx := context.Background() + + require.NoError(t, db.DB(ctx).Model(&model.SystemConfig{}). + Where("key = ?", model.ConfigKeyPagesMaxHistoryCount). + Update("value", "1").Error) + require.NoError(t, repository.InvalidateSystemConfigCache(ctx, model.ConfigKeyPagesMaxHistoryCount)) + + project, err := CreateProject(ctx, Input{Name: "Single History", Slug: "single-history", Enabled: true}) + require.NoError(t, err) + active, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "v1.zip", testPagesZip(t, map[string]string{ + "index.html": "v1", + })), "user:1") + require.NoError(t, err) + _, err = ActivateDeployment(ctx, project.ID, active.ID) + require.NoError(t, err) + + oldCandidate, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "v2.zip", testPagesZip(t, map[string]string{ + "index.html": "v2", + })), "user:1") + require.NoError(t, err) + deployments, err := model.ListPagesDeployments(ctx, project.ID) + require.NoError(t, err) + require.Len(t, deployments, 2) + + newCandidate, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "v3.zip", testPagesZip(t, map[string]string{ + "index.html": "v3", + })), "user:1") + require.NoError(t, err) + deployments, err = model.ListPagesDeployments(ctx, project.ID) + require.NoError(t, err) + require.Len(t, deployments, 2) + kept := map[uint]bool{} + for _, deployment := range deployments { + kept[deployment.ID] = true + } + assert.True(t, kept[active.ID]) + assert.True(t, kept[newCandidate.ID]) + assert.False(t, kept[oldCandidate.ID]) + var removedUpload model.Upload + require.NoError(t, db.DB(ctx).First(&removedUpload, oldCandidate.UploadID).Error) + assert.Equal(t, model.UploadStatusDeleted, removedUpload.Status) + + _, err = ActivateDeployment(ctx, project.ID, newCandidate.ID) + require.NoError(t, err) + deployments, err = model.ListPagesDeployments(ctx, project.ID) + require.NoError(t, err) + require.Len(t, deployments, 1) + assert.Equal(t, newCandidate.ID, deployments[0].ID) +} + +func TestPruneUsesLockTimeNewestCandidateInsteadOfStaleCaller(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + ctx := context.Background() + + project, err := CreateProject(ctx, Input{Name: "Concurrent Candidate", Slug: "concurrent-candidate", Enabled: true}) + require.NoError(t, err) + active, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "v1.zip", testPagesZip(t, map[string]string{ + "index.html": "v1", + })), "user:1") + require.NoError(t, err) + _, err = ActivateDeployment(ctx, project.ID, active.ID) + require.NoError(t, err) + staleCandidate, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "v2.zip", testPagesZip(t, map[string]string{ + "index.html": "v2", + })), "user:1") + require.NoError(t, err) + newCandidate, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "v3.zip", testPagesZip(t, map[string]string{ + "index.html": "v3", + })), "user:1") + require.NoError(t, err) + + deleted, err := pruneProjectDeploymentHistoryOnce(ctx, project.ID, 1, staleCandidate.ID) + require.NoError(t, err) + assert.Equal(t, 1, deleted) + deployments, err := model.ListPagesDeployments(ctx, project.ID) + require.NoError(t, err) + require.Len(t, deployments, 2) + kept := map[uint]bool{} + for _, deployment := range deployments { + kept[deployment.ID] = true + } + assert.True(t, kept[active.ID]) + assert.True(t, kept[newCandidate.ID]) + assert.False(t, kept[staleCandidate.ID]) +} + +func TestDeleteDeploymentAndProjectSoftDeleteUnreferencedArtifacts(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + ctx := context.Background() + + project, err := CreateProject(ctx, Input{Name: "Delete Artifacts", Slug: "delete-artifacts", Enabled: true}) + require.NoError(t, err) + first, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "first.zip", testPagesZip(t, map[string]string{ + "index.html": "first", + })), "user:1") + require.NoError(t, err) + second, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "second.zip", testPagesZip(t, map[string]string{ + "index.html": "second", + })), "user:1") + require.NoError(t, err) + + require.NoError(t, DeleteDeployment(ctx, project.ID, second.ID)) + var secondUpload model.Upload + require.NoError(t, db.DB(ctx).First(&secondUpload, second.UploadID).Error) + assert.Equal(t, model.UploadStatusDeleted, secondUpload.Status) + + require.NoError(t, DeleteProject(ctx, project.ID)) + var firstUpload model.Upload + require.NoError(t, db.DB(ctx).First(&firstUpload, first.UploadID).Error) + assert.Equal(t, model.UploadStatusDeleted, firstUpload.Status) + _, err = model.GetPagesProjectByID(ctx, project.ID) + assert.Error(t, err) +} + func testPagesTarGz(t *testing.T, files map[string]string) []byte { t.Helper() diff --git a/internal/apps/openflare/pages/package_metadata.go b/internal/apps/openflare/pages/package_metadata.go new file mode 100644 index 00000000..0e99c358 --- /dev/null +++ b/internal/apps/openflare/pages/package_metadata.go @@ -0,0 +1,64 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "errors" + "fmt" + "strings" + + "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/model" +) + +// ProjectLatestPackageMetadata describes the active package limits published to Agents. +type ProjectLatestPackageMetadata struct { + DeploymentID uint + Hash string + PackageSize int64 + FileCount int + TotalSize int64 +} + +// GetProjectLatestPackageMetadata returns one coherent metadata snapshot for a +// project's currently active deployment. +func GetProjectLatestPackageMetadata(ctx context.Context, projectID uint) (*ProjectLatestPackageMetadata, error) { + deployment, err := resolveProjectActiveDeploymentForAgent(ctx, projectID) + if err != nil { + return nil, err + } + if deployment.UploadID == 0 { + if err := ensureDeploymentUploadRecord(ctx, deployment); err != nil { + return nil, err + } + deployment, err = model.GetPagesDeploymentByID(ctx, deployment.ID) + if err != nil { + return nil, err + } + } + if deployment.UploadID == 0 { + return nil, errors.New(errPagesDeploymentNotFound) + } + + uploadRecord, err := upload.GetActiveUpload(ctx, deployment.UploadID) + if err != nil { + return nil, fmt.Errorf("pages 部署包不存在: %w", err) + } + hash := strings.TrimSpace(uploadRecord.Hash) + if hash == "" { + hash = strings.TrimSpace(deployment.Checksum) + } + if hash == "" { + return nil, errors.New(errPagesDeploymentHashMissing) + } + + return &ProjectLatestPackageMetadata{ + DeploymentID: deployment.ID, + Hash: hash, + PackageSize: uploadRecord.FileSize, + FileCount: deployment.FileCount, + TotalSize: deployment.TotalSize, + }, nil +} diff --git a/internal/apps/openflare/pages/package_metadata_test.go b/internal/apps/openflare/pages/package_metadata_test.go new file mode 100644 index 00000000..d192a8c9 --- /dev/null +++ b/internal/apps/openflare/pages/package_metadata_test.go @@ -0,0 +1,81 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "testing" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" +) + +func TestGetProjectLatestPackageMetadataPreservesZeroTotalSize(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + + ctx := context.Background() + project, err := CreateProject(ctx, Input{ + Name: "Empty Files", + Slug: "empty-files", + Enabled: true, + EntryFile: "index.html", + }) + if err != nil { + t.Fatalf("CreateProject() error = %v", err) + } + packageBytes := testPagesZip(t, map[string]string{ + "index.html": "", + ".gitkeep": "", + }) + deployment, err := UploadDeployment( + ctx, + project.ID, + testPagesMultipartFile(t, "empty-files.zip", packageBytes), + "test", + ) + if err != nil { + t.Fatalf("UploadDeployment() error = %v", err) + } + if _, err := ActivateDeployment(ctx, project.ID, deployment.ID); err != nil { + t.Fatalf("ActivateDeployment() error = %v", err) + } + if err := db.DB(ctx).Create(&model.ConfigVersion{ + Version: "v-package-metadata", + SnapshotJSON: fmt.Sprintf( + `{"routes":[{"upstream_type":"pages","pages_project_id":%d}]}`, + project.ID, + ), + SupportFilesJSON: "[]", + Checksum: "package-metadata-config", + IsActive: true, + CreatedBy: "test", + }).Error; err != nil { + t.Fatalf("create active ConfigVersion error = %v", err) + } + + got, err := GetProjectLatestPackageMetadata(ctx, project.ID) + if err != nil { + t.Fatalf("GetProjectLatestPackageMetadata(%d) error = %v", project.ID, err) + } + wantHashBytes := sha256.Sum256(packageBytes) + wantHash := hex.EncodeToString(wantHashBytes[:]) + if got.DeploymentID != deployment.ID || got.Hash != wantHash { + t.Errorf("GetProjectLatestPackageMetadata(%d) identity = (%d, %q), want (%d, %q)", + project.ID, got.DeploymentID, got.Hash, deployment.ID, wantHash) + } + if got.PackageSize != int64(len(packageBytes)) { + t.Errorf("GetProjectLatestPackageMetadata(%d).PackageSize = %d, want %d", + project.ID, got.PackageSize, len(packageBytes)) + } + if got.FileCount != 2 || got.TotalSize != 0 { + t.Errorf("GetProjectLatestPackageMetadata(%d) content = (%d files, %d bytes), want (2 files, 0 bytes)", + project.ID, got.FileCount, got.TotalSize) + } +} diff --git a/internal/apps/openflare/pages/rebind.go b/internal/apps/openflare/pages/rebind.go index d1950c9e..d26d34d8 100644 --- a/internal/apps/openflare/pages/rebind.go +++ b/internal/apps/openflare/pages/rebind.go @@ -7,6 +7,7 @@ import ( "context" "encoding/json" "fmt" + "path" "strings" "github.com/Rain-kl/Wavelet/internal/model" @@ -102,7 +103,10 @@ func rebindPagesRouteMaps(ctx context.Context, routes []map[string]json.RawMessa if err != nil { return false, err } - deployment := buildLivePagesDeployment(project, activeDeployment) + deployment, err := buildLivePagesDeployment(project, activeDeployment) + if err != nil { + return false, err + } projectIDCopy := project.ID originURL := fmt.Sprintf("openflare-pages://project/%d", project.ID) @@ -181,15 +185,26 @@ func loadActivePagesProject(ctx context.Context, projectID uint, siteName string return project, activeDeployment, nil } -func buildLivePagesDeployment(project *model.PagesProject, active *model.PagesDeployment) *openrestyrender.PagesDeployment { - entryFile := strings.TrimSpace(project.EntryFile) - if entryFile == "" { - entryFile = defaultPagesEntryFile +func buildLivePagesDeployment( + project *model.PagesProject, + active *model.PagesDeployment, +) (*openrestyrender.PagesDeployment, error) { + rootDir, err := validateAndNormalizePagesRootDir(project.RootDir) + if err != nil { + return nil, err + } + entryFile, err := validateAndNormalizePagesEntryFile(project.EntryFile) + if err != nil { + return nil, err } fallbackPath := strings.TrimSpace(project.SPAFallbackPath) if fallbackPath == "" { fallbackPath = defaultPagesFallbackPath } + localRoot := openrestyrender.PagesProjectLocalRoot(project.ID) + if rootDir != "" { + localRoot = path.Join(localRoot, rootDir) + } return &openrestyrender.PagesDeployment{ ProjectID: project.ID, ProjectSlug: strings.TrimSpace(project.Slug), @@ -203,8 +218,8 @@ func buildLivePagesDeployment(project *model.PagesProject, active *model.PagesDe APIProxyPath: strings.TrimSpace(project.APIProxyPath), APIProxyPass: strings.TrimSpace(project.APIProxyPass), APIProxyRewrite: strings.TrimSpace(project.APIProxyRewrite), - LocalRoot: openrestyrender.PagesProjectLocalRoot(project.ID), - } + LocalRoot: localRoot, + }, nil } func rawJSONString(raw json.RawMessage) (string, bool) { diff --git a/internal/apps/openflare/pages/rebind_test.go b/internal/apps/openflare/pages/rebind_test.go index c37602d1..94dfed4d 100644 --- a/internal/apps/openflare/pages/rebind_test.go +++ b/internal/apps/openflare/pages/rebind_test.go @@ -20,9 +20,11 @@ func TestRebindSnapshotPagesToCurrentActive(t *testing.T) { ctx := context.Background() project, err := CreateProject(ctx, Input{ - Name: "Rebind Site", - Slug: "rebind-site", - Enabled: true, + Name: "Rebind Site", + Slug: "rebind-site", + Enabled: true, + RootDir: "public/site", + EntryFile: "index.html", }) require.NoError(t, err) @@ -84,4 +86,5 @@ func TestRebindSnapshotPagesToCurrentActive(t *testing.T) { deployment := route["pages_deployment"].(map[string]any) assert.EqualValues(t, active.ID, deployment["deployment_id"]) assert.Equal(t, "new-checksum", deployment["checksum"]) + assert.Equal(t, "__OPENFLARE_PAGES_DIR__/projects/1/current/public/site", deployment["local_root"]) } diff --git a/internal/apps/openflare/pages/routers.go b/internal/apps/openflare/pages/routers.go index 5e117fb3..6767f478 100644 --- a/internal/apps/openflare/pages/routers.go +++ b/internal/apps/openflare/pages/routers.go @@ -4,11 +4,14 @@ package pages import ( + "fmt" "net/http" "strconv" + "github.com/Rain-kl/Wavelet/internal/apps/oauth" "github.com/Rain-kl/Wavelet/internal/apps/openflare/apiutil" "github.com/Rain-kl/Wavelet/internal/common/response" + "github.com/Rain-kl/Wavelet/internal/model" "github.com/gin-gonic/gin" ) @@ -33,6 +36,15 @@ func deploymentIDParam(c *gin.Context) (uint, bool) { return uint(id64), true } +func currentPagesActor(c *gin.Context) (string, bool) { + user, ok := oauth.GetFromContext[*model.User](c, oauth.UserObjKey) + if !ok || user == nil || user.ID == 0 { + response.AbortUnauthorized(c, errPagesActorMissing) + return "", false + } + return fmt.Sprintf("user:%d", user.ID), true +} + // ListProjectsHandler 列出全部 Pages 项目。 // @Summary 列出 Pages 项目 // @Description 返回全部 OpenFlare Pages 项目，需要管理员权限 @@ -214,7 +226,11 @@ func UploadDeploymentHandler(c *gin.Context) { response.AbortBadRequest(c, errPagesPackageMissing) return } - deployment, err := UploadDeployment(c.Request.Context(), id, file, "") + actor, ok := currentPagesActor(c) + if !ok { + return + } + deployment, err := UploadDeployment(c.Request.Context(), id, file, actor) if handleLogicError(c, err) { return } @@ -247,7 +263,11 @@ func UploadDeploymentFromURLHandler(c *gin.Context) { response.AbortBadRequest(c, errPagesPackageURLRequired) return } - deployment, err := UploadDeploymentFromURL(c.Request.Context(), id, req.URL, "") + actor, ok := currentPagesActor(c) + if !ok { + return + } + deployment, err := UploadDeploymentFromURL(c.Request.Context(), id, req.URL, actor) if handleLogicError(c, err) { return } diff --git a/internal/apps/openflare/pages/routers_test.go b/internal/apps/openflare/pages/routers_test.go new file mode 100644 index 00000000..c73a55c5 --- /dev/null +++ b/internal/apps/openflare/pages/routers_test.go @@ -0,0 +1,100 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "bytes" + "encoding/json" + "fmt" + "mime/multipart" + "net/http" + "net/http/httptest" + "strconv" + "testing" + + "github.com/Rain-kl/Wavelet/internal/apps/oauth" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/gin-gonic/gin" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestUploadDeploymentHandlerRecordsCurrentUserActor(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + + project, err := CreateProject(t.Context(), Input{Name: "Actor Upload", Slug: "actor-upload", Enabled: true}) + require.NoError(t, err) + packageBytes := testPagesZip(t, map[string]string{"index.html": "ok"}) + + var requestBody bytes.Buffer + writer := multipart.NewWriter(&requestBody) + part, err := writer.CreateFormFile("package", "site.zip") + require.NoError(t, err) + _, err = part.Write(packageBytes) + require.NoError(t, err) + require.NoError(t, writer.Close()) + + req := httptest.NewRequest(http.MethodPost, "/api/v1/d/pages/1/deployments/upload", &requestBody) + req.Header.Set("Content-Type", writer.FormDataContentType()) + recorder := httptest.NewRecorder() + c, _ := gin.CreateTestContext(recorder) + c.Request = req + c.Params = gin.Params{{Key: "id", Value: strconv.FormatUint(uint64(project.ID), 10)}} + oauth.SetToContext(c, oauth.UserObjKey, &model.User{ID: 42}) + + UploadDeploymentHandler(c) + assert.Equal(t, http.StatusOK, recorder.Code) + deployments, err := model.ListPagesDeployments(t.Context(), project.ID) + require.NoError(t, err) + require.Len(t, deployments, 1) + assert.Equal(t, "user:42", deployments[0].CreatedBy) +} + +func TestUploadDeploymentFromURLHandlerRecordsCurrentUserActor(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + + project, err := CreateProject(t.Context(), Input{Name: "Actor URL", Slug: "actor-url", Enabled: true}) + require.NoError(t, err) + packageBytes := testPagesZip(t, map[string]string{"index.html": "ok"}) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/zip") + w.Header().Set("Content-Disposition", `attachment; filename="site.zip"`) + w.WriteHeader(http.StatusOK) + _, _ = w.Write(packageBytes) + })) + defer server.Close() + + body, err := json.Marshal(UploadFromURLInput{URL: server.URL + "/site.zip"}) + require.NoError(t, err) + req := httptest.NewRequest(http.MethodPost, "/api/v1/d/pages/1/deployments/upload-from-url", bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + recorder := httptest.NewRecorder() + c, _ := gin.CreateTestContext(recorder) + c.Request = req + c.Params = gin.Params{{Key: "id", Value: fmt.Sprint(project.ID)}} + oauth.SetToContext(c, oauth.UserObjKey, &model.User{ID: 77}) + + UploadDeploymentFromURLHandler(c) + assert.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String()) + deployments, err := model.ListPagesDeployments(t.Context(), project.ID) + require.NoError(t, err) + require.Len(t, deployments, 1) + assert.Equal(t, "user:77", deployments[0].CreatedBy) +} + +func TestCurrentPagesActorRejectsMissingUser(t *testing.T) { + recorder := httptest.NewRecorder() + c, _ := gin.CreateTestContext(recorder) + + actor, ok := currentPagesActor(c) + assert.False(t, ok) + assert.Empty(t, actor) + assert.True(t, c.IsAborted()) +} diff --git a/internal/apps/openflare/proxy_route/logics.go b/internal/apps/openflare/proxy_route/logics.go index 1b781a84..1570be32 100644 --- a/internal/apps/openflare/proxy_route/logics.go +++ b/internal/apps/openflare/proxy_route/logics.go @@ -6,12 +6,14 @@ package proxy_route import ( "context" "errors" + "sort" "strings" "time" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" "gorm.io/gorm" + "gorm.io/gorm/clause" ) // CustomHeaderInput 自定义响应头。 @@ -122,6 +124,9 @@ func CreateProxyRoute(ctx context.Context, input Input) (*View, error) { return nil, err } if err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + if err := lockPagesProjectsForRouteMutation(tx, 0, route); err != nil { + return err + } if err := tx.Create(route).Error; err != nil { return err } @@ -141,11 +146,15 @@ func UpdateProxyRoute(ctx context.Context, id uint, input Input) (*View, error) if err != nil { return nil, err } + previousPagesProjectID := pagesProjectIDForRoute(route) route, _, err = buildProxyRoute(ctx, route, input) if err != nil { return nil, err } if err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + if err := lockPagesProjectsForRouteMutation(tx, previousPagesProjectID, route); err != nil { + return err + } if err := updateProxyRouteRecord(tx, route); err != nil { return err } @@ -159,6 +168,58 @@ func UpdateProxyRoute(ctx context.Context, id uint, input Input) (*View, error) return buildProxyRouteView(ctx, route) } +func pagesProjectIDForRoute(route *model.ProxyRoute) uint { + if route == nil || route.UpstreamType != proxyRouteUpstreamTypePages || route.PagesProjectID == nil { + return 0 + } + return *route.PagesProjectID +} + +func lockPagesProjectsForRouteMutation(tx *gorm.DB, previousProjectID uint, route *model.ProxyRoute) error { + nextProjectID := pagesProjectIDForRoute(route) + var projectIDs []uint + if previousProjectID != 0 { + projectIDs = append(projectIDs, previousProjectID) + } + if nextProjectID != 0 && nextProjectID != previousProjectID { + projectIDs = append(projectIDs, nextProjectID) + } + sort.Slice(projectIDs, func(i int, j int) bool { return projectIDs[i] < projectIDs[j] }) + + for _, projectID := range projectIDs { + var project model.PagesProject + err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).First(&project, projectID).Error + if errors.Is(err, gorm.ErrRecordNotFound) && projectID != nextProjectID { + continue + } + if errors.Is(err, gorm.ErrRecordNotFound) { + return errors.New(errProxyRoutePagesNotFound) + } + if err != nil { + return err + } + if projectID == nextProjectID { + if err := validateLockedPagesRouteProject(&project); err != nil { + return err + } + } + } + return nil +} + +func validateLockedPagesRouteProject(project *model.PagesProject) error { + if project == nil { + return errors.New(errProxyRoutePagesNotFound) + } + if !project.Enabled { + return errors.New(errProxyRoutePagesDisabled) + } + if project.ActiveDeploymentID == nil || *project.ActiveDeploymentID == 0 { + return errors.New(errProxyRoutePagesNoDeploy) + } + return nil +} + // DeleteProxyRoute 删除代理规则。 func DeleteProxyRoute(ctx context.Context, id uint) error { if _, err := model.GetProxyRouteByID(ctx, id); err != nil { diff --git a/internal/apps/openflare/proxy_route/logics_test.go b/internal/apps/openflare/proxy_route/logics_test.go index 1af61011..2267a089 100644 --- a/internal/apps/openflare/proxy_route/logics_test.go +++ b/internal/apps/openflare/proxy_route/logics_test.go @@ -19,7 +19,14 @@ func setupProxyRouteTestDB(t *testing.T) func() { t.Helper() sqliteDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{DisableForeignKeyConstraintWhenMigrating: true}) require.NoError(t, err) - require.NoError(t, sqliteDB.AutoMigrate(&model.ProxyRoute{}, &model.Origin{}, &model.Zone{}, &model.ZoneDomain{}, &model.TLSCertificate{})) + require.NoError(t, sqliteDB.AutoMigrate( + &model.ProxyRoute{}, + &model.Origin{}, + &model.Zone{}, + &model.ZoneDomain{}, + &model.TLSCertificate{}, + &model.PagesProject{}, + )) db.SetDB(sqliteDB) return func() { db.SetDB(nil) } } @@ -81,6 +88,52 @@ func TestCreateProxyRouteHTTPSRequiresCoveringCertificate(t *testing.T) { require.EqualError(t, err, errProxyRouteCertRequired) } +func TestPagesRouteLocksAndRevalidatesTargetProject(t *testing.T) { + cleanup := setupProxyRouteTestDB(t) + defer cleanup() + ctx := context.Background() + domain := createZoneDomain(t, ctx, "pages.example.com", nil) + activeDeploymentID := uint(99) + project := &model.PagesProject{ + Name: "Pages Site", + Slug: "pages-site", + Enabled: true, + ActiveDeploymentID: &activeDeploymentID, + } + require.NoError(t, db.DB(ctx).Create(project).Error) + + view, err := CreateProxyRoute(ctx, Input{ + SiteName: "pages", + ZoneDomainIDs: []uint{domain.ID}, + UpstreamType: proxyRouteUpstreamTypePages, + PagesProjectID: &project.ID, + Enabled: true, + }) + require.NoError(t, err) + require.NotNil(t, view.PagesProjectID) + assert.Equal(t, project.ID, *view.PagesProjectID) + + require.NoError(t, db.DB(ctx).Delete(&model.PagesProject{}, project.ID).Error) + route := &model.ProxyRoute{UpstreamType: proxyRouteUpstreamTypePages, PagesProjectID: &project.ID} + err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + return lockPagesProjectsForRouteMutation(tx, 0, route) + }) + require.EqualError(t, err, errProxyRoutePagesNotFound) +} + +func TestRouteCanMoveAwayFromAlreadyMissingPagesProject(t *testing.T) { + cleanup := setupProxyRouteTestDB(t) + defer cleanup() + ctx := context.Background() + missingProjectID := uint(404) + route := &model.ProxyRoute{UpstreamType: "direct"} + + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + return lockPagesProjectsForRouteMutation(tx, missingProjectID, route) + }) + require.NoError(t, err) +} + func TestNormalizeCachePolicyDefaultsAndLegacy(t *testing.T) { assert.Equal(t, "", normalizeCachePolicy(false, "static")) // Empty/url on write = legacy all (compat); UI sends static explicitly for new default. diff --git a/internal/apps/upload/exports.go b/internal/apps/upload/exports.go index edbd7713..322acddf 100644 --- a/internal/apps/upload/exports.go +++ b/internal/apps/upload/exports.go @@ -8,6 +8,7 @@ import ( "github.com/Rain-kl/Wavelet/internal/apps/upload/filesrv" "github.com/Rain-kl/Wavelet/internal/apps/upload/handler" "github.com/Rain-kl/Wavelet/internal/apps/upload/ingest" + "github.com/Rain-kl/Wavelet/internal/apps/upload/shared" uploadstats "github.com/Rain-kl/Wavelet/internal/apps/upload/stats" uploadtask "github.com/Rain-kl/Wavelet/internal/apps/upload/task" "github.com/Rain-kl/Wavelet/internal/apps/upload/util" @@ -31,15 +32,17 @@ var ( // Programmatic ingest API var ( - Ingest = ingest.Ingest - Remove = ingest.Remove - RemoveOwned = ingest.RemoveOwned - FindByHash = ingest.FindByHash - GetActiveUpload = ingest.GetActive - OpenStoredUpload = ingest.OpenActiveObject - ActiveUploadHash = ingest.ActiveHash - ResolveLocalFile = ingest.ResolveLocalFile - IngestFromLocalPath = ingest.FromLocalPath + Ingest = ingest.Ingest + Remove = ingest.Remove + RemoveOwned = ingest.RemoveOwned + RemoveLockedTx = ingest.RemoveLockedTx + InvalidateUploadMetaCache = ingest.InvalidateUploadMetaCache + FindByHash = ingest.FindByHash + GetActiveUpload = ingest.GetActive + OpenStoredUpload = ingest.OpenActiveObject + ActiveUploadHash = ingest.ActiveHash + ResolveLocalFile = ingest.ResolveLocalFile + IngestFromLocalPath = ingest.FromLocalPath ) type ( @@ -54,6 +57,8 @@ const ( PolicyCreate = ingest.PolicyCreate PolicyDedupNewRecord = ingest.PolicyDedupNewRecord PolicyResolveExisting = ingest.PolicyResolveExisting + // ReservedPagesDeploymentType is managed exclusively by the Pages domain. + ReservedPagesDeploymentType = shared.ReservedPagesDeploymentType ) type ( @@ -69,6 +74,7 @@ type ( var ( ErrIngestForbidden = ingest.ErrForbidden ErrIngestStorageReadOnly = ingest.ErrStorageReadOnly + ErrReservedUploadType = ingest.ErrReservedUploadType ) // Cache management diff --git a/internal/apps/upload/handler/file_management.go b/internal/apps/upload/handler/file_management.go index 075d72e8..6bb69d46 100644 --- a/internal/apps/upload/handler/file_management.go +++ b/internal/apps/upload/handler/file_management.go @@ -4,6 +4,7 @@ package handler import ( + "errors" "net/http" "strconv" @@ -96,6 +97,7 @@ func ListFiles(c *gin.Context) { // @Success 200 {object} response.Any "删除成功" // @Failure 403 {object} response.Any "无权操作" // @Failure 404 {object} response.Any "文件不存在" +// @Failure 409 {object} response.Any "系统保留类型或存储只读" // @Router /api/v1/admin/uploads/{id} [delete] func DeleteFile(c *gin.Context) { ctx := c.Request.Context() @@ -111,6 +113,10 @@ func DeleteFile(c *gin.Context) { } if _, err := softDeleteUpload(ctx, uploadID); err != nil { + if errors.Is(err, ingest.ErrReservedUploadType) { + response.AbortConflict(c, shared.ErrReservedUploadType) + return + } if isRecordNotFound(err) { response.AbortNotFound(c, "文件记录未找到") return @@ -216,6 +222,7 @@ func ListMyFiles(c *gin.Context) { // @Success 200 {object} response.Any "删除成功" // @Failure 403 {object} response.Any "无权操作" // @Failure 404 {object} response.Any "文件不存在" +// @Failure 409 {object} response.Any "系统保留类型或存储只读" // @Router /api/v1/upload/{id} [delete] func DeleteMyFile(c *gin.Context) { currUser, _ := oauth.GetFromContext[*model.User](c, oauth.UserObjKey) @@ -232,11 +239,15 @@ func DeleteMyFile(c *gin.Context) { } if _, err := softDeleteOwnedUpload(ctx, currUser.ID, uploadID); err != nil { + if errors.Is(err, ingest.ErrReservedUploadType) { + response.AbortConflict(c, shared.ErrReservedUploadType) + return + } if isRecordNotFound(err) { response.AbortNotFound(c, "文件记录未找到") return } - if err == ingest.ErrForbidden { + if errors.Is(err, ingest.ErrForbidden) { response.AbortForbidden(c, "无权操作") return } diff --git a/internal/apps/upload/handler/routers.go b/internal/apps/upload/handler/routers.go index 025052c4..348f8f71 100644 --- a/internal/apps/upload/handler/routers.go +++ b/internal/apps/upload/handler/routers.go @@ -53,6 +53,7 @@ type batchDownloadRequest struct { // @Success 200 {object} response.Any{data=model.Upload} "上传成功" // @Failure 400 {object} response.Any "请求参数错误或文件受限" // @Failure 401 {object} response.Any "未登录" +// @Failure 409 {object} response.Any "系统保留类型或存储只读" // @Failure 500 {object} response.Any "内部错误" // @Router /api/v1/upload [post] // @@ -107,6 +108,10 @@ func UploadFile(c *gin.Context) { } uploadType := c.DefaultPostForm("type", "generic") + if uploadType == shared.ReservedPagesDeploymentType { + response.AbortConflict(c, shared.ErrReservedUploadType) + return + } accessMode, errMsg := resolveUploadAccessMode(c, uploadType) if errMsg != "" { diff --git a/internal/apps/upload/handler/routers_test.go b/internal/apps/upload/handler/routers_test.go index e8ddce20..75459ff8 100644 --- a/internal/apps/upload/handler/routers_test.go +++ b/internal/apps/upload/handler/routers_test.go @@ -226,6 +226,42 @@ func TestUploadFile(t *testing.T) { } }) + t.Run("upload rejects Pages reserved type", func(t *testing.T) { + putCountBefore := putCount + imgContent := []byte("\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01") + contentType, body := createMultipartRequest(t, "file", "pages.png", imgContent, map[string]string{ + "type": shared.ReservedPagesDeploymentType, + }) + req, _ := http.NewRequest("POST", "/api/v1/upload", body) + req.Header.Set("Content-Type", contentType) + + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + + if w.Code != http.StatusConflict { + t.Fatalf("expected status 409, got %d. Body: %s", w.Code, w.Body.String()) + } + var resp testResponse + if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { + t.Fatalf("unmarshal reserved type response: %v", err) + } + if resp.ErrorMsg != shared.ErrReservedUploadType { + t.Fatalf("reserved type error = %q, want %q", resp.ErrorMsg, shared.ErrReservedUploadType) + } + if putCount != putCountBefore { + t.Fatalf("reserved upload wrote storage object: put count %d -> %d", putCountBefore, putCount) + } + var count int64 + if err := dbConn.Model(&model.Upload{}). + Where("type = ?", shared.ReservedPagesDeploymentType). + Count(&count).Error; err != nil { + t.Fatalf("count reserved uploads: %v", err) + } + if count != 0 { + t.Fatalf("reserved upload record count = %d, want 0", count) + } + }) + t.Run("instant upload deduplication (秒传)", func(t *testing.T) { putCount = 0 imgContent := []byte("\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01") @@ -993,6 +1029,62 @@ func TestUserUploadManagement(t *testing.T) { }) } +func TestDeleteReservedUploadType(t *testing.T) { + dbConn, _, cleanup := testhelper.SetupTestEnvironment(t) + defer cleanup() + + authUser := &model.User{ID: 1001, Username: "reserved_owner"} + router := setupTestRouter(authUser) + reserved := model.Upload{ + ID: 4101, + UserID: authUser.ID, + FileName: "pages.zip", + FilePath: "uploads/pages.zip", + FileSize: 128, + MimeType: "application/zip", + Extension: "zip", + Hash: "pages-reserved-hash", + Type: shared.ReservedPagesDeploymentType, + Status: model.UploadStatusUsed, + CreatedAt: time.Now(), + } + if err := dbConn.Create(&reserved).Error; err != nil { + t.Fatalf("seed reserved upload: %v", err) + } + + for _, tc := range []struct { + name string + path string + }{ + {name: "admin delete", path: "/api/v1/admin/uploads/4101"}, + {name: "owner delete", path: "/api/v1/upload/4101"}, + } { + t.Run(tc.name, func(t *testing.T) { + req, _ := http.NewRequest(http.MethodDelete, tc.path, nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + if w.Code != http.StatusConflict { + t.Fatalf("expected status 409, got %d. Body: %s", w.Code, w.Body.String()) + } + var resp testResponse + if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { + t.Fatalf("unmarshal delete response: %v", err) + } + if resp.ErrorMsg != shared.ErrReservedUploadType { + t.Fatalf("reserved delete error = %q, want %q", resp.ErrorMsg, shared.ErrReservedUploadType) + } + }) + } + + var persisted model.Upload + if err := dbConn.First(&persisted, reserved.ID).Error; err != nil { + t.Fatalf("reload reserved upload: %v", err) + } + if persisted.Status != model.UploadStatusUsed { + t.Fatalf("reserved upload status = %s, want used", persisted.Status) + } +} + func configureLocalStorageRoot(t *testing.T, dbConn *gorm.DB, tempDir string) { var sc model.SystemConfig if err := dbConn.Where("key = ?", model.ConfigKeyStorageConfig).First(&sc).Error; err != nil { diff --git a/internal/apps/upload/ingest/errors.go b/internal/apps/upload/ingest/errors.go index 6e501977..59ad4ed8 100644 --- a/internal/apps/upload/ingest/errors.go +++ b/internal/apps/upload/ingest/errors.go @@ -12,5 +12,8 @@ import ( // ErrForbidden indicates the caller is not allowed to mutate the upload record. var ErrForbidden = errors.New("upload forbidden") +// ErrReservedUploadType indicates that a generic mutation targeted a domain-reserved upload type. +var ErrReservedUploadType = errors.New(shared.ErrReservedUploadType) + // ErrStorageReadOnly indicates the storage backend is in migration read-only mode. var ErrStorageReadOnly = errors.New(shared.ErrStorageReadOnly) diff --git a/internal/apps/upload/ingest/helpers.go b/internal/apps/upload/ingest/helpers.go index 00580a90..b4b094ed 100644 --- a/internal/apps/upload/ingest/helpers.go +++ b/internal/apps/upload/ingest/helpers.go @@ -100,13 +100,10 @@ func storeObject(ctx context.Context, objectKey string, reader io.Reader, size i return result.Key, nil } -func persistUploadRecord(ctx context.Context, upload *model.Upload, objectKey string) error { +func persistUploadRecord(ctx context.Context, upload *model.Upload, objectKey string, storedByRequest bool) error { if err := createUploadWithStats(ctx, upload); err != nil { - _, backend, backendErr := storage.Active(ctx) - if backendErr == nil { - if deleteErr := backend.Delete(ctx, objectKey); deleteErr != nil { - logger.WarnF(ctx, "清理未写入数据库的上传对象失败: %v", deleteErr) - } + if storedByRequest { + cleanupUnpersistedObject(ctx, objectKey) } return err } @@ -114,6 +111,16 @@ func persistUploadRecord(ctx context.Context, upload *model.Upload, objectKey st return nil } +func cleanupUnpersistedObject(ctx context.Context, objectKey string) { + _, backend, err := storage.Active(ctx) + if err != nil { + return + } + if err := backend.Delete(ctx, objectKey); err != nil { + logger.WarnF(ctx, "清理未写入数据库的上传对象失败: %v", err) + } +} + func createUploadWithStats(ctx context.Context, upload *model.Upload) error { return db.DB(ctx).Transaction(func(tx *gorm.DB) error { if err := repository.CreateUploadTx(tx, upload); err != nil { @@ -125,6 +132,8 @@ func createUploadWithStats(ctx context.Context, upload *model.Upload) error { func createDedupRecord(ctx context.Context, existing model.Upload, req Request) (Result, error) { accessMode := resolveAccessMode(req.Type, req.AccessMode) + metadata := req.Metadata + metadata.Bucket = existing.Metadata.Bucket newUpload := model.Upload{ ID: idgen.NextUint64ID(), UserID: req.UserID, @@ -137,9 +146,9 @@ func createDedupRecord(ctx context.Context, existing model.Upload, req Request) Type: req.Type, Status: req.Status, AccessMode: accessMode, - Metadata: existing.Metadata, + Metadata: metadata, } - if err := persistUploadRecord(ctx, &newUpload, existing.FilePath); err != nil { + if err := persistUploadRecord(ctx, &newUpload, existing.FilePath, false); err != nil { return Result{}, err } logger.InfoF(ctx, "文件触发秒传成功! ID: %d, Path: %s", newUpload.ID, existing.FilePath) @@ -186,7 +195,7 @@ func createNewUpload(ctx context.Context, req Request) (Result, error) { AccessMode: accessMode, Metadata: req.Metadata, } - if err := persistUploadRecord(ctx, &upload, storedKey); err != nil { + if err := persistUploadRecord(ctx, &upload, storedKey, true); err != nil { return Result{}, err } diff --git a/internal/apps/upload/ingest/ingest_test.go b/internal/apps/upload/ingest/ingest_test.go index d70bedc7..4853d926 100644 --- a/internal/apps/upload/ingest/ingest_test.go +++ b/internal/apps/upload/ingest/ingest_test.go @@ -8,15 +8,20 @@ import ( "context" "crypto/sha256" "encoding/hex" + "errors" "io" "os" + "sync" "testing" "time" + uploadcache "github.com/Rain-kl/Wavelet/internal/apps/upload/cache" + "github.com/Rain-kl/Wavelet/internal/apps/upload/shared" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" "github.com/Rain-kl/Wavelet/internal/storage" "github.com/Rain-kl/Wavelet/internal/testhelper" + "gorm.io/gorm" ) func TestIngestPolicyCreateIncrementsStats(t *testing.T) { @@ -141,7 +146,11 @@ func TestIngestPolicyDedupNewRecordCreatesSecondRecord(t *testing.T) { Extension: "png", Hash: hashStr, Type: "avatar", - Policy: PolicyDedupNewRecord, + Metadata: model.UploadMetadata{ + UserAgent: "first-agent", + Extra: map[string]any{"record": "first"}, + }, + Policy: PolicyDedupNewRecord, }) if err != nil { t.Fatalf("first Ingest returned error: %v", err) @@ -149,6 +158,10 @@ func TestIngestPolicyDedupNewRecordCreatesSecondRecord(t *testing.T) { if putCount != 1 { t.Fatalf("putCount after first ingest = %d, want 1", putCount) } + first.Upload.Metadata.Bucket = "shared-bucket" + if err := dbConn.Save(&first.Upload).Error; err != nil { + t.Fatalf("update first upload metadata failed: %v", err) + } second, err := Ingest(ctx, Request{ UserID: 1002, @@ -159,7 +172,12 @@ func TestIngestPolicyDedupNewRecordCreatesSecondRecord(t *testing.T) { Extension: "png", Hash: hashStr, Type: "avatar", - Policy: PolicyDedupNewRecord, + Metadata: model.UploadMetadata{ + UserAgent: "second-agent", + Bucket: "caller-bucket-must-not-survive", + Extra: map[string]any{"record": "second"}, + }, + Policy: PolicyDedupNewRecord, }) if err != nil { t.Fatalf("second Ingest returned error: %v", err) @@ -173,6 +191,15 @@ func TestIngestPolicyDedupNewRecordCreatesSecondRecord(t *testing.T) { if first.Upload.ID == second.Upload.ID { t.Fatal("dedup records should have unique IDs") } + if second.Upload.Metadata.Bucket != "shared-bucket" { + t.Fatalf("dedup bucket = %q, want inherited shared-bucket", second.Upload.Metadata.Bucket) + } + if second.Upload.Metadata.UserAgent != "second-agent" { + t.Fatalf("dedup user agent = %q, want caller metadata", second.Upload.Metadata.UserAgent) + } + if second.Upload.Metadata.Extra["record"] != "second" { + t.Fatalf("dedup extra metadata = %#v, want caller metadata", second.Upload.Metadata.Extra) + } var count int64 if err := dbConn.Model(&model.Upload{}).Where("hash = ?", hashStr).Count(&count).Error; err != nil { @@ -183,6 +210,77 @@ func TestIngestPolicyDedupNewRecordCreatesSecondRecord(t *testing.T) { } } +func TestDedupRecordFailureDoesNotDeleteSharedObject(t *testing.T) { + dbConn, _, cleanup := testhelper.SetupTestEnvironment(t) + defer cleanup() + ctx := context.Background() + + content := []byte("\x89PNG\r\n\x1a\nshared-object") + hash := sha256.Sum256(content) + hashStr := hex.EncodeToString(hash[:]) + deleteCount := 0 + restoreStorage, disableStorage := setupMockStorageWithDeleteCount(t, nil, &deleteCount) + defer restoreStorage() + defer disableStorage() + + first, err := Ingest(ctx, Request{ + UserID: 1001, + Reader: bytes.NewReader(content), + Size: int64(len(content)), + FileName: "shared.png", + MimeType: "image/png", + Extension: "png", + Hash: hashStr, + Type: "avatar", + Policy: PolicyDedupNewRecord, + }) + if err != nil { + t.Fatalf("first Ingest returned error: %v", err) + } + + const callbackName = "test:reject_dedup_upload_record" + if err := dbConn.Callback().Create().Before("gorm:create").Register(callbackName, func(tx *gorm.DB) { + upload, ok := tx.Statement.Dest.(*model.Upload) + if ok && upload.FileName == "dedup-fail.png" { + tx.AddError(errors.New("injected upload create failure")) + } + }); err != nil { + t.Fatalf("register create failure callback: %v", err) + } + defer func() { _ = dbConn.Callback().Create().Remove(callbackName) }() + + _, err = Ingest(ctx, Request{ + UserID: 1002, + Reader: bytes.NewReader(content), + Size: int64(len(content)), + FileName: "dedup-fail.png", + MimeType: "image/png", + Extension: "png", + Hash: hashStr, + Type: "avatar", + Metadata: model.UploadMetadata{ + Extra: map[string]any{"record": "dedup-failure"}, + }, + Policy: PolicyDedupNewRecord, + }) + if err == nil { + t.Fatal("dedup Ingest expected injected persistence error") + } + if deleteCount != 0 { + t.Fatalf("shared object delete count = %d, want 0", deleteCount) + } + + _, backend, err := storage.Active(ctx) + if err != nil { + t.Fatalf("load active storage: %v", err) + } + obj, err := backend.Get(ctx, first.Upload.FilePath) + if err != nil { + t.Fatalf("shared object became unreadable after dedup failure: %v", err) + } + _ = obj.Body.Close() +} + func TestCreateUploadWithStatsRollsBackOnCreateFailure(t *testing.T) { dbConn, _, cleanup := testhelper.SetupTestEnvironment(t) defer cleanup() @@ -259,6 +357,18 @@ func TestRemoveDecrementsStats(t *testing.T) { if _, err := Remove(ctx, result.Upload.ID); err != nil { t.Fatalf("Remove(%d) returned error: %v", result.Upload.ID, err) } + stale := result.Upload + uploadcache.SetUploadMetaCache(ctx, &stale) + removedAgain, err := Remove(ctx, result.Upload.ID) + if err != nil { + t.Fatalf("second Remove(%d) returned error: %v", result.Upload.ID, err) + } + if removedAgain.Status != model.UploadStatusDeleted { + t.Fatalf("second Remove status = %s, want deleted", removedAgain.Status) + } + if _, err := uploadcache.GetUploadByID(ctx, result.Upload.ID); !errors.Is(err, gorm.ErrRecordNotFound) { + t.Fatalf("cache lookup after idempotent Remove error = %v, want record not found", err) + } stats, err := loadTotalStats(ctx) if err != nil { @@ -269,6 +379,120 @@ func TestRemoveDecrementsStats(t *testing.T) { } } +func TestConcurrentRemoveDecrementsStatsOnce(t *testing.T) { + _, _, cleanup := testhelper.SetupTestEnvironment(t) + defer cleanup() + ctx := context.Background() + + content := []byte("\x89PNG\r\n\x1a\nconcurrent-remove") + hash := sha256.Sum256(content) + restoreStorage, disableStorage := setupMockStorage(t, nil) + defer restoreStorage() + defer disableStorage() + + result, err := Ingest(ctx, Request{ + UserID: 1001, + Reader: bytes.NewReader(content), + Size: int64(len(content)), + FileName: "concurrent.png", + MimeType: "image/png", + Extension: "png", + Hash: hex.EncodeToString(hash[:]), + Type: "generic", + Policy: PolicyCreate, + }) + if err != nil { + t.Fatalf("Ingest returned error: %v", err) + } + + const workers = 8 + start := make(chan struct{}) + errs := make(chan error, workers) + var wg sync.WaitGroup + for i := 0; i < workers; i++ { + wg.Add(1) + go func() { + defer wg.Done() + <-start + _, removeErr := Remove(ctx, result.Upload.ID) + errs <- removeErr + }() + } + close(start) + wg.Wait() + close(errs) + for removeErr := range errs { + if removeErr != nil { + t.Fatalf("concurrent Remove returned error: %v", removeErr) + } + } + + stats, err := loadTotalStats(ctx) + if err != nil { + t.Fatalf("loadTotalStats returned error: %v", err) + } + if stats.TotalCount != 0 || stats.TotalSize != 0 { + t.Fatalf("stats after concurrent remove = count %d size %d, want zero", stats.TotalCount, stats.TotalSize) + } +} + +func TestRemoveOwnedAndReservedTypeBoundaries(t *testing.T) { + dbConn, _, cleanup := testhelper.SetupTestEnvironment(t) + defer cleanup() + ctx := context.Background() + + ordinary := model.Upload{ + ID: 99101, + UserID: 1001, + FileName: "owned.txt", + FilePath: "uploads/owned.txt", + FileSize: 16, + MimeType: "text/plain", + Extension: "txt", + Hash: "owned-hash", + Type: "generic", + Status: model.UploadStatusUsed, + CreatedAt: time.Now(), + } + reserved := model.Upload{ + ID: 99102, + UserID: 1001, + FileName: "pages.zip", + FilePath: "uploads/pages.zip", + FileSize: 32, + MimeType: "application/zip", + Extension: "zip", + Hash: "reserved-hash", + Type: shared.ReservedPagesDeploymentType, + Status: model.UploadStatusUsed, + CreatedAt: time.Now(), + } + if err := dbConn.Create(&ordinary).Error; err != nil { + t.Fatalf("seed ordinary upload: %v", err) + } + if err := dbConn.Create(&reserved).Error; err != nil { + t.Fatalf("seed reserved upload: %v", err) + } + + if _, err := RemoveOwned(ctx, 2002, ordinary.ID); !errors.Is(err, ErrForbidden) { + t.Fatalf("RemoveOwned non-owner error = %v, want ErrForbidden", err) + } + if _, err := Remove(ctx, reserved.ID); !errors.Is(err, ErrReservedUploadType) { + t.Fatalf("Remove reserved error = %v, want ErrReservedUploadType", err) + } + if _, err := RemoveOwned(ctx, reserved.UserID, reserved.ID); !errors.Is(err, ErrReservedUploadType) { + t.Fatalf("RemoveOwned reserved error = %v, want ErrReservedUploadType", err) + } + + var persisted model.Upload + if err := dbConn.First(&persisted, reserved.ID).Error; err != nil { + t.Fatalf("reload reserved upload: %v", err) + } + if persisted.Status != model.UploadStatusUsed { + t.Fatalf("reserved upload status = %s, want used", persisted.Status) + } +} + type totalStatsSnapshot struct { TotalCount int64 TotalSize int64 @@ -289,6 +513,10 @@ func loadTotalStats(ctx context.Context) (totalStatsSnapshot, error) { } func setupMockStorage(t *testing.T, putCount *int) (restore func(), disable func()) { + return setupMockStorageWithDeleteCount(t, putCount, nil) +} + +func setupMockStorageWithDeleteCount(t *testing.T, putCount, deleteCount *int) (restore func(), disable func()) { t.Helper() mockFiles := make(map[string][]byte) restore = storage.MockStorage( @@ -316,6 +544,9 @@ func setupMockStorage(t *testing.T, putCount *int) (restore func(), disable func }, func(ctx context.Context, key string) error { delete(mockFiles, key) + if deleteCount != nil { + *deleteCount++ + } return nil }, ) diff --git a/internal/apps/upload/ingest/remove.go b/internal/apps/upload/ingest/remove.go index 902ab5ef..05c6fb60 100644 --- a/internal/apps/upload/ingest/remove.go +++ b/internal/apps/upload/ingest/remove.go @@ -7,52 +7,76 @@ import ( "context" uploadcache "github.com/Rain-kl/Wavelet/internal/apps/upload/cache" + "github.com/Rain-kl/Wavelet/internal/apps/upload/shared" uploadstats "github.com/Rain-kl/Wavelet/internal/apps/upload/stats" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" "github.com/Rain-kl/Wavelet/internal/repository" "gorm.io/gorm" + "gorm.io/gorm/clause" ) -// Remove soft-deletes an upload and decrements incremental stats. +// Remove soft-deletes an ordinary upload and decrements incremental stats once. func Remove(ctx context.Context, uploadID uint64) (model.Upload, error) { - upload, err := repository.GetActiveUploadByID(ctx, uploadID) + upload, err := remove(ctx, 0, uploadID, false) if err != nil { return model.Upload{}, err } - if err := softDeleteUploadWithStats(ctx, &upload); err != nil { - return model.Upload{}, err - } - upload.Status = model.UploadStatusDeleted return upload, nil } -// RemoveOwned soft-deletes an upload owned by userID and decrements incremental stats. +// RemoveOwned soft-deletes an ordinary upload owned by userID and decrements incremental stats once. func RemoveOwned(ctx context.Context, userID, uploadID uint64) (model.Upload, error) { - upload, err := repository.GetActiveUploadByID(ctx, uploadID) + upload, err := remove(ctx, userID, uploadID, true) if err != nil { return model.Upload{}, err } - if upload.UserID != userID { - return model.Upload{}, ErrForbidden - } - if err := softDeleteUploadWithStats(ctx, &upload); err != nil { - return model.Upload{}, err - } - upload.Status = model.UploadStatusDeleted return upload, nil } -func softDeleteUploadWithStats(ctx context.Context, upload *model.Upload) error { - statsSnapshot := *upload +func remove(ctx context.Context, userID, uploadID uint64, owned bool) (model.Upload, error) { + var upload model.Upload if err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { - if err := repository.SoftDeleteUploadTx(tx, upload); err != nil { + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ?", uploadID). + First(&upload).Error; err != nil { return err } - return uploadstats.ApplyUploadStatsDeltaTx(tx, &statsSnapshot, -1) - }); err != nil { + if owned && upload.UserID != userID { + return ErrForbidden + } + if upload.Type == shared.ReservedPagesDeploymentType { + return ErrReservedUploadType + } + _, err := RemoveLockedTx(tx, &upload) return err + }); err != nil { + return model.Upload{}, err } - uploadcache.InvalidateUploadMetaCache(ctx, upload.ID) - return nil + + InvalidateUploadMetaCache(ctx, uploadID) + upload.Status = model.UploadStatusDeleted + return upload, nil +} + +// RemoveLockedTx performs the idempotent active-to-deleted transition for a row +// that the caller has already locked in its surrounding transaction. +func RemoveLockedTx(tx *gorm.DB, upload *model.Upload) (bool, error) { + rowsAffected, err := repository.SoftDeleteUploadTx(tx, upload) + if err != nil { + return false, err + } + if rowsAffected == 0 { + return false, nil + } + if err := uploadstats.ApplyUploadStatsDeltaTx(tx, upload, -1); err != nil { + return false, err + } + upload.Status = model.UploadStatusDeleted + return true, nil +} + +// InvalidateUploadMetaCache invalidates upload metadata after the caller commits its transaction. +func InvalidateUploadMetaCache(ctx context.Context, uploadID uint64) { + uploadcache.InvalidateUploadMetaCache(ctx, uploadID) } diff --git a/internal/apps/upload/shared/constants.go b/internal/apps/upload/shared/constants.go index 14117af3..f3382112 100644 --- a/internal/apps/upload/shared/constants.go +++ b/internal/apps/upload/shared/constants.go @@ -17,4 +17,6 @@ const ( FileStatsTrendDays = 7 MaxS3KeyLength = 1024 AccessCacheTTL = 5 // seconds; multiplied by time.Second at use site + // ReservedPagesDeploymentType is managed exclusively by the Pages domain. + ReservedPagesDeploymentType = "openflare_pages_deployment" ) diff --git a/internal/apps/upload/shared/errs.go b/internal/apps/upload/shared/errs.go index c37c864c..7f031341 100644 --- a/internal/apps/upload/shared/errs.go +++ b/internal/apps/upload/shared/errs.go @@ -27,6 +27,7 @@ const ( ErrQueryFileCountFailed = "查询文件数量失败" ErrQueryFileListFailed = "查询文件列表失败" ErrDeleteFileFailed = "删除文件失败" + ErrReservedUploadType = "系统保留的文件类型不能通过通用文件接口操作" ErrStorageReadOnly = "存储迁移维护中，当前仅允许读取文件" ErrS3KeyRequired = "s3 key must not be empty" ErrS3KeyTooLongFormat = "s3 key exceeds maximum length of %d" diff --git a/internal/apps/upload/task/cleanup.go b/internal/apps/upload/task/cleanup.go index 0d70e35d..36b3f5b1 100644 --- a/internal/apps/upload/task/cleanup.go +++ b/internal/apps/upload/task/cleanup.go @@ -10,16 +10,15 @@ import ( "fmt" "time" - uploadcache "github.com/Rain-kl/Wavelet/internal/apps/upload/cache" + "github.com/Rain-kl/Wavelet/internal/apps/upload/ingest" "github.com/Rain-kl/Wavelet/internal/apps/upload/shared" - uploadstats "github.com/Rain-kl/Wavelet/internal/apps/upload/stats" uploadstorage "github.com/Rain-kl/Wavelet/internal/apps/upload/storage" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" - "github.com/Rain-kl/Wavelet/internal/storage" "github.com/Rain-kl/Wavelet/internal/task" "github.com/Rain-kl/Wavelet/pkg/logger" "gorm.io/gorm" + "gorm.io/gorm/clause" ) const ( @@ -77,32 +76,31 @@ func (h *SystemCleanupHandler) Execute(ctx context.Context, _ []byte) (*task.Tas for _, u := range unusedUploads { totalProcessed++ + transitioned := false if err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { - if err := tx.Model(&model.Upload{}). - Where("id = ? AND status = ?", u.ID, model.UploadStatusPending). - Update("status", model.UploadStatusDeleted).Error; err != nil { + var locked model.Upload + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ?", u.ID). + First(&locked).Error; err != nil { return err } - - _, backend, err := storage.Active(ctx) - if err != nil { - return err + if locked.Status != model.UploadStatusPending || !locked.CreatedAt.Before(oneHourAgo) { + return nil } - if err := backend.Delete(ctx, u.FilePath); err != nil { - return err - } - - return nil + var err error + transitioned, err = ingest.RemoveLockedTx(tx, &locked) + return err }); err != nil { task.AppendLog(ctx, "清理上传文件失败 [ID:%d]: %v", u.ID, err) lastID = u.ID continue } - uploadstats.RecordUploadStatsRemove(ctx, &u) - uploadcache.InvalidateUploadMetaCache(ctx, u.ID) - totalDeleted++ + ingest.InvalidateUploadMetaCache(ctx, u.ID) + if transitioned { + totalDeleted++ + } lastID = u.ID } } diff --git a/internal/apps/upload/task/tasks_test.go b/internal/apps/upload/task/tasks_test.go index 90572c8e..f75bec6b 100644 --- a/internal/apps/upload/task/tasks_test.go +++ b/internal/apps/upload/task/tasks_test.go @@ -19,6 +19,7 @@ import ( "github.com/Rain-kl/Wavelet/internal/apps/upload/filesrv" "github.com/Rain-kl/Wavelet/internal/apps/upload/shared" + uploadstats "github.com/Rain-kl/Wavelet/internal/apps/upload/stats" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/diskcache" "github.com/Rain-kl/Wavelet/internal/model" @@ -33,13 +34,17 @@ func TestSystemCleanupHandler_Execute(t *testing.T) { _, _, cleanup := testhelper.SetupTestEnvironment(t) defer cleanup() - // Mock S3 存储（让 DeleteObject 总是成功） + deleteCount := 0 + // Mock S3 存储并记录 Delete，cleanup 不应物理删除共享对象。 storageMock := storage.MockStorage( func(ctx context.Context, key string, body io.Reader, size int64, contentType string) error { return nil }, func(ctx context.Context, key string) (*storage.Object, error) { return nil, nil }, - func(ctx context.Context, key string) error { return nil }, + func(ctx context.Context, key string) error { + deleteCount++ + return nil + }, ) defer storageMock() storage.IsEnabledFunc = func() bool { return true } @@ -86,6 +91,7 @@ func TestSystemCleanupHandler_Execute(t *testing.T) { for _, r := range records { err := db.DB(ctx).Create(r).Error require.NoError(t, err) + require.NoError(t, uploadstats.ApplyUploadStatsAdd(ctx, r)) } // 准备推送历史测试数据：1个旧的（应删除），1个新的（应保留） @@ -147,6 +153,26 @@ func TestSystemCleanupHandler_Execute(t *testing.T) { var usedCount int64 db.DB(ctx).Model(&model.Upload{}).Where("status = ?", model.UploadStatusUsed).Count(&usedCount) assert.Equal(t, int64(1), usedCount, "used 状态的文件不应受影响") + assert.Equal(t, 0, deleteCount, "记录级 cleanup 不应调用 storage backend Delete") + + var totalStats model.UploadStat + err = db.DB(ctx). + Where("dimension = ? AND stat_key = ?", model.UploadStatDimensionTotal, ""). + First(&totalStats).Error + require.NoError(t, err) + assert.Equal(t, int64(2), totalStats.FileCount, "cleanup 后统计只应保留 used 与最近 pending 记录") + assert.Equal(t, int64(768), totalStats.FileSize, "cleanup 后统计大小应只扣减一次") + + _, err = handler.Execute(ctx, nil) + require.NoError(t, err) + var statsAfterSecondRun model.UploadStat + err = db.DB(ctx). + Where("dimension = ? AND stat_key = ?", model.UploadStatDimensionTotal, ""). + First(&statsAfterSecondRun).Error + require.NoError(t, err) + assert.Equal(t, totalStats.FileCount, statsAfterSecondRun.FileCount, "重复 cleanup 不应再次扣减统计") + assert.Equal(t, totalStats.FileSize, statsAfterSecondRun.FileSize, "重复 cleanup 不应再次扣减统计大小") + assert.Equal(t, 0, deleteCount, "重复 cleanup 仍不应调用 storage backend Delete") // 验证推送历史数据状态：10天前的应被删除，今天的应保留 var pushCount int64 diff --git a/internal/repository/upload.go b/internal/repository/upload.go index 34c8a96f..599f66c4 100644 --- a/internal/repository/upload.go +++ b/internal/repository/upload.go @@ -62,15 +62,22 @@ func GetActiveUploadByID(ctx context.Context, id uint64) (model.Upload, error) { return upload, nil } -// SoftDeleteUpload marks an upload as deleted. +// SoftDeleteUpload marks an active upload as deleted and reports whether the row transitioned. // External modules must use upload.Remove or upload.RemoveOwned; only internal/apps/upload may call this. -func SoftDeleteUpload(ctx context.Context, upload *model.Upload) error { +func SoftDeleteUpload(ctx context.Context, upload *model.Upload) (int64, error) { return SoftDeleteUploadTx(db.DB(ctx), upload) } -// SoftDeleteUploadTx marks an upload as deleted within an existing transaction. -func SoftDeleteUploadTx(tx *gorm.DB, upload *model.Upload) error { - return tx.Model(upload).Update("status", model.UploadStatusDeleted).Error +// SoftDeleteUploadTx marks an active upload as deleted within an existing transaction. +// RowsAffected is one only for the single successful active-to-deleted transition. +func SoftDeleteUploadTx(tx *gorm.DB, upload *model.Upload) (int64, error) { + result := tx.Model(&model.Upload{}). + Where("id = ? AND status IN ?", upload.ID, []model.UploadStatus{ + model.UploadStatusPending, + model.UploadStatusUsed, + }). + Update("status", model.UploadStatusDeleted) + return result.RowsAffected, result.Error } // UpdateUpload applies partial field updates to an upload record. diff --git a/pkg/pagesarchive/entry.go b/pkg/pagesarchive/entry.go index 59b350aa..ce19734b 100644 --- a/pkg/pagesarchive/entry.go +++ b/pkg/pagesarchive/entry.go @@ -4,6 +4,7 @@ package pagesarchive import ( + "errors" "fmt" "io" "math" @@ -45,39 +46,50 @@ type Entry struct { IsDir bool // IsSymlink marks symbolic links (unsupported for Pages). IsSymlink bool - // Size is the declared uncompressed size when known; 0 means empty or unknown. + // IsHardlink marks hard links (unsupported for Pages). + IsHardlink bool + // IsSpecial marks device, FIFO, socket, and other non-regular entries. + IsSpecial bool + // Size is the archive-declared uncompressed size; 0 means an empty member. Size uint64 // Open returns a reader for the entry body. Caller must Close it. - // May be unavailable for inspect-only tar listings (body not materialized). Open func() (io.ReadCloser, error) } -// copyLimited copies src to dst. -// When maxBytes <= 0, size limits are not enforced (trusted extract path). -func copyLimited(dst io.Writer, src io.Reader, declaredSize uint64, maxBytes int64) (int64, error) { - if maxBytes <= 0 { - if declaredSize > 0 { - if declaredSize > uint64(math.MaxInt64) { - return 0, fmt.Errorf("pages file size out of bounds") - } - //nolint:gosec // declaredSize is bounded to MaxInt64 above - return io.CopyN(dst, src, int64(declaredSize)) - } +// copyLimited copies actual bytes from src. maxBytes < 0 disables the byte cap; +// maxBytes == 0 permits only an empty stream. +func copyLimited(dst io.Writer, src io.Reader, maxBytes int64) (int64, error) { + if maxBytes < 0 { return io.Copy(dst, src) } - if declaredSize > uint64(maxBytes) || declaredSize > uint64(math.MaxInt64) { //nolint:gosec // maxBytes positive - return 0, fmt.Errorf("pages file size out of bounds") + + readLimit := maxBytes + if maxBytes < math.MaxInt64 { + readLimit++ } - if declaredSize > 0 { - //nolint:gosec // declaredSize is bounded to MaxInt64 above - return io.CopyN(dst, src, int64(declaredSize)) + written, err := io.Copy(dst, io.LimitReader(src, readLimit)) + if err != nil { + return written, err } - limited := io.LimitReader(src, maxBytes+1) - written, err := io.Copy(dst, limited) if written > maxBytes { return written, fmt.Errorf("pages file size out of bounds") } - return written, err + return written, nil +} + +func copyAndVerifySize(dst io.Writer, src io.Reader, declaredSize uint64, maxBytes int64) (int64, error) { + if declaredSize > uint64(math.MaxInt64) { + return 0, fmt.Errorf("pages file size out of bounds") + } + written, err := copyLimited(dst, src, maxBytes) + if err != nil { + return written, err + } + //nolint:gosec // declaredSize is bounded to MaxInt64 above + if written != int64(declaredSize) { + return written, fmt.Errorf("pages declared size %d does not match actual %d", declaredSize, written) + } + return written, nil } func writeEntryFile(targetPath string, src io.Reader, declaredSize uint64, maxBytes int64, perm os.FileMode) (int64, error) { @@ -88,6 +100,11 @@ func writeEntryFile(targetPath string, src io.Reader, declaredSize uint64, maxBy if err != nil { return 0, err } - defer func() { _ = target.Close() }() - return copyLimited(target, src, declaredSize, maxBytes) + written, copyErr := copyAndVerifySize(target, src, declaredSize, maxBytes) + closeErr := target.Close() + if err := errors.Join(copyErr, closeErr); err != nil { + _ = os.Remove(targetPath) + return written, err + } + return written, nil } diff --git a/pkg/pagesarchive/extract.go b/pkg/pagesarchive/extract.go index 25335264..c8eb1426 100644 --- a/pkg/pagesarchive/extract.go +++ b/pkg/pagesarchive/extract.go @@ -4,7 +4,9 @@ package pagesarchive import ( + "archive/tar" "bytes" + "errors" "fmt" "io" "os" @@ -16,14 +18,12 @@ const formatDetectHeadBytes = 512 // ExtractOptions controls package extraction. type ExtractOptions struct { - // Limits bounds files and sizes during extraction when EnforceLimits is true. + // Limits bounds actual files and sizes during extraction when EnforceLimits is true. Limits Limits // StripCommonRoot strips a single shared top-level directory when present. StripCommonRoot bool // EnforceLimits enables MaxFiles / MaxFileBytes / MaxTotalBytes checks. - // When false, the caller is assumed to have already validated the package - // (e.g. Agent trusts control-plane inspection). Path-escape and symlink - // guards still apply so local extraction cannot leave destDir. + // Path, member type, and declared/actual-size validation always remain enabled. EnforceLimits bool } @@ -36,16 +36,11 @@ func ExtractBytes(data []byte, format Format, destDir string, opts ExtractOption return err } } - entries, err := listEntriesAt(bytes.NewReader(data), int64(len(data)), format, true) - if err != nil { - return err - } - return extractEntries(entries, destDir, opts) + return extractFromReaderAt(bytes.NewReader(data), int64(len(data)), format, destDir, opts) } // ExtractFile opens path and extracts it into destDir without buffering the -// whole archive as an intermediate []byte for zip/7z (ReaderAt). Tar-family -// formats still materialize member bodies so random Open works for extract. +// whole archive or tar member bodies in memory. func ExtractFile(filePath string, format Format, destDir string, opts ExtractOptions) error { file, err := os.Open(filePath) //nolint:gosec // controlled path if err != nil { @@ -68,7 +63,14 @@ func ExtractFile(filePath string, format Format, destDir string, opts ExtractOpt return err } } - entries, err := listEntriesAt(file, info.Size(), format, true) + return extractFromReaderAt(file, info.Size(), format, destDir, opts) +} + +func extractFromReaderAt(ra io.ReaderAt, size int64, format Format, destDir string, opts ExtractOptions) error { + if isTarFamily(format) { + return extractTarFamilyAt(ra, size, format, destDir, opts) + } + entries, err := listRandomAccessEntriesAt(ra, size, format) if err != nil { return err } @@ -80,88 +82,159 @@ func extractEntries(entries []Entry, destDir string, opts ExtractOptions) error if opts.EnforceLimits { limits = normalizeLimits(opts.Limits) } - commonPrefix := "" - if opts.StripCommonRoot { - commonPrefix = FindCommonRootPrefix(collectFileNames(entries)) + commonPrefix, err := commonRootForEntries(entries, opts.StripCommonRoot) + if err != nil { + return err } - var totalSize int64 - var fileCount int + measured := &measuredArchive{files: make([]measuredFile, 0, len(entries))} for _, entry := range entries { - written, counted, err := extractSingleEntry(entry, destDir, commonPrefix, limits, opts.EnforceLimits) + normalizedPath, skip, err := validateArchiveEntry(entry) if err != nil { return err } - if !counted { + if skip { continue } - fileCount++ - if opts.EnforceLimits && fileCount > limits.MaxFiles { - return fmt.Errorf("pages deployment file count exceeds %d", limits.MaxFiles) + normalizedPath = StripPrefix(normalizedPath, commonPrefix) + if normalizedPath == "" { + continue } - totalSize += written - if opts.EnforceLimits && totalSize > limits.MaxTotalBytes { - return fmt.Errorf("pages extracted size exceeds limit") + if err := prepareMeasuredFile(measured, normalizedPath, entry.Size, limits, opts.EnforceLimits); err != nil { + return err } + if entry.Open == nil { + return fmt.Errorf("%s: pages archive member cannot be opened", normalizedPath) + } + src, err := entry.Open() + if err != nil { + return fmt.Errorf("%s: %w", normalizedPath, err) + } + maxBytes := effectiveFileLimit(limits, measured.totalSize, opts.EnforceLimits) + targetPath, err := safeExtractionTarget(destDir, normalizedPath) + if err != nil { + _ = src.Close() + return err + } + actual, writeErr := writeEntryFile(targetPath, src, entry.Size, maxBytes, filePerm) + closeErr := src.Close() + if err := errors.Join(writeErr, closeErr); err != nil { + return fmt.Errorf("%s: %w", normalizedPath, err) + } + appendMeasuredFile(measured, normalizedPath, actual) } - if fileCount == 0 { + if measured.fileCount == 0 { return fmt.Errorf("pages package is empty") } return nil } -func extractSingleEntry( - entry Entry, - destDir, commonPrefix string, +func extractTarFamilyAt(ra io.ReaderAt, size int64, format Format, destDir string, opts ExtractOptions) error { + limits := Limits{} + if opts.EnforceLimits { + limits = normalizeLimits(opts.Limits) + } + firstPass, err := scanTarFamilyAt(ra, size, format, limits, opts.EnforceLimits) + if err != nil { + return err + } + if firstPass.fileCount == 0 { + return fmt.Errorf("pages package is empty") + } + commonPrefix := "" + if opts.StripCommonRoot { + paths := make([]string, 0, len(firstPass.files)) + for _, file := range firstPass.files { + paths = append(paths, file.path) + } + commonPrefix = FindCommonRootPrefix(paths) + } + + tarReader, closeReader, err := openTarFamilyReader(io.NewSectionReader(ra, 0, size), format) + if err != nil { + return err + } + secondPass, extractErr := extractTarReader(tarReader, destDir, commonPrefix, limits, opts.EnforceLimits) + if closeErr := closeReader(); closeErr != nil { + extractErr = errors.Join(extractErr, closeErr) + } + if extractErr != nil { + return extractErr + } + if secondPass.fileCount != firstPass.fileCount || secondPass.totalSize != firstPass.totalSize { + return fmt.Errorf("pages tar package changed between validation and extraction") + } + return nil +} + +func extractTarReader( + tarReader *tar.Reader, + destDir string, + commonPrefix string, limits Limits, enforceLimits bool, -) (written int64, counted bool, err error) { - relativePath, skip, err := NormalizeEntryPath(entry.Name) - if err != nil { - return 0, false, err +) (*measuredArchive, error) { + measured := &measuredArchive{files: make([]measuredFile, 0)} + for { + header, err := tarReader.Next() + if err == io.EOF { + break + } + if err != nil { + return nil, fmt.Errorf("read tar pages package: %w", err) + } + entry := entryFromTarHeader(header) + normalizedPath, skip, err := validateArchiveEntry(entry) + if err != nil { + return nil, err + } + if skip { + continue + } + normalizedPath = StripPrefix(normalizedPath, commonPrefix) + if normalizedPath == "" { + continue + } + if err := prepareMeasuredFile(measured, normalizedPath, entry.Size, limits, enforceLimits); err != nil { + return nil, err + } + targetPath, err := safeExtractionTarget(destDir, normalizedPath) + if err != nil { + return nil, err + } + maxBytes := effectiveFileLimit(limits, measured.totalSize, enforceLimits) + actual, err := writeEntryFile(targetPath, tarReader, entry.Size, maxBytes, filePerm) + if err != nil { + return nil, fmt.Errorf("%s: %w", normalizedPath, err) + } + appendMeasuredFile(measured, normalizedPath, actual) } - if skip { - return 0, false, nil - } - if commonPrefix != "" { - relativePath = StripPrefix(relativePath, commonPrefix) - if relativePath == "" { - return 0, false, nil + return measured, nil +} + +func commonRootForEntries(entries []Entry, strip bool) (string, error) { + paths := make([]string, 0, len(entries)) + for _, entry := range entries { + normalizedPath, skip, err := validateArchiveEntry(entry) + if err != nil { + return "", err + } + if !skip { + paths = append(paths, normalizedPath) } } - if entry.IsSymlink { - return 0, false, fmt.Errorf("pages package contains unsupported symlink: %s", relativePath) + if !strip { + return "", nil } + return FindCommonRootPrefix(paths), nil +} +func safeExtractionTarget(destDir, relativePath string) (string, error) { targetPath := filepath.Join(destDir, filepath.FromSlash(relativePath)) if !isWithinDir(destDir, targetPath) { - return 0, false, fmt.Errorf("pages package path escapes directory: %s", entry.Name) + return "", fmt.Errorf("pages package path escapes directory: %s", relativePath) } - - if entry.IsDir { - if err := os.MkdirAll(targetPath, dirPerm); err != nil { - return 0, false, err - } - return 0, false, nil - } - - maxFileBytes := int64(0) // unlimited when not enforcing - if enforceLimits { - if exceedsFileByteLimit(entry.Size, limits.MaxFileBytes) { - return 0, false, fmt.Errorf("pages file too large: %s", relativePath) - } - maxFileBytes = limits.MaxFileBytes - } - src, err := entry.Open() - if err != nil { - return 0, false, fmt.Errorf("%s: %w", relativePath, err) - } - written, writeErr := writeEntryFile(targetPath, src, entry.Size, maxFileBytes, filePerm) - _ = src.Close() - if writeErr != nil { - return 0, false, fmt.Errorf("%s: %w", relativePath, writeErr) - } - return written, true, nil + return targetPath, nil } func isWithinDir(baseDir, targetPath string) bool { diff --git a/pkg/pagesarchive/inspect.go b/pkg/pagesarchive/inspect.go index 6c3211d7..ae3a3125 100644 --- a/pkg/pagesarchive/inspect.go +++ b/pkg/pagesarchive/inspect.go @@ -4,13 +4,14 @@ package pagesarchive import ( + "archive/tar" "bytes" + "errors" "fmt" "io" "math" "os" "path" - "strings" ) // InspectOptions controls package inspection. @@ -19,17 +20,26 @@ type InspectOptions struct { RootDir string // EntryFile is the required entry file name (e.g. index.html). EntryFile string - // Limits bounds files and sizes. + // Limits bounds files and actual extracted sizes. Limits Limits - // VerifySizes, when true, streams each regular file and compares the actual - // byte count against the archive-declared size (no content hashing). - // Default false: trust zip central directory / tar header sizes. + // VerifySizes is retained for source compatibility. Inspection now always + // streams regular members and verifies actual bytes against declared sizes. VerifySizes bool } +type measuredFile struct { + path string + size int64 +} + +type measuredArchive struct { + files []measuredFile + fileCount int + totalSize int64 +} + // InspectFile opens path and inspects it as a Pages deployment package without -// loading the whole archive into memory. File inventory uses declared sizes; -// per-file content hashes are not computed. +// loading the whole archive or any tar member body into memory. func InspectFile(filePath string, format Format, opts InspectOptions) (*Manifest, error) { file, err := os.Open(filePath) //nolint:gosec // filePath is a controlled temp upload path if err != nil { @@ -68,55 +78,137 @@ func InspectBytes(data []byte, format Format, opts InspectOptions) (*Manifest, e } func inspectFromReaderAt(ra io.ReaderAt, size int64, format Format, opts InspectOptions) (*Manifest, error) { - // Default: zip/7z use central directory only; tar streams headers and discards bodies. - // VerifySizes needs openable tar bodies, so materialize only when requested. - entries, err := listEntriesAt(ra, size, format, opts.VerifySizes) + limits := normalizeLimits(opts.Limits) + var ( + measured *measuredArchive + err error + ) + if isTarFamily(format) { + measured, err = scanTarFamilyAt(ra, size, format, limits, true) + } else { + var entries []Entry + entries, err = listRandomAccessEntriesAt(ra, size, format) + if err == nil { + measured, err = inspectRandomAccessEntries(entries, limits) + } + } if err != nil { return nil, err } - return buildManifest(entries, opts) + return buildMeasuredManifest(measured, opts) } -func buildManifest(entries []Entry, opts InspectOptions) (*Manifest, error) { - limits := normalizeLimits(opts.Limits) - commonPrefix := FindCommonRootPrefix(collectFileNames(entries)) - targetEntryPath := resolveTargetEntryPath(opts.RootDir, opts.EntryFile) - - manifest := &Manifest{Files: make([]FileEntry, 0)} - entrySeen := false - +func inspectRandomAccessEntries(entries []Entry, limits Limits) (*measuredArchive, error) { + measured := &measuredArchive{files: make([]measuredFile, 0, len(entries))} for _, entry := range entries { - normalizedPath, skip, err := prepareEntryPath(entry, commonPrefix) + normalizedPath, skip, err := validateArchiveEntry(entry) if err != nil { return nil, err } if skip { continue } - if exceedsFileByteLimit(entry.Size, limits.MaxFileBytes) { - return nil, fmt.Errorf("pages file too large: %s", normalizedPath) + if err := prepareMeasuredFile(measured, normalizedPath, entry.Size, limits, true); err != nil { + return nil, err } + if entry.Open == nil { + return nil, fmt.Errorf("%s: pages archive member cannot be opened", normalizedPath) + } + src, err := entry.Open() + if err != nil { + return nil, fmt.Errorf("%s: %w", normalizedPath, err) + } + maxBytes := effectiveFileLimit(limits, measured.totalSize, true) + actual, copyErr := copyAndVerifySize(io.Discard, src, entry.Size, maxBytes) + closeErr := src.Close() + if err := errors.Join(copyErr, closeErr); err != nil { + return nil, fmt.Errorf("%s: %w", normalizedPath, err) + } + appendMeasuredFile(measured, normalizedPath, actual) + } + return measured, nil +} - fileEntry, err := inspectRegularFile(entry, normalizedPath, limits, opts.VerifySizes) +func scanTarFamilyAt( + ra io.ReaderAt, + size int64, + format Format, + limits Limits, + enforceLimits bool, +) (*measuredArchive, error) { + if size < 0 { + return nil, fmt.Errorf("invalid pages package size") + } + tarReader, closeReader, err := openTarFamilyReader(io.NewSectionReader(ra, 0, size), format) + if err != nil { + return nil, err + } + measured, scanErr := scanTarReader(tarReader, limits, enforceLimits) + if closeErr := closeReader(); closeErr != nil { + scanErr = errors.Join(scanErr, closeErr) + } + return measured, scanErr +} + +func scanTarReader(tarReader *tar.Reader, limits Limits, enforceLimits bool) (*measuredArchive, error) { + measured := &measuredArchive{files: make([]measuredFile, 0)} + for { + header, err := tarReader.Next() + if err == io.EOF { + break + } + if err != nil { + return nil, fmt.Errorf("read tar pages package: %w", err) + } + entry := entryFromTarHeader(header) + normalizedPath, skip, err := validateArchiveEntry(entry) if err != nil { return nil, err } - manifest.FileCount++ - if manifest.FileCount > limits.MaxFiles { - return nil, fmt.Errorf("pages deployment file count exceeds %d", limits.MaxFiles) + if skip { + continue } - manifest.TotalSize += fileEntry.Size - if manifest.TotalSize > limits.MaxTotalBytes { - return nil, fmt.Errorf("pages extracted size exceeds limit") + if err := prepareMeasuredFile(measured, normalizedPath, entry.Size, limits, enforceLimits); err != nil { + return nil, err } + maxBytes := effectiveFileLimit(limits, measured.totalSize, enforceLimits) + actual, err := copyAndVerifySize(io.Discard, tarReader, entry.Size, maxBytes) + if err != nil { + return nil, fmt.Errorf("%s: %w", normalizedPath, err) + } + appendMeasuredFile(measured, normalizedPath, actual) + } + return measured, nil +} + +func buildMeasuredManifest(measured *measuredArchive, opts InspectOptions) (*Manifest, error) { + if measured == nil || measured.fileCount == 0 { + return nil, fmt.Errorf("pages package is empty") + } + targetEntryPath, err := resolveTargetEntryPath(opts.RootDir, opts.EntryFile) + if err != nil { + return nil, err + } + paths := make([]string, 0, len(measured.files)) + for _, file := range measured.files { + paths = append(paths, file.path) + } + commonPrefix := FindCommonRootPrefix(paths) + manifest := &Manifest{ + Files: make([]FileEntry, 0, measured.fileCount), + FileCount: measured.fileCount, + TotalSize: measured.totalSize, + } + entrySeen := false + for _, file := range measured.files { + normalizedPath := StripPrefix(file.path, commonPrefix) if normalizedPath == targetEntryPath { entrySeen = true } - manifest.Files = append(manifest.Files, fileEntry) - } - - if manifest.FileCount == 0 { - return nil, fmt.Errorf("pages package is empty") + manifest.Files = append(manifest.Files, FileEntry{ + Path: normalizedPath, + Size: file.size, + }) } if !entrySeen { return nil, fmt.Errorf("pages package is missing entry file %s", targetEntryPath) @@ -124,82 +216,86 @@ func buildManifest(entries []Entry, opts InspectOptions) (*Manifest, error) { return manifest, nil } -func collectFileNames(entries []Entry) []string { - names := make([]string, 0, len(entries)) - for _, entry := range entries { - if entry.IsDir || entry.IsSymlink { - continue - } - names = append(names, entry.Name) +func prepareMeasuredFile(measured *measuredArchive, normalizedPath string, declaredSize uint64, limits Limits, enforceLimits bool) error { + if declaredSize > uint64(math.MaxInt64) { + return fmt.Errorf("%s: pages file size out of bounds", normalizedPath) } - return names + if !enforceLimits { + return nil + } + if measured.fileCount >= limits.MaxFiles { + return fmt.Errorf("pages deployment file count exceeds %d", limits.MaxFiles) + } + if exceedsFileByteLimit(declaredSize, limits.MaxFileBytes) { + return fmt.Errorf("pages file too large: %s", normalizedPath) + } + remaining := limits.MaxTotalBytes - measured.totalSize + if remaining < 0 || declaredSize > uint64(remaining) { //nolint:gosec // remaining is checked non-negative + return fmt.Errorf("pages extracted size exceeds limit") + } + return nil } -func resolveTargetEntryPath(rootDir, entryFile string) string { - normalizedEntry := strings.TrimSpace(entryFile) - if normalizedEntry == "" { - normalizedEntry = "index.html" +func appendMeasuredFile(measured *measuredArchive, normalizedPath string, actual int64) { + measured.files = append(measured.files, measuredFile{path: normalizedPath, size: actual}) + measured.fileCount++ + measured.totalSize += actual +} + +func effectiveFileLimit(limits Limits, totalSize int64, enforceLimits bool) int64 { + if !enforceLimits { + return -1 + } + remaining := limits.MaxTotalBytes - totalSize + if remaining < limits.MaxFileBytes { + return remaining + } + return limits.MaxFileBytes +} + +func resolveTargetEntryPath(rootDir, entryFile string) (string, error) { + normalizedRoot, err := NormalizeLogicalPath(rootDir, true) + if err != nil { + return "", fmt.Errorf("invalid pages root directory: %w", err) + } + if entryFile == "" { + entryFile = "index.html" + } + normalizedEntry, err := NormalizeLogicalPath(entryFile, false) + if err != nil { + return "", fmt.Errorf("invalid pages entry file: %w", err) } - normalizedRoot := strings.Trim(strings.TrimSpace(rootDir), "/") if normalizedRoot == "" { - return normalizedEntry + return normalizedEntry, nil } - return path.Join(normalizedRoot, normalizedEntry) + return path.Join(normalizedRoot, normalizedEntry), nil } -func prepareEntryPath(entry Entry, commonPrefix string) (string, bool, error) { +func validateArchiveEntry(entry Entry) (string, bool, error) { normalizedPath, skip, err := NormalizeEntryPath(entry.Name) if err != nil { return "", false, err } - if skip || entry.IsDir { - return "", true, nil - } - normalizedPath = StripPrefix(normalizedPath, commonPrefix) if entry.IsSymlink { return "", false, fmt.Errorf("pages package contains unsupported symlink: %s", normalizedPath) } + if entry.IsHardlink { + return "", false, fmt.Errorf("pages package contains unsupported hardlink: %s", normalizedPath) + } + if entry.IsSpecial { + return "", false, fmt.Errorf("pages package contains unsupported special entry: %s", normalizedPath) + } + if skip || entry.IsDir { + return normalizedPath, true, nil + } return normalizedPath, false, nil } -func inspectRegularFile(entry Entry, normalizedPath string, limits Limits, verifySizes bool) (FileEntry, error) { - if entry.Size > uint64(math.MaxInt64) { - return FileEntry{}, fmt.Errorf("%s: pages file size out of bounds", normalizedPath) +func isTarFamily(format Format) bool { + switch format { + case FormatTar, FormatTarGz, FormatTarXz, FormatTarBz2: + return true + default: + return false } - //nolint:gosec // bounded to MaxInt64 above - declaredSize := int64(entry.Size) - - if !verifySizes { - return FileEntry{ - Path: normalizedPath, - Size: declaredSize, - Checksum: "", - }, nil - } - - if entry.Open == nil { - return FileEntry{}, fmt.Errorf("%s: cannot verify size without entry open", normalizedPath) - } - src, err := entry.Open() - if err != nil { - return FileEntry{}, fmt.Errorf("%s: %w", normalizedPath, err) - } - actual, measureErr := measureReader(src, entry.Size, limits.MaxFileBytes) - _ = src.Close() - if measureErr != nil { - return FileEntry{}, fmt.Errorf("%s: %w", normalizedPath, measureErr) - } - if declaredSize > 0 && actual != declaredSize { - return FileEntry{}, fmt.Errorf("%s: declared size %d does not match actual %d", normalizedPath, declaredSize, actual) - } - return FileEntry{ - Path: normalizedPath, - Size: actual, - Checksum: "", - }, nil -} - -// measureReader counts bytes without hashing, enforcing maxBytes when positive. -func measureReader(src io.Reader, declaredSize uint64, maxBytes int64) (int64, error) { - return copyLimited(io.Discard, src, declaredSize, maxBytes) } diff --git a/pkg/pagesarchive/list.go b/pkg/pagesarchive/list.go index 16f2fc8f..e5781f08 100644 --- a/pkg/pagesarchive/list.go +++ b/pkg/pagesarchive/list.go @@ -6,7 +6,6 @@ package pagesarchive import ( "archive/tar" "archive/zip" - "bytes" "compress/bzip2" "compress/gzip" "fmt" @@ -19,8 +18,8 @@ import ( type archiveFile interface { Name() string + Mode() os.FileMode IsDir() bool - IsSymlink() bool Size() uint64 Open() (io.ReadCloser, error) } @@ -29,12 +28,10 @@ type zipArchiveFile struct { file *zip.File } -func (z zipArchiveFile) Name() string { return z.file.Name } -func (z zipArchiveFile) IsDir() bool { return z.file.FileInfo().IsDir() } -func (z zipArchiveFile) IsSymlink() bool { - return z.file.Mode()&os.ModeSymlink != 0 -} -func (z zipArchiveFile) Size() uint64 { return z.file.UncompressedSize64 } +func (z zipArchiveFile) Name() string { return z.file.Name } +func (z zipArchiveFile) Mode() os.FileMode { return z.file.Mode() } +func (z zipArchiveFile) IsDir() bool { return z.file.FileInfo().IsDir() } +func (z zipArchiveFile) Size() uint64 { return z.file.UncompressedSize64 } func (z zipArchiveFile) Open() (io.ReadCloser, error) { return z.file.Open() } @@ -43,65 +40,27 @@ type sevenZipArchiveFile struct { file *sevenzip.File } -func (z sevenZipArchiveFile) Name() string { return z.file.Name } -func (z sevenZipArchiveFile) IsDir() bool { return z.file.FileInfo().IsDir() } -func (z sevenZipArchiveFile) IsSymlink() bool { - return z.file.Mode()&os.ModeSymlink != 0 -} -func (z sevenZipArchiveFile) Size() uint64 { return z.file.UncompressedSize } +func (z sevenZipArchiveFile) Name() string { return z.file.Name } +func (z sevenZipArchiveFile) Mode() os.FileMode { return z.file.Mode() } +func (z sevenZipArchiveFile) IsDir() bool { return z.file.FileInfo().IsDir() } +func (z sevenZipArchiveFile) Size() uint64 { return z.file.UncompressedSize } func (z sevenZipArchiveFile) Open() (io.ReadCloser, error) { return z.file.Open() } -// listEntriesAt lists archive members from a random-access source. -// When materializeBodies is true, tar-family streams buffer regular-file bodies so Entry.Open works. -// When false (inspect path), tar bodies are discarded after reading headers; zip/7z only use central directory metadata. -func listEntriesAt(ra io.ReaderAt, size int64, format Format, materializeBodies bool) ([]Entry, error) { +// listRandomAccessEntriesAt lists zip/7z members without reading their bodies. +// Tar-family archives use the sequential streaming paths in inspect.go/extract.go. +func listRandomAccessEntriesAt(ra io.ReaderAt, size int64, format Format) ([]Entry, error) { if size < 0 { return nil, fmt.Errorf("invalid pages package size") } switch format { case FormatZip: return listZipEntriesAt(ra, size) - case FormatTar: - return listTarFamily(io.NewSectionReader(ra, 0, size), FormatTar, materializeBodies) - case FormatTarGz: - return listTarFamily(io.NewSectionReader(ra, 0, size), FormatTarGz, materializeBodies) - case FormatTarXz: - return listTarFamily(io.NewSectionReader(ra, 0, size), FormatTarXz, materializeBodies) - case FormatTarBz2: - return listTarFamily(io.NewSectionReader(ra, 0, size), FormatTarBz2, materializeBodies) case FormatSevenZip: return listSevenZipEntriesAt(ra, size) default: - return nil, fmt.Errorf("unsupported pages package format: %s", format) - } -} - -func listTarFamily(r io.Reader, format Format, materializeBodies bool) ([]Entry, error) { - switch format { - case FormatTar: - if materializeBodies { - return listTarEntries(r, true) - } - return listTarEntries(r, false) - case FormatTarGz: - gzReader, err := gzip.NewReader(r) - if err != nil { - return nil, fmt.Errorf("open gzip pages package: %w", err) - } - defer func() { _ = gzReader.Close() }() - return listTarEntries(gzReader, materializeBodies) - case FormatTarXz: - xzReader, err := xz.NewReader(r) - if err != nil { - return nil, fmt.Errorf("open xz pages package: %w", err) - } - return listTarEntries(xzReader, materializeBodies) - case FormatTarBz2: - return listTarEntries(bzip2.NewReader(r), materializeBodies) - default: - return nil, fmt.Errorf("unsupported tar family format: %s", format) + return nil, fmt.Errorf("unsupported random-access pages package format: %s", format) } } @@ -109,10 +68,15 @@ func entriesFromArchiveFiles(files []archiveFile) []Entry { entries := make([]Entry, 0, len(files)) for _, item := range files { file := item + mode := file.Mode() + isDir := file.IsDir() + isSymlink := mode&os.ModeSymlink != 0 + isSpecial := !isDir && !isSymlink && !mode.IsRegular() entries = append(entries, Entry{ Name: file.Name(), - IsDir: file.IsDir(), - IsSymlink: file.IsSymlink(), + IsDir: isDir, + IsSymlink: isSymlink, + IsSpecial: isSpecial, Size: file.Size(), Open: file.Open, }) @@ -144,132 +108,45 @@ func listSevenZipEntriesAt(ra io.ReaderAt, size int64) ([]Entry, error) { return entriesFromArchiveFiles(files), nil } -func listTarEntries(r io.Reader, materializeBodies bool) ([]Entry, error) { - tarReader := tar.NewReader(r) - type materialised struct { - header *tar.Header - body []byte - } - items := make([]materialised, 0) - for { - header, err := tarReader.Next() - if err == io.EOF { - break - } +func openTarFamilyReader(r io.Reader, format Format) (*tar.Reader, func() error, error) { + switch format { + case FormatTar: + return tar.NewReader(r), func() error { return nil }, nil + case FormatTarGz: + gzReader, err := gzip.NewReader(r) if err != nil { - return nil, fmt.Errorf("read tar pages package: %w", err) + return nil, nil, fmt.Errorf("open gzip pages package: %w", err) } - item, skip, err := readTarHeader(tarReader, header, materializeBodies) + return tar.NewReader(gzReader), gzReader.Close, nil + case FormatTarXz: + xzReader, err := xz.NewReader(r) if err != nil { - return nil, err + return nil, nil, fmt.Errorf("open xz pages package: %w", err) } - if skip { - continue - } - items = append(items, item) - } - - entries := make([]Entry, 0, len(items)) - for _, item := range items { - entries = append(entries, tarEntryFromHeader(item.header, item.body, materializeBodies)) - } - return entries, nil -} - -func readTarHeader(tarReader *tar.Reader, header *tar.Header, materializeBodies bool) (item struct { - header *tar.Header - body []byte -}, skip bool, err error) { - switch header.Typeflag { - case tar.TypeDir, tar.TypeSymlink, tar.TypeLink: - return struct { - header *tar.Header - body []byte - }{header: header}, false, nil - case tar.TypeReg, tar.TypeRegA: //nolint:staticcheck // TypeRegA still appears in older archives - if !materializeBodies { - if err := discardTarBody(tarReader, header); err != nil { - return item, false, err - } - return struct { - header *tar.Header - body []byte - }{header: header}, false, nil - } - body, readErr := readTarBody(tarReader, header) - if readErr != nil { - return item, false, readErr - } - return struct { - header *tar.Header - body []byte - }{header: header, body: body}, false, nil + return tar.NewReader(xzReader), func() error { return nil }, nil + case FormatTarBz2: + return tar.NewReader(bzip2.NewReader(r)), func() error { return nil }, nil default: - if header.Size > 0 { - if _, copyErr := io.CopyN(io.Discard, tarReader, header.Size); copyErr != nil { - return item, false, fmt.Errorf("skip tar entry %s: %w", header.Name, copyErr) - } - } - return item, true, nil + return nil, nil, fmt.Errorf("unsupported tar family format: %s", format) } } -func discardTarBody(tarReader *tar.Reader, header *tar.Header) error { - if header.Size <= 0 { - _, err := io.Copy(io.Discard, tarReader) - if err != nil { - return fmt.Errorf("discard tar entry %s: %w", header.Name, err) - } - return nil - } - if _, err := io.CopyN(io.Discard, tarReader, header.Size); err != nil { - return fmt.Errorf("discard tar entry %s: %w", header.Name, err) - } - return nil -} - -func readTarBody(tarReader *tar.Reader, header *tar.Header) ([]byte, error) { +func entryFromTarHeader(header *tar.Header) Entry { + entry := Entry{Name: header.Name} if header.Size > 0 { - body := make([]byte, header.Size) - if _, err := io.ReadFull(tarReader, body); err != nil { - return nil, fmt.Errorf("read tar entry %s: %w", header.Name, err) - } - return body, nil + entry.Size = uint64(header.Size) //nolint:gosec // archive/tar rejects negative sizes } - body, err := io.ReadAll(tarReader) - if err != nil { - return nil, fmt.Errorf("read tar entry %s: %w", header.Name, err) - } - return body, nil -} - -func tarEntryFromHeader(header *tar.Header, body []byte, materializeBodies bool) Entry { - size := header.Size - if materializeBodies && int64(len(body)) > size { - size = int64(len(body)) - } - entry := Entry{ - Name: header.Name, - IsDir: header.Typeflag == tar.TypeDir, - IsSymlink: header.Typeflag == tar.TypeSymlink || header.Typeflag == tar.TypeLink, - Size: uint64(size), //nolint:gosec // non-negative sizes - } - if entry.IsDir || entry.IsSymlink { - entry.Open = func() (io.ReadCloser, error) { - return io.NopCloser(bytes.NewReader(nil)), nil - } - return entry - } - if materializeBodies { - bodyCopy := body - entry.Open = func() (io.ReadCloser, error) { - return io.NopCloser(bytes.NewReader(bodyCopy)), nil - } - return entry - } - // Inspect path: body not retained; Open is unavailable. - entry.Open = func() (io.ReadCloser, error) { - return nil, fmt.Errorf("tar entry body not materialized: %s", header.Name) + switch header.Typeflag { + case tar.TypeReg, tar.TypeRegA: //nolint:staticcheck // TypeRegA appears in older archives + // Regular file. + case tar.TypeDir: + entry.IsDir = true + case tar.TypeSymlink: + entry.IsSymlink = true + case tar.TypeLink: + entry.IsHardlink = true + default: + entry.IsSpecial = true } return entry } diff --git a/pkg/pagesarchive/path.go b/pkg/pagesarchive/path.go index 4735ff86..94562c8b 100644 --- a/pkg/pagesarchive/path.go +++ b/pkg/pagesarchive/path.go @@ -6,33 +6,89 @@ package pagesarchive import ( "fmt" "path" - "path/filepath" "strings" + "unicode" + "unicode/utf8" ) +// NormalizeLogicalPath validates and normalizes a relative POSIX path. +// Empty input is returned unchanged only when allowEmpty is true. +func NormalizeLogicalPath(raw string, allowEmpty bool) (string, error) { + if raw == "" { + if allowEmpty { + return "", nil + } + return "", fmt.Errorf("pages path is required") + } + if err := validateLogicalPathText(raw); err != nil { + return "", err + } + + cleaned := path.Clean(raw) + if cleaned == "." || cleaned == "" { + if allowEmpty { + return "", nil + } + return "", fmt.Errorf("pages path is required") + } + if strings.HasPrefix(cleaned, "/") || cleaned == ".." || strings.HasPrefix(cleaned, "../") { + return "", fmt.Errorf("pages path escapes directory: %s", raw) + } + return cleaned, nil +} + +func validateLogicalPathText(raw string) error { + if !utf8.ValidString(raw) { + return fmt.Errorf("pages path is not valid UTF-8") + } + if strings.Contains(raw, "\\") { + return fmt.Errorf("pages path must use POSIX separators: %s", raw) + } + if strings.HasPrefix(raw, "/") || path.IsAbs(raw) { + return fmt.Errorf("pages path must be relative: %s", raw) + } + if err := validateLogicalPathRunes(raw); err != nil { + return err + } + return validateLogicalPathSegments(raw) +} + +func validateLogicalPathRunes(raw string) error { + for _, r := range raw { + if r == 0 || unicode.IsControl(r) { + return fmt.Errorf("pages path contains a control character") + } + if r == '\'' || r == '"' || r == ';' { + return fmt.Errorf("pages path contains an unsupported character: %s", raw) + } + } + return nil +} + +func validateLogicalPathSegments(raw string) error { + for _, segment := range strings.Split(raw, "/") { + if len(segment) >= 2 && segment[1] == ':' { + return fmt.Errorf("pages path contains a Windows drive: %s", raw) + } + if segment == "." || segment == ".." { + return fmt.Errorf("pages path escapes directory or contains a dot segment: %s", raw) + } + } + return nil +} + // NormalizeEntryPath cleans an archive entry path and rejects zip-slip / absolute paths. // skip=true means the entry should be ignored (empty path or directory marker). func NormalizeEntryPath(raw string) (cleaned string, skip bool, err error) { - name := strings.TrimSpace(filepath.ToSlash(raw)) - if name == "" { + if raw == "" { return "", true, nil } - if strings.HasSuffix(name, "/") { - return "", true, nil + cleanedPath, normalizeErr := NormalizeLogicalPath(raw, false) + if normalizeErr != nil { + return "", false, fmt.Errorf("invalid pages package path %q: %w", raw, normalizeErr) } - if strings.HasPrefix(name, "/") || path.IsAbs(name) { - return "", false, fmt.Errorf("pages package contains absolute path: %s", raw) - } - // Reject Windows drive / UNC-style paths that may appear after ToSlash. - if len(name) >= 2 && name[1] == ':' { - return "", false, fmt.Errorf("pages package contains absolute path: %s", raw) - } - cleanedPath := path.Clean(name) - if cleanedPath == "." { - return "", true, nil - } - if cleanedPath == ".." || strings.HasPrefix(cleanedPath, "../") || strings.Contains(cleanedPath, "/../") { - return "", false, fmt.Errorf("pages package path escapes directory: %s", raw) + if strings.HasSuffix(raw, "/") { + return cleanedPath, true, nil } return cleanedPath, false, nil } diff --git a/pkg/pagesarchive/security_test.go b/pkg/pagesarchive/security_test.go new file mode 100644 index 00000000..9194991e --- /dev/null +++ b/pkg/pagesarchive/security_test.go @@ -0,0 +1,499 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "archive/tar" + "archive/zip" + "bytes" + "encoding/base64" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +var testLimits = Limits{ + MaxFiles: 100, + MaxFileBytes: 1 << 20, + MaxTotalBytes: 1 << 20, +} + +func TestNormalizeLogicalPathStrict(t *testing.T) { + t.Parallel() + + valid := []struct { + name string + raw string + allowEmpty bool + want string + }{ + {name: "empty root", allowEmpty: true}, + {name: "single file", raw: "index.html", want: "index.html"}, + {name: "nested posix", raw: "public/assets/app.js", want: "public/assets/app.js"}, + {name: "unicode", raw: "静态/首页.html", want: "静态/首页.html"}, + {name: "repeated separator is normalized", raw: "public//app.js", want: "public/app.js"}, + } + for _, tt := range valid { + t.Run(tt.name, func(t *testing.T) { + got, err := NormalizeLogicalPath(tt.raw, tt.allowEmpty) + require.NoError(t, err) + assert.Equal(t, tt.want, got) + }) + } + + invalidUTF8 := string([]byte{'a', '/', 0xff}) + invalid := []struct { + name string + raw string + }{ + {name: "empty entry"}, + {name: "absolute", raw: "/etc/passwd"}, + {name: "unc", raw: "//server/share"}, + {name: "windows drive", raw: "C:/site/index.html"}, + {name: "nested windows drive", raw: "site/C:/index.html"}, + {name: "windows separator", raw: `site\index.html`}, + {name: "windows unc", raw: `\\server\share`}, + {name: "parent segment", raw: "../index.html"}, + {name: "nested parent segment", raw: "site/../index.html"}, + {name: "current segment", raw: "site/./index.html"}, + {name: "nul", raw: "site/\x00index.html"}, + {name: "newline", raw: "site/\nindex.html"}, + {name: "delete control", raw: "site/\x7findex.html"}, + {name: "single quote", raw: "site/'index.html"}, + {name: "double quote", raw: `site/"index.html`}, + {name: "semicolon", raw: "site/;index.html"}, + {name: "invalid utf8", raw: invalidUTF8}, + } + for _, tt := range invalid { + t.Run(tt.name, func(t *testing.T) { + _, err := NormalizeLogicalPath(tt.raw, false) + require.Error(t, err) + }) + } + + cleaned, skip, err := NormalizeEntryPath("") + require.NoError(t, err) + assert.Empty(t, cleaned) + assert.True(t, skip) + + cleaned, skip, err = NormalizeEntryPath("assets/") + require.NoError(t, err) + assert.Equal(t, "assets", cleaned) + assert.True(t, skip) +} + +func TestSupportedFormatsInspectAndExtract(t *testing.T) { + t.Parallel() + + sevenZipData := decodeFixture(t, "N3q8ryccAASgR6WICAAAAAAAAABmAAAAAAAAAN2R8/FiYXIKZm9vCgEEBgACCQQEAAcLAgABAQABAQAMBAQACAoB6bOiBKhlMn4AAAUCGQUAAAAAABERAGIAYQByAAAAZgBvAG8AAAAZAgAAFBIBAACFM3PyY9YBAFgCcvJj1gEVCgEAIICkgSCApIEAAA==") + bzipTarData := decodeFixture(t, "QlpoOTFBWSZTWYp5f6EAAHV//P64A8RQAf/iOm/9cO/v/9AAAgBADlAABAADAAgwAU1RIZJpNCaammmnqbSGTI9Q0BoBpp6mjIaGmmRoaHGRpkxNBkyYTTIGQ0BoDTJoYATQGG1KCntExT01MhoAABoAAHqAAPU9QacVDtN45fA6MmuGVQlWowrijpZgwASITYSPUcpJpoQGMkKq69jMkUR6L86R5j0IySUaZEjazEqhQ9E8vuuxsmWZQLCA84jNsobYNEzuEB1eCPhw8nc2AOz+xrCY5hVxQW1IIokpfSRKi+McvXU+QoYuEg6BD4w8x3K0imi+bULpkLCylCZ4lzoGlTQgibvG67sQcrTCRBTbBCVL7zC0q0qULmK/WOneu94s9cs4s4K98SjY2YvpdZvl42kwtxvvPMheorYQ2pcxyF4sNQYvd4+bgqm5gKXElqnGF3jhxGTeXp9eCUxWVlbi9ikxAik4xxATl7cJrISVWnHwUFiLdhEnKWw0Lhm3ZyKlX7P5Wj7b9TLAmWBaAwH/F3JFOFCQinl/oQ==") + + cases := []struct { + name string + format Format + data []byte + entryFile string + wantPath string + }{ + {name: "zip", format: FormatZip, data: testZip(t, map[string]string{"bundle/index.html": "zip"}), entryFile: "index.html", wantPath: "index.html"}, + {name: "tar", format: FormatTar, data: testTar(t, map[string]string{"bundle/index.html": "tar"}), entryFile: "index.html", wantPath: "index.html"}, + {name: "tar gzip", format: FormatTarGz, data: testTarGz(t, map[string]string{"bundle/index.html": "gzip"}), entryFile: "index.html", wantPath: "index.html"}, + {name: "tar xz", format: FormatTarXz, data: testTarXz(t, map[string]string{"bundle/index.html": "xz"}), entryFile: "index.html", wantPath: "index.html"}, + {name: "tar bzip2", format: FormatTarBz2, data: bzipTarData, entryFile: "index.html", wantPath: "index.html"}, + {name: "7z", format: FormatSevenZip, data: sevenZipData, entryFile: "foo", wantPath: "foo"}, + } + for _, tt := range cases { + t.Run(tt.name, func(t *testing.T) { + manifest, err := InspectBytes(tt.data, tt.format, InspectOptions{ + EntryFile: tt.entryFile, + Limits: testLimits, + }) + require.NoError(t, err) + assert.Positive(t, manifest.FileCount) + assertManifestContains(t, manifest, tt.wantPath) + + destDir := t.TempDir() + require.NoError(t, ExtractBytes(tt.data, tt.format, destDir, ExtractOptions{ + Limits: testLimits, + StripCommonRoot: true, + EnforceLimits: true, + })) + _, err = os.Stat(filepath.Join(destDir, filepath.FromSlash(tt.wantPath))) + require.NoError(t, err) + }) + } +} + +func TestExtractFilePreservesCommonRootAndEnforcesLimits(t *testing.T) { + t.Parallel() + + data := testTarGz(t, map[string]string{ + "repository/dist/index.html": "pages", + "repository/dist/app.js": "javascript", + }) + archivePath := filepath.Join(t.TempDir(), "site.tar.gz") + require.NoError(t, os.WriteFile(archivePath, data, 0o600)) + + manifest, err := InspectFile(archivePath, FormatTarGz, InspectOptions{ + RootDir: "dist", + EntryFile: "index.html", + Limits: testLimits, + }) + require.NoError(t, err) + assertManifestContains(t, manifest, "dist/index.html") + + destDir := t.TempDir() + require.NoError(t, ExtractFile(archivePath, FormatTarGz, destDir, ExtractOptions{ + Limits: testLimits, + StripCommonRoot: true, + EnforceLimits: true, + })) + body, err := os.ReadFile(filepath.Join(destDir, "dist", "index.html")) //nolint:gosec + require.NoError(t, err) + assert.Equal(t, "pages", string(body)) + + err = ExtractFile(archivePath, FormatTarGz, t.TempDir(), ExtractOptions{ + Limits: Limits{ + MaxFiles: 10, + MaxFileBytes: 5, + MaxTotalBytes: 1 << 20, + }, + StripCommonRoot: true, + EnforceLimits: true, + }) + require.ErrorContains(t, err, "file too large") + + err = ExtractFile(archivePath, FormatTarGz, t.TempDir(), ExtractOptions{ + Limits: Limits{ + MaxFiles: 10, + MaxFileBytes: 1 << 20, + MaxTotalBytes: int64(len("pages") + len("javascript") - 1), + }, + StripCommonRoot: true, + EnforceLimits: true, + }) + require.ErrorContains(t, err, "extracted size exceeds limit") +} + +func TestRandomAccessMembersVerifyActualSize(t *testing.T) { + t.Parallel() + + entry := Entry{ + Name: "index.html", + Size: 1, + Open: func() (io.ReadCloser, error) { + return io.NopCloser(strings.NewReader("actual-body")), nil + }, + } + _, err := inspectRandomAccessEntries([]Entry{entry}, testLimits) + require.ErrorContains(t, err, "declared size 1 does not match actual 11") + + destDir := t.TempDir() + err = extractEntries([]Entry{entry}, destDir, ExtractOptions{ + Limits: testLimits, + EnforceLimits: true, + }) + require.ErrorContains(t, err, "declared size 1 does not match actual 11") + _, statErr := os.Stat(filepath.Join(destDir, "index.html")) + assert.ErrorIs(t, statErr, os.ErrNotExist, "failed extraction must remove the partial file") +} + +func TestActualByteLimitAbortsReaderEarly(t *testing.T) { + t.Parallel() + + reader := &countingFillReader{remaining: 1 << 30} + entry := Entry{ + Name: "index.html", + Size: 0, + Open: func() (io.ReadCloser, error) { + return io.NopCloser(reader), nil + }, + } + _, err := inspectRandomAccessEntries([]Entry{entry}, Limits{ + MaxFiles: 1, + MaxFileBytes: 32, + MaxTotalBytes: 32, + }) + require.ErrorContains(t, err, "size out of bounds") + assert.LessOrEqual(t, reader.read, int64(33), "inspection must stop after limit+1 actual bytes") + + tarData := tarWithDeclaredBodyOnly(t, "index.html", 1<<30) + _, err = InspectBytes(tarData, FormatTar, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{ + MaxFiles: 1, + MaxFileBytes: 32, + MaxTotalBytes: 32, + }, + }) + require.ErrorContains(t, err, "file too large") +} + +func TestArchiveLimitsUseFilesAndActualTotals(t *testing.T) { + t.Parallel() + + data := testZip(t, map[string]string{ + "index.html": "1234", + "app.js": "5678", + }) + _, err := InspectBytes(data, FormatZip, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{ + MaxFiles: 1, + MaxFileBytes: 8, + MaxTotalBytes: 16, + }, + }) + require.ErrorContains(t, err, "file count exceeds") + + _, err = InspectBytes(data, FormatZip, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{ + MaxFiles: 2, + MaxFileBytes: 8, + MaxTotalBytes: 7, + }, + }) + require.ErrorContains(t, err, "extracted size exceeds limit") + + sevenZipData := decodeFixture(t, "N3q8ryccAASgR6WICAAAAAAAAABmAAAAAAAAAN2R8/FiYXIKZm9vCgEEBgACCQQEAAcLAgABAQABAQAMBAQACAoB6bOiBKhlMn4AAAUCGQUAAAAAABERAGIAYQByAAAAZgBvAG8AAAAZAgAAFBIBAACFM3PyY9YBAFgCcvJj1gEVCgEAIICkgSCApIEAAA==") + _, err = InspectBytes(sevenZipData, FormatSevenZip, InspectOptions{ + EntryFile: "foo", + Limits: Limits{ + MaxFiles: 10, + MaxFileBytes: 3, + MaxTotalBytes: 32, + }, + }) + require.ErrorContains(t, err, "file too large") +} + +func TestRejectUnsupportedTarMemberTypes(t *testing.T) { + t.Parallel() + + cases := []struct { + name string + header tar.Header + wantErr string + }{ + {name: "symlink", header: tar.Header{Name: "link", Typeflag: tar.TypeSymlink, Linkname: "index.html"}, wantErr: "unsupported symlink"}, + {name: "hardlink", header: tar.Header{Name: "hard", Typeflag: tar.TypeLink, Linkname: "index.html"}, wantErr: "unsupported hardlink"}, + {name: "fifo", header: tar.Header{Name: "pipe", Typeflag: tar.TypeFifo, Mode: 0o600}, wantErr: "unsupported special entry"}, + {name: "character device", header: tar.Header{Name: "tty", Typeflag: tar.TypeChar, Mode: 0o600, Devmajor: 1, Devminor: 3}, wantErr: "unsupported special entry"}, + } + for _, tt := range cases { + t.Run(tt.name, func(t *testing.T) { + data := tarWithSpecialEntry(t, &tt.header) + _, err := InspectBytes(data, FormatTar, InspectOptions{ + EntryFile: "index.html", + Limits: testLimits, + }) + require.ErrorContains(t, err, tt.wantErr) + + destDir := t.TempDir() + err = ExtractBytes(data, FormatTar, destDir, ExtractOptions{ + Limits: testLimits, + EnforceLimits: true, + }) + require.ErrorContains(t, err, tt.wantErr) + _, statErr := os.Stat(filepath.Join(destDir, "index.html")) + assert.ErrorIs(t, statErr, os.ErrNotExist, "tar validation pass must reject before writing files") + }) + } +} + +func TestRejectUnsupportedZipMemberTypes(t *testing.T) { + t.Parallel() + + cases := []struct { + name string + mode os.FileMode + wantErr string + }{ + {name: "symlink", mode: os.ModeSymlink | 0o777, wantErr: "unsupported symlink"}, + {name: "named pipe", mode: os.ModeNamedPipe | 0o600, wantErr: "unsupported special entry"}, + {name: "device", mode: os.ModeDevice | 0o600, wantErr: "unsupported special entry"}, + } + for _, tt := range cases { + t.Run(tt.name, func(t *testing.T) { + data := zipWithSpecialEntry(t, tt.mode) + _, err := InspectBytes(data, FormatZip, InspectOptions{ + EntryFile: "index.html", + Limits: testLimits, + }) + require.ErrorContains(t, err, tt.wantErr) + + err = ExtractBytes(data, FormatZip, t.TempDir(), ExtractOptions{ + Limits: testLimits, + EnforceLimits: true, + }) + require.ErrorContains(t, err, tt.wantErr) + }) + } +} + +func TestTarMetadataHeadersRemainTransparent(t *testing.T) { + t.Parallel() + + for _, format := range []tar.Format{tar.FormatPAX, tar.FormatGNU} { + format := format + t.Run(format.String(), func(t *testing.T) { + data := tarWithLongMetadata(t, format) + manifest, err := InspectBytes(data, FormatTar, InspectOptions{ + EntryFile: "index.html", + Limits: testLimits, + }) + require.NoError(t, err) + assert.Equal(t, 2, manifest.FileCount) + assertManifestContains(t, manifest, "index.html") + + destDir := t.TempDir() + require.NoError(t, ExtractBytes(data, FormatTar, destDir, ExtractOptions{ + Limits: testLimits, + EnforceLimits: true, + })) + _, err = os.Stat(filepath.Join(destDir, "index.html")) + require.NoError(t, err) + }) + } +} + +type countingFillReader struct { + remaining int64 + read int64 +} + +func (r *countingFillReader) Read(p []byte) (int, error) { + if r.remaining == 0 { + return 0, io.EOF + } + if int64(len(p)) > r.remaining { + p = p[:r.remaining] + } + for i := range p { + p[i] = 'x' + } + r.remaining -= int64(len(p)) + r.read += int64(len(p)) + return len(p), nil +} + +func decodeFixture(t *testing.T, encoded string) []byte { + t.Helper() + data, err := base64.StdEncoding.DecodeString(encoded) + require.NoError(t, err) + return data +} + +func assertManifestContains(t *testing.T, manifest *Manifest, path string) { + t.Helper() + for _, file := range manifest.Files { + if file.Path == path { + return + } + } + require.Failf(t, "manifest path missing", "path %q not found in %#v", path, manifest.Files) +} + +func testTar(t *testing.T, files map[string]string) []byte { + t.Helper() + var buffer bytes.Buffer + writer := tar.NewWriter(&buffer) + for name, content := range files { + require.NoError(t, writer.WriteHeader(&tar.Header{ + Name: name, + Mode: 0o644, + Size: int64(len(content)), + })) + _, err := writer.Write([]byte(content)) + require.NoError(t, err) + } + require.NoError(t, writer.Close()) + return buffer.Bytes() +} + +func tarWithDeclaredBodyOnly(t *testing.T, name string, size int64) []byte { + t.Helper() + var buffer bytes.Buffer + writer := tar.NewWriter(&buffer) + require.NoError(t, writer.WriteHeader(&tar.Header{ + Name: name, + Mode: 0o644, + Size: size, + })) + // Deliberately omit the body and trailer. The limit must reject from the + // header before archive/tar attempts to stream the declared body. + return buffer.Bytes() +} + +func tarWithSpecialEntry(t *testing.T, special *tar.Header) []byte { + t.Helper() + var buffer bytes.Buffer + writer := tar.NewWriter(&buffer) + require.NoError(t, writer.WriteHeader(&tar.Header{ + Name: "index.html", + Mode: 0o644, + Size: 2, + })) + _, err := writer.Write([]byte("ok")) + require.NoError(t, err) + require.NoError(t, writer.WriteHeader(special)) + require.NoError(t, writer.Close()) + return buffer.Bytes() +} + +func zipWithSpecialEntry(t *testing.T, mode os.FileMode) []byte { + t.Helper() + var buffer bytes.Buffer + writer := zip.NewWriter(&buffer) + index, err := writer.Create("index.html") + require.NoError(t, err) + _, err = index.Write([]byte("ok")) + require.NoError(t, err) + + header := &zip.FileHeader{Name: "special"} + header.SetMode(mode) + special, err := writer.CreateHeader(header) + require.NoError(t, err) + if mode&os.ModeSymlink != 0 { + _, err = special.Write([]byte("index.html")) + require.NoError(t, err) + } + require.NoError(t, writer.Close()) + return buffer.Bytes() +} + +func tarWithLongMetadata(t *testing.T, format tar.Format) []byte { + t.Helper() + var buffer bytes.Buffer + writer := tar.NewWriter(&buffer) + longName := strings.Repeat("long-segment-", 12) + "asset.js" + header := &tar.Header{ + Name: longName, + Mode: 0o644, + Size: 1, + Format: format, + } + if format == tar.FormatPAX { + header.PAXRecords = map[string]string{"comment": "metadata is not a deployable member"} + } + require.NoError(t, writer.WriteHeader(header)) + _, err := writer.Write([]byte("x")) + require.NoError(t, err) + require.NoError(t, writer.WriteHeader(&tar.Header{ + Name: "index.html", + Mode: 0o644, + Size: 2, + Format: format, + })) + _, err = writer.Write([]byte("ok")) + require.NoError(t, err) + require.NoError(t, writer.Close()) + return buffer.Bytes() +} diff --git a/pkg/protocol/agent.go b/pkg/protocol/agent.go index 26572286..cd0ba6ee 100644 --- a/pkg/protocol/agent.go +++ b/pkg/protocol/agent.go @@ -1,3 +1,6 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + // Package protocol defines the communication protocol between OpenFlare server, agent, and relay components. package protocol @@ -218,4 +221,7 @@ type PagesProjectLatestHashResponse struct { ProjectID uint `json:"project_id"` DeploymentID uint `json:"deployment_id"` Hash string `json:"hash"` + PackageSize int64 `json:"package_size"` + FileCount int `json:"file_count"` + TotalSize int64 `json:"total_size"` } diff --git a/pkg/protocol/agent_test.go b/pkg/protocol/agent_test.go index 4a0f71ab..0a1c99d4 100644 --- a/pkg/protocol/agent_test.go +++ b/pkg/protocol/agent_test.go @@ -1,3 +1,6 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + package protocol import ( @@ -47,6 +50,18 @@ func TestAgentProtocolJSONTags(t *testing.T) { "AccessToken": "agent_token", }, }, + { + name: "PagesProjectLatestHashResponse", + value: PagesProjectLatestHashResponse{}, + expected: map[string]string{ + "ProjectID": "project_id", + "DeploymentID": "deployment_id", + "Hash": "hash", + "PackageSize": "package_size", + "FileCount": "file_count", + "TotalSize": "total_size", + }, + }, } for _, tc := range cases { From 38b0516937db89b7c7037fb135887e4415017177 Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 17:36:51 +0800 Subject: [PATCH 3/8] =?UTF-8?q?feat(pages):=20=E6=94=AF=E6=8C=81=20Remote?= =?UTF-8?q?=20=E9=83=A8=E7=BD=B2=E6=BA=90=E5=90=8C=E6=AD=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增部署源配置与运行态模型、安全下载、租约续期、原子激活和失败补偿。 接入内部任务与脱敏前端交互，并阻止数据库 Trace 和日志展开敏感查询参数。 --- docs/docs.go | 541 ++++++++++++- docs/swagger.json | 541 ++++++++++++- docs/swagger.yaml | 336 +++++++- .../components/deployment-upload-dialog.tsx | 268 +++---- .../(main)/pages/components/pages-utils.ts | 4 + .../components/deployment-files-panel.tsx | 86 +++ .../detail/components/deployment-history.tsx | 300 ++++++++ .../detail/components/pages-source-card.tsx | 393 ++++++++++ .../detail/components/pages-source-dialog.tsx | 376 +++++++++ .../app/(main)/pages/detail/page-client.tsx | 505 ------------- frontend/app/(main)/pages/detail/page.tsx | 307 +++++++- frontend/lib/services/index.ts | 3 + frontend/lib/services/openflare/index.ts | 11 + .../lib/services/openflare/pages.service.ts | 43 +- frontend/lib/services/openflare/types.ts | 81 +- .../tests/openflare/pages-service.test.ts | 105 ++- .../tests/openflare/pages-source-ui.test.tsx | 227 ++++++ internal/apps/admin/task/routers.go | 18 +- internal/apps/admin/task/routers_test.go | 133 ++++ internal/apps/openflare/pages/download_url.go | 266 +------ .../apps/openflare/pages/download_url_test.go | 17 +- internal/apps/openflare/pages/errs.go | 16 + internal/apps/openflare/pages/helpers.go | 25 +- internal/apps/openflare/pages/logics.go | 347 +++++++-- internal/apps/openflare/pages/logics_test.go | 2 + internal/apps/openflare/pages/routers.go | 219 +++++- .../openflare/pages/routers_source_test.go | 224 ++++++ internal/apps/openflare/pages/source.go | 495 ++++++++++++ .../apps/openflare/pages/source_provider.go | 549 ++++++++++++++ .../openflare/pages/source_provider_test.go | 459 +++++++++++ .../apps/openflare/pages/source_runtime.go | 309 ++++++++ .../openflare/pages/source_runtime_test.go | 350 +++++++++ internal/apps/openflare/pages/source_sync.go | 715 ++++++++++++++++++ .../apps/openflare/pages/source_sync_test.go | 604 +++++++++++++++ internal/apps/openflare/pages/source_tasks.go | 275 +++++++ .../apps/openflare/pages/source_tasks_test.go | 166 ++++ internal/apps/openflare/pages/source_test.go | 386 ++++++++++ .../202607190001_add_pages_source_runtime.sql | 82 ++ .../202607190001_add_pages_source_runtime.sql | 151 ++++ internal/db/migrator/migrator_test.go | 6 +- .../migrator/pages_source_migration_test.go | 322 ++++++++ internal/db/postgres.go | 37 +- internal/db/postgres_logger.go | 5 + internal/db/postgres_logger_test.go | 76 ++ internal/db/postgres_tracing_test.go | 69 ++ internal/model/openflare_pages.go | 43 +- internal/model/openflare_pages_source.go | 59 ++ internal/model/openflare_pages_source_test.go | 80 ++ .../router/v1/openflare/register_pages.go | 5 + internal/task/executor.go | 6 +- internal/task/executor_test.go | 51 ++ internal/task/handlers/register.go | 5 + internal/task/meta.go | 12 +- internal/task/meta_test.go | 25 + internal/task/permanent_error.go | 35 + internal/task/permanent_error_test.go | 27 + pkg/httppool/httppool.go | 72 +- pkg/httppool/httppool_test.go | 63 ++ 58 files changed, 9858 insertions(+), 1075 deletions(-) create mode 100644 frontend/app/(main)/pages/detail/components/deployment-files-panel.tsx create mode 100644 frontend/app/(main)/pages/detail/components/deployment-history.tsx create mode 100644 frontend/app/(main)/pages/detail/components/pages-source-card.tsx create mode 100644 frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx delete mode 100644 frontend/app/(main)/pages/detail/page-client.tsx create mode 100644 frontend/tests/openflare/pages-source-ui.test.tsx create mode 100644 internal/apps/openflare/pages/routers_source_test.go create mode 100644 internal/apps/openflare/pages/source.go create mode 100644 internal/apps/openflare/pages/source_provider.go create mode 100644 internal/apps/openflare/pages/source_provider_test.go create mode 100644 internal/apps/openflare/pages/source_runtime.go create mode 100644 internal/apps/openflare/pages/source_runtime_test.go create mode 100644 internal/apps/openflare/pages/source_sync.go create mode 100644 internal/apps/openflare/pages/source_sync_test.go create mode 100644 internal/apps/openflare/pages/source_tasks.go create mode 100644 internal/apps/openflare/pages/source_tasks_test.go create mode 100644 internal/apps/openflare/pages/source_test.go create mode 100644 internal/db/migrator/goose/postgres/202607190001_add_pages_source_runtime.sql create mode 100644 internal/db/migrator/goose/sqlite/202607190001_add_pages_source_runtime.sql create mode 100644 internal/db/migrator/pages_source_migration_test.go create mode 100644 internal/db/postgres_tracing_test.go create mode 100644 internal/model/openflare_pages_source.go create mode 100644 internal/model/openflare_pages_source_test.go create mode 100644 internal/task/permanent_error.go create mode 100644 internal/task/permanent_error_test.go diff --git a/docs/docs.go b/docs/docs.go index df744b22..6947b49a 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -8584,7 +8584,7 @@ const docTemplate = `{ "SessionCookie": [] } ], - "description": "从用户提供的 HTTP(S) 链接下载部署包并创建部署记录；服务端使用浏览器伪装请求头拉取，允许内网地址与不安全 TLS 证书，需要管理员权限", + "description": "已弃用的一次性 URL 导入；使用 trusted_internal 策略兼容内网与自签名证书，不创建持久部署源", "consumes": [ "application/json" ], @@ -8595,6 +8595,7 @@ const docTemplate = `{ "openflare-pages" ], "summary": "从 URL 导入 Pages 部署包", + "deprecated": true, "parameters": [ { "type": "integer", @@ -8801,6 +8802,391 @@ const docTemplate = `{ } } }, + "/api/v1/d/pages/{id}/source": { + "get": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "返回脱敏后的项目部署源配置与运行状态，需要管理员权限", + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "获取 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "部署源", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceView" + } + } + } + ] + } + }, + "400": { + "description": "参数错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "项目或部署源不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/check": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约", + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "检查 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "任务回执", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceActionReceipt" + } + } + } + ] + } + }, + "400": { + "description": "当前来源不支持检查", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "部署源不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "409": { + "description": "来源任务正在执行", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/delete": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "幂等删除持久部署源；已有部署历史与当前生产部署保持不变", + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "删除 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "手动来源视图", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceView" + } + } + } + ] + } + }, + "400": { + "description": "参数错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "项目不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/sync": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "异步下载、校验并原子激活来源部署包；空请求体与空 JSON 对象均有效", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "同步并发布 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + }, + { + "description": "同步参数", + "name": "request", + "in": "body", + "schema": { + "$ref": "#/definitions/pages.SourceSyncInput" + } + } + ], + "responses": { + "200": { + "description": "任务回执", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceActionReceipt" + } + } + } + ] + } + }, + "400": { + "description": "参数或来源类型无效", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "部署源不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "409": { + "description": "来源任务正在执行", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/update": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "更新 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + }, + { + "description": "部署源配置", + "name": "request", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/pages.SourceUpdateInput" + } + } + ], + "responses": { + "200": { + "description": "更新结果", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceUpdateResult" + } + } + } + ] + } + }, + "400": { + "description": "配置无效", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "项目不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, "/api/v1/d/pages/{id}/update": { "post": { "security": [ @@ -17329,12 +17715,21 @@ const docTemplate = `{ "project_id": { "type": "integer" }, + "source_label": { + "type": "string" + }, + "source_type": { + "type": "string" + }, "status": { "type": "string" }, "total_size": { "type": "integer" }, + "trigger_type": { + "type": "string" + }, "upload_id": { "type": "string", "example": "0" @@ -17382,6 +17777,150 @@ const docTemplate = `{ } } }, + "pages.SourceActionReceipt": { + "type": "object", + "properties": { + "action": { + "type": "string" + }, + "execution_id": { + "type": "string" + }, + "task_id": { + "type": "string" + } + } + }, + "pages.SourceRevisionView": { + "type": "object", + "properties": { + "asset_name": { + "type": "string" + }, + "label": { + "type": "string" + }, + "revision": { + "type": "string" + } + } + }, + "pages.SourceSyncInput": { + "type": "object", + "properties": { + "confirmed_revision": { + "type": "string" + } + } + }, + "pages.SourceUpdateInput": { + "type": "object", + "properties": { + "asset_name": { + "type": "string" + }, + "auto_update_enabled": { + "type": "boolean" + }, + "check_interval_minutes": { + "type": "integer" + }, + "release_selector": { + "type": "string" + }, + "release_tag": { + "type": "string" + }, + "remote_network_policy": { + "type": "string" + }, + "remote_url": { + "type": "string" + }, + "remote_url_set": { + "type": "boolean" + }, + "repository_url": { + "type": "string" + }, + "source_type": { + "type": "string" + } + } + }, + "pages.SourceUpdateResult": { + "type": "object", + "properties": { + "check_task": { + "$ref": "#/definitions/pages.SourceActionReceipt" + }, + "source": { + "$ref": "#/definitions/pages.SourceView" + }, + "warning": { + "type": "string" + } + } + }, + "pages.SourceView": { + "type": "object", + "properties": { + "asset_name": { + "type": "string" + }, + "auto_update_enabled": { + "type": "boolean" + }, + "check_interval_minutes": { + "type": "integer" + }, + "display_url": { + "type": "string" + }, + "github_repository": { + "type": "string" + }, + "has_remote_url": { + "type": "boolean" + }, + "last_applied": { + "$ref": "#/definitions/pages.SourceRevisionView" + }, + "last_checked_at": { + "type": "string" + }, + "last_error": { + "type": "string" + }, + "last_seen": { + "$ref": "#/definitions/pages.SourceRevisionView" + }, + "last_synced_at": { + "type": "string" + }, + "next_check_at": { + "type": "string" + }, + "release_selector": { + "type": "string" + }, + "release_tag": { + "type": "string" + }, + "remote_network_policy": { + "type": "string" + }, + "source_type": { + "type": "string" + }, + "sync_status": { + "type": "string" + }, + "update_available": { + "type": "boolean" + } + } + }, "pages.UploadFromURLInput": { "type": "object", "properties": { diff --git a/docs/swagger.json b/docs/swagger.json index efdab2a3..5c81cc5e 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -8577,7 +8577,7 @@ "SessionCookie": [] } ], - "description": "从用户提供的 HTTP(S) 链接下载部署包并创建部署记录；服务端使用浏览器伪装请求头拉取，允许内网地址与不安全 TLS 证书，需要管理员权限", + "description": "已弃用的一次性 URL 导入；使用 trusted_internal 策略兼容内网与自签名证书，不创建持久部署源", "consumes": [ "application/json" ], @@ -8588,6 +8588,7 @@ "openflare-pages" ], "summary": "从 URL 导入 Pages 部署包", + "deprecated": true, "parameters": [ { "type": "integer", @@ -8794,6 +8795,391 @@ } } }, + "/api/v1/d/pages/{id}/source": { + "get": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "返回脱敏后的项目部署源配置与运行状态，需要管理员权限", + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "获取 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "部署源", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceView" + } + } + } + ] + } + }, + "400": { + "description": "参数错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "项目或部署源不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/check": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约", + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "检查 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "任务回执", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceActionReceipt" + } + } + } + ] + } + }, + "400": { + "description": "当前来源不支持检查", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "部署源不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "409": { + "description": "来源任务正在执行", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/delete": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "幂等删除持久部署源；已有部署历史与当前生产部署保持不变", + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "删除 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "手动来源视图", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceView" + } + } + } + ] + } + }, + "400": { + "description": "参数错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "项目不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/sync": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "异步下载、校验并原子激活来源部署包；空请求体与空 JSON 对象均有效", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "同步并发布 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + }, + { + "description": "同步参数", + "name": "request", + "in": "body", + "schema": { + "$ref": "#/definitions/pages.SourceSyncInput" + } + } + ], + "responses": { + "200": { + "description": "任务回执", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceActionReceipt" + } + } + } + ] + } + }, + "400": { + "description": "参数或来源类型无效", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "部署源不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "409": { + "description": "来源任务正在执行", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, + "/api/v1/d/pages/{id}/source/update": { + "post": { + "security": [ + { + "SessionCookie": [] + } + ], + "description": "Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "openflare-pages" + ], + "summary": "更新 Pages 部署源", + "parameters": [ + { + "type": "integer", + "description": "项目 ID", + "name": "id", + "in": "path", + "required": true + }, + { + "description": "部署源配置", + "name": "request", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/pages.SourceUpdateInput" + } + } + ], + "responses": { + "200": { + "description": "更新结果", + "schema": { + "allOf": [ + { + "$ref": "#/definitions/response.Any" + }, + { + "type": "object", + "properties": { + "data": { + "$ref": "#/definitions/pages.SourceUpdateResult" + } + } + } + ] + } + }, + "400": { + "description": "配置无效", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "401": { + "description": "未登录", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "404": { + "description": "项目不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, + "500": { + "description": "内部错误", + "schema": { + "$ref": "#/definitions/response.Any" + } + } + } + } + }, "/api/v1/d/pages/{id}/update": { "post": { "security": [ @@ -17322,12 +17708,21 @@ "project_id": { "type": "integer" }, + "source_label": { + "type": "string" + }, + "source_type": { + "type": "string" + }, "status": { "type": "string" }, "total_size": { "type": "integer" }, + "trigger_type": { + "type": "string" + }, "upload_id": { "type": "string", "example": "0" @@ -17375,6 +17770,150 @@ } } }, + "pages.SourceActionReceipt": { + "type": "object", + "properties": { + "action": { + "type": "string" + }, + "execution_id": { + "type": "string" + }, + "task_id": { + "type": "string" + } + } + }, + "pages.SourceRevisionView": { + "type": "object", + "properties": { + "asset_name": { + "type": "string" + }, + "label": { + "type": "string" + }, + "revision": { + "type": "string" + } + } + }, + "pages.SourceSyncInput": { + "type": "object", + "properties": { + "confirmed_revision": { + "type": "string" + } + } + }, + "pages.SourceUpdateInput": { + "type": "object", + "properties": { + "asset_name": { + "type": "string" + }, + "auto_update_enabled": { + "type": "boolean" + }, + "check_interval_minutes": { + "type": "integer" + }, + "release_selector": { + "type": "string" + }, + "release_tag": { + "type": "string" + }, + "remote_network_policy": { + "type": "string" + }, + "remote_url": { + "type": "string" + }, + "remote_url_set": { + "type": "boolean" + }, + "repository_url": { + "type": "string" + }, + "source_type": { + "type": "string" + } + } + }, + "pages.SourceUpdateResult": { + "type": "object", + "properties": { + "check_task": { + "$ref": "#/definitions/pages.SourceActionReceipt" + }, + "source": { + "$ref": "#/definitions/pages.SourceView" + }, + "warning": { + "type": "string" + } + } + }, + "pages.SourceView": { + "type": "object", + "properties": { + "asset_name": { + "type": "string" + }, + "auto_update_enabled": { + "type": "boolean" + }, + "check_interval_minutes": { + "type": "integer" + }, + "display_url": { + "type": "string" + }, + "github_repository": { + "type": "string" + }, + "has_remote_url": { + "type": "boolean" + }, + "last_applied": { + "$ref": "#/definitions/pages.SourceRevisionView" + }, + "last_checked_at": { + "type": "string" + }, + "last_error": { + "type": "string" + }, + "last_seen": { + "$ref": "#/definitions/pages.SourceRevisionView" + }, + "last_synced_at": { + "type": "string" + }, + "next_check_at": { + "type": "string" + }, + "release_selector": { + "type": "string" + }, + "release_tag": { + "type": "string" + }, + "remote_network_policy": { + "type": "string" + }, + "source_type": { + "type": "string" + }, + "sync_status": { + "type": "string" + }, + "update_available": { + "type": "boolean" + } + } + }, "pages.UploadFromURLInput": { "type": "object", "properties": { diff --git a/docs/swagger.yaml b/docs/swagger.yaml index a52e0600..b0e074d9 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -2571,10 +2571,16 @@ definitions: type: integer project_id: type: integer + source_label: + type: string + source_type: + type: string status: type: string total_size: type: integer + trigger_type: + type: string upload_id: example: "0" type: string @@ -2606,6 +2612,100 @@ definitions: spa_fallback_path: type: string type: object + pages.SourceActionReceipt: + properties: + action: + type: string + execution_id: + type: string + task_id: + type: string + type: object + pages.SourceRevisionView: + properties: + asset_name: + type: string + label: + type: string + revision: + type: string + type: object + pages.SourceSyncInput: + properties: + confirmed_revision: + type: string + type: object + pages.SourceUpdateInput: + properties: + asset_name: + type: string + auto_update_enabled: + type: boolean + check_interval_minutes: + type: integer + release_selector: + type: string + release_tag: + type: string + remote_network_policy: + type: string + remote_url: + type: string + remote_url_set: + type: boolean + repository_url: + type: string + source_type: + type: string + type: object + pages.SourceUpdateResult: + properties: + check_task: + $ref: '#/definitions/pages.SourceActionReceipt' + source: + $ref: '#/definitions/pages.SourceView' + warning: + type: string + type: object + pages.SourceView: + properties: + asset_name: + type: string + auto_update_enabled: + type: boolean + check_interval_minutes: + type: integer + display_url: + type: string + github_repository: + type: string + has_remote_url: + type: boolean + last_applied: + $ref: '#/definitions/pages.SourceRevisionView' + last_checked_at: + type: string + last_error: + type: string + last_seen: + $ref: '#/definitions/pages.SourceRevisionView' + last_synced_at: + type: string + next_check_at: + type: string + release_selector: + type: string + release_tag: + type: string + remote_network_policy: + type: string + source_type: + type: string + sync_status: + type: string + update_available: + type: boolean + type: object pages.UploadFromURLInput: properties: url: @@ -9277,7 +9377,8 @@ paths: post: consumes: - application/json - description: 从用户提供的 HTTP(S) 链接下载部署包并创建部署记录；服务端使用浏览器伪装请求头拉取，允许内网地址与不安全 TLS 证书，需要管理员权限 + deprecated: true + description: 已弃用的一次性 URL 导入；使用 trusted_internal 策略兼容内网与自签名证书，不创建持久部署源 parameters: - description: 项目 ID in: path @@ -9323,6 +9424,239 @@ paths: summary: 从 URL 导入 Pages 部署包 tags: - openflare-pages + /api/v1/d/pages/{id}/source: + get: + description: 返回脱敏后的项目部署源配置与运行状态，需要管理员权限 + parameters: + - description: 项目 ID + in: path + name: id + required: true + type: integer + produces: + - application/json + responses: + "200": + description: 部署源 + schema: + allOf: + - $ref: '#/definitions/response.Any' + - properties: + data: + $ref: '#/definitions/pages.SourceView' + type: object + "400": + description: 参数错误 + schema: + $ref: '#/definitions/response.Any' + "401": + description: 未登录 + schema: + $ref: '#/definitions/response.Any' + "404": + description: 项目或部署源不存在 + schema: + $ref: '#/definitions/response.Any' + "500": + description: 内部错误 + schema: + $ref: '#/definitions/response.Any' + security: + - SessionCookie: [] + summary: 获取 Pages 部署源 + tags: + - openflare-pages + /api/v1/d/pages/{id}/source/check: + post: + description: Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约 + parameters: + - description: 项目 ID + in: path + name: id + required: true + type: integer + produces: + - application/json + responses: + "200": + description: 任务回执 + schema: + allOf: + - $ref: '#/definitions/response.Any' + - properties: + data: + $ref: '#/definitions/pages.SourceActionReceipt' + type: object + "400": + description: 当前来源不支持检查 + schema: + $ref: '#/definitions/response.Any' + "401": + description: 未登录 + schema: + $ref: '#/definitions/response.Any' + "404": + description: 部署源不存在 + schema: + $ref: '#/definitions/response.Any' + "409": + description: 来源任务正在执行 + schema: + $ref: '#/definitions/response.Any' + "500": + description: 内部错误 + schema: + $ref: '#/definitions/response.Any' + security: + - SessionCookie: [] + summary: 检查 Pages 部署源 + tags: + - openflare-pages + /api/v1/d/pages/{id}/source/delete: + post: + description: 幂等删除持久部署源；已有部署历史与当前生产部署保持不变 + parameters: + - description: 项目 ID + in: path + name: id + required: true + type: integer + produces: + - application/json + responses: + "200": + description: 手动来源视图 + schema: + allOf: + - $ref: '#/definitions/response.Any' + - properties: + data: + $ref: '#/definitions/pages.SourceView' + type: object + "400": + description: 参数错误 + schema: + $ref: '#/definitions/response.Any' + "401": + description: 未登录 + schema: + $ref: '#/definitions/response.Any' + "404": + description: 项目不存在 + schema: + $ref: '#/definitions/response.Any' + "500": + description: 内部错误 + schema: + $ref: '#/definitions/response.Any' + security: + - SessionCookie: [] + summary: 删除 Pages 部署源 + tags: + - openflare-pages + /api/v1/d/pages/{id}/source/sync: + post: + consumes: + - application/json + description: 异步下载、校验并原子激活来源部署包；空请求体与空 JSON 对象均有效 + parameters: + - description: 项目 ID + in: path + name: id + required: true + type: integer + - description: 同步参数 + in: body + name: request + schema: + $ref: '#/definitions/pages.SourceSyncInput' + produces: + - application/json + responses: + "200": + description: 任务回执 + schema: + allOf: + - $ref: '#/definitions/response.Any' + - properties: + data: + $ref: '#/definitions/pages.SourceActionReceipt' + type: object + "400": + description: 参数或来源类型无效 + schema: + $ref: '#/definitions/response.Any' + "401": + description: 未登录 + schema: + $ref: '#/definitions/response.Any' + "404": + description: 部署源不存在 + schema: + $ref: '#/definitions/response.Any' + "409": + description: 来源任务正在执行 + schema: + $ref: '#/definitions/response.Any' + "500": + description: 内部错误 + schema: + $ref: '#/definitions/response.Any' + security: + - SessionCookie: [] + summary: 同步并发布 Pages 部署源 + tags: + - openflare-pages + /api/v1/d/pages/{id}/source/update: + post: + consumes: + - application/json + description: Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回 + parameters: + - description: 项目 ID + in: path + name: id + required: true + type: integer + - description: 部署源配置 + in: body + name: request + required: true + schema: + $ref: '#/definitions/pages.SourceUpdateInput' + produces: + - application/json + responses: + "200": + description: 更新结果 + schema: + allOf: + - $ref: '#/definitions/response.Any' + - properties: + data: + $ref: '#/definitions/pages.SourceUpdateResult' + type: object + "400": + description: 配置无效 + schema: + $ref: '#/definitions/response.Any' + "401": + description: 未登录 + schema: + $ref: '#/definitions/response.Any' + "404": + description: 项目不存在 + schema: + $ref: '#/definitions/response.Any' + "500": + description: 内部错误 + schema: + $ref: '#/definitions/response.Any' + security: + - SessionCookie: [] + summary: 更新 Pages 部署源 + tags: + - openflare-pages /api/v1/d/pages/{id}/update: post: consumes: diff --git a/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx b/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx index d47b0e83..29c17a94 100644 --- a/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx +++ b/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx @@ -2,7 +2,7 @@ import { useRef, useState } from 'react'; import { useMutation, useQueryClient } from '@tanstack/react-query'; -import { Link2, Loader2, UploadCloud } from 'lucide-react'; +import { UploadCloud } from 'lucide-react'; import { toast } from 'sonner'; import { Button } from '@/components/ui/button'; @@ -14,10 +14,14 @@ import { DialogHeader, DialogTitle, } from '@/components/ui/dialog'; -import { Input } from '@/components/ui/input'; -import { Label } from '@/components/ui/label'; +import { + Field, + FieldDescription, + FieldGroup, + FieldLabel, +} from '@/components/ui/field'; import { Progress } from '@/components/ui/progress'; -import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs'; +import { Spinner } from '@/components/ui/spinner'; import { PagesService } from '@/lib/services/openflare'; import { cn } from '@/lib/utils'; @@ -46,52 +50,51 @@ const PAGES_PACKAGE_EXTENSIONS = [ function isSupportedPagesPackage(fileName: string) { const lower = fileName.toLowerCase(); - return PAGES_PACKAGE_EXTENSIONS.some((ext) => lower.endsWith(ext)); + return PAGES_PACKAGE_EXTENSIONS.some((extension) => + lower.endsWith(extension), + ); +} + +export function pagesEntryPath(rootDir: string, entryFile: string) { + const root = rootDir.trim().replace(/^\/+|\/+$/g, ''); + const entry = entryFile.trim().replace(/^\/+/, ''); + return root ? `${root}/${entry}` : entry; } interface DeploymentUploadDialogProps { open: boolean; onOpenChange: (open: boolean) => void; projectId: number; + rootDir: string; + entryFile: string; } export function DeploymentUploadDialog({ open, onOpenChange, projectId, + rootDir, + entryFile, }: DeploymentUploadDialogProps) { const queryClient = useQueryClient(); const fileInputRef = useRef(null); - const [mode, setMode] = useState<'file' | 'url'>('file'); const [file, setFile] = useState(null); - const [packageURL, setPackageURL] = useState(''); const [isDragActive, setIsDragActive] = useState(false); const [uploadProgress, setUploadProgress] = useState(null); const resetForm = () => { setFile(null); - setPackageURL(''); setIsDragActive(false); setUploadProgress(null); if (fileInputRef.current) fileInputRef.current.value = ''; }; - const handleClose = (nextOpen: boolean) => { + const handleOpenChange = (nextOpen: boolean) => { if (!nextOpen) resetForm(); onOpenChange(nextOpen); }; - const invalidate = async () => { - await Promise.all([ - queryClient.invalidateQueries({ - queryKey: deploymentsQueryKey(projectId), - }), - queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), - queryClient.invalidateQueries({ queryKey: projectsQueryKey }), - ]); - }; - - const uploadFileMutation = useMutation({ + const uploadMutation = useMutation({ mutationFn: () => { if (!file) throw new Error('请选择部署包'); return PagesService.uploadDeployment(projectId, { @@ -101,8 +104,14 @@ export function DeploymentUploadDialog({ }, onSuccess: async () => { toast.success('部署包上传成功'); - await invalidate(); - handleClose(false); + await Promise.all([ + queryClient.invalidateQueries({ + queryKey: deploymentsQueryKey(projectId), + }), + queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), + queryClient.invalidateQueries({ queryKey: projectsQueryKey }), + ]); + handleOpenChange(false); }, onError: (error) => { toast.error(error instanceof Error ? error.message : '上传失败'); @@ -110,27 +119,6 @@ export function DeploymentUploadDialog({ }, }); - const uploadURLMutation = useMutation({ - mutationFn: () => { - const url = packageURL.trim(); - if (!url) throw new Error('请填写部署包下载链接'); - if (!/^https?:\/\//i.test(url)) { - throw new Error('链接必须以 http:// 或 https:// 开头'); - } - return PagesService.uploadDeploymentFromURL(projectId, { url }); - }, - onSuccess: async () => { - toast.success('已从链接下载并创建部署'); - await invalidate(); - handleClose(false); - }, - onError: (error) => { - toast.error(error instanceof Error ? error.message : '从链接导入失败'); - }, - }); - - const isPending = uploadFileMutation.isPending || uploadURLMutation.isPending; - const handleFileSelect = (selected: File | null) => { if (!selected) return; if (!isSupportedPagesPackage(selected.name)) { @@ -140,150 +128,112 @@ export function DeploymentUploadDialog({ setFile(selected); }; - const handleSubmit = () => { - if (mode === 'file') { - uploadFileMutation.mutate(); - return; - } - uploadURLMutation.mutate(); - }; - - const canSubmit = - mode === 'file' ? Boolean(file) : packageURL.trim().length > 0; - return ( - + 上传部署包 - 支持本地上传或从 URL 下载静态资源压缩包（zip / tar.gz / tar.xz / - tar.bz2 / tar / 7z），创建部署后可在列表中激活。 + 上传不可变的静态资源压缩包，完成后可在部署历史中激活。 - setMode(value as 'file' | 'url')} - className='w-full' - > - - 本地上传 - 从 URL 下载 - - - -
+ + 本地部署包 + - handleFileSelect(e.target.files?.[0] ?? null)} - /> -
- + + + 拖拽部署包到此处，或点击选择文件 + + + zip、tar.gz、tar.xz、tar.bz2、tar、7z + + + + handleFileSelect(event.target.files?.[0] ?? null) + } + /> {file ? ( -
-

{file.name}

-

- {formatBytes(file.size)} -

-
- ) : null} + + 已选择 {file.name}（{formatBytes(file.size)}） + + ) : ( + 请选择一个受支持的压缩包。 + )} + - {uploadProgress !== null ? ( -
-
- - {uploadProgress >= 100 ? '服务端处理中' : '上传进度'} - - - {uploadProgress >= 100 ? '请稍候' : `${uploadProgress}%`} - -
- = 100 ? 100 : uploadProgress} - /> -
- ) : null} - - - -
- -
- - setPackageURL(e.target.value)} - disabled={isPending} - /> -
-

- 服务端将使用浏览器环境请求头从该链接下载压缩包，支持内网地址与自签证书 - HTTPS。 -

+ + 部署入口 +
+ {pagesEntryPath(rootDir, entryFile)}
- - + + 入口来自项目设置；部署包上传不会覆盖该配置。 + + -
- - -
+ {uploadProgress !== null ? ( + +
+ + {uploadProgress >= 100 ? '服务端处理中' : '上传进度'} + + + {uploadProgress >= 100 ? '请稍候' : `${uploadProgress}%`} + +
+ + + ) : null} + - - diff --git a/frontend/app/(main)/pages/components/pages-utils.ts b/frontend/app/(main)/pages/components/pages-utils.ts index 50581b84..f43b3fac 100644 --- a/frontend/app/(main)/pages/components/pages-utils.ts +++ b/frontend/app/(main)/pages/components/pages-utils.ts @@ -17,6 +17,10 @@ export function deploymentsQueryKey(projectId: number) { return ['openflare', 'pages', 'deployments', projectId] as const; } +export function sourceQueryKey(projectId: number) { + return ['openflare', 'pages', 'source', projectId] as const; +} + export function deploymentFilesQueryKey( projectId: number, deploymentId: number, diff --git a/frontend/app/(main)/pages/detail/components/deployment-files-panel.tsx b/frontend/app/(main)/pages/detail/components/deployment-files-panel.tsx new file mode 100644 index 00000000..7b0b4ad9 --- /dev/null +++ b/frontend/app/(main)/pages/detail/components/deployment-files-panel.tsx @@ -0,0 +1,86 @@ +'use client'; + +import { useQuery } from '@tanstack/react-query'; + +import { EmptyInline } from '@/components/layout/empty'; +import { ErrorInline } from '@/components/layout/error'; +import { Skeleton } from '@/components/ui/skeleton'; +import { + Table, + TableBody, + TableCell, + TableHead, + TableHeader, + TableRow, +} from '@/components/ui/table'; +import { PagesService } from '@/lib/services/openflare'; + +import { + deploymentFilesQueryKey, + formatBytes, +} from '../../components/pages-utils'; + +interface DeploymentFilesPanelProps { + projectId: number; + deploymentId: number; +} + +export function DeploymentFilesPanel({ + projectId, + deploymentId, +}: DeploymentFilesPanelProps) { + const filesQuery = useQuery({ + queryKey: deploymentFilesQueryKey(projectId, deploymentId), + queryFn: () => PagesService.listDeploymentFiles(deploymentId), + }); + + if (filesQuery.isLoading) { + return ( +
+ + +
+ ); + } + + if (filesQuery.isError) { + return ( +
+ void filesQuery.refetch()} + /> +
+ ); + } + + const files = filesQuery.data ?? []; + if (files.length === 0) { + return ; + } + + return ( + + + + 路径 + 大小 + + + + {files.map((file) => ( + + {file.path} + + {formatBytes(file.size)} + + + ))} + +
+ ); +} diff --git a/frontend/app/(main)/pages/detail/components/deployment-history.tsx b/frontend/app/(main)/pages/detail/components/deployment-history.tsx new file mode 100644 index 00000000..bd046500 --- /dev/null +++ b/frontend/app/(main)/pages/detail/components/deployment-history.tsx @@ -0,0 +1,300 @@ +'use client'; + +import { useMemo, useState } from 'react'; +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { ChevronDown, ChevronRight } from 'lucide-react'; +import { toast } from 'sonner'; + +import { EmptyStateWithBorder } from '@/components/layout/empty'; +import { ErrorInline } from '@/components/layout/error'; +import { LoadingStateWithBorder } from '@/components/layout/loading'; +import { + AlertDialog, + AlertDialogAction, + AlertDialogCancel, + AlertDialogContent, + AlertDialogDescription, + AlertDialogFooter, + AlertDialogHeader, + AlertDialogTitle, +} from '@/components/ui/alert-dialog'; +import { Badge } from '@/components/ui/badge'; +import { Button } from '@/components/ui/button'; +import { + Card, + CardContent, + CardDescription, + CardHeader, + CardTitle, +} from '@/components/ui/card'; +import { Spinner } from '@/components/ui/spinner'; +import { type PagesDeployment, PagesService } from '@/lib/services/openflare'; +import { formatDateTime } from '@/lib/utils'; + +import { + deploymentFilesQueryKey, + deploymentsQueryKey, + formatBytes, + projectQueryKey, + projectsQueryKey, + sourceQueryKey, +} from '../../components/pages-utils'; +import { DeploymentFilesPanel } from './deployment-files-panel'; + +const SOURCE_LABELS: Record = { + manual_upload: '本地上传', + manual_url: 'URL 导入', + remote_url: 'Remote URL', + github_release: 'GitHub Release', +}; + +const TRIGGER_LABELS: Record = { + manual_upload: '手动上传', + manual_url: '手动导入', + manual_sync: '手动同步', + scheduled_auto_update: '定时更新', +}; + +interface DeploymentHistoryProps { + projectId: number; + activeDeploymentId?: number | null; +} + +type PendingAction = { + type: 'activate' | 'delete'; + deployment: PagesDeployment; +}; + +function deploymentSnapshot(deployment: PagesDeployment) { + return [ + SOURCE_LABELS[deployment.source_type], + deployment.source_label, + TRIGGER_LABELS[deployment.trigger_type], + ] + .filter(Boolean) + .join(' · '); +} + +export function DeploymentHistory({ + projectId, + activeDeploymentId, +}: DeploymentHistoryProps) { + const queryClient = useQueryClient(); + const [expandedDeploymentId, setExpandedDeploymentId] = useState< + number | null + >(null); + const [pendingAction, setPendingAction] = useState( + null, + ); + + const deploymentsQuery = useQuery({ + queryKey: deploymentsQueryKey(projectId), + queryFn: () => PagesService.listDeployments(projectId), + }); + + const deployments = useMemo(() => { + const records = [...(deploymentsQuery.data ?? [])]; + return records.sort((left, right) => { + const leftActive = + left.id === activeDeploymentId || left.status === 'active'; + const rightActive = + right.id === activeDeploymentId || right.status === 'active'; + if (leftActive !== rightActive) return leftActive ? -1 : 1; + return right.deployment_number - left.deployment_number; + }); + }, [activeDeploymentId, deploymentsQuery.data]); + + const invalidateDeploymentState = async () => { + await Promise.all([ + queryClient.invalidateQueries({ + queryKey: deploymentsQueryKey(projectId), + }), + queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), + queryClient.invalidateQueries({ queryKey: sourceQueryKey(projectId) }), + queryClient.invalidateQueries({ queryKey: projectsQueryKey }), + ]); + }; + + const activateMutation = useMutation({ + mutationFn: (deploymentId: number) => + PagesService.activateDeployment(projectId, deploymentId), + onSuccess: async () => { + toast.success('部署已激活'); + await invalidateDeploymentState(); + setPendingAction(null); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '激活失败'); + }, + }); + + const deleteMutation = useMutation({ + mutationFn: (deploymentId: number) => + PagesService.deleteDeployment(projectId, deploymentId), + onSuccess: async (_, deploymentId) => { + toast.success('部署已删除'); + queryClient.removeQueries({ + queryKey: deploymentFilesQueryKey(projectId, deploymentId), + }); + await invalidateDeploymentState(); + setPendingAction(null); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '删除失败'); + }, + }); + + const actionPending = activateMutation.isPending || deleteMutation.isPending; + + return ( + <> + + + 部署历史 + + 部署记录不可变，来源信息是创建部署时的安全快照。 + + + + {deploymentsQuery.isLoading ? ( + + ) : deploymentsQuery.isError ? ( +
+ void deploymentsQuery.refetch()} + /> +
+ ) : deployments.length === 0 ? ( + + ) : ( + deployments.map((deployment) => { + const active = + deployment.id === activeDeploymentId || + deployment.status === 'active'; + const expanded = expandedDeploymentId === deployment.id; + + return ( +
+
+
+ +
+
+ + 部署 #{deployment.deployment_number} + + + {active ? '当前生产部署' : '历史部署'} + + + {deploymentSnapshot(deployment)} + +
+

+ {deployment.checksum.slice(0, 16)} ·{' '} + {deployment.file_count} 个文件 ·{' '} + {formatBytes(deployment.total_size)} +

+

+ 创建于 {formatDateTime(deployment.created_at)} +

+
+
+
+ + +
+
+ {expanded ? ( + + ) : null} +
+ ); + }) + )} + + + + { + if (!open && !actionPending) setPendingAction(null); + }} + > + + + + {pendingAction?.type === 'activate' ? '激活历史部署' : '删除部署'} + + + {pendingAction?.type === 'activate' + ? '激活其它历史部署会终止当前来源任务；若已开启自动更新，将同时关闭自动更新。' + : `确认删除部署 #${pendingAction?.deployment.deployment_number} 吗？此操作不可恢复。`} + + + + 取消 + { + event.preventDefault(); + if (!pendingAction) return; + if (pendingAction.type === 'activate') { + activateMutation.mutate(pendingAction.deployment.id); + } else { + deleteMutation.mutate(pendingAction.deployment.id); + } + }} + > + {actionPending ? : null} + 确认 + + + + + + ); +} diff --git a/frontend/app/(main)/pages/detail/components/pages-source-card.tsx b/frontend/app/(main)/pages/detail/components/pages-source-card.tsx new file mode 100644 index 00000000..eced9522 --- /dev/null +++ b/frontend/app/(main)/pages/detail/components/pages-source-card.tsx @@ -0,0 +1,393 @@ +'use client'; + +import { useEffect, useRef, useState } from 'react'; +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { Download, Pencil, RefreshCw, RotateCcw } from 'lucide-react'; +import { toast } from 'sonner'; + +import { ErrorInline } from '@/components/layout/error'; +import { Badge } from '@/components/ui/badge'; +import { Button } from '@/components/ui/button'; +import { + Card, + CardAction, + CardContent, + CardDescription, + CardFooter, + CardHeader, + CardTitle, +} from '@/components/ui/card'; +import { Skeleton } from '@/components/ui/skeleton'; +import { Spinner } from '@/components/ui/spinner'; +import { AdminTaskService } from '@/lib/services/admin'; +import { + type PagesSource, + type PagesSourceActionReceipt, + type PagesSourceStatus, + PagesService, +} from '@/lib/services/openflare'; +import { formatDateTime } from '@/lib/utils'; + +import { + deploymentsQueryKey, + projectQueryKey, + projectsQueryKey, + sourceQueryKey, +} from '../../components/pages-utils'; +import { PagesSourceDialog } from './pages-source-dialog'; + +const ACTION_POLL_INTERVAL = 2_000; +const ACTION_MAX_WAIT = 16 * 60 * 1_000; + +const SOURCE_STATUS: Record< + PagesSourceStatus, + { + label: string; + variant: 'default' | 'secondary' | 'destructive' | 'outline'; + } +> = { + idle: { label: '空闲', variant: 'outline' }, + checking: { label: '检查中', variant: 'secondary' }, + update_available: { label: '有可用更新', variant: 'default' }, + syncing: { label: '同步中', variant: 'secondary' }, + failed: { label: '最近同步失败', variant: 'destructive' }, + attention: { label: '需要确认', variant: 'destructive' }, +}; + +interface ActiveAction { + receipt: PagesSourceActionReceipt; + startedAt: number; +} + +function revisionSummary(source: PagesSource) { + if (source.source_type === 'manual' || !source.last_applied) + return '尚未应用'; + return `${source.last_applied.label} · ${source.last_applied.revision.slice(0, 12)}`; +} + +export function PagesSourceCard({ projectId }: { projectId: number }) { + const queryClient = useQueryClient(); + const handledExecutionID = useRef(null); + const sourcePollingStartedAt = useRef(null); + const [dialogOpen, setDialogOpen] = useState(false); + const [dialogMode, setDialogMode] = useState<'manual' | 'remote_url'>( + 'manual', + ); + const [activeAction, setActiveAction] = useState(null); + const [actionTimedOut, setActionTimedOut] = useState(false); + + const sourceQuery = useQuery({ + queryKey: sourceQueryKey(projectId), + queryFn: () => PagesService.getSource(projectId), + refetchInterval: (query) => { + const source = query.state.data; + if ( + source && + source.source_type !== 'manual' && + (source.sync_status === 'checking' || source.sync_status === 'syncing') + ) { + sourcePollingStartedAt.current ??= Date.now(); + return Date.now() - sourcePollingStartedAt.current < ACTION_MAX_WAIT + ? ACTION_POLL_INTERVAL + : false; + } + sourcePollingStartedAt.current = null; + return false; + }, + }); + + const executionQuery = useQuery({ + queryKey: [ + 'admin', + 'task-execution', + activeAction?.receipt.execution_id ?? '', + ], + queryFn: () => + AdminTaskService.getTaskExecution(activeAction!.receipt.execution_id), + enabled: Boolean(activeAction) && !actionTimedOut, + refetchInterval: (query) => { + if (actionTimedOut) return false; + const status = query.state.data?.status; + return status === 'pending' || status === 'running' + ? ACTION_POLL_INTERVAL + : false; + }, + }); + + useEffect(() => { + if (!activeAction || actionTimedOut) return; + const elapsed = Date.now() - activeAction.startedAt; + const remaining = Math.max(0, ACTION_MAX_WAIT - elapsed); + const timeout = window.setTimeout(() => setActionTimedOut(true), remaining); + return () => window.clearTimeout(timeout); + }, [actionTimedOut, activeAction]); + + useEffect(() => { + const execution = executionQuery.data; + if (!execution || !['succeeded', 'failed'].includes(execution.status)) { + return; + } + if (handledExecutionID.current === execution.id) return; + handledExecutionID.current = execution.id; + + void Promise.all([ + queryClient.invalidateQueries({ queryKey: sourceQueryKey(projectId) }), + queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), + queryClient.invalidateQueries({ + queryKey: deploymentsQueryKey(projectId), + }), + queryClient.invalidateQueries({ + queryKey: ['openflare', 'pages', 'deployment-files', projectId], + }), + queryClient.invalidateQueries({ queryKey: projectsQueryKey }), + ]); + + if (execution.status === 'succeeded') { + toast.success('部署源同步并发布完成'); + } else { + toast.error(execution.error_message || '部署源同步失败'); + } + setActiveAction(null); + setActionTimedOut(false); + }, [executionQuery.data, projectId, queryClient]); + + const syncMutation = useMutation({ + mutationFn: () => PagesService.syncSource(projectId, {}), + onSuccess: async (receipt) => { + handledExecutionID.current = null; + setActiveAction({ receipt, startedAt: Date.now() }); + setActionTimedOut(false); + await queryClient.invalidateQueries({ + queryKey: sourceQueryKey(projectId), + }); + toast.success('同步任务已提交'); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '同步任务提交失败'); + }, + }); + + const source = sourceQuery.data; + const executionBusy = + activeAction !== null && + (executionQuery.data?.status === undefined || + executionQuery.data.status === 'pending' || + executionQuery.data.status === 'running'); + const sourceBusy = + source?.source_type !== 'manual' && + (source?.sync_status === 'checking' || source?.sync_status === 'syncing'); + const actionsDisabled = syncMutation.isPending || executionBusy || sourceBusy; + + const openSourceDialog = (mode: 'manual' | 'remote_url') => { + setDialogMode(mode); + setDialogOpen(true); + }; + + if (sourceQuery.isLoading) { + return ( + + + 部署源 + 加载来源配置... + + + + + + + ); + } + + if (sourceQuery.isError || !source) { + return ( + + + 部署源 + 来源配置与部署历史相互独立。 + + + void sourceQuery.refetch()} + /> + + + ); + } + + const status = + source.source_type === 'manual' + ? null + : SOURCE_STATUS[source.sync_status ?? 'idle']; + + return ( + <> + + + 部署源 + + 来源配置负责发现内容，发布结果记录在独立的部署历史中。 + + + {status ? ( + {status.label} + ) : ( + 手动部署 + )} + + + + + {source.source_type === 'manual' ? ( +
+

本地部署包

+

+ 当前没有持久化远端来源。上传部署包后，再从部署历史显式激活。 +

+
+ ) : source.source_type === 'remote_url' ? ( +
+
+ 脱敏地址 + {source.display_url} +
+
+ 网络策略 + + {source.remote_network_policy === 'trusted_internal' + ? '受信内网模式' + : '公网安全模式'} + +
+
+ 最近同步 + + {source.last_synced_at + ? formatDateTime(source.last_synced_at) + : '尚未同步'} + +
+
+ + 已应用 revision + + + {revisionSummary(source)} + +
+ {source.last_error ? ( +
+ +
+ ) : null} +
+ ) : ( +
+ 当前版本暂不提供该来源类型的编辑界面。 +
+ )} + + {executionQuery.isError ? ( + void executionQuery.refetch()} + /> + ) : null} + {actionTimedOut ? ( +
+ + 自动等待已停止，任务可能仍在后台运行。 + + +
+ ) : null} + + + + {source.source_type === 'manual' ? ( + + ) : source.source_type === 'remote_url' ? ( + <> + + + + + ) : null} + + + + + + + ); +} diff --git a/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx b/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx new file mode 100644 index 00000000..c3bdb0a1 --- /dev/null +++ b/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx @@ -0,0 +1,376 @@ +'use client'; + +import { useEffect, useState } from 'react'; +import { useMutation, useQueryClient } from '@tanstack/react-query'; +import { toast } from 'sonner'; + +import { + AlertDialog, + AlertDialogAction, + AlertDialogCancel, + AlertDialogContent, + AlertDialogDescription, + AlertDialogFooter, + AlertDialogHeader, + AlertDialogTitle, +} from '@/components/ui/alert-dialog'; +import { Button } from '@/components/ui/button'; +import { + Dialog, + DialogContent, + DialogDescription, + DialogFooter, + DialogHeader, + DialogTitle, +} from '@/components/ui/dialog'; +import { + Field, + FieldDescription, + FieldGroup, + FieldLabel, + FieldTitle, +} from '@/components/ui/field'; +import { Input } from '@/components/ui/input'; +import { Spinner } from '@/components/ui/spinner'; +import { ToggleGroup, ToggleGroupItem } from '@/components/ui/toggle-group'; +import { + type PagesRemoteNetworkPolicy, + type PagesSource, + PagesService, +} from '@/lib/services/openflare'; + +import { + deploymentsQueryKey, + projectQueryKey, + projectsQueryKey, + sourceQueryKey, +} from '../../components/pages-utils'; + +type SourceMode = 'manual' | 'remote_url'; +type Confirmation = 'trusted_internal' | 'manual' | null; + +interface PagesSourceDialogProps { + open: boolean; + onOpenChange: (open: boolean) => void; + projectId: number; + source: PagesSource; + initialMode?: SourceMode; +} + +export function PagesSourceDialog({ + open, + onOpenChange, + projectId, + source, + initialMode, +}: PagesSourceDialogProps) { + const queryClient = useQueryClient(); + const [mode, setMode] = useState('manual'); + const [networkPolicy, setNetworkPolicy] = + useState('public'); + const [replaceURL, setReplaceURL] = useState(false); + const [remoteURL, setRemoteURL] = useState(''); + const [urlError, setURLError] = useState(''); + const [confirmation, setConfirmation] = useState(null); + + useEffect(() => { + if (!open) return; + const nextMode = + initialMode ?? + (source.source_type === 'remote_url' ? 'remote_url' : 'manual'); + setMode(nextMode); + setNetworkPolicy( + source.source_type === 'remote_url' + ? source.remote_network_policy + : 'public', + ); + setReplaceURL(source.source_type !== 'remote_url'); + setRemoteURL(''); + setURLError(''); + setConfirmation(null); + }, [initialMode, open, source]); + + const invalidateSourceState = async () => { + await Promise.all([ + queryClient.invalidateQueries({ queryKey: sourceQueryKey(projectId) }), + queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), + queryClient.invalidateQueries({ + queryKey: deploymentsQueryKey(projectId), + }), + queryClient.invalidateQueries({ queryKey: projectsQueryKey }), + ]); + }; + + const updateMutation = useMutation({ + mutationFn: () => + PagesService.updateSource(projectId, { + source_type: 'remote_url', + remote_url_set: replaceURL, + remote_url: replaceURL ? remoteURL.trim() : '', + remote_network_policy: networkPolicy, + }), + onSuccess: async (result) => { + queryClient.setQueryData(sourceQueryKey(projectId), result.source); + await invalidateSourceState(); + toast.success('部署源已更新'); + if (result.warning) toast.warning(result.warning); + setConfirmation(null); + onOpenChange(false); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '部署源更新失败'); + }, + }); + + const deleteMutation = useMutation({ + mutationFn: () => PagesService.deleteSource(projectId), + onSuccess: async (manualSource) => { + queryClient.setQueryData(sourceQueryKey(projectId), manualSource); + await invalidateSourceState(); + toast.success('已切换回手动部署'); + setConfirmation(null); + onOpenChange(false); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '切换失败'); + }, + }); + + const isPending = updateMutation.isPending || deleteMutation.isPending; + + const submitRemote = () => { + if (replaceURL) { + const value = remoteURL.trim(); + if (!value) { + setURLError('请输入 Remote URL'); + return; + } + try { + const parsed = new URL(value); + if (!['http:', 'https:'].includes(parsed.protocol)) throw new Error(); + } catch { + setURLError('请输入有效的 HTTP(S) URL'); + return; + } + } + setURLError(''); + if (networkPolicy === 'trusted_internal') { + setConfirmation('trusted_internal'); + return; + } + updateMutation.mutate(); + }; + + const handleSubmit = () => { + if (mode === 'manual') { + if (source.source_type === 'manual') { + onOpenChange(false); + } else { + setConfirmation('manual'); + } + return; + } + submitRemote(); + }; + + return ( + <> + { + if (!isPending) onOpenChange(nextOpen); + }} + > + + + 部署源设置 + + 手动部署与 Remote URL + 各自保持独立配置；后续仓库构建来源会作为新的来源类型接入。 + + + + + + 来源类型 + { + if (value === 'manual' || value === 'remote_url') { + setMode(value); + } + }} + > + + 手动部署 + + + Remote URL + + + + 手动部署由管理员上传本地包；Remote URL 通过显式同步下载并发布。 + + + + {mode === 'manual' ? ( + + 手动部署 +
+ 保留现有部署与当前生产版本，后续通过“上传部署包”创建新部署。 +
+ + ) : ( + <> + + Remote URL + {source.source_type === 'remote_url' && !replaceURL ? ( +
+ + {source.display_url} + + +
+ ) : ( + { + setRemoteURL(event.target.value); + setURLError(''); + }} + /> + )} + + {urlError || + (replaceURL + ? '保存后不会回显原始地址或 query token。' + : '界面只显示脱敏地址；留空表示保留当前地址。')} + + {source.source_type === 'remote_url' && replaceURL ? ( + + ) : null} + + + + 网络策略 + { + if (value === 'public' || value === 'trusted_internal') { + setNetworkPolicy(value); + } + }} + > + + 公网安全模式 + + + 受信内网模式 + + + + {networkPolicy === 'public' + ? '阻止内网地址、代理与不安全 TLS。' + : '允许访问内网地址与自签名证书，仅用于可信来源。'} + + + + )} + + + + + + + + + + { + if (!nextOpen && !isPending) setConfirmation(null); + }} + > + + + + {confirmation === 'manual' + ? '切换回手动部署' + : '启用受信内网模式'} + + + {confirmation === 'manual' + ? '当前来源配置将被删除，但已有部署与当前生产版本会保留。' + : '该模式允许访问私有网络并接受自签名证书。请确认此地址属于可信内部来源，且不会被非可信用户控制。'} + + + + 取消 + { + event.preventDefault(); + if (confirmation === 'manual') { + deleteMutation.mutate(); + } else if (confirmation === 'trusted_internal') { + updateMutation.mutate(); + } + }} + > + {isPending ? : null} + 确认 + + + + + + ); +} diff --git a/frontend/app/(main)/pages/detail/page-client.tsx b/frontend/app/(main)/pages/detail/page-client.tsx deleted file mode 100644 index a1115b68..00000000 --- a/frontend/app/(main)/pages/detail/page-client.tsx +++ /dev/null @@ -1,505 +0,0 @@ -'use client'; - -import Link from 'next/link'; -import { useMemo, useState } from 'react'; -import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; -import { useSearchParams } from 'next/navigation'; -import { - ArrowLeft, - ChevronDown, - ChevronRight, - FileText, - Loader2, - Trash2, - Upload, -} from 'lucide-react'; -import { toast } from 'sonner'; - -import { - AlertDialog, - AlertDialogAction, - AlertDialogCancel, - AlertDialogContent, - AlertDialogDescription, - AlertDialogFooter, - AlertDialogHeader, - AlertDialogTitle, -} from '@/components/ui/alert-dialog'; -import { EmptyStateWithBorder } from '@/components/layout/empty'; -import { ErrorInline } from '@/components/layout/error'; -import { LoadingStateWithBorder } from '@/components/layout/loading'; -import { Badge } from '@/components/ui/badge'; -import { Button } from '@/components/ui/button'; -import { - Table, - TableBody, - TableCell, - TableHead, - TableHeader, - TableRow, -} from '@/components/ui/table'; -import { type PagesDeployment, PagesService } from '@/lib/services/openflare'; -import { formatDateTime } from '@/lib/utils'; - -import { DeploymentUploadDialog } from '../components/deployment-upload-dialog'; -import { ProjectEditorDialog } from '../components/project-editor-dialog'; -import { - deploymentFilesQueryKey, - deploymentsQueryKey, - formatBytes, - projectQueryKey, - projectsQueryKey, -} from '../components/pages-utils'; - -function DeploymentFilesPanel({ - projectId, - deployment, -}: { - projectId: number; - deployment: PagesDeployment; -}) { - const filesQuery = useQuery({ - queryKey: deploymentFilesQueryKey(projectId, deployment.id), - queryFn: () => PagesService.listDeploymentFiles(deployment.id), - }); - - if (filesQuery.isLoading) { - return ( -

加载文件清单...

- ); - } - - if (filesQuery.isError) { - return ( -

- {filesQuery.error instanceof Error - ? filesQuery.error.message - : '加载失败'} -

- ); - } - - const files = filesQuery.data ?? []; - if (files.length === 0) { - return ( -

暂无文件记录

- ); - } - - return ( -
- - - - 路径 - 大小 - - - - {files.map((file) => ( - - {file.path} - - {formatBytes(file.size)} - - - ))} - -
-
- ); -} - -export function PagesDetailPageClient() { - const searchParams = useSearchParams(); - const queryClient = useQueryClient(); - const projectId = searchParams.get('id')?.trim() ?? ''; - const parsedProjectId = Number(projectId); - - const [editorOpen, setEditorOpen] = useState(false); - const [uploadOpen, setUploadOpen] = useState(false); - const [expandedDeploymentId, setExpandedDeploymentId] = useState< - number | null - >(null); - const [deleteProjectOpen, setDeleteProjectOpen] = useState(false); - const [pendingDeploymentAction, setPendingDeploymentAction] = useState<{ - type: 'activate' | 'delete'; - deployment: PagesDeployment; - } | null>(null); - - const enabled = projectId !== '' && Number.isFinite(parsedProjectId); - - const projectQuery = useQuery({ - queryKey: projectQueryKey(projectId), - queryFn: () => PagesService.getProject(parsedProjectId), - enabled, - }); - - const deploymentsQuery = useQuery({ - queryKey: deploymentsQueryKey(parsedProjectId), - queryFn: () => PagesService.listDeployments(parsedProjectId), - enabled, - }); - - const activateMutation = useMutation({ - mutationFn: (deploymentId: number) => - PagesService.activateDeployment(parsedProjectId, deploymentId), - onSuccess: async () => { - toast.success('部署已激活'); - await Promise.all([ - queryClient.invalidateQueries({ - queryKey: deploymentsQueryKey(parsedProjectId), - }), - queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), - queryClient.invalidateQueries({ queryKey: projectsQueryKey }), - ]); - setPendingDeploymentAction(null); - }, - onError: (error) => { - toast.error(error instanceof Error ? error.message : '激活失败'); - }, - }); - - const deleteDeploymentMutation = useMutation({ - mutationFn: (deploymentId: number) => - PagesService.deleteDeployment(parsedProjectId, deploymentId), - onSuccess: async () => { - toast.success('部署已删除'); - await Promise.all([ - queryClient.invalidateQueries({ - queryKey: deploymentsQueryKey(parsedProjectId), - }), - queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), - queryClient.invalidateQueries({ queryKey: projectsQueryKey }), - ]); - setPendingDeploymentAction(null); - }, - onError: (error) => { - toast.error(error instanceof Error ? error.message : '删除失败'); - }, - }); - - const deleteProjectMutation = useMutation({ - mutationFn: () => PagesService.deleteProject(parsedProjectId), - onSuccess: async () => { - toast.success('项目已删除'); - await queryClient.invalidateQueries({ queryKey: projectsQueryKey }); - window.location.href = '/pages'; - }, - onError: (error) => { - toast.error(error instanceof Error ? error.message : '删除失败'); - }, - }); - - const project = projectQuery.data; - const deployments = useMemo( - () => deploymentsQuery.data ?? [], - [deploymentsQuery.data], - ); - - if (!enabled) { - return ( -
- -
- ); - } - - if (projectQuery.isLoading) { - return ( -
- -
- ); - } - - if (projectQuery.isError) { - return ( -
- void projectQuery.refetch()} - /> -
- ); - } - - if (!project) { - return ( -
- - -
- ); - } - - return ( -
-
-
- -
- -

- {project.name} -

-
-

- {project.slug} · {project.deployment_count} 个部署 -

-
-
- - - -
-
- -
-
-

- 状态 -

- - {project.enabled ? '已启用' : '已停用'} - -
-
-

- 当前激活 -

-

- {project.active_deployment - ? `#${project.active_deployment.deployment_number}` - : '暂无'} -

-
-
-

- 更新时间 -

-

{formatDateTime(project.updated_at)}

-
-
- -
-
-

部署历史

-

- 部署不可变；激活后发布配置，Agent 才会拉取并切换静态资源。 -

-
- - {deploymentsQuery.isLoading ? ( - - ) : deployments.length === 0 ? ( - - ) : ( -
- {deployments.map((deployment) => { - const expanded = expandedDeploymentId === deployment.id; - return ( -
-
-
- -
-
-

- #{deployment.deployment_number} -

- {deployment.status === 'active' ? ( - - 已激活 - - ) : null} -
-

- {deployment.checksum.slice(0, 16)} ·{' '} - {deployment.file_count} files ·{' '} - {formatBytes(deployment.total_size)} -

-

- 创建于 {formatDateTime(deployment.created_at)} -

-
-
-
- - -
-
- {expanded ? ( - - ) : null} -
- ); - })} -
- )} -
- - - - - - - - 删除 Pages 项目 - - 确认删除项目 {project.name} 吗？此操作不可恢复。 - - - - 取消 - deleteProjectMutation.mutate()} - > - {deleteProjectMutation.isPending ? ( - - ) : ( - '确认删除' - )} - - - - - - { - if (!open) setPendingDeploymentAction(null); - }} - > - - - - {pendingDeploymentAction?.type === 'activate' - ? '激活部署' - : '删除部署'} - - - {pendingDeploymentAction?.type === 'activate' - ? `确认激活部署 #${pendingDeploymentAction.deployment.deployment_number} 吗？` - : `确认删除部署 #${pendingDeploymentAction?.deployment.deployment_number} 吗？`} - - - - 取消 - { - if (!pendingDeploymentAction) return; - if (pendingDeploymentAction.type === 'activate') { - activateMutation.mutate( - pendingDeploymentAction.deployment.id, - ); - } else { - deleteDeploymentMutation.mutate( - pendingDeploymentAction.deployment.id, - ); - } - }} - > - 确认 - - - - -
- ); -} diff --git a/frontend/app/(main)/pages/detail/page.tsx b/frontend/app/(main)/pages/detail/page.tsx index a24e6fca..e653471b 100644 --- a/frontend/app/(main)/pages/detail/page.tsx +++ b/frontend/app/(main)/pages/detail/page.tsx @@ -1,23 +1,318 @@ -import { Suspense } from 'react'; +'use client'; +import Link from 'next/link'; +import { Suspense, useState } from 'react'; +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { useRouter, useSearchParams } from 'next/navigation'; +import { ArrowLeft, FileText, Pencil, Trash2, Upload } from 'lucide-react'; +import { toast } from 'sonner'; + +import { EmptyStateWithBorder } from '@/components/layout/empty'; +import { ErrorInline } from '@/components/layout/error'; +import { LoadingStateWithBorder } from '@/components/layout/loading'; +import { + AlertDialog, + AlertDialogAction, + AlertDialogCancel, + AlertDialogContent, + AlertDialogDescription, + AlertDialogFooter, + AlertDialogHeader, + AlertDialogTitle, +} from '@/components/ui/alert-dialog'; +import { Badge } from '@/components/ui/badge'; +import { Button } from '@/components/ui/button'; +import { + Card, + CardAction, + CardContent, + CardDescription, + CardHeader, + CardTitle, +} from '@/components/ui/card'; import { Skeleton } from '@/components/ui/skeleton'; +import { Spinner } from '@/components/ui/spinner'; +import { PagesService } from '@/lib/services/openflare'; +import { formatDateTime } from '@/lib/utils'; -import { PagesDetailPageClient } from './page-client'; +import { + DeploymentUploadDialog, + pagesEntryPath, +} from '../components/deployment-upload-dialog'; +import { ProjectEditorDialog } from '../components/project-editor-dialog'; +import { + projectQueryKey, + projectsQueryKey, + sourceQueryKey, +} from '../components/pages-utils'; +import { DeploymentHistory } from './components/deployment-history'; +import { PagesSourceCard } from './components/pages-source-card'; function PagesDetailPageFallback() { return ( -
- - +
+ + +
+ + +
); } +function PagesDetailRoute() { + const searchParams = useSearchParams(); + const router = useRouter(); + const queryClient = useQueryClient(); + const [editorOpen, setEditorOpen] = useState(false); + const [uploadOpen, setUploadOpen] = useState(false); + const [deleteProjectOpen, setDeleteProjectOpen] = useState(false); + + const rawProjectId = searchParams.get('id')?.trim() ?? ''; + const projectId = Number(rawProjectId); + const validProjectId = + rawProjectId !== '' && Number.isInteger(projectId) && projectId > 0; + + const projectQuery = useQuery({ + queryKey: projectQueryKey(projectId), + queryFn: () => PagesService.getProject(projectId), + enabled: validProjectId, + }); + + const deleteProjectMutation = useMutation({ + mutationFn: () => PagesService.deleteProject(projectId), + onSuccess: async () => { + toast.success('项目已删除'); + await queryClient.invalidateQueries({ queryKey: projectsQueryKey }); + router.push('/pages'); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '删除失败'); + }, + }); + + if (!validProjectId) { + return ( +
+ +
+ ); + } + + if (projectQuery.isLoading) { + return ( +
+ +
+ ); + } + + if (projectQuery.isError) { + return ( +
+
+ void projectQuery.refetch()} + /> +
+
+ ); + } + + const project = projectQuery.data; + if (!project) { + return ( +
+ + +
+ ); + } + + const activeDeployment = project.active_deployment; + const entryPath = pagesEntryPath(project.root_dir ?? '', project.entry_file); + + return ( +
+
+ +
+
+
+ +

+ {project.name} +

+
+

+ {project.slug} · {project.deployment_count} 个部署 +

+
+
+ + + +
+
+
+ +
+ + + 当前生产部署 + + Agent 当前应拉取并提供服务的不可变部署。 + + + + {activeDeployment ? '生产中' : '未发布'} + + + + + {activeDeployment ? ( + <> +

+ 部署 #{activeDeployment.deployment_number} +

+

+ {activeDeployment.checksum.slice(0, 20)} +

+

+ 激活于{' '} + {activeDeployment.activated_at + ? formatDateTime(activeDeployment.activated_at) + : '未知时间'} +

+ + ) : ( +

+ 尚无生产部署。上传或同步来源后，从部署历史激活一个版本。 +

+ )} + + + + + + 站点入口 + + 解包校验、发布快照与 Agent 切换共同使用此路径。 + + + + {project.enabled ? '项目已启用' : '项目已停用'} + + + + + + {entryPath} + +

+ 项目更新于 {formatDateTime(project.updated_at)} +

+ + +
+ + + + + { + setEditorOpen(nextOpen); + if (!nextOpen) { + void queryClient.invalidateQueries({ + queryKey: sourceQueryKey(projectId), + }); + } + }} + project={project} + /> + + + + + + 删除 Pages 项目 + + 确认删除项目 {project.name} 吗？此操作不可恢复。 + + + + + 取消 + + { + event.preventDefault(); + deleteProjectMutation.mutate(); + }} + > + {deleteProjectMutation.isPending ? ( + + ) : null} + 确认删除 + + + + +
+ ); +} + export default function PagesDetailPage() { return ( }> - + ); } diff --git a/frontend/lib/services/index.ts b/frontend/lib/services/index.ts index 6184f565..69e79fa1 100644 --- a/frontend/lib/services/index.ts +++ b/frontend/lib/services/index.ts @@ -251,6 +251,9 @@ export type { DnsAccountItem, PagesProject, PagesDeployment, + PagesSource, + PagesSourceActionReceipt, + PagesSourceStatus, OriginItem, OriginDetail, AccessLogList, diff --git a/frontend/lib/services/openflare/index.ts b/frontend/lib/services/openflare/index.ts index 4ef01046..7bca6843 100644 --- a/frontend/lib/services/openflare/index.ts +++ b/frontend/lib/services/openflare/index.ts @@ -109,8 +109,19 @@ export type { OriginMutationPayload, PagesDeployment, PagesDeploymentFile, + PagesGitHubReleaseSource, + PagesManualSource, PagesProject, PagesProjectPayload, + PagesRemoteNetworkPolicy, + PagesRemoteSourceUpdatePayload, + PagesRemoteURLSource, + PagesSource, + PagesSourceActionPayload, + PagesSourceActionReceipt, + PagesSourceRevision, + PagesSourceStatus, + PagesSourceUpdateResult, AcmeAccountItem, DnsAccountItem, DnsAccountMutationPayload, diff --git a/frontend/lib/services/openflare/pages.service.ts b/frontend/lib/services/openflare/pages.service.ts index e220a168..e45b0543 100644 --- a/frontend/lib/services/openflare/pages.service.ts +++ b/frontend/lib/services/openflare/pages.service.ts @@ -12,6 +12,11 @@ import type { PagesDeploymentUploadPayload, PagesProject, PagesProjectPayload, + PagesRemoteSourceUpdatePayload, + PagesSource, + PagesSourceActionPayload, + PagesSourceActionReceipt, + PagesSourceUpdateResult, } from './types'; export class PagesService extends OpenFlareBaseService { @@ -40,6 +45,41 @@ export class PagesService extends OpenFlareBaseService { return this.post(`/${id}/delete`); } + static getSource(projectId: number): Promise { + return this.get(`/${projectId}/source`); + } + + static updateSource( + projectId: number, + payload: PagesRemoteSourceUpdatePayload, + ): Promise { + return this.post( + `/${projectId}/source/update`, + payload, + ); + } + + static deleteSource(projectId: number): Promise { + return this.post(`/${projectId}/source/delete`); + } + + static checkSource(projectId: number): Promise { + return this.post( + `/${projectId}/source/check`, + {}, + ); + } + + static syncSource( + projectId: number, + payload: PagesSourceActionPayload = {}, + ): Promise { + return this.post( + `/${projectId}/source/sync`, + payload, + ); + } + static listDeployments(projectId: number): Promise { return this.get(`/${projectId}/deployments`); } @@ -58,8 +98,6 @@ export class PagesService extends OpenFlareBaseService { ): Promise { const formData = new FormData(); formData.append('package', payload.file); - formData.append('root_dir', payload.rootDir ?? ''); - formData.append('entry_file', payload.entryFile ?? 'index.html'); return this.postFormData( `/${projectId}/deployments/upload`, @@ -75,6 +113,7 @@ export class PagesService extends OpenFlareBaseService { return this.post( `/${projectId}/deployments/upload-from-url`, payload, + { timeout: apiConfig.uploadTimeout } as InternalAxiosRequestConfig, ); } diff --git a/frontend/lib/services/openflare/types.ts b/frontend/lib/services/openflare/types.ts index feede7b6..0b9be41c 100644 --- a/frontend/lib/services/openflare/types.ts +++ b/frontend/lib/services/openflare/types.ts @@ -394,9 +394,11 @@ export interface PagesDeployment { status: 'uploaded' | 'active'; file_count: number; total_size: number; - root_dir?: string; - entry_file: string; created_by: string; + source_type: 'manual_upload' | 'manual_url' | 'remote_url' | 'github_release'; + source_label: string; + trigger_type: + 'manual_upload' | 'manual_url' | 'manual_sync' | 'scheduled_auto_update'; created_at: string; activated_at?: string | null; } @@ -448,8 +450,6 @@ export interface PagesProjectPayload { export interface PagesDeploymentUploadPayload { file: File; - rootDir?: string; - entryFile?: string; onProgress?: (percent: number) => void; } @@ -457,6 +457,79 @@ export interface PagesDeploymentUploadFromURLPayload { url: string; } +export type PagesSourceStatus = + 'idle' | 'checking' | 'update_available' | 'syncing' | 'failed' | 'attention'; + +export type PagesRemoteNetworkPolicy = 'public' | 'trusted_internal'; + +export interface PagesSourceRevision { + revision: string; + label: string; + asset_name?: string; +} + +interface PagesSourceRuntimeView { + sync_status?: PagesSourceStatus; + update_available?: boolean; + last_seen?: PagesSourceRevision; + last_applied?: PagesSourceRevision; + last_checked_at?: string | null; + last_synced_at?: string | null; + next_check_at?: string | null; + last_error?: string; +} + +export interface PagesManualSource { + source_type: 'manual'; +} + +export interface PagesRemoteURLSource extends PagesSourceRuntimeView { + source_type: 'remote_url'; + has_remote_url: boolean; + display_url: string; + remote_network_policy: PagesRemoteNetworkPolicy; +} + +export interface PagesGitHubReleaseSource extends PagesSourceRuntimeView { + source_type: 'github_release'; + github_repository: string; + release_selector: 'latest' | 'tag'; + release_tag: string; + asset_name: string; + auto_update_enabled: boolean; + check_interval_minutes: number; +} + +/** + * 部署源使用判别联合，后续仓库构建来源只需增加独立 variant， + * 不需要向 Remote 或 GitHub Release 填入构建字段。 + */ +export type PagesSource = + PagesManualSource | PagesRemoteURLSource | PagesGitHubReleaseSource; + +export interface PagesRemoteSourceUpdatePayload { + source_type: 'remote_url'; + remote_url_set: boolean; + remote_url: string; + remote_network_policy: PagesRemoteNetworkPolicy; +} + +export interface PagesSourceActionPayload { + confirmed_revision?: string; +} + +export interface PagesSourceActionReceipt { + task_id: string; + execution_id: string; + action: 'check' | 'sync'; +} + +export interface PagesSourceUpdateResult { + source: PagesSource; + check_task: PagesSourceActionReceipt | null; + warning: string; +} + // ==================== Origins ==================== export interface OriginItem { diff --git a/frontend/tests/openflare/pages-service.test.ts b/frontend/tests/openflare/pages-service.test.ts index c38303be..516e5596 100644 --- a/frontend/tests/openflare/pages-service.test.ts +++ b/frontend/tests/openflare/pages-service.test.ts @@ -2,27 +2,34 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import type { AxiosResponse } from 'axios'; import apiClient from '@/lib/services/core/api-client'; +import { apiConfig } from '@/lib/services/core/config'; import { PagesService } from '@/lib/services/openflare/pages.service'; vi.mock('@/lib/services/core/api-client', () => ({ default: { get: vi.fn(), + post: vi.fn(), }, })); +function response(data: T) { + return { + data: { error_msg: '', data }, + status: 200, + statusText: 'OK', + headers: {}, + config: { headers: {} }, + } as AxiosResponse; +} + describe('PagesService', () => { beforeEach(() => { vi.mocked(apiClient.get).mockReset(); + vi.mocked(apiClient.post).mockReset(); }); it('requests deployment files using the backend deployment route', async () => { - vi.mocked(apiClient.get).mockResolvedValue({ - data: { error_msg: '', data: [] }, - status: 200, - statusText: 'OK', - headers: {}, - config: { headers: {} }, - } as AxiosResponse); + vi.mocked(apiClient.get).mockResolvedValue(response([])); await PagesService.listDeploymentFiles(7); @@ -31,4 +38,88 @@ describe('PagesService', () => { expect.objectContaining({ params: undefined }), ); }); + + it('connects all source endpoints with explicit action payloads', async () => { + vi.mocked(apiClient.get).mockResolvedValue( + response({ source_type: 'manual' }), + ); + vi.mocked(apiClient.post).mockResolvedValue( + response({ source_type: 'manual' }), + ); + + await PagesService.getSource(12); + await PagesService.updateSource(12, { + source_type: 'remote_url', + remote_url_set: true, + remote_url: 'https://example.com/site.zip?token=secret', + remote_network_policy: 'public', + }); + await PagesService.deleteSource(12); + await PagesService.checkSource(12); + await PagesService.syncSource(12); + + expect(apiClient.get).toHaveBeenCalledWith( + '/api/v1/d/pages/12/source', + expect.objectContaining({ params: undefined }), + ); + expect(apiClient.post).toHaveBeenNthCalledWith( + 1, + '/api/v1/d/pages/12/source/update', + expect.objectContaining({ + source_type: 'remote_url', + remote_url_set: true, + }), + undefined, + ); + expect(apiClient.post).toHaveBeenNthCalledWith( + 2, + '/api/v1/d/pages/12/source/delete', + undefined, + undefined, + ); + expect(apiClient.post).toHaveBeenNthCalledWith( + 3, + '/api/v1/d/pages/12/source/check', + {}, + undefined, + ); + expect(apiClient.post).toHaveBeenNthCalledWith( + 4, + '/api/v1/d/pages/12/source/sync', + {}, + undefined, + ); + }); + + it('uploads only the package multipart field', async () => { + vi.mocked(apiClient.post).mockResolvedValue(response({})); + const file = new File(['site'], 'site.zip', { + type: 'application/zip', + }); + + await PagesService.uploadDeployment(8, { file }); + + const formData = vi.mocked(apiClient.post).mock.calls[0]?.[1]; + expect(formData).toBeInstanceOf(FormData); + expect(Array.from((formData as FormData).keys())).toEqual(['package']); + expect(apiClient.post).toHaveBeenCalledWith( + '/api/v1/d/pages/8/deployments/upload', + formData, + expect.objectContaining({ timeout: apiConfig.uploadTimeout }), + ); + }); + + it('keeps the compatibility URL import on the long upload timeout', async () => { + vi.mocked(apiClient.post).mockResolvedValue(response({})); + + await PagesService.uploadDeploymentFromURL(8, { + url: 'https://example.com/site.zip', + }); + + expect(apiClient.post).toHaveBeenCalledWith( + '/api/v1/d/pages/8/deployments/upload-from-url', + { url: 'https://example.com/site.zip' }, + expect.objectContaining({ timeout: apiConfig.uploadTimeout }), + ); + }); }); diff --git a/frontend/tests/openflare/pages-source-ui.test.tsx b/frontend/tests/openflare/pages-source-ui.test.tsx new file mode 100644 index 00000000..b9773638 --- /dev/null +++ b/frontend/tests/openflare/pages-source-ui.test.tsx @@ -0,0 +1,227 @@ +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { DeploymentUploadDialog } from '@/app/(main)/pages/components/deployment-upload-dialog'; +import { DeploymentHistory } from '@/app/(main)/pages/detail/components/deployment-history'; +import { PagesSourceCard } from '@/app/(main)/pages/detail/components/pages-source-card'; +import { AdminTaskService } from '@/lib/services/admin'; +import { + type PagesRemoteURLSource, + PagesService, +} from '@/lib/services/openflare'; + +vi.mock('@/lib/services/openflare', async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + PagesService: { + getSource: vi.fn(), + updateSource: vi.fn(), + deleteSource: vi.fn(), + syncSource: vi.fn(), + listDeployments: vi.fn(), + listDeploymentFiles: vi.fn(), + activateDeployment: vi.fn(), + deleteDeployment: vi.fn(), + uploadDeployment: vi.fn(), + }, + }; +}); + +vi.mock('@/lib/services/admin', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + AdminTaskService: { + getTaskExecution: vi.fn(), + }, + }; +}); + +function renderWithQuery(ui: React.ReactNode) { + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false, gcTime: 0 }, + mutations: { retry: false }, + }, + }); + return render( + {ui}, + ); +} + +const remoteSource: PagesRemoteURLSource = { + source_type: 'remote_url', + has_remote_url: true, + display_url: 'https://artifacts.example.com/site.zip?***', + remote_network_policy: 'public', + sync_status: 'idle', + last_applied: { + revision: 'a'.repeat(64), + label: 'site.zip', + }, + last_synced_at: '2026-07-19T10:00:00Z', + last_error: '', +}; + +describe('Pages source UI', () => { + beforeEach(() => { + vi.mocked(PagesService.getSource).mockReset(); + vi.mocked(PagesService.updateSource).mockReset(); + vi.mocked(PagesService.deleteSource).mockReset(); + vi.mocked(PagesService.syncSource).mockReset(); + vi.mocked(PagesService.listDeployments).mockReset(); + vi.mocked(PagesService.listDeploymentFiles).mockReset(); + vi.mocked(PagesService.activateDeployment).mockReset(); + vi.mocked(PagesService.deleteDeployment).mockReset(); + vi.mocked(PagesService.uploadDeployment).mockReset(); + vi.mocked(AdminTaskService.getTaskExecution).mockReset(); + }); + + it('keeps Phase 1 manual source focused on upload and Remote URL', async () => { + vi.mocked(PagesService.getSource).mockResolvedValue({ + source_type: 'manual', + }); + + renderWithQuery(); + + expect(await screen.findByText('本地部署包')).toBeVisible(); + expect( + screen.getByRole('button', { name: '配置 Remote URL' }), + ).toBeVisible(); + expect(screen.queryByText('检查更新')).not.toBeInTheDocument(); + expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); + }); + + it('never reuses the masked URL as an editable value', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(remoteSource); + vi.mocked(PagesService.updateSource).mockResolvedValue({ + source: remoteSource, + check_task: null, + warning: '', + }); + + renderWithQuery(); + + expect( + await screen.findByText('https://artifacts.example.com/site.zip?***'), + ).toBeVisible(); + expect(screen.queryByText(/token=secret/)).not.toBeInTheDocument(); + + await user.click(screen.getByRole('button', { name: '编辑来源' })); + await user.click(screen.getByRole('button', { name: '更换地址' })); + + const input = screen.getByPlaceholderText( + 'https://artifacts.example.com/site.zip?token=...', + ); + expect(input).toHaveValue(''); + await user.type(input, 'https://new.example.com/site.zip?token=new'); + await user.click(screen.getByRole('button', { name: '保存 Remote 来源' })); + + await waitFor(() => { + expect(PagesService.updateSource).toHaveBeenCalledWith(9, { + source_type: 'remote_url', + remote_url_set: true, + remote_url: 'https://new.example.com/site.zip?token=new', + remote_network_policy: 'public', + }); + }); + }); + + it('requires a second confirmation for trusted internal networking', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(remoteSource); + vi.mocked(PagesService.updateSource).mockResolvedValue({ + source: { ...remoteSource, remote_network_policy: 'trusted_internal' }, + check_task: null, + warning: '', + }); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '编辑来源' })); + await user.click(screen.getByRole('radio', { name: '受信内网模式' })); + await user.click(screen.getByRole('button', { name: '保存 Remote 来源' })); + + expect(await screen.findByText('启用受信内网模式')).toBeVisible(); + expect(PagesService.updateSource).not.toHaveBeenCalled(); + + await user.click(screen.getByRole('button', { name: '确认' })); + await waitFor(() => { + expect(PagesService.updateSource).toHaveBeenCalledWith(9, { + source_type: 'remote_url', + remote_url_set: false, + remote_url: '', + remote_network_policy: 'trusted_internal', + }); + }); + }); + + it('polls the existing task execution detail after sync dispatch', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(remoteSource); + vi.mocked(PagesService.syncSource).mockResolvedValue({ + task_id: 'manual_of_pages_source_action_1', + execution_id: '42', + action: 'sync', + }); + vi.mocked(AdminTaskService.getTaskExecution).mockResolvedValue({ + id: '42', + task_id: 'manual_of_pages_source_action_1', + task_type: 'of_pages_source_action', + task_name: 'Pages 来源动作', + status: 'succeeded', + retryable: false, + max_retry: 0, + retry_count: 0, + log: '', + error_message: '', + result: '', + duration: 1, + payload: '', + triggered_by: 'admin:1', + created_at: '2026-07-19T10:00:00Z', + updated_at: '2026-07-19T10:00:01Z', + }); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '同步并发布' })); + + await waitFor(() => { + expect(PagesService.syncSource).toHaveBeenCalledWith(9, {}); + expect(AdminTaskService.getTaskExecution).toHaveBeenCalledWith('42'); + }); + }); + + it('shows the actual project entry and no one-off URL upload tab', () => { + renderWithQuery( + , + ); + + expect(screen.getByText('dist/site/home.html')).toBeVisible(); + expect(screen.queryByText('从 URL 下载')).not.toBeInTheDocument(); + expect(screen.queryByText('部署包下载链接')).not.toBeInTheDocument(); + }); + + it('renders a deployment query failure instead of an empty history', async () => { + vi.mocked(PagesService.listDeployments).mockRejectedValue( + new Error('部署历史暂时不可用'), + ); + + renderWithQuery(); + + expect(await screen.findByText('部署历史暂时不可用')).toBeVisible(); + expect(screen.queryByText('暂无部署')).not.toBeInTheDocument(); + }); +}); diff --git a/internal/apps/admin/task/routers.go b/internal/apps/admin/task/routers.go index ed969a7d..594f1eb2 100644 --- a/internal/apps/admin/task/routers.go +++ b/internal/apps/admin/task/routers.go @@ -66,7 +66,7 @@ func DispatchTask(c *gin.Context) { return } - meta := task.GetTaskMeta(req.TaskType) + meta := getAdminTaskMeta(req.TaskType) if meta == nil { response.AbortBadRequest(c, InvalidTaskType) return @@ -256,7 +256,7 @@ func CreateSchedule(c *gin.Context) { } // 校验关联的异步任务类型 - meta := task.GetTaskMeta(req.TaskType) + meta := getAdminTaskMeta(req.TaskType) if meta == nil { response.AbortBadRequest(c, InvalidTaskType) return @@ -338,6 +338,10 @@ func UpdateSchedule(c *gin.Context) { response.AbortNotFound(c, ScheduleNotFound) return } + if existingMeta := task.GetTaskMeta(schedule.TaskType); existingMeta != nil && existingMeta.InternalOnly { + response.AbortBadRequest(c, InvalidTaskType) + return + } // 校验 Cron 表达式 if _, err := cron.ParseStandard(req.Cron); err != nil { @@ -346,7 +350,7 @@ func UpdateSchedule(c *gin.Context) { } // 校验关联的异步任务类型 - meta := task.GetTaskMeta(req.TaskType) + meta := getAdminTaskMeta(req.TaskType) if meta == nil { response.AbortBadRequest(c, InvalidTaskType) return @@ -382,6 +386,14 @@ func UpdateSchedule(c *gin.Context) { c.JSON(http.StatusOK, response.OK(schedule)) } +func getAdminTaskMeta(taskType string) *task.TaskMeta { + meta := task.GetTaskMeta(taskType) + if meta == nil || meta.InternalOnly { + return nil + } + return meta +} + // DeleteSchedule 删除定时任务 // @Summary 删除定时任务 // @Description 删除指定的定时任务配置，并触发调度器热加载，需要管理员权限 diff --git a/internal/apps/admin/task/routers_test.go b/internal/apps/admin/task/routers_test.go index db63ab8b..694861e7 100644 --- a/internal/apps/admin/task/routers_test.go +++ b/internal/apps/admin/task/routers_test.go @@ -29,6 +29,20 @@ import ( "github.com/Rain-kl/Wavelet/internal/common/response" ) +const ( + testInternalOnlyTaskType = "test_internal_only_admin" + testInternalOnlyAsynqTask = "test:internal_only_admin" +) + +func registerInternalOnlyTaskMeta() { + task.RegisterTaskMeta(task.TaskMeta{ + Type: testInternalOnlyTaskType, + AsynqTask: testInternalOnlyAsynqTask, + Name: "内部测试任务", + InternalOnly: true, + }) +} + func setupTaskTestEnvironment(t *testing.T) func() { _, mr, cleanup := testhelper.SetupTestEnvironment(t) bootstrap.RegisterTasks() @@ -61,12 +75,15 @@ func setupTestRouter(authUser *model.User) *gin.Engine { adminGroup.GET("/tasks/executions", ListTaskExecutions) adminGroup.GET("/tasks/executions/:id", GetTaskExecution) adminGroup.POST("/tasks/executions/:id/retry", RetryTask) + adminGroup.POST("/tasks/schedules", CreateSchedule) + adminGroup.PUT("/tasks/schedules/:id", UpdateSchedule) return r } func TestListTaskTypes(t *testing.T) { cleanup := setupTaskTestEnvironment(t) defer cleanup() + registerInternalOnlyTaskMeta() adminUser := &model.User{ID: 1001, Username: "admin", IsAdmin: true} router := setupTestRouter(adminUser) @@ -93,6 +110,9 @@ func TestListTaskTypes(t *testing.T) { foundCleanup := false foundWarmImageCache := false for _, m := range taskMetas { + if m.Type == testInternalOnlyTaskType { + t.Errorf("internal-only task type %s must not be listed", testInternalOnlyTaskType) + } if m.Type == uploadtask.TaskTypeSystemCleanup { foundCleanup = true } @@ -108,6 +128,119 @@ func TestListTaskTypes(t *testing.T) { } } +func TestInternalOnlyTaskAdminBoundaries(t *testing.T) { + cleanup := setupTaskTestEnvironment(t) + defer cleanup() + registerInternalOnlyTaskMeta() + + adminUser := &model.User{ID: 1001, Username: "admin", IsAdmin: true} + router := setupTestRouter(adminUser) + ctx := context.Background() + + t.Run("dispatch rejects internal-only task", func(t *testing.T) { + body, err := json.Marshal(DispatchTaskRequest{TaskType: testInternalOnlyTaskType}) + require.NoError(t, err) + req := httptest.NewRequest(http.MethodPost, "/api/v1/admin/tasks/dispatch", bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusBadRequest, w.Code) + var resp response.Any + require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp)) + assert.Equal(t, InvalidTaskType, resp.ErrorMsg) + }) + + t.Run("create schedule rejects internal-only task", func(t *testing.T) { + isActive := true + body, err := json.Marshal(CreateScheduleRequest{ + Name: "内部任务排程", + TaskType: testInternalOnlyTaskType, + Cron: "0 * * * *", + IsActive: &isActive, + }) + require.NoError(t, err) + req := httptest.NewRequest(http.MethodPost, "/api/v1/admin/tasks/schedules", bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusBadRequest, w.Code) + var resp response.Any + require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp)) + assert.Equal(t, InvalidTaskType, resp.ErrorMsg) + }) + + t.Run("update cannot convert existing internal schedule to public task", func(t *testing.T) { + schedule := &model.Schedule{ + Name: "系统内部排程", + TaskType: testInternalOnlyTaskType, + Cron: "0 * * * *", + IsActive: true, + } + require.NoError(t, model.CreateSchedule(ctx, schedule)) + isActive := false + body, err := json.Marshal(UpdateScheduleRequest{ + Name: "尝试修改内部排程", + TaskType: uploadtask.TaskTypeSystemCleanup, + Cron: "5 * * * *", + IsActive: &isActive, + }) + require.NoError(t, err) + req := httptest.NewRequest( + http.MethodPut, + fmt.Sprintf("/api/v1/admin/tasks/schedules/%d", schedule.ID), + bytes.NewReader(body), + ) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusBadRequest, w.Code) + unchanged, err := model.GetScheduleByID(ctx, schedule.ID) + require.NoError(t, err) + assert.Equal(t, "系统内部排程", unchanged.Name) + assert.Equal(t, testInternalOnlyTaskType, unchanged.TaskType) + assert.True(t, unchanged.IsActive) + }) + + t.Run("update public schedule rejects internal-only target task", func(t *testing.T) { + schedule := &model.Schedule{ + Name: "公开排程", + TaskType: uploadtask.TaskTypeSystemCleanup, + Cron: "0 * * * *", + IsActive: true, + } + require.NoError(t, model.CreateSchedule(ctx, schedule)) + isActive := true + body, err := json.Marshal(UpdateScheduleRequest{ + Name: "尝试切入内部任务", + TaskType: testInternalOnlyTaskType, + Cron: "10 * * * *", + IsActive: &isActive, + }) + require.NoError(t, err) + req := httptest.NewRequest( + http.MethodPut, + fmt.Sprintf("/api/v1/admin/tasks/schedules/%d", schedule.ID), + bytes.NewReader(body), + ) + req.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusBadRequest, w.Code) + unchanged, err := model.GetScheduleByID(ctx, schedule.ID) + require.NoError(t, err) + assert.Equal(t, "公开排程", unchanged.Name) + assert.Equal(t, uploadtask.TaskTypeSystemCleanup, unchanged.TaskType) + }) +} + func TestDispatchTask(t *testing.T) { cleanup := setupTaskTestEnvironment(t) defer cleanup() diff --git a/internal/apps/openflare/pages/download_url.go b/internal/apps/openflare/pages/download_url.go index 5959d7e7..09e0a93f 100644 --- a/internal/apps/openflare/pages/download_url.go +++ b/internal/apps/openflare/pages/download_url.go @@ -5,211 +5,38 @@ package pages import ( "context" - "crypto/sha256" - "crypto/tls" - "encoding/hex" "errors" - "fmt" - "io" - "mime" - "net" - "net/http" "net/url" - "os" - "path" - "path/filepath" "strings" - "time" "github.com/Rain-kl/Wavelet/pkg/pagesarchive" ) -const ( - pagesURLDownloadTimeout = 10 * time.Minute - pagesURLMaxRedirects = 5 - pagesMagicSniffBytes = 512 - pagesURLDialTimeout = 30 * time.Second - pagesURLTLSHandshake = 15 * time.Second - pagesBrowserUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" - pagesBrowserAccept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" - pagesBrowserAcceptLang = "zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7" - pagesBrowserSecCHUA = `"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"` - pagesBrowserSecCHUAMobile = "?0" - pagesBrowserSecCHUAPlat = `"Windows"` -) - -// downloadPagesPackageFromURL fetches a remote archive with browser-like headers -// and writes it to a temp file. Allows private/LAN hosts and insecure TLS certs -// (self-signed / internal CA) so operators can pull from internal artifact stores. -func downloadPagesPackageFromURL(ctx context.Context, rawURL string, maxPackageBytes int64) (tempPath string, checksum string, size int64, format pagesarchive.Format, fileName string, err error) { - parsed, err := parseAndValidatePagesDownloadURL(rawURL) - if err != nil { +// downloadPagesPackageFromURL is the deprecated one-shot URL adapter. It uses +// the same bounded downloader as persisted sources, with the legacy trusted +// network policy that permits operator-managed internal artifact services. +func downloadPagesPackageFromURL( + ctx context.Context, + rawURL string, + maxPackageBytes int64, +) (tempPath string, checksum string, size int64, format pagesarchive.Format, fileName string, err error) { + if _, err := parseAndValidatePagesDownloadURL(rawURL); err != nil { return "", "", 0, "", "", err } - - resp, err := doBrowserDownload(ctx, parsed) + candidate, err := FetchRemoteSource(ctx, RemoteSourceRequest{ + URL: strings.TrimSpace(rawURL), + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: maxPackageBytes, + }) if err != nil { - return "", "", 0, "", "", err - } - defer func() { _ = resp.Body.Close() }() - if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return "", "", 0, "", "", fmt.Errorf("%s: HTTP %d", errPagesPackageURLDownloadFailed, resp.StatusCode) - } - if resp.ContentLength > 0 && resp.ContentLength > maxPackageBytes { - return "", "", 0, "", "", errors.New(errPagesPackageURLTooLarge) - } - - fileName = fileNameFromDownload(resp, parsed) - format, _ = pagesarchive.DetectFormatFromName(fileName) - - tempPath, checksum, size, err = writeLimitedPackageTemp(resp.Body, format, maxPackageBytes) - if err != nil { - return "", "", 0, "", "", err - } - format, fileName, err = ensurePackageFormat(tempPath, format, fileName) - if err != nil { - _ = os.Remove(tempPath) - return "", "", 0, "", "", err - } - return tempPath, checksum, size, format, fileName, nil -} - -func newPagesURLDownloadClient() *http.Client { - transport := &http.Transport{ - Proxy: http.ProxyFromEnvironment, - DialContext: (&net.Dialer{ - Timeout: pagesURLDialTimeout, - KeepAlive: pagesURLDialTimeout, - }).DialContext, - ForceAttemptHTTP2: true, - MaxIdleConns: 32, - IdleConnTimeout: 90 * time.Second, - TLSHandshakeTimeout: pagesURLTLSHandshake, - ExpectContinueTimeout: time.Second, - // Allow self-signed / internal certificates for artifact hosts. - TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // intentional for internal/dev artifact URLs - } - client := &http.Client{ - Timeout: pagesURLDownloadTimeout, - Transport: transport, - } - client.CheckRedirect = func(req *http.Request, via []*http.Request) error { - if len(via) >= pagesURLMaxRedirects { - return errors.New(errPagesPackageURLDownloadFailed) + if strings.Contains(err.Error(), errPagesSourceRemoteURLInvalid) { + return "", "", 0, "", "", errors.New(errPagesPackageURLInvalid) } - if err := validatePagesDownloadURLValue(req.URL); err != nil { - return err - } - applyBrowserDownloadHeaders(req, via[0].URL.String()) - return nil - } - return client -} - -func doBrowserDownload(ctx context.Context, parsed *url.URL) (*http.Response, error) { - client := newPagesURLDownloadClient() - req, err := http.NewRequestWithContext(ctx, http.MethodGet, parsed.String(), nil) - if err != nil { - return nil, errors.New(errPagesPackageURLInvalid) - } - applyBrowserDownloadHeaders(req, "") - resp, err := client.Do(req) //nolint:gosec // scheme validated; private hosts and insecure TLS intentionally allowed - if err != nil { - return nil, fmt.Errorf("%s: %w", errPagesPackageURLDownloadFailed, err) - } - return resp, nil -} - -func writeLimitedPackageTemp(body io.Reader, format pagesarchive.Format, maxPackageBytes int64) (tempPath, checksum string, size int64, err error) { - temp, err := os.CreateTemp("", "openflare-pages-url-*."+safeTempSuffixOrBin(format)) - if err != nil { - return "", "", 0, err - } - tempPath = temp.Name() - defer func() { - _ = temp.Close() - if err != nil { - _ = os.Remove(tempPath) - } - }() - - hash := sha256.New() - written, copyErr := io.Copy(io.MultiWriter(temp, hash), io.LimitReader(body, maxPackageBytes+1)) - if copyErr != nil { - err = fmt.Errorf("%s: %w", errPagesPackageURLDownloadFailed, copyErr) - return "", "", 0, err - } - if written > maxPackageBytes { - err = errors.New(errPagesPackageURLTooLarge) - return "", "", 0, err - } - if written == 0 { - err = errors.New(errPagesPackageEmpty) - return "", "", 0, err - } - return tempPath, hex.EncodeToString(hash.Sum(nil)), written, nil -} - -func ensurePackageFormat(tempPath string, format pagesarchive.Format, fileName string) (pagesarchive.Format, string, error) { - if format != "" { - return format, fileName, nil - } - detected, ok := sniffPackageFormat(tempPath) - if !ok { - return "", fileName, errors.New(errPagesPackageUnsupported) - } - if !strings.Contains(strings.ToLower(fileName), ".") { - fileName = fileName + "." + pagesarchive.Extension(detected) - } - return detected, fileName, nil -} - -func sniffPackageFormat(tempPath string) (pagesarchive.Format, bool) { - file, err := os.Open(tempPath) //nolint:gosec // temp path created by us - if err != nil { - return "", false - } - defer func() { _ = file.Close() }() - head := make([]byte, pagesMagicSniffBytes) - n, _ := io.ReadFull(file, head) - if n <= 0 { - return "", false - } - return pagesarchive.DetectFormatFromBytes(head[:n]) -} - -func safeTempSuffixOrBin(format pagesarchive.Format) string { - if format == "" { - return "bin" - } - return safeTempSuffix(format) -} - -func applyBrowserDownloadHeaders(req *http.Request, referer string) { - if req == nil { - return - } - req.Header.Set("User-Agent", pagesBrowserUserAgent) - req.Header.Set("Accept", pagesBrowserAccept) - req.Header.Set("Accept-Language", pagesBrowserAcceptLang) - req.Header.Set("Cache-Control", "no-cache") - req.Header.Set("Pragma", "no-cache") - req.Header.Set("Upgrade-Insecure-Requests", "1") - req.Header.Set("Sec-Fetch-Dest", "document") - req.Header.Set("Sec-Fetch-Mode", "navigate") - req.Header.Set("Sec-Fetch-Site", "none") - req.Header.Set("Sec-Fetch-User", "?1") - req.Header.Set("Sec-Ch-Ua", pagesBrowserSecCHUA) - req.Header.Set("Sec-Ch-Ua-Mobile", pagesBrowserSecCHUAMobile) - req.Header.Set("Sec-Ch-Ua-Platform", pagesBrowserSecCHUAPlat) - if referer != "" { - req.Header.Set("Referer", referer) - req.Header.Set("Sec-Fetch-Site", "cross-site") - return - } - if req.URL != nil { - req.Header.Set("Referer", req.URL.Scheme+"://"+req.URL.Host+"/") + return "", "", 0, "", "", err } + // Ownership transfers to the existing one-shot caller, which removes the + // temporary file after the candidate deployment has been created. + return candidate.TempPath, candidate.Checksum, candidate.PackageSize, candidate.Format, candidate.SafeLabel, nil } func parseAndValidatePagesDownloadURL(raw string) (*url.URL, error) { @@ -218,57 +45,12 @@ func parseAndValidatePagesDownloadURL(raw string) (*url.URL, error) { return nil, errors.New(errPagesPackageURLRequired) } parsed, err := url.Parse(value) - if err != nil { + if err != nil || parsed.User != nil || parsed.Fragment != "" || parsed.Opaque != "" { return nil, errors.New(errPagesPackageURLInvalid) } - if err := validatePagesDownloadURLValue(parsed); err != nil { - return nil, err + scheme := strings.ToLower(strings.TrimSpace(parsed.Scheme)) + if (scheme != remoteSourceSchemeHTTP && scheme != remoteSourceSchemeHTTPS) || strings.TrimSpace(parsed.Hostname()) == "" { + return nil, errors.New(errPagesPackageURLInvalid) } return parsed, nil } - -func validatePagesDownloadURLValue(parsed *url.URL) error { - if parsed == nil { - return errors.New(errPagesPackageURLInvalid) - } - scheme := strings.ToLower(strings.TrimSpace(parsed.Scheme)) - if scheme != "http" && scheme != "https" { - return errors.New(errPagesPackageURLInvalid) - } - if strings.TrimSpace(parsed.Hostname()) == "" { - return errors.New(errPagesPackageURLInvalid) - } - return nil -} - -func fileNameFromDownload(resp *http.Response, parsed *url.URL) string { - if name := fileNameFromContentDisposition(resp); name != "" { - return name - } - if parsed != nil { - base := path.Base(parsed.Path) - if base != "" && base != "." && base != "/" { - return base - } - } - return "package.bin" -} - -func fileNameFromContentDisposition(resp *http.Response) string { - if resp == nil { - return "" - } - cd := resp.Header.Get("Content-Disposition") - if cd == "" { - return "" - } - _, params, err := mime.ParseMediaType(cd) - if err != nil { - return "" - } - name := strings.TrimSpace(params["filename"]) - if name == "" { - return "" - } - return path.Base(filepath.ToSlash(name)) -} diff --git a/internal/apps/openflare/pages/download_url_test.go b/internal/apps/openflare/pages/download_url_test.go index 918cf962..fc46e65f 100644 --- a/internal/apps/openflare/pages/download_url_test.go +++ b/internal/apps/openflare/pages/download_url_test.go @@ -10,7 +10,6 @@ import ( "net/http" "net/http/httptest" "os" - "strings" "testing" "github.com/stretchr/testify/assert" @@ -48,10 +47,10 @@ func TestDownloadPagesPackageFromURLAllowsPrivateHost(t *testing.T) { require.NoError(t, zw.Close()) zipBytes := body.Bytes() - var sawBrowserUA bool + var sawProviderUA bool server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if strings.Contains(r.Header.Get("User-Agent"), "Mozilla") { - sawBrowserUA = true + if r.Header.Get("User-Agent") == remoteSourceUserAgent { + sawProviderUA = true } w.Header().Set("Content-Disposition", `attachment; filename="remote-site.zip"`) w.Header().Set("Content-Type", "application/zip") @@ -59,12 +58,6 @@ func TestDownloadPagesPackageFromURLAllowsPrivateHost(t *testing.T) { })) t.Cleanup(server.Close) - req, err := http.NewRequest(http.MethodGet, server.URL+"/pkg.zip", nil) - require.NoError(t, err) - applyBrowserDownloadHeaders(req, "") - assert.Contains(t, req.Header.Get("User-Agent"), "Mozilla") - assert.Contains(t, req.Header.Get("Sec-Fetch-Mode"), "navigate") - tempPath, checksum, size, format, fileName, err := downloadPagesPackageFromURL( context.Background(), server.URL+"/pkg.zip", @@ -72,11 +65,11 @@ func TestDownloadPagesPackageFromURLAllowsPrivateHost(t *testing.T) { ) require.NoError(t, err) t.Cleanup(func() { _ = os.Remove(tempPath) }) - assert.True(t, sawBrowserUA) + assert.True(t, sawProviderUA) assert.NotEmpty(t, checksum) assert.Positive(t, size) assert.Equal(t, "zip", string(format)) - assert.Equal(t, "remote-site.zip", fileName) + assert.Equal(t, "pkg.zip", fileName) } func TestUploadDeploymentFromURLPrivateHost(t *testing.T) { diff --git a/internal/apps/openflare/pages/errs.go b/internal/apps/openflare/pages/errs.go index 602825a1..15b2c270 100644 --- a/internal/apps/openflare/pages/errs.go +++ b/internal/apps/openflare/pages/errs.go @@ -36,4 +36,20 @@ const ( errPagesInvalidSnapshotFormat = "配置快照格式无效" errPagesActorMissing = "无法识别当前用户" errPagesEntryFileMissing = "当前激活部署中不存在指定入口文件" + errPagesSourceNotFound = "pages 部署源不存在" + errPagesSourceTypeRequired = "请选择 pages 部署源类型" + errPagesSourceTypeUnsupported = "当前阶段仅支持远程地址部署源" + errPagesSourceRemoteFields = "远程地址来源不能包含 GitHub 或自动更新配置" + errPagesSourceRemoteURLRequired = "请提供远程部署包地址" + errPagesSourceRemoteURLMode = "remote_url_set 与 remote_url 参数不匹配" + errPagesSourceRemoteURLInvalid = "远程部署包地址无效，仅支持不含用户信息和片段的 http/https 地址" + errPagesSourceNetworkPolicy = "远程地址网络策略仅支持 public 或 trusted_internal" + errPagesSourceCheckUnsupported = "远程地址来源不支持检查更新，请使用立即同步" + errPagesSourceActionBusy = "pages 部署源任务正在执行" + errPagesSourceActionInvalid = "pages 部署源任务参数无效" + errPagesSourceActionStale = "pages 部署源配置已变化，本次任务已跳过" + errPagesSourceLeaseLost = "pages 部署源任务执行权已失效" + errPagesSourceSyncFailed = "pages 部署源同步失败" + errPagesSourceTaskDispatchFailed = "pages 部署源任务入队失败" + errPagesSourceInternal = "pages 部署源操作失败，请稍后重试" ) diff --git a/internal/apps/openflare/pages/helpers.go b/internal/apps/openflare/pages/helpers.go index 90db9996..8d1b2f69 100644 --- a/internal/apps/openflare/pages/helpers.go +++ b/internal/apps/openflare/pages/helpers.go @@ -38,6 +38,7 @@ const ( pagesIngestMarkerKey = "pages_ingest_marker" pagesIngestMarkerV2 = "pages_deployment_v2" pagesProjectIDMetadataKey = "pages_project_id" + pagesSourceIDMetadataKey = "pages_source_id" pagesMaxPathLength = 512 bytesPerMiB = 1024 * 1024 pagesExtractedSizeMultiplier = 4 @@ -263,10 +264,29 @@ func ingestPagesDeploymentPackage( projectID uint, fileName string, format pagesarchive.Format, +) (upload.IngestResult, error) { + return ingestPagesDeploymentPackageWithSource(ctx, localPath, checksum, projectID, 0, fileName, format) +} + +func ingestPagesDeploymentPackageWithSource( + ctx context.Context, + localPath string, + checksum string, + projectID uint, + sourceID uint, + fileName string, + format pagesarchive.Format, ) (upload.IngestResult, error) { systemUser := repository.GetSystemUser(ctx) accessMode := 0 extension := pagesarchive.NormalizeNameExtension(fileName, format) + extra := map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: strconv.FormatUint(uint64(projectID), 10), + } + if sourceID != 0 { + extra[pagesSourceIDMetadataKey] = strconv.FormatUint(uint64(sourceID), 10) + } return upload.IngestFromLocalPath(ctx, localPath, upload.IngestRequest{ UserID: systemUser.ID, FileName: fileName, @@ -278,10 +298,7 @@ func ingestPagesDeploymentPackage( SkipExtensionCheck: true, Policy: upload.PolicyDedupNewRecord, Metadata: model.UploadMetadata{ - Extra: map[string]any{ - pagesIngestMarkerKey: pagesIngestMarkerV2, - pagesProjectIDMetadataKey: strconv.FormatUint(uint64(projectID), 10), - }, + Extra: extra, }, }) } diff --git a/internal/apps/openflare/pages/logics.go b/internal/apps/openflare/pages/logics.go index 622564bc..3a0f0a49 100644 --- a/internal/apps/openflare/pages/logics.go +++ b/internal/apps/openflare/pages/logics.go @@ -60,6 +60,9 @@ type DeploymentView struct { FileCount int `json:"file_count"` TotalSize int64 `json:"total_size"` CreatedBy string `json:"created_by"` + SourceType string `json:"source_type"` + SourceLabel string `json:"source_label"` + TriggerType string `json:"trigger_type"` CreatedAt time.Time `json:"created_at"` ActivatedAt *time.Time `json:"activated_at"` } @@ -151,13 +154,14 @@ func UpdateProject(ctx context.Context, id uint, input Input) (*View, error) { if err != nil { return err } - if (existing.RootDir != project.RootDir || existing.EntryFile != project.EntryFile) && + contentConfigChanged := existing.RootDir != project.RootDir || existing.EntryFile != project.EntryFile + if contentConfigChanged && existing.ActiveDeploymentID != nil && *existing.ActiveDeploymentID != 0 { if err := ensureDeploymentEntry(tx, *existing.ActiveDeploymentID, project.RootDir, project.EntryFile); err != nil { return err } } - return tx.Model(&existing).Updates(map[string]any{ + updates := map[string]any{ "name": project.Name, "slug": project.Slug, "description": project.Description, @@ -170,7 +174,23 @@ func UpdateProject(ctx context.Context, id uint, input Input) (*View, error) { "api_proxy_rewrite": project.APIProxyRewrite, "root_dir": project.RootDir, "entry_file": project.EntryFile, - }).Error + } + if contentConfigChanged { + updates["content_config_version"] = existing.ContentConfigVersion + 1 + var source model.PagesProjectSource + sourceErr := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("project_id = ?", existing.ID). + First(&source).Error + if sourceErr != nil && !errors.Is(sourceErr, gorm.ErrRecordNotFound) { + return sourceErr + } + if sourceErr == nil { + if err := fenceAndNormalizeRuntime(tx, source.ID); err != nil { + return err + } + } + } + return tx.Model(&existing).Updates(updates).Error }) if err != nil { if isUniqueConstraintError(err) { @@ -221,6 +241,28 @@ func DeleteProject(ctx context.Context, id uint) error { return errors.New(errPagesDeleteReferenced) } } + var source model.PagesProjectSource + sourceErr := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("project_id = ?", project.ID). + First(&source).Error + if sourceErr != nil && !errors.Is(sourceErr, gorm.ErrRecordNotFound) { + return sourceErr + } + if sourceErr == nil { + var runtime model.PagesProjectSourceRuntime + runtimeErr := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", source.ID). + First(&runtime).Error + if runtimeErr != nil && !errors.Is(runtimeErr, gorm.ErrRecordNotFound) { + return runtimeErr + } + if err := tx.Where("source_id = ?", source.ID).Delete(&model.PagesProjectSourceRuntime{}).Error; err != nil { + return err + } + if err := tx.Delete(&source).Error; err != nil { + return err + } + } if err := tx.Where("project_id = ?", project.ID).Find(&deployments).Error; err != nil { return err } @@ -233,7 +275,7 @@ func DeleteProject(ctx context.Context, id uint) error { if err := tx.Where("project_id = ?", project.ID).Delete(&model.PagesDeployment{}).Error; err != nil { return err } - if err := tx.Delete(project).Error; err != nil { + if err := tx.Delete(&lockedProject).Error; err != nil { return err } return nil @@ -301,7 +343,18 @@ func UploadDeployment(ctx context.Context, projectID uint, fileHeader *multipart return nil, err } defer func() { _ = os.Remove(tempPath) }() - return createDeploymentFromTempPackage(ctx, project, tempPath, checksum, format, fileHeader.Filename, createdBy, limits) + return createDeploymentFromTempPackage( + ctx, + project, + tempPath, + checksum, + format, + fileHeader.Filename, + createdBy, + "manual_upload", + "manual_upload", + limits, + ) } // UploadFromURLInput is the request body for downloading a deployment package from a remote URL. @@ -321,7 +374,18 @@ func UploadDeploymentFromURL(ctx context.Context, projectID uint, rawURL string, return nil, err } defer func() { _ = os.Remove(tempPath) }() - return createDeploymentFromTempPackage(ctx, project, tempPath, checksum, format, fileName, createdBy, limits) + return createDeploymentFromTempPackage( + ctx, + project, + tempPath, + checksum, + format, + fileName, + createdBy, + "manual_url", + "manual_url", + limits, + ) } func createDeploymentFromTempPackage( @@ -332,6 +396,8 @@ func createDeploymentFromTempPackage( format pagesarchive.Format, fileName string, createdBy string, + sourceType string, + triggerType string, limits pagesLimits, ) (*DeploymentView, error) { if project == nil { @@ -377,6 +443,15 @@ func createDeploymentFromTempPackage( if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&lockedProject, project.ID).Error; err != nil { return err } + var uploadRecord model.Upload + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ?", ingestResult.Upload.ID). + First(&uploadRecord).Error; err != nil { + return errors.New(errPagesPackageUploadMissing) + } + if uploadRecord.Status != model.UploadStatusUsed || uploadRecord.Type != upload.ReservedPagesDeploymentType { + return errors.New(errPagesPackageUploadMissing) + } var maxNumber int if err := tx.Model(&model.PagesDeployment{}). Where("project_id = ?", project.ID). @@ -393,6 +468,9 @@ func createDeploymentFromTempPackage( FileCount: manifest.FileCount, TotalSize: manifest.TotalSize, CreatedBy: strings.TrimSpace(createdBy), + SourceType: sourceType, + SourceLabel: safeRemoteSourceLabel(fileName), + TriggerType: triggerType, } if err := tx.Create(deployment).Error; err != nil { return err @@ -582,72 +660,27 @@ func selectDeploymentsToPrune( // ActivateDeployment 激活 Pages 部署。 func ActivateDeployment(ctx context.Context, projectID uint, deploymentID uint) (*View, error) { - project, err := model.GetPagesProjectByID(ctx, projectID) + return ActivateDeploymentAs(ctx, projectID, deploymentID, "system:pages-manual-activation") +} + +// ActivateDeploymentAs activates a historical deployment and fences any +// configured source when the active deployment actually changes. +func ActivateDeploymentAs(ctx context.Context, projectID uint, deploymentID uint, actor string) (*View, error) { + if err := ensureActivationDeploymentUpload(ctx, projectID, deploymentID); err != nil { + return nil, err + } + audit, err := activateDeploymentTransaction(ctx, projectID, deploymentID, time.Now()) if err != nil { return nil, err } - deployment, err := model.GetPagesDeploymentByID(ctx, deploymentID) - if err != nil { - return nil, err - } - if deployment.ProjectID != project.ID { - return nil, errors.New(errPagesDeploymentMismatch) - } - if deployment.UploadID == 0 { - if err = ensureDeploymentUploadRecord(ctx, deployment); err != nil { - return nil, err - } - } - now := time.Now() - if err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { - var project model.PagesProject - if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { - return err - } - var deployment model.PagesDeployment - if err := tx.First(&deployment, deploymentID).Error; err != nil { - return err - } - if deployment.ProjectID != project.ID { - return errors.New(errPagesDeploymentMismatch) - } - rootDir, err := validateAndNormalizePagesRootDir(project.RootDir) - if err != nil { - return err - } - entryFile, err := validateAndNormalizePagesEntryFile(project.EntryFile) - if err != nil { - return err - } - if err := ensureDeploymentEntry(tx, deployment.ID, rootDir, entryFile); err != nil { - return err - } - var uploadRecord model.Upload - if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). - Where("id = ?", deployment.UploadID). - First(&uploadRecord).Error; err != nil { - return errors.New(errPagesPackageUploadMissing) - } - if uploadRecord.Status != model.UploadStatusUsed || uploadRecord.Type != upload.ReservedPagesDeploymentType { - return errors.New(errPagesPackageUploadMissing) - } - if err := tx.Model(&model.PagesDeployment{}). - Where("project_id = ?", project.ID). - Update("status", model.PagesDeploymentStatusUploaded).Error; err != nil { - return err - } - if err := tx.Model(&deployment).Updates(map[string]any{ - "status": model.PagesDeploymentStatusActive, - "activated_at": &now, - }).Error; err != nil { - return err - } - return tx.Model(&project).Updates(map[string]any{ - "active_deployment_id": deployment.ID, - }).Error - }); err != nil { - return nil, err + if audit.Noop { + return GetProject(ctx, projectID) } + logger.InfoF(ctx, + "[Pages] manual activation: actor=%s project_id=%d old_deployment_id=%d new_deployment_id=%d source_type=%s source_identity=%s auto_disabled=%t", + strings.TrimSpace(actor), projectID, audit.OldDeploymentID, deploymentID, + audit.SourceType, audit.SourceIdentity, audit.AutoDisabled, + ) limits := resolvePagesLimits(ctx) if pruneErr := pruneProjectDeploymentHistory(ctx, projectID, limits.HistoryCount, 0); pruneErr != nil { logger.ErrorF(ctx, @@ -658,6 +691,179 @@ func ActivateDeployment(ctx context.Context, projectID uint, deploymentID uint) return GetProject(ctx, projectID) } +type deploymentActivationAudit struct { + OldDeploymentID uint + SourceType string + SourceIdentity string + AutoDisabled bool + Noop bool +} + +type deploymentActivationSource struct { + Source *model.PagesProjectSource + Runtime *model.PagesProjectSourceRuntime +} + +func ensureActivationDeploymentUpload(ctx context.Context, projectID uint, deploymentID uint) error { + deployment, err := model.GetPagesDeploymentByID(ctx, deploymentID) + if err != nil { + return err + } + if deployment.ProjectID != projectID { + return errors.New(errPagesDeploymentMismatch) + } + if deployment.UploadID != 0 { + return nil + } + return ensureDeploymentUploadRecord(ctx, deployment) +} + +func activateDeploymentTransaction( + ctx context.Context, + projectID uint, + deploymentID uint, + now time.Time, +) (deploymentActivationAudit, error) { + audit := deploymentActivationAudit{} + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { + return err + } + if project.ActiveDeploymentID != nil { + audit.OldDeploymentID = *project.ActiveDeploymentID + } + if audit.OldDeploymentID == deploymentID { + audit.Noop = true + return nil + } + + sourceState, err := lockDeploymentActivationSource(tx, project.ID) + if err != nil { + return err + } + deployment, err := loadDeploymentActivationTarget(tx, &project, deploymentID) + if err != nil { + return err + } + if err := fenceDeploymentActivationSource(tx, sourceState, deployment, &audit); err != nil { + return err + } + return switchActiveDeploymentTx(tx, &project, deployment, now) + }) + return audit, err +} + +func lockDeploymentActivationSource(tx *gorm.DB, projectID uint) (*deploymentActivationSource, error) { + var source model.PagesProjectSource + err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("project_id = ?", projectID). + First(&source).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil, nil + } + if err != nil { + return nil, err + } + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", source.ID). + First(&runtime).Error; err != nil { + return nil, err + } + return &deploymentActivationSource{Source: &source, Runtime: &runtime}, nil +} + +func loadDeploymentActivationTarget( + tx *gorm.DB, + project *model.PagesProject, + deploymentID uint, +) (*model.PagesDeployment, error) { + var deployment model.PagesDeployment + if err := tx.First(&deployment, deploymentID).Error; err != nil { + return nil, err + } + if deployment.ProjectID != project.ID { + return nil, errors.New(errPagesDeploymentMismatch) + } + rootDir, err := validateAndNormalizePagesRootDir(project.RootDir) + if err != nil { + return nil, err + } + entryFile, err := validateAndNormalizePagesEntryFile(project.EntryFile) + if err != nil { + return nil, err + } + if err := ensureDeploymentEntry(tx, deployment.ID, rootDir, entryFile); err != nil { + return nil, err + } + var uploadRecord model.Upload + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ?", deployment.UploadID). + First(&uploadRecord).Error; err != nil { + return nil, errors.New(errPagesPackageUploadMissing) + } + if uploadRecord.Status != model.UploadStatusUsed || uploadRecord.Type != upload.ReservedPagesDeploymentType { + return nil, errors.New(errPagesPackageUploadMissing) + } + return &deployment, nil +} + +func fenceDeploymentActivationSource( + tx *gorm.DB, + state *deploymentActivationSource, + deployment *model.PagesDeployment, + audit *deploymentActivationAudit, +) error { + if state == nil { + return nil + } + audit.SourceType = state.Source.SourceType + audit.SourceIdentity = state.Source.SourceIdentity + audit.AutoDisabled = state.Source.AutoUpdateEnabled + if err := tx.Model(state.Source).Updates(map[string]any{ + "config_version": state.Source.ConfigVersion + 1, + "auto_update_enabled": false, + }).Error; err != nil { + return err + } + if deployment.SourceIdentity != nil && *deployment.SourceIdentity == state.Source.SourceIdentity && + deployment.SourceRevision != nil { + state.Runtime.LastAppliedRevision = *deployment.SourceRevision + state.Runtime.LastAppliedDetail = deployment.SourceMeta + } else { + state.Runtime.LastAppliedRevision = "" + state.Runtime.LastAppliedDetail = "" + } + return tx.Model(state.Runtime).Updates(map[string]any{ + "last_applied_revision": state.Runtime.LastAppliedRevision, + "last_applied_detail": state.Runtime.LastAppliedDetail, + sourceRuntimeColumnSyncStatus: normalizedSourceRuntimeStatus(state.Runtime), + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + }).Error +} + +func switchActiveDeploymentTx( + tx *gorm.DB, + project *model.PagesProject, + deployment *model.PagesDeployment, + now time.Time, +) error { + if err := tx.Model(&model.PagesDeployment{}). + Where("project_id = ?", project.ID). + Update("status", model.PagesDeploymentStatusUploaded).Error; err != nil { + return err + } + if err := tx.Model(deployment).Updates(map[string]any{ + "status": model.PagesDeploymentStatusActive, + "activated_at": &now, + }).Error; err != nil { + return err + } + return tx.Model(project).Update("active_deployment_id", deployment.ID).Error +} + // GetDeploymentPackageHash returns the upload SHA-256 hash of the deployment package. // Prefer GetProjectLatestPackageHash for Agent latest-pointer pulls. func GetDeploymentPackageHash(ctx context.Context, deploymentID uint) (string, error) { @@ -1104,7 +1310,7 @@ func buildProject(existing *model.PagesProject, input Input) (*model.PagesProjec return nil, errors.New(errPagesAPIProxyPassRequired) } parsedURL, err := url.Parse(apiProxyPass) - if err != nil || (parsedURL.Scheme != "http" && parsedURL.Scheme != "https") || parsedURL.Host == "" { + if err != nil || (parsedURL.Scheme != remoteSourceSchemeHTTP && parsedURL.Scheme != remoteSourceSchemeHTTPS) || parsedURL.Host == "" { return nil, errors.New(errPagesAPIProxyPassInvalid) } } @@ -1177,6 +1383,9 @@ func buildDeploymentView(deployment *model.PagesDeployment) DeploymentView { FileCount: deployment.FileCount, TotalSize: deployment.TotalSize, CreatedBy: deployment.CreatedBy, + SourceType: deployment.SourceType, + SourceLabel: deployment.SourceLabel, + TriggerType: deployment.TriggerType, CreatedAt: deployment.CreatedAt, ActivatedAt: deployment.ActivatedAt, } diff --git a/internal/apps/openflare/pages/logics_test.go b/internal/apps/openflare/pages/logics_test.go index 230f3352..346aba55 100644 --- a/internal/apps/openflare/pages/logics_test.go +++ b/internal/apps/openflare/pages/logics_test.go @@ -43,6 +43,8 @@ func setupPagesTestDB(t *testing.T) func() { &model.PagesProject{}, &model.PagesDeployment{}, &model.PagesDeploymentFile{}, + &model.PagesProjectSource{}, + &model.PagesProjectSourceRuntime{}, &model.ConfigVersion{}, &model.SystemConfig{}, )) diff --git a/internal/apps/openflare/pages/routers.go b/internal/apps/openflare/pages/routers.go index 6767f478..4ad4c862 100644 --- a/internal/apps/openflare/pages/routers.go +++ b/internal/apps/openflare/pages/routers.go @@ -4,7 +4,10 @@ package pages import ( + "encoding/json" + "errors" "fmt" + "io" "net/http" "strconv" @@ -12,7 +15,9 @@ import ( "github.com/Rain-kl/Wavelet/internal/apps/openflare/apiutil" "github.com/Rain-kl/Wavelet/internal/common/response" "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/pkg/logger" "github.com/gin-gonic/gin" + "gorm.io/gorm" ) func handleLogicError(c *gin.Context, err error) bool { @@ -22,6 +27,53 @@ func handleLogicError(c *gin.Context, err error) bool { return apiutil.AbortNotFoundIfMissing(c, err, errPagesProjectNotFound) } +func handleSourceLogicError(c *gin.Context, err error) bool { + if err == nil { + return false + } + if errors.Is(err, gorm.ErrRecordNotFound) || err.Error() == errPagesSourceNotFound { + response.AbortNotFound(c, errPagesSourceNotFound) + return true + } + switch err.Error() { + case errPagesSourceActionBusy: + response.AbortConflict(c, errPagesSourceActionBusy) + case errPagesSourceTypeRequired, + errPagesSourceTypeUnsupported, + errPagesSourceRemoteFields, + errPagesSourceRemoteURLRequired, + errPagesSourceRemoteURLMode, + errPagesSourceRemoteURLInvalid, + errPagesSourceNetworkPolicy, + errPagesSourceCheckUnsupported, + errPagesSourceActionInvalid: + response.AbortBadRequest(c, err.Error()) + case errPagesSourceTaskDispatchFailed: + response.AbortInternal(c, errPagesSourceInternal) + default: + logger.ErrorF(c.Request.Context(), "[PagesSource] API operation failed: error=%v", err) + response.AbortInternal(c, errPagesSourceInternal) + } + return true +} + +func decodeStrictJSON(c *gin.Context, target any, allowEmpty bool) bool { + decoder := json.NewDecoder(c.Request.Body) + decoder.DisallowUnknownFields() + if err := decoder.Decode(target); err != nil { + if allowEmpty && errors.Is(err, io.EOF) { + return true + } + response.AbortBadRequest(c, errPagesSourceActionInvalid) + return false + } + if err := ensureJSONEOF(decoder); err != nil { + response.AbortBadRequest(c, errPagesSourceActionInvalid) + return false + } + return true +} + func deploymentIDParam(c *gin.Context) (uint, bool) { raw := c.Param("deployment_id") if raw == "" { @@ -174,6 +226,164 @@ func DeleteProjectHandler(c *gin.Context) { c.JSON(http.StatusOK, response.OKNil()) } +// GetSourceHandler 获取 Pages 项目的部署源。 +// @Summary 获取 Pages 部署源 +// @Description 返回脱敏后的项目部署源配置与运行状态，需要管理员权限 +// @Tags openflare-pages +// @Produce json +// @Security SessionCookie +// @Param id path int true "项目 ID" +// @Success 200 {object} response.Any{data=pages.SourceView} "部署源" +// @Failure 400 {object} response.Any "参数错误" +// @Failure 401 {object} response.Any "未登录" +// @Failure 404 {object} response.Any "项目或部署源不存在" +// @Failure 500 {object} response.Any "内部错误" +// @Router /api/v1/d/pages/{id}/source [get] +func GetSourceHandler(c *gin.Context) { + projectID, ok := apiutil.IDParam(c) + if !ok { + return + } + source, err := GetSource(c.Request.Context(), projectID) + if handleSourceLogicError(c, err) { + return + } + c.JSON(http.StatusOK, response.OK(source)) +} + +// UpdateSourceHandler 创建或更新 Pages 项目部署源。 +// @Summary 更新 Pages 部署源 +// @Description Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回 +// @Tags openflare-pages +// @Accept json +// @Produce json +// @Security SessionCookie +// @Param id path int true "项目 ID" +// @Param request body pages.SourceUpdateInput true "部署源配置" +// @Success 200 {object} response.Any{data=pages.SourceUpdateResult} "更新结果" +// @Failure 400 {object} response.Any "配置无效" +// @Failure 401 {object} response.Any "未登录" +// @Failure 404 {object} response.Any "项目不存在" +// @Failure 500 {object} response.Any "内部错误" +// @Router /api/v1/d/pages/{id}/source/update [post] +func UpdateSourceHandler(c *gin.Context) { + projectID, ok := apiutil.IDParam(c) + if !ok { + return + } + var input SourceUpdateInput + if !decodeStrictJSON(c, &input, false) { + return + } + result, err := UpdateSource(c.Request.Context(), projectID, input) + if handleSourceLogicError(c, err) { + return + } + c.JSON(http.StatusOK, response.OK(result)) +} + +// DeleteSourceHandler 将 Pages 项目切换回手动部署模式。 +// @Summary 删除 Pages 部署源 +// @Description 幂等删除持久部署源；已有部署历史与当前生产部署保持不变 +// @Tags openflare-pages +// @Produce json +// @Security SessionCookie +// @Param id path int true "项目 ID" +// @Success 200 {object} response.Any{data=pages.SourceView} "手动来源视图" +// @Failure 400 {object} response.Any "参数错误" +// @Failure 401 {object} response.Any "未登录" +// @Failure 404 {object} response.Any "项目不存在" +// @Failure 500 {object} response.Any "内部错误" +// @Router /api/v1/d/pages/{id}/source/delete [post] +func DeleteSourceHandler(c *gin.Context) { + projectID, ok := apiutil.IDParam(c) + if !ok { + return + } + source, err := DeleteSource(c.Request.Context(), projectID) + if handleSourceLogicError(c, err) { + return + } + c.JSON(http.StatusOK, response.OK(source)) +} + +// CheckSourceHandler 请求检查 Pages 部署源。 +// @Summary 检查 Pages 部署源 +// @Description Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约 +// @Tags openflare-pages +// @Produce json +// @Security SessionCookie +// @Param id path int true "项目 ID" +// @Success 200 {object} response.Any{data=pages.SourceActionReceipt} "任务回执" +// @Failure 400 {object} response.Any "当前来源不支持检查" +// @Failure 401 {object} response.Any "未登录" +// @Failure 404 {object} response.Any "部署源不存在" +// @Failure 409 {object} response.Any "来源任务正在执行" +// @Failure 500 {object} response.Any "内部错误" +// @Router /api/v1/d/pages/{id}/source/check [post] +func CheckSourceHandler(c *gin.Context) { + projectID, ok := apiutil.IDParam(c) + if !ok { + return + } + actor, ok := currentPagesActor(c) + if !ok { + return + } + receipt, err := DispatchSourceAction(c.Request.Context(), projectID, sourceActionCheck, actor, "") + if handleSourceLogicError(c, err) { + return + } + c.JSON(http.StatusOK, response.OK(receipt)) +} + +// SourceSyncInput is the optional source sync action payload. +type SourceSyncInput struct { + ConfirmedRevision string `json:"confirmed_revision"` +} + +// SyncSourceHandler 请求同步并发布 Pages 部署源。 +// @Summary 同步并发布 Pages 部署源 +// @Description 异步下载、校验并原子激活来源部署包；空请求体与空 JSON 对象均有效 +// @Tags openflare-pages +// @Accept json +// @Produce json +// @Security SessionCookie +// @Param id path int true "项目 ID" +// @Param request body pages.SourceSyncInput false "同步参数" +// @Success 200 {object} response.Any{data=pages.SourceActionReceipt} "任务回执" +// @Failure 400 {object} response.Any "参数或来源类型无效" +// @Failure 401 {object} response.Any "未登录" +// @Failure 404 {object} response.Any "部署源不存在" +// @Failure 409 {object} response.Any "来源任务正在执行" +// @Failure 500 {object} response.Any "内部错误" +// @Router /api/v1/d/pages/{id}/source/sync [post] +func SyncSourceHandler(c *gin.Context) { + projectID, ok := apiutil.IDParam(c) + if !ok { + return + } + var input SourceSyncInput + if !decodeStrictJSON(c, &input, true) { + return + } + actor, ok := currentPagesActor(c) + if !ok { + return + } + receipt, err := DispatchSourceAction( + c.Request.Context(), + projectID, + sourceActionSync, + actor, + input.ConfirmedRevision, + ) + if handleSourceLogicError(c, err) { + return + } + c.JSON(http.StatusOK, response.OK(receipt)) +} + // ListDeploymentsHandler 列出项目的全部部署。 // @Summary 列出 Pages 部署 // @Description 返回指定项目的全部部署记录，需要管理员权限 @@ -239,7 +449,8 @@ func UploadDeploymentHandler(c *gin.Context) { // UploadDeploymentFromURLHandler 从 URL 下载并创建 Pages 部署。 // @Summary 从 URL 导入 Pages 部署包 -// @Description 从用户提供的 HTTP(S) 链接下载部署包并创建部署记录；服务端使用浏览器伪装请求头拉取，允许内网地址与不安全 TLS 证书，需要管理员权限 +// @Description 已弃用的一次性 URL 导入；使用 trusted_internal 策略兼容内网与自签名证书，不创建持久部署源 +// @Deprecated // @Tags openflare-pages // @Accept json // @Produce json @@ -298,7 +509,11 @@ func ActivateDeploymentHandler(c *gin.Context) { if !ok { return } - project, err := ActivateDeployment(c.Request.Context(), projectID, deploymentID) + actor, ok := currentPagesActor(c) + if !ok { + return + } + project, err := ActivateDeploymentAs(c.Request.Context(), projectID, deploymentID, actor) if handleLogicError(c, err) { return } diff --git a/internal/apps/openflare/pages/routers_source_test.go b/internal/apps/openflare/pages/routers_source_test.go new file mode 100644 index 00000000..6bd96346 --- /dev/null +++ b/internal/apps/openflare/pages/routers_source_test.go @@ -0,0 +1,224 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "bytes" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/apps/oauth" + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/task" + "github.com/Rain-kl/Wavelet/internal/testhelper" + "github.com/alicebob/miniredis/v2" + "github.com/gin-gonic/gin" + "github.com/hibiken/asynq" + "github.com/redis/go-redis/v9" +) + +type sourceHandlerEnvelope struct { + ErrorMsg string `json:"error_msg"` + Data json.RawMessage `json:"data"` +} + +func newPagesSourceTestRouter(userID uint64) *gin.Engine { + router := testhelper.NewTestGinEngine(func(ctx *gin.Context) { + oauth.SetToContext(ctx, oauth.UserObjKey, &model.User{ID: userID}) + ctx.Next() + }) + router.GET("/api/v1/d/pages/:id/source", GetSourceHandler) + router.POST("/api/v1/d/pages/:id/source/update", UpdateSourceHandler) + router.POST("/api/v1/d/pages/:id/source/delete", DeleteSourceHandler) + router.POST("/api/v1/d/pages/:id/source/check", CheckSourceHandler) + router.POST("/api/v1/d/pages/:id/source/sync", SyncSourceHandler) + return router +} + +func performPagesSourceRequest( + t *testing.T, + router http.Handler, + method string, + path string, + body []byte, +) (int, sourceHandlerEnvelope) { + t.Helper() + request := httptest.NewRequest(method, path, bytes.NewReader(body)) + if body != nil { + request.Header.Set("Content-Type", "application/json") + } + recorder := httptest.NewRecorder() + router.ServeHTTP(recorder, request) + var envelope sourceHandlerEnvelope + if err := json.Unmarshal(recorder.Body.Bytes(), &envelope); err != nil { + t.Fatalf("json.Unmarshal(%s %s response %q) error = %v, want nil", method, path, recorder.Body.String(), err) + } + return recorder.Code, envelope +} + +func setupPagesSourceDispatchTest(t *testing.T) { + t.Helper() + miniRedis, err := miniredis.Run() + if err != nil { + t.Fatalf("miniredis.Run() error = %v, want nil", err) + } + redisClient := redis.NewClient(&redis.Options{Addr: miniRedis.Addr()}) + asynqClient := asynq.NewClient(asynq.RedisClientOpt{Addr: miniRedis.Addr()}) + previousRedis := db.Redis + previousAsynqClient := task.AsynqClient + db.Redis = redisClient + task.AsynqClient = asynqClient + task.RegisterTaskMeta(PagesSourceActionMeta) + t.Cleanup(func() { + _ = asynqClient.Close() + _ = redisClient.Close() + miniRedis.Close() + task.AsynqClient = previousAsynqClient + db.Redis = previousRedis + }) +} + +func TestPagesSourceHandlersReturnStableActionErrors(t *testing.T) { + ctx := setupPagesSourceTest(t) + router := newPagesSourceTestRouter(42) + + manualProject := mustCreatePagesSourceProject(t, ctx, "handler-no-source") + code, envelope := performPagesSourceRequest( + t, + router, + http.MethodPost, + fmt.Sprintf("/api/v1/d/pages/%d/source/sync", manualProject.ID), + nil, + ) + if got, want := code, http.StatusNotFound; got != want { + t.Errorf("POST source/sync without source status = %d, want %d", got, want) + } + if got, want := envelope.ErrorMsg, errPagesSourceNotFound; got != want { + t.Errorf("POST source/sync without source error = %q, want %q", got, want) + } + + remoteProject := mustCreatePagesSourceProject(t, ctx, "handler-check") + _, _ = mustConfigureRemoteSource( + t, + ctx, + remoteProject.ID, + "https://example.com/site.zip?token=handler-secret", + RemoteNetworkPolicyPublic, + ) + code, envelope = performPagesSourceRequest( + t, + router, + http.MethodPost, + fmt.Sprintf("/api/v1/d/pages/%d/source/check", remoteProject.ID), + nil, + ) + if got, want := code, http.StatusBadRequest; got != want { + t.Errorf("POST remote source/check status = %d, want %d", got, want) + } + if got, want := envelope.ErrorMsg, errPagesSourceCheckUnsupported; got != want { + t.Errorf("POST remote source/check error = %q, want %q", got, want) + } + if strings.Contains(string(envelope.Data), "handler-secret") || strings.Contains(envelope.ErrorMsg, "handler-secret") { + t.Errorf("POST remote source/check response = %+v, want no URL secret", envelope) + } + + busyProject := mustCreatePagesSourceProject(t, ctx, "handler-busy") + busySource, _ := mustConfigureRemoteSource( + t, + ctx, + busyProject.ID, + "https://example.com/site.zip", + RemoteNetworkPolicyPublic, + ) + future := time.Now().Add(time.Minute) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", busySource.ID). + Updates(map[string]any{ + "sync_status": pagesSourceStatusSyncing, + "lease_token": "busy-owner", + "lease_expires_at": &future, + }).Error; err != nil { + t.Fatalf("seed busy source runtime error = %v, want nil", err) + } + code, envelope = performPagesSourceRequest( + t, + router, + http.MethodPost, + fmt.Sprintf("/api/v1/d/pages/%d/source/sync", busyProject.ID), + []byte(`{}`), + ) + if got, want := code, http.StatusConflict; got != want { + t.Errorf("POST busy source/sync status = %d, want %d", got, want) + } + if got, want := envelope.ErrorMsg, errPagesSourceActionBusy; got != want { + t.Errorf("POST busy source/sync error = %q, want %q", got, want) + } +} + +func TestSyncSourceHandlerAcceptsEmptyBodyAndEmptyObject(t *testing.T) { + ctx := setupPagesSourceTest(t) + setupPagesSourceDispatchTest(t) + router := newPagesSourceTestRouter(77) + project := mustCreatePagesSourceProject(t, ctx, "handler-empty-sync") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip?token=dispatch-secret", + RemoteNetworkPolicyPublic, + ) + path := fmt.Sprintf("/api/v1/d/pages/%d/source/sync", project.ID) + + for _, test := range []struct { + name string + body []byte + }{ + {name: "empty body", body: nil}, + {name: "empty object", body: []byte(`{}`)}, + } { + t.Run(test.name, func(t *testing.T) { + code, envelope := performPagesSourceRequest(t, router, http.MethodPost, path, test.body) + if got, want := code, http.StatusOK; got != want { + t.Fatalf("POST source/sync (%s) status = %d, want %d; error=%q", test.name, got, want, envelope.ErrorMsg) + } + if envelope.ErrorMsg != "" { + t.Errorf("POST source/sync (%s) error = %q, want empty", test.name, envelope.ErrorMsg) + } + var receipt SourceActionReceipt + if err := json.Unmarshal(envelope.Data, &receipt); err != nil { + t.Fatalf("json.Unmarshal(source/sync %s receipt) error = %v, want nil", test.name, err) + } + if receipt.TaskID == "" || receipt.ExecutionID == "" || receipt.Action != sourceActionSync { + t.Errorf("POST source/sync (%s) receipt = %+v, want task/execution IDs and action %q", test.name, receipt, sourceActionSync) + } + }) + } + + var executions []model.TaskExecution + if err := db.DB(ctx).Where("task_type = ?", PagesSourceActionTask).Order("id asc").Find(&executions).Error; err != nil { + t.Fatalf("list Pages source task executions error = %v, want nil", err) + } + if got, want := len(executions), 2; got != want { + t.Fatalf("Pages source task execution count = %d, want %d", got, want) + } + for _, execution := range executions { + if strings.Contains(execution.Payload, "dispatch-secret") || strings.Contains(execution.Payload, "http") { + t.Errorf("task execution %q payload = %s, want no Remote URL secret", execution.TaskID, execution.Payload) + } + var payload SourceActionPayload + if err := json.Unmarshal([]byte(execution.Payload), &payload); err != nil { + t.Errorf("json.Unmarshal(task execution %q payload) error = %v, want nil", execution.TaskID, err) + continue + } + if payload.SourceID != source.ID || payload.ConfigVersion != source.ConfigVersion || payload.Actor != "user:77" { + t.Errorf("task execution %q payload = %+v, want source=%d config=%d actor=user:77", execution.TaskID, payload, source.ID, source.ConfigVersion) + } + } +} diff --git a/internal/apps/openflare/pages/source.go b/internal/apps/openflare/pages/source.go new file mode 100644 index 00000000..9bc46217 --- /dev/null +++ b/internal/apps/openflare/pages/source.go @@ -0,0 +1,495 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "net" + "net/url" + "path" + "strings" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +const ( + // PagesSourceTypeManual represents projects without a persisted source row. + PagesSourceTypeManual = "manual" + // PagesSourceTypeRemoteURL represents a persisted artifact URL. + PagesSourceTypeRemoteURL = "remote_url" + // PagesSourceTypeGitHubRelease is reserved for Phase 2. + PagesSourceTypeGitHubRelease = "github_release" + + pagesSourceStatusIdle = "idle" + pagesSourceStatusChecking = "checking" + pagesSourceStatusUpdateAvailable = "update_available" + pagesSourceStatusSyncing = "syncing" + pagesSourceStatusFailed = "failed" + pagesSourceStatusAttention = "attention" + + defaultRemoteAssetLabel = "pages-package" +) + +// SourceUpdateInput is the discriminated source configuration payload. +// GitHub fields are accepted by the decoder so mode-incompatible values can be +// rejected deterministically; GitHub itself is enabled in Phase 2. +type SourceUpdateInput struct { + SourceType string `json:"source_type"` + RemoteURLSet bool `json:"remote_url_set"` + RemoteURL string `json:"remote_url"` + RemoteNetworkPolicy string `json:"remote_network_policy"` + RepositoryURL string `json:"repository_url"` + ReleaseSelector string `json:"release_selector"` + ReleaseTag string `json:"release_tag"` + AssetName string `json:"asset_name"` + AutoUpdateEnabled bool `json:"auto_update_enabled"` + CheckIntervalMinutes int `json:"check_interval_minutes"` +} + +// SourceRevisionView is a credential-free source cursor. +type SourceRevisionView struct { + Revision string `json:"revision"` + Label string `json:"label"` + AssetName string `json:"asset_name,omitempty"` +} + +// SourceView is the safe discriminated source view returned to the console. +type SourceView struct { + SourceType string `json:"source_type"` + HasRemoteURL bool `json:"has_remote_url,omitempty"` + DisplayURL string `json:"display_url,omitempty"` + RemoteNetworkPolicy string `json:"remote_network_policy,omitempty"` + GitHubRepository string `json:"github_repository,omitempty"` + ReleaseSelector string `json:"release_selector,omitempty"` + ReleaseTag string `json:"release_tag,omitempty"` + AssetName string `json:"asset_name,omitempty"` + AutoUpdateEnabled bool `json:"auto_update_enabled,omitempty"` + CheckIntervalMinutes int `json:"check_interval_minutes,omitempty"` + SyncStatus string `json:"sync_status,omitempty"` + UpdateAvailable bool `json:"update_available,omitempty"` + LastSeen *SourceRevisionView `json:"last_seen,omitempty"` + LastApplied *SourceRevisionView `json:"last_applied,omitempty"` + LastCheckedAt *time.Time `json:"last_checked_at,omitempty"` + LastSyncedAt *time.Time `json:"last_synced_at,omitempty"` + NextCheckAt *time.Time `json:"next_check_at,omitempty"` + LastError string `json:"last_error,omitempty"` +} + +// SourceActionReceipt identifies the internal task execution created by an action API. +type SourceActionReceipt struct { + TaskID string `json:"task_id"` + ExecutionID string `json:"execution_id"` + Action string `json:"action"` +} + +// SourceUpdateResult is returned after persisting source configuration. +type SourceUpdateResult struct { + Source *SourceView `json:"source"` + CheckTask *SourceActionReceipt `json:"check_task"` + Warning string `json:"warning"` +} + +type sourceDetail struct { + Provider string `json:"provider"` + Label string `json:"label"` + AssetName string `json:"asset_name,omitempty"` + ReleaseID string `json:"release_id,omitempty"` +} + +type remoteSourceConfig struct { + URL string + Policy string + Identity string +} + +// GetSource returns the current persisted source or a manual discriminator. +func GetSource(ctx context.Context, projectID uint) (*SourceView, error) { + if _, err := model.GetPagesProjectByID(ctx, projectID); err != nil { + return nil, err + } + source, runtime, err := loadSourceByProject(ctx, projectID) + if errors.Is(err, gorm.ErrRecordNotFound) { + return &SourceView{SourceType: PagesSourceTypeManual}, nil + } + if err != nil { + return nil, err + } + return buildSourceView(source, runtime) +} + +// UpdateSource creates or updates a Remote URL source and its 1:1 runtime row. +func UpdateSource(ctx context.Context, projectID uint, input SourceUpdateInput) (*SourceUpdateResult, error) { + if err := validateRemoteSourceInput(input); err != nil { + return nil, err + } + + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + return updateRemoteSourceTx(tx, projectID, input) + }) + if err != nil { + return nil, err + } + + view, err := GetSource(ctx, projectID) + if err != nil { + return nil, err + } + return &SourceUpdateResult{Source: view, Warning: ""}, nil +} + +func updateRemoteSourceTx(tx *gorm.DB, projectID uint, input SourceUpdateInput) error { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { + return err + } + existing, hasExisting, err := loadProjectSourceForUpdate(tx, projectID) + if err != nil { + return err + } + config, err := buildRemoteSourceConfig(existing, hasExisting, input) + if err != nil { + return err + } + if !hasExisting { + return createRemoteSourceTx(tx, projectID, config) + } + return updateExistingRemoteSourceTx(tx, existing, config) +} + +func loadProjectSourceForUpdate(tx *gorm.DB, projectID uint) (*model.PagesProjectSource, bool, error) { + var source model.PagesProjectSource + err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("project_id = ?", projectID). + First(&source).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + return &source, false, nil + } + if err != nil { + return nil, false, err + } + return &source, true, nil +} + +func buildRemoteSourceConfig( + existing *model.PagesProjectSource, + hasExisting bool, + input SourceUpdateInput, +) (remoteSourceConfig, error) { + remoteURL, err := resolveUpdatedRemoteURL(existing, hasExisting, input) + if err != nil { + return remoteSourceConfig{}, err + } + parsedURL, err := parseRemoteSourceURL(remoteURL) + if err != nil { + return remoteSourceConfig{}, err + } + policy := strings.TrimSpace(input.RemoteNetworkPolicy) + if policy == "" { + policy = RemoteNetworkPolicyPublic + } + return remoteSourceConfig{ + URL: remoteURL, + Policy: policy, + Identity: remoteSourceIdentity(parsedURL), + }, nil +} + +func createRemoteSourceTx(tx *gorm.DB, projectID uint, config remoteSourceConfig) error { + source := &model.PagesProjectSource{ + ProjectID: projectID, + SourceType: PagesSourceTypeRemoteURL, + RemoteURL: config.URL, + RemoteNetworkPolicy: config.Policy, + AutoUpdateEnabled: false, + CheckIntervalMinutes: 0, + ConfigVersion: 1, + SourceIdentity: config.Identity, + } + if err := tx.Create(source).Error; err != nil { + return err + } + return tx.Create(&model.PagesProjectSourceRuntime{ + SourceID: source.ID, + SyncStatus: pagesSourceStatusIdle, + }).Error +} + +func updateExistingRemoteSourceTx( + tx *gorm.DB, + existing *model.PagesProjectSource, + config remoteSourceConfig, +) error { + if !remoteSourceConfigChanged(existing, config) { + return nil + } + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", existing.ID). + First(&runtime).Error; err != nil { + return err + } + identityChanged := existing.SourceIdentity != config.Identity + if err := tx.Model(existing).Updates(map[string]any{ + "source_type": PagesSourceTypeRemoteURL, + "remote_url": config.URL, + "remote_network_policy": config.Policy, + "github_repository": "", + "release_selector": "", + "release_tag": "", + "asset_name": "", + "auto_update_enabled": false, + "check_interval_minutes": 0, + "config_version": existing.ConfigVersion + 1, + "source_identity": config.Identity, + }).Error; err != nil { + return err + } + return resetRuntimeAfterSourceUpdate(tx, &runtime, identityChanged) +} + +func remoteSourceConfigChanged(existing *model.PagesProjectSource, config remoteSourceConfig) bool { + return existing.SourceType != PagesSourceTypeRemoteURL || + existing.RemoteURL != config.URL || + existing.RemoteNetworkPolicy != config.Policy || + existing.GitHubRepository != "" || + existing.ReleaseSelector != "" || + existing.ReleaseTag != "" || + existing.AssetName != "" || + existing.AutoUpdateEnabled || + existing.CheckIntervalMinutes != 0 +} + +// DeleteSource idempotently switches a project back to manual mode. +func DeleteSource(ctx context.Context, projectID uint) (*SourceView, error) { + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { + return err + } + var source model.PagesProjectSource + err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("project_id = ?", projectID). + First(&source).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil + } + if err != nil { + return err + } + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", source.ID). + First(&runtime).Error; err != nil && !errors.Is(err, gorm.ErrRecordNotFound) { + return err + } + if err := tx.Where("source_id = ?", source.ID).Delete(&model.PagesProjectSourceRuntime{}).Error; err != nil { + return err + } + return tx.Delete(&source).Error + }) + if err != nil { + return nil, err + } + return &SourceView{SourceType: PagesSourceTypeManual}, nil +} + +func validateRemoteSourceInput(input SourceUpdateInput) error { + sourceType := strings.TrimSpace(input.SourceType) + if sourceType == "" { + return errors.New(errPagesSourceTypeRequired) + } + if sourceType != PagesSourceTypeRemoteURL { + return errors.New(errPagesSourceTypeUnsupported) + } + if strings.TrimSpace(input.RepositoryURL) != "" || strings.TrimSpace(input.ReleaseSelector) != "" || + strings.TrimSpace(input.ReleaseTag) != "" || strings.TrimSpace(input.AssetName) != "" || + input.AutoUpdateEnabled || input.CheckIntervalMinutes != 0 { + return errors.New(errPagesSourceRemoteFields) + } + policy := strings.TrimSpace(input.RemoteNetworkPolicy) + if policy != "" && policy != RemoteNetworkPolicyPublic && policy != RemoteNetworkPolicyTrustedInternal { + return errors.New(errPagesSourceNetworkPolicy) + } + if !input.RemoteURLSet && strings.TrimSpace(input.RemoteURL) != "" { + return errors.New(errPagesSourceRemoteURLMode) + } + if input.RemoteURLSet && strings.TrimSpace(input.RemoteURL) == "" { + return errors.New(errPagesSourceRemoteURLRequired) + } + return nil +} + +func resolveUpdatedRemoteURL(existing *model.PagesProjectSource, hasExisting bool, input SourceUpdateInput) (string, error) { + if input.RemoteURLSet { + return strings.TrimSpace(input.RemoteURL), nil + } + if !hasExisting || existing.SourceType != PagesSourceTypeRemoteURL || strings.TrimSpace(existing.RemoteURL) == "" { + return "", errors.New(errPagesSourceRemoteURLRequired) + } + return existing.RemoteURL, nil +} + +func parseRemoteSourceURL(raw string) (*url.URL, error) { + parsed, err := url.Parse(strings.TrimSpace(raw)) + if err != nil || parsed.Host == "" || parsed.User != nil || parsed.Fragment != "" { + return nil, errors.New(errPagesSourceRemoteURLInvalid) + } + parsed.Scheme = strings.ToLower(parsed.Scheme) + if parsed.Scheme != remoteSourceSchemeHTTP && parsed.Scheme != remoteSourceSchemeHTTPS { + return nil, errors.New(errPagesSourceRemoteURLInvalid) + } + if strings.TrimSpace(parsed.Hostname()) == "" { + return nil, errors.New(errPagesSourceRemoteURLInvalid) + } + return parsed, nil +} + +func remoteSourceIdentity(parsed *url.URL) string { + hostname := strings.ToLower(parsed.Hostname()) + port := parsed.Port() + if (parsed.Scheme == "https" && port == "443") || (parsed.Scheme == "http" && port == "80") { + port = "" + } + host := hostname + if port != "" { + host = net.JoinHostPort(hostname, port) + } else if strings.Contains(hostname, ":") { + host = "[" + hostname + "]" + } + canonicalPath := parsed.EscapedPath() + if canonicalPath == "" { + canonicalPath = "/" + } + canonicalPath = path.Clean("/" + strings.TrimPrefix(canonicalPath, "/")) + canonical := parsed.Scheme + "://" + host + canonicalPath + sum := sha256.Sum256([]byte("remote_url|" + canonical)) + return hex.EncodeToString(sum[:]) +} + +func displayRemoteSourceURL(raw string) string { + parsed, err := parseRemoteSourceURL(raw) + if err != nil { + return "" + } + hadQuery := parsed.RawQuery != "" + parsed.RawQuery = "" + display := parsed.String() + if hadQuery { + display += "?***" + } + return display +} + +func loadSourceByProject(ctx context.Context, projectID uint) (*model.PagesProjectSource, *model.PagesProjectSourceRuntime, error) { + var source model.PagesProjectSource + if err := db.DB(ctx).Where("project_id = ?", projectID).First(&source).Error; err != nil { + return nil, nil, err + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&runtime).Error; err != nil { + return nil, nil, err + } + return &source, &runtime, nil +} + +func buildSourceView(source *model.PagesProjectSource, runtime *model.PagesProjectSourceRuntime) (*SourceView, error) { + if source == nil || runtime == nil { + return nil, errors.New(errPagesSourceNotFound) + } + view := &SourceView{ + SourceType: source.SourceType, + SyncStatus: runtime.SyncStatus, + LastSyncedAt: runtime.LastSyncedAt, + LastError: runtime.LastError, + } + if runtime.LastAppliedRevision != "" { + view.LastApplied = revisionView(runtime.LastAppliedRevision, runtime.LastAppliedDetail) + } + switch source.SourceType { + case PagesSourceTypeRemoteURL: + view.HasRemoteURL = strings.TrimSpace(source.RemoteURL) != "" + view.DisplayURL = displayRemoteSourceURL(source.RemoteURL) + view.RemoteNetworkPolicy = source.RemoteNetworkPolicy + case PagesSourceTypeGitHubRelease: + view.LastCheckedAt = runtime.LastCheckedAt + view.NextCheckAt = runtime.NextCheckAt + view.UpdateAvailable = runtime.LastSeenRevision != "" && runtime.LastSeenRevision != runtime.LastAppliedRevision + if runtime.LastSeenRevision != "" { + view.LastSeen = revisionView(runtime.LastSeenRevision, runtime.LastSeenDetail) + } + view.GitHubRepository = source.GitHubRepository + view.ReleaseSelector = source.ReleaseSelector + view.ReleaseTag = source.ReleaseTag + view.AssetName = source.AssetName + view.AutoUpdateEnabled = source.AutoUpdateEnabled + view.CheckIntervalMinutes = source.CheckIntervalMinutes + default: + return nil, errors.New(errPagesSourceTypeUnsupported) + } + return view, nil +} + +func revisionView(revision string, detailJSON string) *SourceRevisionView { + detail := sourceDetail{} + _ = unmarshalSourceDetail(detailJSON, &detail) + label := strings.TrimSpace(detail.Label) + if label == "" { + label = defaultRemoteAssetLabel + } + return &SourceRevisionView{ + Revision: revision, + Label: label, + AssetName: detail.AssetName, + } +} + +func unmarshalSourceDetail(raw string, detail *sourceDetail) error { + if detail == nil || strings.TrimSpace(raw) == "" { + return nil + } + return json.Unmarshal([]byte(raw), detail) +} + +func resetRuntimeAfterSourceUpdate(tx *gorm.DB, runtime *model.PagesProjectSourceRuntime, identityChanged bool) error { + updates := map[string]any{ + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + sourceRuntimeColumnLastError: "", + } + if identityChanged { + updates["etag"] = "" + updates["last_seen_revision"] = "" + updates["last_seen_detail"] = "" + updates["last_applied_revision"] = "" + updates["last_applied_detail"] = "" + updates["last_checked_at"] = nil + updates["last_synced_at"] = nil + updates["next_check_at"] = nil + updates[sourceRuntimeColumnSyncStatus] = pagesSourceStatusIdle + } else { + updates[sourceRuntimeColumnSyncStatus] = normalizedSourceRuntimeStatus(runtime) + } + return tx.Model(runtime).Updates(updates).Error +} + +func normalizedSourceRuntimeStatus(runtime *model.PagesProjectSourceRuntime) string { + if runtime == nil { + return pagesSourceStatusIdle + } + if sourceHasSameReleaseReplacement(runtime) { + return pagesSourceStatusAttention + } + if runtime.LastSeenRevision != "" && runtime.LastSeenRevision != runtime.LastAppliedRevision { + return pagesSourceStatusUpdateAvailable + } + return pagesSourceStatusIdle +} diff --git a/internal/apps/openflare/pages/source_provider.go b/internal/apps/openflare/pages/source_provider.go new file mode 100644 index 00000000..77a242a5 --- /dev/null +++ b/internal/apps/openflare/pages/source_provider.go @@ -0,0 +1,549 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "crypto/tls" + "encoding/hex" + "errors" + "fmt" + "io" + "math" + "net" + "net/http" + "net/netip" + "net/url" + "os" + "path" + "strings" + "time" + + "github.com/Rain-kl/Wavelet/pkg/httppool" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" +) + +const ( + // RemoteNetworkPolicyPublic only permits publicly routable targets and + // performs DNS validation again for every connection. + RemoteNetworkPolicyPublic = "public" + // RemoteNetworkPolicyTrustedInternal permits private targets and self-signed + // TLS certificates. It is an explicit administrator trust boundary. + RemoteNetworkPolicyTrustedInternal = "trusted_internal" + + remoteSourceDownloadTimeout = 10 * time.Minute + remoteSourceResponseHeaderTimeout = 30 * time.Second + remoteSourceDialTimeout = 30 * time.Second + remoteSourceDialKeepAlive = 30 * time.Second + remoteSourceMaxRedirects = 5 + remoteSourceMagicSniffBytes = 512 + remoteSourceMaxSafeLabelBytes = 255 + remoteSourceFallbackLabel = "package" + remoteSourceUserAgent = "OpenFlare Pages Source/2" + remoteSourceSchemeHTTP = "http" + remoteSourceSchemeHTTPS = "https" +) + +type remoteProviderError string + +func (providerError remoteProviderError) Error() string { + return string(providerError) +} + +const ( + errRemoteProviderInvalidPolicy remoteProviderError = "远程来源网络策略无效" + errRemoteProviderInvalidLimit remoteProviderError = "远程来源部署包大小限制无效" + errRemoteProviderBlockedAddress remoteProviderError = "远程来源 public 策略禁止访问非公网地址" + errRemoteProviderResolveFailed remoteProviderError = "远程来源地址解析失败" + errRemoteProviderRedirectLimit remoteProviderError = "远程来源重定向次数超过限制" + errRemoteProviderDownloadFailed remoteProviderError = errPagesPackageURLDownloadFailed + errRemoteProviderTooLarge remoteProviderError = errPagesPackageURLTooLarge + errRemoteProviderEmpty remoteProviderError = errPagesPackageEmpty + errRemoteProviderUnsupported remoteProviderError = errPagesPackageUnsupported + errRemoteProviderCleanupFailed remoteProviderError = "清理远程来源临时文件失败" +) + +var remoteSourceNonPublicPrefixes = []netip.Prefix{ + // IPv4 special-use, private, link-local, documentation, multicast and + // reserved ranges. A conservative deny list is intentional for SSRF safety. + netip.MustParsePrefix("0.0.0.0/8"), + netip.MustParsePrefix("10.0.0.0/8"), + netip.MustParsePrefix("100.64.0.0/10"), + netip.MustParsePrefix("127.0.0.0/8"), + netip.MustParsePrefix("169.254.0.0/16"), + netip.MustParsePrefix("172.16.0.0/12"), + netip.MustParsePrefix("192.0.0.0/24"), + netip.MustParsePrefix("192.0.2.0/24"), + netip.MustParsePrefix("192.88.99.0/24"), + netip.MustParsePrefix("192.168.0.0/16"), + netip.MustParsePrefix("198.18.0.0/15"), + netip.MustParsePrefix("198.51.100.0/24"), + netip.MustParsePrefix("203.0.113.0/24"), + netip.MustParsePrefix("224.0.0.0/4"), + netip.MustParsePrefix("240.0.0.0/4"), + // IPv6 protocol-assignment, documentation and transition ranges that are + // not acceptable as direct public artifact origins. + netip.MustParsePrefix("2001::/23"), + netip.MustParsePrefix("2001:db8::/32"), + netip.MustParsePrefix("2002::/16"), + netip.MustParsePrefix("3fff::/20"), +} + +var remoteSourcePublicIPv6Prefix = netip.MustParsePrefix("2000::/3") + +// RemoteSourceRequest describes one immutable Remote URL package fetch. +type RemoteSourceRequest struct { + URL string + NetworkPolicy string + MaxPackageBytes int64 +} + +// SourceCandidate is a constrained, immutable archive downloaded to a +// provider-owned temporary file. The caller owns the file after a successful +// fetch and must call Cleanup when processing finishes. +type SourceCandidate struct { + TempPath string + Checksum string + PackageSize int64 + Format pagesarchive.Format + SafeLabel string +} + +// Cleanup removes the candidate temporary file. It is safe to call repeatedly. +func (candidate *SourceCandidate) Cleanup() error { + if candidate == nil || candidate.TempPath == "" { + return nil + } + tempPath := candidate.TempPath + err := os.Remove(tempPath) + if err == nil || errors.Is(err, os.ErrNotExist) { + candidate.TempPath = "" + return nil + } + return errRemoteProviderCleanupFailed +} + +type remoteSourceResolver interface { + LookupNetIP(context.Context, string, string) ([]netip.Addr, error) +} + +type remoteSourceDependencies struct { + resolver remoteSourceResolver + dialContext func(context.Context, string, string) (net.Conn, error) + createTemp func(string, string) (*os.File, error) +} + +// FetchRemoteSource downloads a Remote URL package without writing deployment +// state. Errors are reduced to safe domain messages and never contain the raw +// URL, query, response headers or response body. +func FetchRemoteSource(ctx context.Context, request RemoteSourceRequest) (*SourceCandidate, error) { + dialer := &net.Dialer{ + Timeout: remoteSourceDialTimeout, + KeepAlive: remoteSourceDialKeepAlive, + } + dependencies := remoteSourceDependencies{ + resolver: net.DefaultResolver, + dialContext: dialer.DialContext, + createTemp: os.CreateTemp, + } + return fetchRemoteSource(ctx, request, dependencies) +} + +func fetchRemoteSource(ctx context.Context, request RemoteSourceRequest, dependencies remoteSourceDependencies) (*SourceCandidate, error) { + if request.MaxPackageBytes <= 0 { + return nil, errRemoteProviderInvalidLimit + } + if dependencies.dialContext == nil || dependencies.createTemp == nil { + return nil, errRemoteProviderDownloadFailed + } + policy, err := normalizeRemoteNetworkPolicy(request.NetworkPolicy) + if err != nil { + return nil, err + } + parsed, err := parseRemoteSourceURL(request.URL) + if err != nil { + return nil, err + } + if err := validateRemoteSourceTarget(ctx, parsed, policy, dependencies.resolver); err != nil { + return nil, sanitizeRemoteProviderError(ctx, err) + } + + safeLabel, namedFormat := remoteSourceLabel(parsed) + client := newRemoteSourceClient(policy, dependencies) + defer client.CloseIdleConnections() + response, err := requestRemoteSource(ctx, client, parsed) + if err != nil { + return nil, err + } + defer func() { _ = response.Body.Close() }() + + if response.StatusCode < http.StatusOK || response.StatusCode >= http.StatusMultipleChoices { + return nil, fmt.Errorf("%w: HTTP %d", errRemoteProviderDownloadFailed, response.StatusCode) + } + if response.ContentLength > request.MaxPackageBytes { + return nil, errRemoteProviderTooLarge + } + + tempPath, checksum, packageSize, err := streamRemoteSourcePackage( + response.Body, + request.MaxPackageBytes, + dependencies.createTemp, + ) + if err != nil { + return nil, sanitizeRemoteProviderError(ctx, err) + } + format, safeLabel, err := detectRemoteSourceFormat(tempPath, safeLabel, namedFormat) + if err != nil { + if removeErr := os.Remove(tempPath); removeErr != nil && !errors.Is(removeErr, os.ErrNotExist) { + return nil, errRemoteProviderCleanupFailed + } + return nil, err + } + + return &SourceCandidate{ + TempPath: tempPath, + Checksum: checksum, + PackageSize: packageSize, + Format: format, + SafeLabel: safeLabel, + }, nil +} + +func normalizeRemoteNetworkPolicy(policy string) (string, error) { + switch strings.TrimSpace(policy) { + case "", RemoteNetworkPolicyPublic: + return RemoteNetworkPolicyPublic, nil + case RemoteNetworkPolicyTrustedInternal: + return RemoteNetworkPolicyTrustedInternal, nil + default: + return "", errRemoteProviderInvalidPolicy + } +} + +func newRemoteSourceClient(policy string, dependencies remoteSourceDependencies) *http.Client { + tlsConfig := &tls.Config{MinVersion: tls.VersionTLS12} + dialContext := dependencies.dialContext + if policy == RemoteNetworkPolicyPublic { + dialContext = newPublicRemoteSourceDialer(dependencies.resolver, dependencies.dialContext) + } else { + // trusted_internal is an explicit administrator-selected boundary for + // private artifact services using an internal CA or self-signed cert. + tlsConfig.InsecureSkipVerify = true //nolint:gosec // required trusted_internal semantics + } + + client := &http.Client{ + Timeout: remoteSourceDownloadTimeout, + Transport: httppool.NewTransport(httppool.TransportOptions{ + Proxy: nil, + DialContext: dialContext, + TLSClientConfig: tlsConfig, + ResponseHeaderTimeout: remoteSourceResponseHeaderTimeout, + TraceFilter: remoteSourceTraceFilter, + }), + } + client.CheckRedirect = func(next *http.Request, previous []*http.Request) error { + if len(previous) > remoteSourceMaxRedirects { + return errRemoteProviderRedirectLimit + } + stripRemoteSourceRedirectHeaders(next) + if err := validateRemoteSourceTarget(next.Context(), next.URL, policy, dependencies.resolver); err != nil { + return err + } + applyRemoteSourceHeaders(next) + return nil + } + return client +} + +func requestRemoteSource(ctx context.Context, client *http.Client, parsed *url.URL) (*http.Response, error) { + request, err := http.NewRequestWithContext(ctx, http.MethodGet, parsed.String(), nil) + if err != nil { + return nil, errors.New(errPagesSourceRemoteURLInvalid) + } + applyRemoteSourceHeaders(request) + response, err := client.Do(request) //nolint:gosec // scheme and every dial target are validated above + if err == nil { + return response, nil + } + if response != nil && response.Body != nil { + _ = response.Body.Close() + } + return nil, sanitizeRemoteProviderError(ctx, err) +} + +func applyRemoteSourceHeaders(request *http.Request) { + request.Header.Set("User-Agent", remoteSourceUserAgent) + request.Header.Set("Accept", "application/octet-stream,application/zip,application/x-tar,application/gzip,*/*;q=0.1") + // Preserve the artifact bytes exactly as stored. Automatic HTTP gzip + // decompression would change the checksum, size and archive format. + request.Header.Set("Accept-Encoding", "identity") +} + +func stripRemoteSourceRedirectHeaders(request *http.Request) { + request.Header.Del("Authorization") + request.Header.Del("Cookie") + request.Header.Del("Proxy-Authorization") + request.Header.Del("Referer") +} + +func remoteSourceTraceFilter(request *http.Request) bool { + // otelhttp records url.full. Signed query strings must never enter traces. + return request.URL == nil || request.URL.RawQuery == "" +} + +func validateRemoteSourceTarget(ctx context.Context, target *url.URL, policy string, resolver remoteSourceResolver) error { + if target == nil || target.User != nil || target.Fragment != "" || target.Opaque != "" { + return errors.New(errPagesSourceRemoteURLInvalid) + } + scheme := strings.ToLower(strings.TrimSpace(target.Scheme)) + if (scheme != remoteSourceSchemeHTTP && scheme != remoteSourceSchemeHTTPS) || strings.TrimSpace(target.Hostname()) == "" { + return errors.New(errPagesSourceRemoteURLInvalid) + } + if policy != RemoteNetworkPolicyPublic { + return nil + } + _, err := resolvePublicRemoteSourceIPs(ctx, resolver, target.Hostname()) + return err +} + +func newPublicRemoteSourceDialer( + resolver remoteSourceResolver, + directDial func(context.Context, string, string) (net.Conn, error), +) func(context.Context, string, string) (net.Conn, error) { + return func(ctx context.Context, network string, address string) (net.Conn, error) { + host, port, err := net.SplitHostPort(address) + if err != nil { + return nil, errRemoteProviderDownloadFailed + } + addresses, err := resolvePublicRemoteSourceIPs(ctx, resolver, host) + if err != nil { + return nil, err + } + for _, address := range addresses { + if !remoteSourceIPMatchesNetwork(address, network) { + continue + } + connection, dialErr := directDial(ctx, network, net.JoinHostPort(address.String(), port)) + if dialErr == nil { + return connection, nil + } + } + return nil, errRemoteProviderDownloadFailed + } +} + +func resolvePublicRemoteSourceIPs(ctx context.Context, resolver remoteSourceResolver, host string) ([]netip.Addr, error) { + if strings.Contains(host, "%") { + return nil, errRemoteProviderBlockedAddress + } + if literal, parseErr := netip.ParseAddr(host); parseErr == nil { + if !isPublicRemoteSourceIP(literal) { + return nil, errRemoteProviderBlockedAddress + } + return []netip.Addr{literal}, nil + } + if resolver == nil { + return nil, errRemoteProviderResolveFailed + } + addresses, err := resolver.LookupNetIP(ctx, "ip", host) + if err != nil || len(addresses) == 0 { + return nil, errRemoteProviderResolveFailed + } + for _, address := range addresses { + if !isPublicRemoteSourceIP(address) { + return nil, errRemoteProviderBlockedAddress + } + } + return addresses, nil +} + +func isPublicRemoteSourceIP(address netip.Addr) bool { + if !address.IsValid() || address.Zone() != "" { + return false + } + address = address.Unmap() + if !address.IsGlobalUnicast() { + return false + } + if address.Is6() && !remoteSourcePublicIPv6Prefix.Contains(address) { + return false + } + for _, prefix := range remoteSourceNonPublicPrefixes { + if prefix.Contains(address) { + return false + } + } + return true +} + +func remoteSourceIPMatchesNetwork(address netip.Addr, network string) bool { + switch network { + case "tcp4": + return address.Unmap().Is4() + case "tcp6": + return address.Unmap().Is6() + default: + return true + } +} + +func streamRemoteSourcePackage( + body io.Reader, + maxPackageBytes int64, + createTemp func(string, string) (*os.File, error), +) (tempPath string, checksum string, packageSize int64, err error) { + if createTemp == nil { + return "", "", 0, errRemoteProviderDownloadFailed + } + tempFile, err := createTemp("", "openflare-pages-source-*") + if err != nil { + return "", "", 0, errRemoteProviderDownloadFailed + } + createdTempPath := tempFile.Name() + tempPath = createdTempPath + defer func() { + closeErr := tempFile.Close() + if err == nil && closeErr != nil { + err = errRemoteProviderDownloadFailed + } + if err != nil { + if removeErr := os.Remove(createdTempPath); removeErr != nil && !errors.Is(removeErr, os.ErrNotExist) { + err = errRemoteProviderCleanupFailed + } + } + }() + + hasher := sha256.New() + readLimit := maxPackageBytes + if readLimit < math.MaxInt64 { + readLimit++ + } + packageSize, err = io.Copy(io.MultiWriter(tempFile, hasher), io.LimitReader(body, readLimit)) + if err != nil { + return "", "", 0, errRemoteProviderDownloadFailed + } + if packageSize > maxPackageBytes { + return "", "", 0, errRemoteProviderTooLarge + } + if packageSize == 0 { + return "", "", 0, errRemoteProviderEmpty + } + checksum = hex.EncodeToString(hasher.Sum(nil)) + return tempPath, checksum, packageSize, nil +} + +func detectRemoteSourceFormat( + tempPath string, + safeLabel string, + namedFormat pagesarchive.Format, +) (pagesarchive.Format, string, error) { + if namedFormat != "" { + return namedFormat, safeLabel, nil + } + tempFile, err := os.Open(tempPath) //nolint:gosec // path is a provider-created temporary file + if err != nil { + return "", safeLabel, errRemoteProviderDownloadFailed + } + defer func() { _ = tempFile.Close() }() + + head := make([]byte, remoteSourceMagicSniffBytes) + readBytes, readErr := io.ReadFull(tempFile, head) + if readErr != nil && !errors.Is(readErr, io.EOF) && !errors.Is(readErr, io.ErrUnexpectedEOF) { + return "", safeLabel, errRemoteProviderDownloadFailed + } + format, ok := pagesarchive.DetectFormatFromBytes(head[:readBytes]) + if !ok { + return "", safeLabel, errRemoteProviderUnsupported + } + return format, appendRemoteSourceLabelExtension(safeLabel, format), nil +} + +func remoteSourceLabel(parsed *url.URL) (string, pagesarchive.Format) { + baseName := path.Base(parsed.Path) + if baseName == "" || baseName == "." || baseName == "/" { + baseName = remoteSourceFallbackLabel + } + safeLabel := sanitizeRemoteSourceLabel(baseName) + format, _ := pagesarchive.DetectFormatFromName(safeLabel) + return limitRemoteSourceLabel(safeLabel, format), format +} + +func sanitizeRemoteSourceLabel(label string) string { + var builder strings.Builder + lastReplacement := false + for _, character := range label { + if isRemoteSourceLabelCharacter(character) { + builder.WriteRune(character) + lastReplacement = false + continue + } + if !lastReplacement { + builder.WriteByte('-') + lastReplacement = true + } + } + safeLabel := strings.TrimSpace(builder.String()) + if safeLabel == "" || strings.Trim(safeLabel, "._-") == "" { + return remoteSourceFallbackLabel + } + return safeLabel +} + +func isRemoteSourceLabelCharacter(character rune) bool { + return character >= 'a' && character <= 'z' || + character >= 'A' && character <= 'Z' || + character >= '0' && character <= '9' || + character == '.' || character == '-' || character == '_' +} + +func limitRemoteSourceLabel(label string, format pagesarchive.Format) string { + if len(label) <= remoteSourceMaxSafeLabelBytes { + return label + } + if format == "" { + return strings.TrimRight(label[:remoteSourceMaxSafeLabelBytes], ".-_") + } + extension := "." + pagesarchive.Extension(format) + prefixLength := remoteSourceMaxSafeLabelBytes - len(extension) + prefix := strings.TrimRight(label[:prefixLength], ".-_") + if prefix == "" { + prefix = remoteSourceFallbackLabel + } + return prefix + extension +} + +func appendRemoteSourceLabelExtension(label string, format pagesarchive.Format) string { + extension := "." + pagesarchive.Extension(format) + maxPrefixLength := remoteSourceMaxSafeLabelBytes - len(extension) + if len(label) > maxPrefixLength { + label = strings.TrimRight(label[:maxPrefixLength], ".-_") + } + if label == "" { + label = remoteSourceFallbackLabel + } + return label + extension +} + +func sanitizeRemoteProviderError(ctx context.Context, err error) error { + if ctxErr := ctx.Err(); ctxErr != nil { + return fmt.Errorf("%w: %w", errRemoteProviderDownloadFailed, ctxErr) + } + for _, safeError := range []error{ + errRemoteProviderInvalidPolicy, + errRemoteProviderInvalidLimit, + errRemoteProviderBlockedAddress, + errRemoteProviderResolveFailed, + errRemoteProviderRedirectLimit, + errRemoteProviderTooLarge, + errRemoteProviderEmpty, + errRemoteProviderUnsupported, + errRemoteProviderCleanupFailed, + } { + if errors.Is(err, safeError) { + return safeError + } + } + return errRemoteProviderDownloadFailed +} diff --git a/internal/apps/openflare/pages/source_provider_test.go b/internal/apps/openflare/pages/source_provider_test.go new file mode 100644 index 00000000..c38f1850 --- /dev/null +++ b/internal/apps/openflare/pages/source_provider_test.go @@ -0,0 +1,459 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "archive/zip" + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "io" + "log" + "net" + "net/http" + "net/http/httptest" + "net/netip" + "os" + "strconv" + "strings" + "sync/atomic" + "testing" +) + +type remoteSourceResolverFunc func(context.Context, string, string) ([]netip.Addr, error) + +func (function remoteSourceResolverFunc) LookupNetIP( + ctx context.Context, + network string, + host string, +) ([]netip.Addr, error) { + return function(ctx, network, host) +} + +func TestFetchRemoteSourceTrustedInternalSelfSignedAndSafeLabel(t *testing.T) { + packageBytes := makeRemoteSourceZIP(t) + server := httptest.NewTLSServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.URL.Query().Get("token") != "source-secret" { + t.Error("signed query did not reach the artifact server") + } + if request.Header.Get("Accept-Encoding") != "identity" { + t.Error("artifact request must disable automatic HTTP decompression") + } + writer.Header().Set("Content-Disposition", `attachment; filename="redirected.tar.gz"`) + _, _ = writer.Write(packageBytes) + })) + server.Config.ErrorLog = log.New(io.Discard, "", 0) + t.Cleanup(server.Close) + + candidate, err := FetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: server.URL + "/original/site.zip?token=source-secret", + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: int64(len(packageBytes) + 1), + }) + if err != nil { + t.Fatalf("FetchRemoteSource() error = %v", err) + } + if candidate.Format != "zip" { + t.Fatalf("Format = %q, want zip", candidate.Format) + } + if candidate.SafeLabel != "site.zip" { + t.Fatalf("SafeLabel = %q, want original path basename", candidate.SafeLabel) + } + if candidate.PackageSize != int64(len(packageBytes)) { + t.Fatalf("PackageSize = %d, want %d", candidate.PackageSize, len(packageBytes)) + } + wantChecksum := sha256.Sum256(packageBytes) + if candidate.Checksum != hex.EncodeToString(wantChecksum[:]) { + t.Fatalf("Checksum = %q, want SHA-256", candidate.Checksum) + } + downloaded, err := os.ReadFile(candidate.TempPath) //nolint:gosec // provider-owned test temp file + if err != nil { + t.Fatalf("ReadFile() error = %v", err) + } + if !bytes.Equal(downloaded, packageBytes) { + t.Fatal("downloaded package differs from response body") + } + tempPath := candidate.TempPath + if err := candidate.Cleanup(); err != nil { + t.Fatalf("Cleanup() error = %v", err) + } + if err := candidate.Cleanup(); err != nil { + t.Fatalf("second Cleanup() error = %v", err) + } + if _, err := os.Stat(tempPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("temporary file still exists: %v", err) + } +} + +func TestFetchRemoteSourceKeepsOriginalLabelAcrossRedirect(t *testing.T) { + packageBytes := makeRemoteSourceZIP(t) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.URL.Path == "/original/site.zip" { + writer.Header().Set("Location", "/delivery/final.tar.gz?token=redirect-secret") + writer.WriteHeader(http.StatusFound) + return + } + if request.Header.Get("Referer") != "" { + t.Error("redirect must not forward a signed source URL as Referer") + } + writer.Header().Set("Content-Disposition", `attachment; filename="response.7z"`) + _, _ = writer.Write(packageBytes) + })) + t.Cleanup(server.Close) + + candidate, err := FetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: server.URL + "/original/site.zip?token=initial-secret", + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: int64(len(packageBytes) + 1), + }) + if err != nil { + t.Fatalf("FetchRemoteSource() error = %v", err) + } + defer func() { _ = candidate.Cleanup() }() + if candidate.SafeLabel != "site.zip" || candidate.Format != "zip" { + t.Fatalf("candidate = label %q format %q, want original site.zip", candidate.SafeLabel, candidate.Format) + } +} + +func TestFetchRemoteSourcePublicRejectsNonPublicAddresses(t *testing.T) { + tests := []string{ + "http://127.0.0.1/site.zip?token=loopback-secret", + "http://[::1]/site.zip?token=ipv6-secret", + "http://100.64.0.1/site.zip?token=cgnat-secret", + "http://192.0.2.1/site.zip?token=documentation-secret", + } + for _, rawURL := range tests { + t.Run(rawURL, func(t *testing.T) { + _, err := FetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: rawURL, + NetworkPolicy: RemoteNetworkPolicyPublic, + MaxPackageBytes: 1024, + }) + if !errors.Is(err, errRemoteProviderBlockedAddress) { + t.Fatalf("FetchRemoteSource() error = %v, want blocked address", err) + } + assertRemoteSourceErrorRedacted(t, err, rawURL, "secret", "token=") + }) + } +} + +func TestFetchRemoteSourcePublicDialsValidatedIP(t *testing.T) { + packageBytes := makeRemoteSourceZIP(t) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write(packageBytes) + })) + t.Cleanup(server.Close) + + var dialedAddress string + dependencies := mappedRemoteSourceDependencies(server.Listener.Addr().String(), staticPublicRemoteSourceResolver()) + dependencies.dialContext = func(ctx context.Context, network string, address string) (net.Conn, error) { + dialedAddress = address + dialer := &net.Dialer{} + return dialer.DialContext(ctx, network, server.Listener.Addr().String()) + } + candidate, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: "http://artifact.example/site.zip", + NetworkPolicy: RemoteNetworkPolicyPublic, + MaxPackageBytes: int64(len(packageBytes) + 1), + }, dependencies) + if err != nil { + t.Fatalf("fetchRemoteSource() error = %v", err) + } + defer func() { _ = candidate.Cleanup() }() + if dialedAddress != "93.184.216.34:80" { + t.Fatalf("direct dial address = %q, want validated IP", dialedAddress) + } +} + +func TestFetchRemoteSourcePublicRejectsPrivateRedirect(t *testing.T) { + var requestCount atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + requestCount.Add(1) + writer.Header().Set("Location", "http://127.0.0.1/private.zip?token=redirect-secret") + writer.WriteHeader(http.StatusFound) + })) + t.Cleanup(server.Close) + + dependencies := mappedRemoteSourceDependencies(server.Listener.Addr().String(), staticPublicRemoteSourceResolver()) + rawURL := "http://artifact.example/start.zip?token=initial-secret" + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: rawURL, + NetworkPolicy: RemoteNetworkPolicyPublic, + MaxPackageBytes: 1024, + }, dependencies) + if !errors.Is(err, errRemoteProviderBlockedAddress) { + t.Fatalf("fetchRemoteSource() error = %v, want blocked redirect", err) + } + if requestCount.Load() != 1 { + t.Fatalf("request count = %d, private redirect must not be requested", requestCount.Load()) + } + assertRemoteSourceErrorRedacted(t, err, rawURL, "initial-secret", "redirect-secret", "token=") +} + +func TestFetchRemoteSourcePublicRejectsDNSRebinding(t *testing.T) { + var lookupCount atomic.Int32 + resolver := remoteSourceResolverFunc(func(context.Context, string, string) ([]netip.Addr, error) { + if lookupCount.Add(1) == 1 { + return []netip.Addr{netip.MustParseAddr("93.184.216.34")}, nil + } + return []netip.Addr{netip.MustParseAddr("127.0.0.1")}, nil + }) + var dialCount atomic.Int32 + dependencies := remoteSourceDependencies{ + resolver: resolver, + dialContext: func(context.Context, string, string) (net.Conn, error) { + dialCount.Add(1) + return nil, errors.New("unexpected dial") + }, + createTemp: os.CreateTemp, + } + rawURL := "http://rebind.example/site.zip?signature=dns-secret" + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: rawURL, + NetworkPolicy: RemoteNetworkPolicyPublic, + MaxPackageBytes: 1024, + }, dependencies) + if !errors.Is(err, errRemoteProviderBlockedAddress) { + t.Fatalf("fetchRemoteSource() error = %v, want DNS rebinding rejection", err) + } + if lookupCount.Load() != 2 { + t.Fatalf("DNS lookup count = %d, want preflight plus dial validation", lookupCount.Load()) + } + if dialCount.Load() != 0 { + t.Fatalf("direct dial count = %d, rebound address must not be dialed", dialCount.Load()) + } + assertRemoteSourceErrorRedacted(t, err, rawURL, "dns-secret", "signature=") +} + +func TestFetchRemoteSourcePublicRejectsSelfSignedTLS(t *testing.T) { + packageBytes := makeRemoteSourceZIP(t) + server := httptest.NewTLSServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write(packageBytes) + })) + server.Config.ErrorLog = log.New(io.Discard, "", 0) + t.Cleanup(server.Close) + + dependencies := mappedRemoteSourceDependencies(server.Listener.Addr().String(), staticPublicRemoteSourceResolver()) + rawURL := "https://artifact.example/site.zip?signature=tls-secret" + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: rawURL, + NetworkPolicy: RemoteNetworkPolicyPublic, + MaxPackageBytes: int64(len(packageBytes) + 1), + }, dependencies) + if !errors.Is(err, errRemoteProviderDownloadFailed) { + t.Fatalf("fetchRemoteSource() error = %v, want strict TLS failure", err) + } + assertRemoteSourceErrorRedacted(t, err, rawURL, "tls-secret", "signature=") +} + +func TestFetchRemoteSourceRejectsChunkedBodyOverLimitAndCleansTemp(t *testing.T) { + const maxPackageBytes = int64(64) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write(bytes.Repeat([]byte{'x'}, int(maxPackageBytes))) + if flusher, ok := writer.(http.Flusher); ok { + flusher.Flush() + } + _, _ = writer.Write([]byte("overflow")) + })) + t.Cleanup(server.Close) + + tempDir := t.TempDir() + dependencies := defaultRemoteSourceDependenciesForTest() + dependencies.createTemp = func(_ string, pattern string) (*os.File, error) { + return os.CreateTemp(tempDir, pattern) + } + rawURL := server.URL + "/site.zip?token=chunk-secret" + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: rawURL, + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: maxPackageBytes, + }, dependencies) + if !errors.Is(err, errRemoteProviderTooLarge) { + t.Fatalf("fetchRemoteSource() error = %v, want actual stream limit", err) + } + assertRemoteSourceTempDirEmpty(t, tempDir) + assertRemoteSourceErrorRedacted(t, err, rawURL, "chunk-secret", "token=") +} + +func TestFetchRemoteSourceRejectsContentLengthBeforeCreatingTemp(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("Content-Length", "4096") + writer.WriteHeader(http.StatusOK) + })) + t.Cleanup(server.Close) + + var createCount atomic.Int32 + dependencies := defaultRemoteSourceDependenciesForTest() + dependencies.createTemp = func(directory string, pattern string) (*os.File, error) { + createCount.Add(1) + return os.CreateTemp(directory, pattern) + } + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: server.URL + "/site.zip", + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: 1024, + }, dependencies) + if !errors.Is(err, errRemoteProviderTooLarge) { + t.Fatalf("fetchRemoteSource() error = %v, want Content-Length rejection", err) + } + if createCount.Load() != 0 { + t.Fatalf("CreateTemp called %d times before Content-Length rejection", createCount.Load()) + } +} + +func TestFetchRemoteSourceSniffsAtLeast512BytesForTar(t *testing.T) { + packageBytes := make([]byte, remoteSourceMagicSniffBytes) + copy(packageBytes[257:], []byte("ustar")) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write(packageBytes) + })) + t.Cleanup(server.Close) + + candidate, err := FetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: server.URL + "/download", + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: int64(len(packageBytes) + 1), + }) + if err != nil { + t.Fatalf("FetchRemoteSource() error = %v", err) + } + defer func() { _ = candidate.Cleanup() }() + if candidate.Format != "tar" { + t.Fatalf("Format = %q, want tar detected at byte 257", candidate.Format) + } + if candidate.SafeLabel != "download.tar" { + t.Fatalf("SafeLabel = %q, want download.tar", candidate.SafeLabel) + } +} + +func TestFetchRemoteSourceRedactsURLHeadersAndBodyFromErrors(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("X-Artifact-Secret", "header-secret") + _, _ = writer.Write([]byte("response-body-secret")) + })) + t.Cleanup(server.Close) + + tempDir := t.TempDir() + dependencies := defaultRemoteSourceDependenciesForTest() + dependencies.createTemp = func(_ string, pattern string) (*os.File, error) { + return os.CreateTemp(tempDir, pattern) + } + rawURL := server.URL + "/download?token=query-secret" + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: rawURL, + NetworkPolicy: RemoteNetworkPolicyTrustedInternal, + MaxPackageBytes: 1024, + }, dependencies) + if !errors.Is(err, errRemoteProviderUnsupported) { + t.Fatalf("fetchRemoteSource() error = %v, want unsupported archive", err) + } + assertRemoteSourceTempDirEmpty(t, tempDir) + assertRemoteSourceErrorRedacted( + t, + err, + rawURL, + "query-secret", + "header-secret", + "response-body-secret", + "token=", + ) +} + +func TestFetchRemoteSourceAllowsFiveRedirectsOnly(t *testing.T) { + packageBytes := makeRemoteSourceZIP(t) + var requestCount atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + requestCount.Add(1) + redirectNumber, _ := strconv.Atoi(strings.TrimPrefix(request.URL.Path, "/")) + if redirectNumber < remoteSourceMaxRedirects+1 { + writer.Header().Set("Location", "/"+strconv.Itoa(redirectNumber+1)) + writer.WriteHeader(http.StatusFound) + return + } + _, _ = writer.Write(packageBytes) + })) + t.Cleanup(server.Close) + + dependencies := mappedRemoteSourceDependencies(server.Listener.Addr().String(), staticPublicRemoteSourceResolver()) + _, err := fetchRemoteSource(t.Context(), RemoteSourceRequest{ + URL: "http://artifact.example/0", + NetworkPolicy: RemoteNetworkPolicyPublic, + MaxPackageBytes: int64(len(packageBytes) + 1), + }, dependencies) + if !errors.Is(err, errRemoteProviderRedirectLimit) { + t.Fatalf("fetchRemoteSource() error = %v, want redirect limit", err) + } + if requestCount.Load() != remoteSourceMaxRedirects+1 { + t.Fatalf("request count = %d, want initial plus five redirects", requestCount.Load()) + } +} + +func staticPublicRemoteSourceResolver() remoteSourceResolver { + return remoteSourceResolverFunc(func(context.Context, string, string) ([]netip.Addr, error) { + return []netip.Addr{netip.MustParseAddr("93.184.216.34")}, nil + }) +} + +func defaultRemoteSourceDependenciesForTest() remoteSourceDependencies { + dialer := &net.Dialer{} + return remoteSourceDependencies{ + resolver: net.DefaultResolver, + dialContext: dialer.DialContext, + createTemp: os.CreateTemp, + } +} + +func mappedRemoteSourceDependencies(targetAddress string, resolver remoteSourceResolver) remoteSourceDependencies { + dialer := &net.Dialer{} + return remoteSourceDependencies{ + resolver: resolver, + dialContext: func(ctx context.Context, network string, _ string) (net.Conn, error) { + return dialer.DialContext(ctx, network, targetAddress) + }, + createTemp: os.CreateTemp, + } +} + +func makeRemoteSourceZIP(t *testing.T) []byte { + t.Helper() + var buffer bytes.Buffer + archive := zip.NewWriter(&buffer) + file, err := archive.Create("index.html") + if err != nil { + t.Fatalf("zip.Create() error = %v", err) + } + if _, err := file.Write([]byte("

OpenFlare

")); err != nil { + t.Fatalf("zip entry Write() error = %v", err) + } + if err := archive.Close(); err != nil { + t.Fatalf("zip.Close() error = %v", err) + } + return buffer.Bytes() +} + +func assertRemoteSourceTempDirEmpty(t *testing.T, directory string) { + t.Helper() + entries, err := os.ReadDir(directory) + if err != nil { + t.Fatalf("ReadDir() error = %v", err) + } + if len(entries) != 0 { + t.Fatalf("temporary directory contains %d leaked files", len(entries)) + } +} + +func assertRemoteSourceErrorRedacted(t *testing.T, err error, sensitiveValues ...string) { + t.Helper() + if err == nil { + t.Fatal("expected an error") + } + message := err.Error() + for _, sensitiveValue := range sensitiveValues { + if sensitiveValue != "" && strings.Contains(message, sensitiveValue) { + t.Fatalf("error %q contains sensitive value %q", message, sensitiveValue) + } + } +} diff --git a/internal/apps/openflare/pages/source_runtime.go b/internal/apps/openflare/pages/source_runtime.go new file mode 100644 index 00000000..0dd8fc0f --- /dev/null +++ b/internal/apps/openflare/pages/source_runtime.go @@ -0,0 +1,309 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "strings" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +const ( + pagesSourceCheckLeaseDuration = 2 * time.Minute + pagesSourceSyncLeaseDuration = 15 * time.Minute + sourceLeaseTokenBytes = 32 + sourceRuntimeErrorMaxBytes = 512 + sourceRevisionHexLength = 64 + + sourceRuntimeColumnSyncStatus = "sync_status" + sourceRuntimeColumnLastError = "last_error" + sourceRuntimeColumnLeaseToken = "lease_token" + sourceRuntimeColumnLeaseExpiresAt = "lease_expires_at" +) + +type sourceLeaseOutcome string + +const ( + sourceLeaseAcquired sourceLeaseOutcome = "acquired" + sourceLeaseBusy sourceLeaseOutcome = "busy" + sourceLeaseStale sourceLeaseOutcome = "stale" +) + +// sourceExecutionSnapshot captures every mutable value that can affect archive +// validation or the atomic activation decision. The queued payload deliberately +// does not carry project content configuration. +type sourceExecutionSnapshot struct { + ProjectID uint + SourceID uint + SourceConfigVersion int + ContentConfigVersion int + SourceType string + SourceIdentity string + RemoteURL string + RemoteNetworkPolicy string + RootDir string + EntryFile string + LeaseToken string + LeaseExpiresAt time.Time +} + +func acquireSourceLease( + ctx context.Context, + sourceID uint, + expectedConfigVersion int, + action string, +) (*sourceExecutionSnapshot, sourceLeaseOutcome, error) { + leaseDuration, status, err := sourceLeaseParameters(action) + if err != nil { + return nil, sourceLeaseStale, err + } + token, err := newSourceLeaseToken() + if err != nil { + return nil, sourceLeaseStale, err + } + now := time.Now() + expiresAt := now.Add(leaseDuration) + + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", sourceID). + Where("lease_expires_at IS NULL OR lease_expires_at <= ?", now). + Where( + "EXISTS (SELECT 1 FROM of_pages_project_sources source WHERE source.id = ? AND source.config_version = ?)", + sourceID, + expectedConfigVersion, + ). + Updates(map[string]any{ + sourceRuntimeColumnLeaseToken: token, + sourceRuntimeColumnLeaseExpiresAt: expiresAt, + sourceRuntimeColumnSyncStatus: status, + sourceRuntimeColumnLastError: "", + }) + if result.Error != nil { + return nil, sourceLeaseStale, result.Error + } + if result.RowsAffected == 0 { + outcome, inspectErr := inspectSourceLeaseMiss(ctx, sourceID, expectedConfigVersion, now) + return nil, outcome, inspectErr + } + + snapshot, err := loadSourceExecutionSnapshot(ctx, sourceID, token) + if err != nil { + if errors.Is(err, errSourceLeaseSnapshotStale) { + return nil, sourceLeaseStale, nil + } + return nil, sourceLeaseStale, err + } + return snapshot, sourceLeaseAcquired, nil +} + +var errSourceLeaseSnapshotStale = errors.New("source lease snapshot stale") + +func loadSourceExecutionSnapshot( + ctx context.Context, + sourceID uint, + token string, +) (*sourceExecutionSnapshot, error) { + var snapshot sourceExecutionSnapshot + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var source model.PagesProjectSource + if err := tx.Where("id = ?", sourceID).First(&source).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return errSourceLeaseSnapshotStale + } + return err + } + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + First(&project, source.ProjectID).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return errSourceLeaseSnapshotStale + } + return err + } + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ?", sourceID). + First(&source).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return errSourceLeaseSnapshotStale + } + return err + } + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", source.ID). + First(&runtime).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return errSourceLeaseSnapshotStale + } + return err + } + // 必须在 runtime 行锁拿到后重新取时间，避免锁等待跨过 + // lease expiry 时仍使用事务开始前的旧时间继续执行。 + now := time.Now() + if runtime.LeaseToken != token || runtime.LeaseExpiresAt == nil || !runtime.LeaseExpiresAt.After(now) { + return errSourceLeaseSnapshotStale + } + snapshot = sourceExecutionSnapshot{ + ProjectID: project.ID, + SourceID: source.ID, + SourceConfigVersion: source.ConfigVersion, + ContentConfigVersion: project.ContentConfigVersion, + SourceType: source.SourceType, + SourceIdentity: source.SourceIdentity, + RemoteURL: source.RemoteURL, + RemoteNetworkPolicy: source.RemoteNetworkPolicy, + RootDir: project.RootDir, + EntryFile: project.EntryFile, + LeaseToken: token, + LeaseExpiresAt: *runtime.LeaseExpiresAt, + } + return nil + }) + if err != nil { + return nil, err + } + return &snapshot, nil +} + +func inspectSourceLeaseMiss( + ctx context.Context, + sourceID uint, + expectedConfigVersion int, + now time.Time, +) (sourceLeaseOutcome, error) { + var source model.PagesProjectSource + if err := db.DB(ctx).Where("id = ?", sourceID).First(&source).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return sourceLeaseStale, nil + } + return sourceLeaseStale, err + } + if source.ConfigVersion != expectedConfigVersion { + return sourceLeaseStale, nil + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", sourceID).First(&runtime).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return sourceLeaseStale, nil + } + return sourceLeaseStale, err + } + if runtime.LeaseExpiresAt != nil && runtime.LeaseExpiresAt.After(now) { + return sourceLeaseBusy, nil + } + return sourceLeaseStale, nil +} + +func sourceLeaseParameters(action string) (time.Duration, string, error) { + switch action { + case sourceActionCheck: + return pagesSourceCheckLeaseDuration, pagesSourceStatusChecking, nil + case sourceActionSync: + return pagesSourceSyncLeaseDuration, pagesSourceStatusSyncing, nil + default: + return 0, "", errors.New(errPagesSourceActionInvalid) + } +} + +func newSourceLeaseToken() (string, error) { + value := make([]byte, sourceLeaseTokenBytes) + if _, err := rand.Read(value); err != nil { + return "", err + } + return hex.EncodeToString(value), nil +} + +func renewSourceLease(ctx context.Context, snapshot *sourceExecutionSnapshot, duration time.Duration) (bool, error) { + if snapshot == nil || snapshot.SourceID == 0 || snapshot.LeaseToken == "" || duration <= 0 { + return false, errors.New(errPagesSourceLeaseLost) + } + now := time.Now() + expiresAt := now.Add(duration) + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", snapshot.SourceID, snapshot.LeaseToken, now). + Updates(map[string]any{sourceRuntimeColumnLeaseExpiresAt: expiresAt}) + if result.Error != nil { + return false, result.Error + } + if result.RowsAffected == 0 { + return false, nil + } + snapshot.LeaseExpiresAt = expiresAt + return true, nil +} + +func failSourceLease(ctx context.Context, snapshot *sourceExecutionSnapshot, message string) error { + if snapshot == nil || snapshot.SourceID == 0 || snapshot.LeaseToken == "" { + return nil + } + message = safeSourceRuntimeError(message) + now := time.Now() + return db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", snapshot.SourceID, snapshot.LeaseToken, now). + Updates(map[string]any{ + sourceRuntimeColumnSyncStatus: pagesSourceStatusFailed, + sourceRuntimeColumnLastError: message, + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + }).Error +} + +func safeSourceRuntimeError(message string) string { + message = strings.TrimSpace(message) + if message == "" { + return errPagesSourceSyncFailed + } + if len(message) > sourceRuntimeErrorMaxBytes { + message = message[:sourceRuntimeErrorMaxBytes] + } + return message +} + +func sourceLeaseIsBusy(ctx context.Context, sourceID uint) (bool, error) { + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", sourceID).First(&runtime).Error; err != nil { + return false, err + } + return runtime.LeaseExpiresAt != nil && runtime.LeaseExpiresAt.After(time.Now()), nil +} + +// fenceAndNormalizeRuntime invalidates in-flight work while preserving safe +// seen/applied cursors. The caller must already hold the source row lock. +func fenceAndNormalizeRuntime(tx *gorm.DB, sourceID uint) error { + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", sourceID). + First(&runtime).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil + } + return err + } + return tx.Model(&runtime).Updates(map[string]any{ + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + sourceRuntimeColumnSyncStatus: normalizedSourceRuntimeStatus(&runtime), + }).Error +} + +func sourceHasSameReleaseReplacement(runtime *model.PagesProjectSourceRuntime) bool { + if runtime == nil || runtime.LastSeenRevision == "" || runtime.LastSeenRevision == runtime.LastAppliedRevision { + return false + } + seen := sourceDetail{} + applied := sourceDetail{} + if unmarshalSourceDetail(runtime.LastSeenDetail, &seen) != nil || + unmarshalSourceDetail(runtime.LastAppliedDetail, &applied) != nil { + return false + } + return seen.ReleaseID != "" && seen.ReleaseID == applied.ReleaseID +} diff --git a/internal/apps/openflare/pages/source_runtime_test.go b/internal/apps/openflare/pages/source_runtime_test.go new file mode 100644 index 00000000..b971c424 --- /dev/null +++ b/internal/apps/openflare/pages/source_runtime_test.go @@ -0,0 +1,350 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "errors" + "strings" + "sync" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" +) + +func TestSourceLeaseHeartbeatRenewsAndCancelsOnOwnershipLoss(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "lease-heartbeat") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip", + RemoteNetworkPolicyPublic, + ) + snapshot, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil || outcome != sourceLeaseAcquired || snapshot == nil { + t.Fatalf("acquireSourceLease(heartbeat) = (%+v, %q, %v), want acquired", snapshot, outcome, err) + } + + workCtx, heartbeat, err := startSourceLeaseHeartbeat(ctx, snapshot, 500*time.Millisecond, 20*time.Millisecond) + if err != nil { + t.Fatalf("startSourceLeaseHeartbeat() error = %v, want nil", err) + } + t.Cleanup(func() { _ = heartbeat.stop() }) + + var initial model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&initial).Error; err != nil { + t.Fatalf("load initial heartbeat runtime error = %v, want nil", err) + } + if initial.LeaseExpiresAt == nil { + t.Fatal("initial heartbeat expiry = nil, want non-nil") + } + deadline := time.Now().Add(2 * time.Second) + for { + var renewedRuntime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&renewedRuntime).Error; err != nil { + t.Fatalf("load renewed heartbeat runtime error = %v, want nil", err) + } + if renewedRuntime.LeaseExpiresAt != nil && renewedRuntime.LeaseExpiresAt.After(*initial.LeaseExpiresAt) { + break + } + if time.Now().After(deadline) { + t.Fatal("heartbeat did not extend lease before deadline") + } + time.Sleep(10 * time.Millisecond) + } + + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Update("lease_token", "replacement-owner").Error; err != nil { + t.Fatalf("replace heartbeat lease owner error = %v, want nil", err) + } + select { + case <-workCtx.Done(): + case <-time.After(2 * time.Second): + t.Fatal("heartbeat work context was not canceled after ownership loss") + } + if err := heartbeat.stop(); !errors.Is(err, errSourceLeaseHeartbeatLost) { + t.Fatalf("heartbeat.stop() error = %v, want %v", err, errSourceLeaseHeartbeatLost) + } + if err := heartbeat.stop(); !errors.Is(err, errSourceLeaseHeartbeatLost) { + t.Fatalf("heartbeat.stop() second error = %v, want stable %v", err, errSourceLeaseHeartbeatLost) + } +} + +func TestAcquireSourceLeaseConcurrentOnlyOneOwner(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "lease-concurrent") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip", + RemoteNetworkPolicyPublic, + ) + type leaseResult struct { + snapshot *sourceExecutionSnapshot + outcome sourceLeaseOutcome + err error + } + results := make(chan leaseResult, 2) + var workers sync.WaitGroup + workers.Add(2) + for range 2 { + go func() { + defer workers.Done() + snapshot, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + results <- leaseResult{snapshot: snapshot, outcome: outcome, err: err} + }() + } + workers.Wait() + close(results) + + acquired := 0 + busy := 0 + for result := range results { + if result.err != nil { + t.Errorf("acquireSourceLease(concurrent) error = %v, want nil", result.err) + continue + } + switch result.outcome { + case sourceLeaseAcquired: + acquired++ + if result.snapshot == nil || result.snapshot.LeaseToken == "" { + t.Errorf("acquireSourceLease(concurrent acquired) snapshot = %+v, want token-bearing snapshot", result.snapshot) + } + case sourceLeaseBusy: + busy++ + if result.snapshot != nil { + t.Errorf("acquireSourceLease(concurrent busy) snapshot = %+v, want nil", result.snapshot) + } + default: + t.Errorf("acquireSourceLease(concurrent) outcome = %q, want acquired or busy", result.outcome) + } + } + if acquired != 1 || busy != 1 { + t.Errorf("concurrent lease outcomes = acquired:%d busy:%d, want 1 and 1", acquired, busy) + } +} + +func TestAcquireSourceLeaseMutualExclusionExpiryAndTerminalOwnership(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "lease-cas") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip", + RemoteNetworkPolicyPublic, + ) + + first, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil { + t.Fatalf("acquireSourceLease(first) error = %v, want nil", err) + } + if got, want := outcome, sourceLeaseAcquired; got != want { + t.Fatalf("acquireSourceLease(first) outcome = %q, want %q", got, want) + } + if first == nil || first.LeaseToken == "" { + t.Fatalf("acquireSourceLease(first) snapshot = %+v, want token-bearing snapshot", first) + } + + second, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil { + t.Fatalf("acquireSourceLease(duplicate) error = %v, want nil", err) + } + if got, want := outcome, sourceLeaseBusy; got != want { + t.Errorf("acquireSourceLease(duplicate) outcome = %q, want %q", got, want) + } + if second != nil { + t.Errorf("acquireSourceLease(duplicate) snapshot = %+v, want nil", second) + } + + past := time.Now().Add(-time.Second) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Update("lease_expires_at", &past).Error; err != nil { + t.Fatalf("expire first lease error = %v, want nil", err) + } + takeover, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil { + t.Fatalf("acquireSourceLease(takeover) error = %v, want nil", err) + } + if got, want := outcome, sourceLeaseAcquired; got != want { + t.Fatalf("acquireSourceLease(takeover) outcome = %q, want %q", got, want) + } + if takeover == nil { + t.Fatal("acquireSourceLease(takeover) snapshot = nil, want non-nil") + } + if takeover.LeaseToken == "" || takeover.LeaseToken == first.LeaseToken { + t.Fatalf("takeover LeaseToken = %q, want non-empty token distinct from %q", takeover.LeaseToken, first.LeaseToken) + } + + renewed, err := renewSourceLease(ctx, first, pagesSourceSyncLeaseDuration) + if err != nil { + t.Fatalf("renewSourceLease(expired owner) error = %v, want nil", err) + } + if renewed { + t.Error("renewSourceLease(expired owner) = true, want false") + } + if err := failSourceLease(ctx, first, "stale worker must not win"); err != nil { + t.Fatalf("failSourceLease(expired owner) error = %v, want nil", err) + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&runtime).Error; err != nil { + t.Fatalf("load runtime after takeover error = %v, want nil", err) + } + if got, want := runtime.LeaseToken, takeover.LeaseToken; got != want { + t.Errorf("runtime LeaseToken after stale terminal write = %q, want %q", got, want) + } + if got, want := runtime.SyncStatus, pagesSourceStatusSyncing; got != want { + t.Errorf("runtime SyncStatus after stale terminal write = %q, want %q", got, want) + } + + renewed, err = renewSourceLease(ctx, takeover, pagesSourceSyncLeaseDuration) + if err != nil { + t.Fatalf("renewSourceLease(current owner) error = %v, want nil", err) + } + if !renewed { + t.Error("renewSourceLease(current owner) = false, want true") + } + if err := failSourceLease(ctx, takeover, errPagesSourceSyncFailed); err != nil { + t.Fatalf("failSourceLease(current owner) error = %v, want nil", err) + } + var failedRuntime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&failedRuntime).Error; err != nil { + t.Fatalf("load failed runtime error = %v, want nil", err) + } + if got, want := failedRuntime.SyncStatus, pagesSourceStatusFailed; got != want { + t.Errorf("failed runtime SyncStatus = %q, want %q", got, want) + } + if failedRuntime.LeaseToken != "" || failedRuntime.LeaseExpiresAt != nil { + t.Errorf("failed runtime lease = (%q, %v), want cleared", failedRuntime.LeaseToken, failedRuntime.LeaseExpiresAt) + } +} + +func TestSourceConfigAndProjectContentChangesFenceLease(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "lease-fence") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip?token=first", + RemoteNetworkPolicyPublic, + ) + + configSnapshot, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil || outcome != sourceLeaseAcquired { + t.Fatalf("acquireSourceLease(config fence) = (%+v, %q, %v), want acquired", configSnapshot, outcome, err) + } + if _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: "https://example.com/site.zip?token=second", + RemoteNetworkPolicy: RemoteNetworkPolicyPublic, + }); err != nil { + t.Fatalf("UpdateSource(config fence) error = %v, want nil", err) + } + renewed, err := renewSourceLease(ctx, configSnapshot, pagesSourceSyncLeaseDuration) + if err != nil { + t.Fatalf("renewSourceLease(after source update) error = %v, want nil", err) + } + if renewed { + t.Error("renewSourceLease(after source update) = true, want false") + } + var updatedSource model.PagesProjectSource + if err := db.DB(ctx).Where("id = ?", source.ID).First(&updatedSource).Error; err != nil { + t.Fatalf("load updated source error = %v, want nil", err) + } + if got, want := updatedSource.ConfigVersion, source.ConfigVersion+1; got != want { + t.Errorf("updated source ConfigVersion = %d, want %d", got, want) + } + if snapshot, staleOutcome, staleErr := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync); staleErr != nil || staleOutcome != sourceLeaseStale || snapshot != nil { + t.Errorf("acquireSourceLease(old config) = (%+v, %q, %v), want (nil, %q, nil)", snapshot, staleOutcome, staleErr, sourceLeaseStale) + } + + contentSnapshot, outcome, err := acquireSourceLease(ctx, source.ID, updatedSource.ConfigVersion, sourceActionSync) + if err != nil || outcome != sourceLeaseAcquired { + t.Fatalf("acquireSourceLease(content fence) = (%+v, %q, %v), want acquired", contentSnapshot, outcome, err) + } + if _, err := UpdateProject(ctx, project.ID, Input{ + Name: project.Name, + Slug: project.Slug, + Enabled: true, + RootDir: "dist", + EntryFile: "index.html", + }); err != nil { + t.Fatalf("UpdateProject(content fence) error = %v, want nil", err) + } + renewed, err = renewSourceLease(ctx, contentSnapshot, pagesSourceSyncLeaseDuration) + if err != nil { + t.Fatalf("renewSourceLease(after content update) error = %v, want nil", err) + } + if renewed { + t.Error("renewSourceLease(after content update) = true, want false") + } + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", project.ID, err) + } + if got, want := storedProject.ContentConfigVersion, project.ContentConfigVersion+1; got != want { + t.Errorf("ContentConfigVersion after RootDir update = %d, want %d", got, want) + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&runtime).Error; err != nil { + t.Fatalf("load fenced runtime error = %v, want nil", err) + } + if runtime.LeaseToken != "" || runtime.LeaseExpiresAt != nil { + t.Errorf("content-fenced runtime lease = (%q, %v), want cleared", runtime.LeaseToken, runtime.LeaseExpiresAt) + } +} + +func TestSourceRuntimeUsesOnlySixDocumentedStates(t *testing.T) { + states := []string{ + pagesSourceStatusIdle, + pagesSourceStatusChecking, + pagesSourceStatusUpdateAvailable, + pagesSourceStatusSyncing, + pagesSourceStatusFailed, + pagesSourceStatusAttention, + } + seen := make(map[string]struct{}, len(states)) + for _, state := range states { + if strings.TrimSpace(state) == "" { + t.Errorf("documented source state = %q, want non-empty", state) + } + if _, exists := seen[state]; exists { + t.Errorf("documented source state %q is duplicated", state) + } + seen[state] = struct{}{} + } + if got, want := len(seen), 6; got != want { + t.Errorf("unique source states = %d, want %d", got, want) + } + + updateRuntime := &model.PagesProjectSourceRuntime{ + LastSeenRevision: strings.Repeat("a", 64), + LastAppliedRevision: strings.Repeat("b", 64), + LastSeenDetail: `{"release_id":"new"}`, + LastAppliedDetail: `{"release_id":"old"}`, + } + if got, want := normalizedSourceRuntimeStatus(updateRuntime), pagesSourceStatusUpdateAvailable; got != want { + t.Errorf("normalizedSourceRuntimeStatus(update) = %q, want %q", got, want) + } + attentionRuntime := &model.PagesProjectSourceRuntime{ + LastSeenRevision: strings.Repeat("a", 64), + LastAppliedRevision: strings.Repeat("b", 64), + LastSeenDetail: `{"release_id":"same"}`, + LastAppliedDetail: `{"release_id":"same"}`, + } + if got, want := normalizedSourceRuntimeStatus(attentionRuntime), pagesSourceStatusAttention; got != want { + t.Errorf("normalizedSourceRuntimeStatus(attention) = %q, want %q", got, want) + } + if got, want := normalizedSourceRuntimeStatus(&model.PagesProjectSourceRuntime{}), pagesSourceStatusIdle; got != want { + t.Errorf("normalizedSourceRuntimeStatus(idle) = %q, want %q", got, want) + } +} diff --git a/internal/apps/openflare/pages/source_sync.go b/internal/apps/openflare/pages/source_sync.go new file mode 100644 index 00000000..78430cb9 --- /dev/null +++ b/internal/apps/openflare/pages/source_sync.go @@ -0,0 +1,715 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "path" + "sort" + "strings" + "sync" + "time" + "unicode/utf8" + + "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/task" + "github.com/Rain-kl/Wavelet/pkg/logger" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +const ( + pagesSourceTriggerManualSync = "manual_sync" + pagesSourceCreatedBySystem = "system:pages-source-sync" + pagesSourceHeartbeatInterval = pagesSourceSyncLeaseDuration / 3 + pagesSourceCleanupTimeout = 15 * time.Second +) + +var ( + errSourceFinalFence = errors.New("pages source final fence rejected") + errSourceLeaseHeartbeatLost = errors.New("pages source lease heartbeat lost") + sourceCommitNow = time.Now +) + +type sourceSyncOutcome struct { + Deployment *DeploymentView + Reused bool + Stale bool +} + +type preparedRemoteSource struct { + Candidate *SourceCandidate + Manifest *deploymentManifest + Detail sourceDetail + DetailJSON string +} + +type sourceIngestState struct { + Result upload.IngestResult + HasIngest bool + Referenced bool +} + +type sourceCommitState struct { + Project *model.PagesProject + Source *model.PagesProjectSource + Runtime *model.PagesProjectSourceRuntime + Now time.Time +} + +type sourceLeaseHeartbeat struct { + cancel context.CancelFunc + done <-chan error + stopOnce sync.Once + stopErr error +} + +func startSourceLeaseHeartbeat( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + leaseDuration time.Duration, + interval time.Duration, +) (context.Context, *sourceLeaseHeartbeat, error) { + if snapshot == nil || leaseDuration <= 0 || interval <= 0 || interval >= leaseDuration { + return nil, nil, errors.New(errPagesSourceLeaseLost) + } + renewed, err := renewSourceLease(ctx, snapshot, leaseDuration) + if err != nil { + return nil, nil, err + } + if !renewed { + return nil, nil, errSourceLeaseHeartbeatLost + } + + workCtx, cancel := context.WithCancel(ctx) + done := make(chan error, 1) + heartbeat := &sourceLeaseHeartbeat{cancel: cancel, done: done} + go func() { + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-workCtx.Done(): + done <- nil + return + case <-ticker.C: + renewed, renewErr := renewSourceLease(workCtx, snapshot, leaseDuration) + if renewErr != nil { + if workCtx.Err() != nil { + done <- nil + return + } + done <- renewErr + cancel() + return + } + if !renewed { + done <- errSourceLeaseHeartbeatLost + cancel() + return + } + } + } + }() + return workCtx, heartbeat, nil +} + +func (heartbeat *sourceLeaseHeartbeat) stop() error { + if heartbeat == nil { + return nil + } + heartbeat.stopOnce.Do(func() { + heartbeat.cancel() + heartbeat.stopErr = <-heartbeat.done + }) + return heartbeat.stopErr +} + +func sourceHeartbeatOutcome(err error) (*sourceSyncOutcome, error) { + if errors.Is(err, errSourceLeaseHeartbeatLost) { + return &sourceSyncOutcome{Stale: true}, nil + } + return nil, err +} + +func recordSourceLeaseFailure(ctx context.Context, snapshot *sourceExecutionSnapshot) { + cleanupCtx, cancel := sourceCleanupContext(ctx) + defer cancel() + if err := failSourceLease(cleanupCtx, snapshot, errPagesSourceSyncFailed); err != nil { + var sourceID uint + if snapshot != nil { + sourceID = snapshot.SourceID + } + logger.WarnF(cleanupCtx, "[PagesSource] record failed runtime state failed: source_id=%d error=%v", sourceID, err) + } +} + +func sourceCleanupContext(ctx context.Context) (context.Context, context.CancelFunc) { + return context.WithTimeout(context.WithoutCancel(ctx), pagesSourceCleanupTimeout) +} + +func syncRemoteSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + actor string, +) (outcome *sourceSyncOutcome, resultErr error) { + if snapshot == nil || snapshot.SourceType != PagesSourceTypeRemoteURL { + return nil, errors.New(errPagesSourceTypeUnsupported) + } + actor = strings.TrimSpace(actor) + if actor == "" { + return nil, errors.New(errPagesSourceActionInvalid) + } + defer func() { + if resultErr != nil { + recordSourceLeaseFailure(ctx, snapshot) + } + }() + + workCtx, heartbeat, err := startSourceLeaseHeartbeat( + ctx, + snapshot, + pagesSourceSyncLeaseDuration, + pagesSourceHeartbeatInterval, + ) + if err != nil { + return sourceHeartbeatOutcome(err) + } + defer func() { + _ = heartbeat.stop() + }() + + limits := resolvePagesLimits(workCtx) + prepared, err := prepareRemoteSource(workCtx, snapshot, limits) + if err != nil { + if heartbeatErr := heartbeat.stop(); heartbeatErr != nil { + return sourceHeartbeatOutcome(heartbeatErr) + } + return nil, err + } + defer func() { + if cleanupErr := prepared.Candidate.Cleanup(); cleanupErr != nil { + logger.WarnF(ctx, "[PagesSource] cleanup temporary package failed: source_id=%d error=%v", snapshot.SourceID, cleanupErr) + } + }() + + ingestState, err := resolveSourceIngest(workCtx, snapshot, prepared) + if err != nil { + if heartbeatErr := heartbeat.stop(); heartbeatErr != nil { + return sourceHeartbeatOutcome(heartbeatErr) + } + return nil, err + } + defer func() { + compensateSourceIngest(ctx, snapshot, ingestState) + }() + + if heartbeatErr := heartbeat.stop(); heartbeatErr != nil { + return sourceHeartbeatOutcome(heartbeatErr) + } + renewed, err := renewSourceLease(ctx, snapshot, pagesSourceSyncLeaseDuration) + if err != nil { + return nil, err + } + if !renewed { + return &sourceSyncOutcome{Stale: true}, nil + } + + task.AppendLog(ctx, "[activate] 正在原子切换生产部署") + deployment, reused, referenced, err := commitSourceDeployment( + ctx, + snapshot, + prepared.Candidate.Checksum, + prepared.Detail, + prepared.DetailJSON, + actor, + prepared.Manifest, + ingestState.Result, + ingestState.HasIngest, + ) + ingestState.Referenced = referenced + if errors.Is(err, errSourceFinalFence) { + return &sourceSyncOutcome{Stale: true}, nil + } + if err != nil { + return nil, err + } + ingestState.Referenced = ingestState.HasIngest && deployment.UploadID == ingestState.Result.Upload.ID + + if pruneErr := pruneProjectDeploymentHistory(ctx, snapshot.ProjectID, limits.HistoryCount, 0); pruneErr != nil { + logger.ErrorF(ctx, + "[PagesSource] strict prune failed: project_id=%d source_id=%d keep=%d error=%v", + snapshot.ProjectID, + snapshot.SourceID, + limits.HistoryCount, + pruneErr, + ) + } + view := buildDeploymentView(deployment) + return &sourceSyncOutcome{Deployment: &view, Reused: reused}, nil +} + +func prepareRemoteSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + limits pagesLimits, +) (*preparedRemoteSource, error) { + task.AppendLog(ctx, "[download] 正在获取远程部署包") + candidate, err := FetchRemoteSource(ctx, RemoteSourceRequest{ + URL: snapshot.RemoteURL, + NetworkPolicy: snapshot.RemoteNetworkPolicy, + MaxPackageBytes: limits.PackageBytes, + }) + if err != nil { + return nil, err + } + if candidate == nil || candidate.TempPath == "" || candidate.Checksum == "" || candidate.Format == "" { + return nil, errors.New(errPagesSourceSyncFailed) + } + rootDir, err := validateAndNormalizePagesRootDir(snapshot.RootDir) + if err != nil { + cleanupFailedRemoteCandidate(ctx, snapshot, candidate) + return nil, err + } + entryFile, err := validateAndNormalizePagesEntryFile(snapshot.EntryFile) + if err != nil { + cleanupFailedRemoteCandidate(ctx, snapshot, candidate) + return nil, err + } + task.AppendLog(ctx, "[verify] 正在校验归档结构与入口文件") + manifest, err := inspectPagesPackage(candidate.TempPath, candidate.Format, rootDir, entryFile, limits) + if err != nil { + cleanupFailedRemoteCandidate(ctx, snapshot, candidate) + return nil, err + } + detail := sourceDetail{Provider: PagesSourceTypeRemoteURL, Label: safeRemoteSourceLabel(candidate.SafeLabel)} + detailJSON, err := json.Marshal(detail) + if err != nil { + cleanupFailedRemoteCandidate(ctx, snapshot, candidate) + return nil, err + } + return &preparedRemoteSource{ + Candidate: candidate, + Manifest: manifest, + Detail: detail, + DetailJSON: string(detailJSON), + }, nil +} + +func cleanupFailedRemoteCandidate( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + candidate *SourceCandidate, +) { + if err := candidate.Cleanup(); err != nil { + logger.WarnF(ctx, "[PagesSource] cleanup failed preparation package: source_id=%d error=%v", snapshot.SourceID, err) + } +} + +func resolveSourceIngest( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + prepared *preparedRemoteSource, +) (*sourceIngestState, error) { + _, err := findSourceDeployment( + ctx, + snapshot.ProjectID, + snapshot.SourceIdentity, + prepared.Candidate.Checksum, + ) + if err == nil { + return &sourceIngestState{}, nil + } + if !errors.Is(err, gorm.ErrRecordNotFound) { + return nil, err + } + task.AppendLog(ctx, "[ingest] 正在保存受管部署包") + result, err := ingestPagesDeploymentPackageWithSource( + ctx, + prepared.Candidate.TempPath, + prepared.Candidate.Checksum, + snapshot.ProjectID, + snapshot.SourceID, + prepared.Detail.Label, + prepared.Candidate.Format, + ) + if err != nil { + return nil, err + } + return &sourceIngestState{Result: result, HasIngest: true}, nil +} + +func compensateSourceIngest( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + state *sourceIngestState, +) { + if state == nil || !state.HasIngest || !state.Result.Created || state.Referenced { + return + } + cleanupCtx, cancel := sourceCleanupContext(ctx) + defer cancel() + task.AppendLog(cleanupCtx, "[cleanup] 正在补偿未引用的部署包记录") + if err := removePagesUploadIfUnreferenced(cleanupCtx, snapshot.ProjectID, state.Result.Upload.ID); err != nil { + logger.ErrorF(cleanupCtx, + "[PagesSource] compensate upload failed: project_id=%d source_id=%d upload_id=%d error=%v", + snapshot.ProjectID, snapshot.SourceID, state.Result.Upload.ID, err, + ) + } +} + +func findSourceDeployment( + ctx context.Context, + projectID uint, + sourceIdentity string, + revision string, +) (*model.PagesDeployment, error) { + var deployment model.PagesDeployment + err := db.DB(ctx). + Where("project_id = ? AND source_identity = ? AND source_revision = ?", projectID, sourceIdentity, revision). + First(&deployment).Error + if err != nil { + return nil, err + } + return &deployment, nil +} + +func commitSourceDeployment( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + revision string, + detail sourceDetail, + detailJSON string, + actor string, + manifest *deploymentManifest, + ingestResult upload.IngestResult, + hasIngest bool, +) (*model.PagesDeployment, bool, bool, error) { + if snapshot == nil || manifest == nil { + return nil, false, false, errors.New(errPagesSourceSyncFailed) + } + var committed model.PagesDeployment + reused := false + ingestReferenced := false + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + state, err := lockSourceCommitState(tx, snapshot) + if err != nil { + return err + } + target, targetReused, err := resolveSourceDeploymentTx( + tx, state, revision, detail, detailJSON, actor, manifest, ingestResult, hasIngest, + ) + if err != nil { + return err + } + if err := lockSourceDeploymentUploadsTx(tx, target, ingestResult, hasIngest); err != nil { + return err + } + if err := ensureDeploymentEntry(tx, target.ID, state.Project.RootDir, state.Project.EntryFile); err != nil { + return err + } + if err := refreshSourceCommitLease(state, snapshot); err != nil { + return err + } + if err := activateSourceDeploymentTx(tx, state, target, revision, detailJSON); err != nil { + return err + } + committed = *target + reused = targetReused + ingestReferenced = hasIngest && target.UploadID == ingestResult.Upload.ID + return nil + }) + if err != nil { + return nil, false, false, err + } + return &committed, reused, ingestReferenced, nil +} + +func lockSourceCommitState(tx *gorm.DB, snapshot *sourceExecutionSnapshot) (*sourceCommitState, error) { + state := &sourceCommitState{} + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + First(&project, snapshot.ProjectID).Error; err != nil { + return nil, sourceFenceRecordError(err) + } + if project.ContentConfigVersion != snapshot.ContentConfigVersion { + return nil, errSourceFinalFence + } + var source model.PagesProjectSource + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id = ? AND project_id = ?", snapshot.SourceID, snapshot.ProjectID). + First(&source).Error; err != nil { + return nil, sourceFenceRecordError(err) + } + if source.ConfigVersion != snapshot.SourceConfigVersion || + source.SourceIdentity != snapshot.SourceIdentity || + source.SourceType != snapshot.SourceType { + return nil, errSourceFinalFence + } + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", source.ID). + First(&runtime).Error; err != nil { + return nil, sourceFenceRecordError(err) + } + state.Project = &project + state.Source = &source + state.Runtime = &runtime + if err := refreshSourceCommitLease(state, snapshot); err != nil { + return nil, err + } + return state, nil +} + +func refreshSourceCommitLease(state *sourceCommitState, snapshot *sourceExecutionSnapshot) error { + if state == nil || state.Runtime == nil || snapshot == nil { + return errSourceFinalFence + } + now := sourceCommitNow() + if state.Runtime.LeaseToken != snapshot.LeaseToken || state.Runtime.LeaseExpiresAt == nil || + !state.Runtime.LeaseExpiresAt.After(now) { + return errSourceFinalFence + } + state.Now = now + return nil +} + +func sourceFenceRecordError(err error) error { + if errors.Is(err, gorm.ErrRecordNotFound) { + return errSourceFinalFence + } + return err +} + +func resolveSourceDeploymentTx( + tx *gorm.DB, + state *sourceCommitState, + revision string, + detail sourceDetail, + detailJSON string, + actor string, + manifest *deploymentManifest, + ingestResult upload.IngestResult, + hasIngest bool, +) (*model.PagesDeployment, bool, error) { + var target model.PagesDeployment + err := tx.Where( + "project_id = ? AND source_identity = ? AND source_revision = ?", + state.Project.ID, + state.Source.SourceIdentity, + revision, + ).First(&target).Error + if err == nil { + return &target, true, nil + } + if !errors.Is(err, gorm.ErrRecordNotFound) { + return nil, false, err + } + if !hasIngest { + return nil, false, errSourceFinalFence + } + return createSourceDeploymentTx( + tx, state, revision, detail, detailJSON, actor, manifest, ingestResult, + ) +} + +func createSourceDeploymentTx( + tx *gorm.DB, + state *sourceCommitState, + revision string, + detail sourceDetail, + detailJSON string, + actor string, + manifest *deploymentManifest, + ingestResult upload.IngestResult, +) (*model.PagesDeployment, bool, error) { + var maxNumber int + if err := tx.Model(&model.PagesDeployment{}). + Where("project_id = ?", state.Project.ID). + Select("COALESCE(MAX(deployment_number), 0)"). + Scan(&maxNumber).Error; err != nil { + return nil, false, err + } + identity := state.Source.SourceIdentity + revisionValue := revision + target := &model.PagesDeployment{ + ProjectID: state.Project.ID, + DeploymentNumber: maxNumber + 1, + Checksum: revision, + Status: model.PagesDeploymentStatusUploaded, + UploadID: ingestResult.Upload.ID, + FileCount: manifest.FileCount, + TotalSize: manifest.TotalSize, + CreatedBy: actor, + SourceType: state.Source.SourceType, + SourceIdentity: &identity, + SourceRevision: &revisionValue, + SourceLabel: detail.Label, + SourceMeta: detailJSON, + TriggerType: pagesSourceTriggerManualSync, + } + result := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(target) + if result.Error != nil { + return nil, false, result.Error + } + if result.RowsAffected == 0 { + return reloadSourceDeploymentTx(tx, state.Project.ID, identity, revision) + } + if err := createSourceDeploymentFilesTx(tx, target.ID, manifest.Files); err != nil { + return nil, false, err + } + return target, false, nil +} + +func reloadSourceDeploymentTx( + tx *gorm.DB, + projectID uint, + identity string, + revision string, +) (*model.PagesDeployment, bool, error) { + var target model.PagesDeployment + err := tx.Where( + "project_id = ? AND source_identity = ? AND source_revision = ?", + projectID, + identity, + revision, + ).First(&target).Error + return &target, true, err +} + +func createSourceDeploymentFilesTx(tx *gorm.DB, deploymentID uint, files []model.PagesDeploymentFile) error { + if len(files) == 0 { + return nil + } + for index := range files { + files[index].DeploymentID = deploymentID + } + return tx.Create(&files).Error +} + +func lockSourceDeploymentUploadsTx( + tx *gorm.DB, + target *model.PagesDeployment, + ingestResult upload.IngestResult, + hasIngest bool, +) error { + uploadIDs := []uint64{target.UploadID} + if hasIngest && ingestResult.Upload.ID != 0 && ingestResult.Upload.ID != target.UploadID { + uploadIDs = append(uploadIDs, ingestResult.Upload.ID) + } + sort.Slice(uploadIDs, func(i, j int) bool { return uploadIDs[i] < uploadIDs[j] }) + var records []model.Upload + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("id IN ?", uploadIDs). + Order("id asc"). + Find(&records).Error; err != nil { + return err + } + for index := range records { + if records[index].ID != target.UploadID { + continue + } + if records[index].Status == model.UploadStatusUsed && records[index].Type == upload.ReservedPagesDeploymentType { + return nil + } + break + } + return errSourceFinalFence +} + +func activateSourceDeploymentTx( + tx *gorm.DB, + state *sourceCommitState, + target *model.PagesDeployment, + revision string, + detailJSON string, +) error { + if err := tx.Model(&model.PagesDeployment{}). + Where("project_id = ?", state.Project.ID). + Update("status", model.PagesDeploymentStatusUploaded).Error; err != nil { + return err + } + if err := tx.Model(target).Updates(map[string]any{ + "status": model.PagesDeploymentStatusActive, + "activated_at": &state.Now, + }).Error; err != nil { + return err + } + if err := tx.Model(state.Project).Update("active_deployment_id", target.ID).Error; err != nil { + return err + } + finishedAt := sourceCommitNow() + result := tx.Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", + state.Runtime.SourceID, + state.Runtime.LeaseToken, + finishedAt, + ). + Updates(map[string]any{ + "last_seen_revision": revision, + "last_seen_detail": detailJSON, + "last_applied_revision": revision, + "last_applied_detail": detailJSON, + sourceRuntimeColumnSyncStatus: pagesSourceStatusIdle, + sourceRuntimeColumnLastError: "", + "last_checked_at": &finishedAt, + "last_synced_at": &finishedAt, + "next_check_at": nil, + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + }) + if result.Error != nil { + return result.Error + } + if result.RowsAffected != 1 { + return errSourceFinalFence + } + return nil +} + +func safeRemoteSourceLabel(raw string) string { + value := strings.ReplaceAll(strings.ToValidUTF8(raw, ""), "\\", "/") + value = path.Base(strings.TrimSpace(value)) + if value == "." || value == "/" { + value = "" + } + var builder strings.Builder + for _, character := range value { + if character >= 0x20 && character != 0x7f { + builder.WriteRune(character) + } + } + value = strings.TrimSpace(builder.String()) + if value == "" { + value = defaultRemoteAssetLabel + } + if len(value) > remoteSourceMaxSafeLabelBytes { + value = value[:remoteSourceMaxSafeLabelBytes] + for !utf8.ValidString(value) { + _, size := utf8.DecodeLastRuneInString(value) + value = value[:len(value)-size] + } + } + return value +} + +func sourceSyncResultDetail(outcome *sourceSyncOutcome) string { + if outcome == nil || outcome.Deployment == nil { + return "" + } + payload := map[string]any{ + "deployment_id": outcome.Deployment.ID, + "reused": outcome.Reused, + } + encoded, err := json.Marshal(payload) + if err != nil { + return fmt.Sprintf(`{"deployment_id":%d}`, outcome.Deployment.ID) + } + return string(encoded) +} diff --git a/internal/apps/openflare/pages/source_sync_test.go b/internal/apps/openflare/pages/source_sync_test.go new file mode 100644 index 00000000..870446a8 --- /dev/null +++ b/internal/apps/openflare/pages/source_sync_test.go @@ -0,0 +1,604 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" + "gorm.io/gorm" +) + +func setupPagesSourceSyncTest(t *testing.T) context.Context { + t.Helper() + ctx := setupPagesSourceTest(t) + _, disableStorage := setupPagesStorageMock(t) + t.Cleanup(disableStorage) + return ctx +} + +func mustAcquireRemoteSyncLease( + t *testing.T, + ctx context.Context, + source *model.PagesProjectSource, +) *sourceExecutionSnapshot { + t.Helper() + snapshot, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil { + t.Fatalf("acquireSourceLease(source=%d) error = %v, want nil", source.ID, err) + } + if got, want := outcome, sourceLeaseAcquired; got != want { + t.Fatalf("acquireSourceLease(source=%d) outcome = %q, want %q", source.ID, got, want) + } + if snapshot == nil { + t.Fatalf("acquireSourceLease(source=%d) snapshot = nil, want non-nil", source.ID) + } + return snapshot +} + +func mustCreateActiveManualDeployment( + t *testing.T, + ctx context.Context, + projectID uint, + content string, +) *model.PagesDeployment { + t.Helper() + view, err := UploadDeployment( + ctx, + projectID, + testPagesMultipartFile(t, "manual.zip", testPagesZip(t, map[string]string{"index.html": content})), + "user:1", + ) + if err != nil { + t.Fatalf("UploadDeployment(project=%d) error = %v, want nil", projectID, err) + } + if _, err := ActivateDeploymentAs(ctx, projectID, view.ID, "user:1"); err != nil { + t.Fatalf("ActivateDeploymentAs(project=%d, deployment=%d) error = %v, want nil", projectID, view.ID, err) + } + deployment, err := model.GetPagesDeploymentByID(ctx, view.ID) + if err != nil { + t.Fatalf("GetPagesDeploymentByID(%d) error = %v, want nil", view.ID, err) + } + return deployment +} + +func newPagesArchiveServer(t *testing.T, status int, body []byte, beforeWrite func() error) *httptest.Server { + t.Helper() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + if beforeWrite != nil { + if err := beforeWrite(); err != nil { + writer.WriteHeader(http.StatusInternalServerError) + return + } + } + writer.Header().Set("Content-Type", "application/zip") + writer.Header().Set("Content-Disposition", `attachment; filename="site.zip"`) + writer.WriteHeader(status) + _, _ = writer.Write(body) + })) + t.Cleanup(server.Close) + return server +} + +func TestSyncRemoteSourceAtomicallyActivatesAndReusesChecksum(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-success") + packageBytes := testPagesZip(t, map[string]string{ + "index.html": "remote-v1", + "assets/app.js": "console.log('ok')", + }) + server := newPagesArchiveServer(t, http.StatusOK, packageBytes, nil) + secret := "sync-query-secret" + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + server.URL+"/site.zip?token="+secret, + RemoteNetworkPolicyTrustedInternal, + ) + + firstSnapshot := mustAcquireRemoteSyncLease(t, ctx, source) + first, err := syncRemoteSource(ctx, firstSnapshot, "user:42") + if err != nil { + t.Fatalf("syncRemoteSource(first) error = %v, want nil", err) + } + if first == nil || first.Stale || first.Reused || first.Deployment == nil { + t.Fatalf("syncRemoteSource(first) = %+v, want new active deployment", first) + } + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", project.ID, err) + } + if storedProject.ActiveDeploymentID == nil || *storedProject.ActiveDeploymentID != first.Deployment.ID { + t.Fatalf("project ActiveDeploymentID = %v, want %d", storedProject.ActiveDeploymentID, first.Deployment.ID) + } + deployment, err := model.GetPagesDeploymentByID(ctx, first.Deployment.ID) + if err != nil { + t.Fatalf("GetPagesDeploymentByID(%d) error = %v, want nil", first.Deployment.ID, err) + } + expectedHash := sha256.Sum256(packageBytes) + if got, want := deployment.Checksum, hex.EncodeToString(expectedHash[:]); got != want { + t.Errorf("deployment Checksum = %q, want %q", got, want) + } + if got, want := deployment.Status, model.PagesDeploymentStatusActive; got != want { + t.Errorf("deployment Status = %q, want %q", got, want) + } + if got, want := deployment.SourceType, PagesSourceTypeRemoteURL; got != want { + t.Errorf("deployment SourceType = %q, want %q", got, want) + } + if deployment.SourceIdentity == nil || *deployment.SourceIdentity != source.SourceIdentity { + t.Errorf("deployment SourceIdentity = %v, want %q", deployment.SourceIdentity, source.SourceIdentity) + } + if deployment.SourceRevision == nil || *deployment.SourceRevision != deployment.Checksum { + t.Errorf("deployment SourceRevision = %v, want %q", deployment.SourceRevision, deployment.Checksum) + } + if got, want := deployment.CreatedBy, "user:42"; got != want { + t.Errorf("deployment CreatedBy = %q, want %q", got, want) + } + if got, want := deployment.TriggerType, pagesSourceTriggerManualSync; got != want { + t.Errorf("deployment TriggerType = %q, want %q", got, want) + } + if strings.Contains(deployment.SourceMeta, secret) || strings.Contains(deployment.SourceLabel, secret) { + t.Errorf("deployment provenance = label:%q meta:%q, want no query secret", deployment.SourceLabel, deployment.SourceMeta) + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&runtime).Error; err != nil { + t.Fatalf("load source runtime error = %v, want nil", err) + } + if got, want := runtime.SyncStatus, pagesSourceStatusIdle; got != want { + t.Errorf("runtime SyncStatus = %q, want %q", got, want) + } + if got, want := runtime.LastAppliedRevision, deployment.Checksum; got != want { + t.Errorf("runtime LastAppliedRevision = %q, want %q", got, want) + } + if runtime.LeaseToken != "" || runtime.LeaseExpiresAt != nil { + t.Errorf("runtime lease = (%q, %v), want cleared", runtime.LeaseToken, runtime.LeaseExpiresAt) + } + var uploadRecord model.Upload + if err := db.DB(ctx).First(&uploadRecord, deployment.UploadID).Error; err != nil { + t.Fatalf("load deployment upload %d error = %v, want nil", deployment.UploadID, err) + } + if got, want := uploadRecord.Status, model.UploadStatusUsed; got != want { + t.Errorf("deployment upload Status = %q, want %q", got, want) + } + if got, want := uploadRecord.Type, upload.ReservedPagesDeploymentType; got != want { + t.Errorf("deployment upload Type = %q, want %q", got, want) + } + if got, want := fmt.Sprint(uploadRecord.Metadata.Extra[pagesSourceIDMetadataKey]), fmt.Sprint(source.ID); got != want { + t.Errorf("deployment upload pages_source_id = %q, want %q", got, want) + } + + secondSnapshot := mustAcquireRemoteSyncLease(t, ctx, source) + second, err := syncRemoteSource(ctx, secondSnapshot, "user:42") + if err != nil { + t.Fatalf("syncRemoteSource(second) error = %v, want nil", err) + } + if second == nil || second.Stale || !second.Reused || second.Deployment == nil { + t.Fatalf("syncRemoteSource(second) = %+v, want reused active deployment", second) + } + if got, want := second.Deployment.ID, first.Deployment.ID; got != want { + t.Errorf("reused deployment ID = %d, want %d", got, want) + } + var deploymentCount, uploadCount int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}).Where("project_id = ?", project.ID).Count(&deploymentCount).Error; err != nil { + t.Fatalf("count source deployments error = %v, want nil", err) + } + if err := db.DB(ctx).Model(&model.Upload{}).Count(&uploadCount).Error; err != nil { + t.Fatalf("count source uploads error = %v, want nil", err) + } + if got, want := deploymentCount, int64(1); got != want { + t.Errorf("deployment count after identical sync = %d, want %d", got, want) + } + if got, want := uploadCount, int64(1); got != want { + t.Errorf("upload count after identical sync = %d, want %d", got, want) + } +} + +func TestSyncRemoteSourceDownloadFailureKeepsOldActive(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-download-fail") + oldActive := mustCreateActiveManualDeployment(t, ctx, project.ID, "old-active") + server := newPagesArchiveServer(t, http.StatusBadGateway, []byte("upstream failed"), nil) + secret := "download-failure-secret" + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + server.URL+"/site.zip?token="+secret, + RemoteNetworkPolicyTrustedInternal, + ) + + _, err := syncRemoteSource(ctx, mustAcquireRemoteSyncLease(t, ctx, source), "user:2") + if err == nil { + t.Fatal("syncRemoteSource(download failure) error = nil, want non-nil") + } + if strings.Contains(err.Error(), secret) { + t.Errorf("syncRemoteSource(download failure) error = %q, want no query secret", err) + } + assertPagesSyncFailureState(t, ctx, project.ID, source.ID, oldActive.ID, 1) +} + +func TestSyncRemoteSourceArchiveFailureKeepsOldActive(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-archive-fail") + oldActive := mustCreateActiveManualDeployment(t, ctx, project.ID, "old-active") + server := newPagesArchiveServer(t, http.StatusOK, []byte("not-a-valid-zip"), nil) + secret := "archive-failure-secret" + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + server.URL+"/site.zip?token="+secret, + RemoteNetworkPolicyTrustedInternal, + ) + + _, err := syncRemoteSource(ctx, mustAcquireRemoteSyncLease(t, ctx, source), "user:3") + if err == nil { + t.Fatal("syncRemoteSource(archive failure) error = nil, want non-nil") + } + if strings.Contains(err.Error(), secret) { + t.Errorf("syncRemoteSource(archive failure) error = %q, want no query secret", err) + } + assertPagesSyncFailureState(t, ctx, project.ID, source.ID, oldActive.ID, 1) +} + +func TestSyncRemoteSourceFinalFenceCompensatesIngest(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-final-fence") + oldActive := mustCreateActiveManualDeployment(t, ctx, project.ID, "old-active") + packageBytes := testPagesZip(t, map[string]string{"index.html": "never-activate"}) + mutationResult := make(chan error, 1) + server := newPagesArchiveServer(t, http.StatusOK, packageBytes, func() error { + err := db.DB(context.Background()).Model(&model.PagesProject{}). + Where("id = ?", project.ID). + Update("content_config_version", gorm.Expr("content_config_version + 1")).Error + mutationResult <- err + return err + }) + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + server.URL+"/site.zip?token=final-fence-secret", + RemoteNetworkPolicyTrustedInternal, + ) + + outcome, err := syncRemoteSource(ctx, mustAcquireRemoteSyncLease(t, ctx, source), "user:4") + if err != nil { + t.Fatalf("syncRemoteSource(final fence) error = %v, want nil stale outcome", err) + } + select { + case mutationErr := <-mutationResult: + if mutationErr != nil { + t.Fatalf("content version mutation error = %v, want nil", mutationErr) + } + case <-time.After(2 * time.Second): + t.Fatal("content version mutation was not observed") + } + if outcome == nil || !outcome.Stale || outcome.Deployment != nil { + t.Fatalf("syncRemoteSource(final fence) = %+v, want stale outcome without deployment", outcome) + } + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", project.ID, err) + } + if storedProject.ActiveDeploymentID == nil || *storedProject.ActiveDeploymentID != oldActive.ID { + t.Errorf("ActiveDeploymentID after final fence = %v, want %d", storedProject.ActiveDeploymentID, oldActive.ID) + } + var deployments []model.PagesDeployment + if err := db.DB(ctx).Where("project_id = ?", project.ID).Find(&deployments).Error; err != nil { + t.Fatalf("list deployments after final fence error = %v, want nil", err) + } + if got, want := len(deployments), 1; got != want { + t.Errorf("deployment count after final fence = %d, want %d", got, want) + } + var uploads []model.Upload + if err := db.DB(ctx).Order("id asc").Find(&uploads).Error; err != nil { + t.Fatalf("list uploads after final fence error = %v, want nil", err) + } + var compensated *model.Upload + for index := range uploads { + if fmt.Sprint(uploads[index].Metadata.Extra[pagesSourceIDMetadataKey]) == fmt.Sprint(source.ID) { + compensated = &uploads[index] + break + } + } + if compensated == nil { + t.Fatalf("source upload after final fence = nil, want compensated upload record") + } + if got, want := compensated.Status, model.UploadStatusDeleted; got != want { + t.Errorf("compensated upload Status = %q, want %q", got, want) + } + var danglingCount int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}). + Where("upload_id = ?", compensated.ID). + Count(&danglingCount).Error; err != nil { + t.Fatalf("count compensated upload references error = %v, want nil", err) + } + if got, want := danglingCount, int64(0); got != want { + t.Errorf("deployments referencing compensated upload = %d, want %d", got, want) + } +} + +func TestCommitSourceDeploymentRechecksLeaseAfterUploadLocks(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-expiry-recheck") + packageBytes := testPagesZip(t, map[string]string{"index.html": "expiry-recheck"}) + server := newPagesArchiveServer(t, http.StatusOK, packageBytes, nil) + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + server.URL+"/site.zip", + RemoteNetworkPolicyTrustedInternal, + ) + first, err := syncRemoteSource(ctx, mustAcquireRemoteSyncLease(t, ctx, source), "user:5") + if err != nil || first == nil || first.Deployment == nil { + t.Fatalf("syncRemoteSource(seed) = (%+v, %v), want deployment", first, err) + } + deployment, err := model.GetPagesDeploymentByID(ctx, first.Deployment.ID) + if err != nil { + t.Fatalf("GetPagesDeploymentByID(%d) error = %v, want nil", first.Deployment.ID, err) + } + if err := db.DB(ctx).Model(&model.PagesProject{}). + Where("id = ?", project.ID). + Update("active_deployment_id", nil).Error; err != nil { + t.Fatalf("clear active deployment error = %v, want nil", err) + } + if err := db.DB(ctx).Model(&model.PagesDeployment{}). + Where("id = ?", deployment.ID). + Update("status", model.PagesDeploymentStatusUploaded).Error; err != nil { + t.Fatalf("reset deployment status error = %v, want nil", err) + } + + snapshot := mustAcquireRemoteSyncLease(t, ctx, source) + expiresAt := time.Now().Add(time.Hour) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ? AND lease_token = ?", source.ID, snapshot.LeaseToken). + Update("lease_expires_at", &expiresAt).Error; err != nil { + t.Fatalf("set deterministic lease expiry error = %v, want nil", err) + } + originalNow := sourceCommitNow + nowCalls := 0 + sourceCommitNow = func() time.Time { + nowCalls++ + if nowCalls == 1 { + return expiresAt.Add(-time.Second) + } + return expiresAt.Add(time.Second) + } + t.Cleanup(func() { sourceCommitNow = originalNow }) + + _, _, _, err = commitSourceDeployment( + ctx, + snapshot, + deployment.Checksum, + sourceDetail{Provider: PagesSourceTypeRemoteURL, Label: deployment.SourceLabel}, + deployment.SourceMeta, + "user:5", + &deploymentManifest{}, + upload.IngestResult{}, + false, + ) + if !errors.Is(err, errSourceFinalFence) { + t.Fatalf("commitSourceDeployment(expired after upload lock) error = %v, want %v", err, errSourceFinalFence) + } + if nowCalls != 2 { + t.Fatalf("sourceCommitNow calls = %d, want runtime-lock and post-upload checks", nowCalls) + } + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", project.ID, err) + } + if storedProject.ActiveDeploymentID != nil { + t.Errorf("ActiveDeploymentID after expiry recheck = %v, want nil", storedProject.ActiveDeploymentID) + } + storedDeployment, err := model.GetPagesDeploymentByID(ctx, deployment.ID) + if err != nil { + t.Fatalf("GetPagesDeploymentByID(%d) after expiry error = %v, want nil", deployment.ID, err) + } + if got, want := storedDeployment.Status, model.PagesDeploymentStatusUploaded; got != want { + t.Errorf("deployment status after expiry recheck = %q, want %q", got, want) + } +} + +func TestCompensateSourceIngestSurvivesCanceledParentContext(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-canceled-compensation") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip", + RemoteNetworkPolicyPublic, + ) + packageBytes := testPagesZip(t, map[string]string{"index.html": "cancel-compensation"}) + packagePath := filepath.Join(t.TempDir(), "site.zip") + if err := os.WriteFile(packagePath, packageBytes, 0o600); err != nil { + t.Fatalf("write test package error = %v, want nil", err) + } + digest := sha256.Sum256(packageBytes) + result, err := ingestPagesDeploymentPackageWithSource( + ctx, + packagePath, + hex.EncodeToString(digest[:]), + project.ID, + source.ID, + "site.zip", + pagesarchive.FormatZip, + ) + if err != nil { + t.Fatalf("ingestPagesDeploymentPackageWithSource() error = %v, want nil", err) + } + if !result.Created { + t.Fatal("ingest result Created = false, want a compensatable record") + } + + canceledCtx, cancel := context.WithCancel(ctx) + cancel() + compensateSourceIngest(canceledCtx, &sourceExecutionSnapshot{ + ProjectID: project.ID, + SourceID: source.ID, + }, &sourceIngestState{ + Result: result, + HasIngest: true, + }) + + var uploadRecord model.Upload + if err := db.DB(ctx).Where("id = ?", result.Upload.ID).First(&uploadRecord).Error; err != nil { + t.Fatalf("load compensated upload error = %v, want nil", err) + } + if got, want := uploadRecord.Status, model.UploadStatusDeleted; got != want { + t.Errorf("compensated upload status = %q, want %q", got, want) + } +} + +func assertPagesSyncFailureState( + t *testing.T, + ctx context.Context, + projectID uint, + sourceID uint, + oldActiveID uint, + wantDeploymentCount int64, +) { + t.Helper() + project, err := model.GetPagesProjectByID(ctx, projectID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", projectID, err) + } + if project.ActiveDeploymentID == nil || *project.ActiveDeploymentID != oldActiveID { + t.Errorf("project %d ActiveDeploymentID = %v, want %d", projectID, project.ActiveDeploymentID, oldActiveID) + } + var deploymentCount int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}). + Where("project_id = ?", projectID). + Count(&deploymentCount).Error; err != nil { + t.Fatalf("count project %d deployments error = %v, want nil", projectID, err) + } + if got, want := deploymentCount, wantDeploymentCount; got != want { + t.Errorf("project %d deployment count = %d, want %d", projectID, got, want) + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", sourceID).First(&runtime).Error; err != nil { + t.Fatalf("load source %d runtime error = %v, want nil", sourceID, err) + } + if got, want := runtime.SyncStatus, pagesSourceStatusFailed; got != want { + t.Errorf("source %d runtime SyncStatus = %q, want %q", sourceID, got, want) + } + if runtime.LeaseToken != "" || runtime.LeaseExpiresAt != nil { + t.Errorf("source %d runtime lease = (%q, %v), want cleared", sourceID, runtime.LeaseToken, runtime.LeaseExpiresAt) + } +} + +func TestCommitSourceDeploymentRejectsDeletedTargetUpload(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "sync-deleted-upload") + packageBytes := testPagesZip(t, map[string]string{"index.html": "content"}) + server := newPagesArchiveServer(t, http.StatusOK, packageBytes, nil) + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + server.URL+"/site.zip", + RemoteNetworkPolicyTrustedInternal, + ) + snapshot := mustAcquireRemoteSyncLease(t, ctx, source) + + // A pre-existing source deployment whose upload was removed must never be + // reactivated into a dangling active pointer. + identity := source.SourceIdentity + revision := strings.Repeat("d", 64) + uploadRecord := &model.Upload{ + ID: 987654321, + UserID: 999, + FileName: "deleted.zip", + FilePath: "deleted.zip", + FileSize: 1, + MimeType: "application/zip", + Extension: "zip", + Hash: revision, + Type: upload.ReservedPagesDeploymentType, + Status: model.UploadStatusDeleted, + AccessMode: 0, + } + if err := db.DB(ctx).Create(uploadRecord).Error; err != nil { + t.Fatalf("create deleted upload error = %v, want nil", err) + } + deployment := &model.PagesDeployment{ + ProjectID: project.ID, + DeploymentNumber: 1, + Checksum: revision, + Status: model.PagesDeploymentStatusUploaded, + UploadID: uploadRecord.ID, + FileCount: 1, + TotalSize: 1, + CreatedBy: "user:1", + SourceType: PagesSourceTypeRemoteURL, + SourceIdentity: &identity, + SourceRevision: &revision, + SourceLabel: "deleted.zip", + SourceMeta: `{"provider":"remote_url","label":"deleted.zip"}`, + TriggerType: pagesSourceTriggerManualSync, + } + if err := db.DB(ctx).Create(deployment).Error; err != nil { + t.Fatalf("create source deployment error = %v, want nil", err) + } + if err := db.DB(ctx).Create(&model.PagesDeploymentFile{ + DeploymentID: deployment.ID, + Path: "index.html", + Size: 1, + Checksum: revision, + }).Error; err != nil { + t.Fatalf("create source deployment file error = %v, want nil", err) + } + manifest := &deploymentManifest{ + FileCount: 1, + TotalSize: 1, + EntryFile: "index.html", + } + _, _, _, err := commitSourceDeployment( + ctx, + snapshot, + revision, + sourceDetail{Provider: PagesSourceTypeRemoteURL, Label: "deleted.zip"}, + `{"provider":"remote_url","label":"deleted.zip"}`, + "user:1", + manifest, + upload.IngestResult{}, + false, + ) + if !errors.Is(err, errSourceFinalFence) { + t.Errorf("commitSourceDeployment(deleted upload) error = %v, want %v", err, errSourceFinalFence) + } + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", project.ID, err) + } + if storedProject.ActiveDeploymentID != nil { + t.Errorf("ActiveDeploymentID after deleted upload rejection = %v, want nil", storedProject.ActiveDeploymentID) + } + var activeCount int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}). + Where("project_id = ? AND status = ?", project.ID, model.PagesDeploymentStatusActive). + Count(&activeCount).Error; err != nil && !errors.Is(err, gorm.ErrRecordNotFound) { + t.Fatalf("count active deployments error = %v, want nil", err) + } + if got, want := activeCount, int64(0); got != want { + t.Errorf("active deployment count after deleted upload rejection = %d, want %d", got, want) + } +} diff --git a/internal/apps/openflare/pages/source_tasks.go b/internal/apps/openflare/pages/source_tasks.go new file mode 100644 index 00000000..5fef7d5b --- /dev/null +++ b/internal/apps/openflare/pages/source_tasks.go @@ -0,0 +1,275 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "bytes" + "context" + "encoding/hex" + "encoding/json" + "errors" + "io" + "strconv" + "strings" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/task" + "github.com/Rain-kl/Wavelet/pkg/logger" + "gorm.io/gorm" +) + +const ( + // PagesSourceActionTask is the private Asynq task type for source actions. + PagesSourceActionTask = "openflare:pages_source_action" + // TaskTypePagesSourceAction is the internal task meta type. + TaskTypePagesSourceAction = "of_pages_source_action" + + sourceActionCheck = "check" + sourceActionSync = "sync" +) + +var errUnexpectedJSONTrailingValue = errors.New("unexpected trailing JSON value") + +// PagesSourceActionMeta is intentionally hidden from generic Admin task APIs. +var PagesSourceActionMeta = task.TaskMeta{ + Type: TaskTypePagesSourceAction, + AsynqTask: PagesSourceActionTask, + Name: "OpenFlare Pages 部署源操作", + Description: "检查或同步 Pages 项目部署源", + SupportsTime: false, + MaxRetry: 2, + Queue: task.QueueDefault, + Retryable: false, + InternalOnly: true, +} + +// SourceActionPayload is the credential-free internal queue contract. +type SourceActionPayload struct { + SourceID uint `json:"source_id"` + ConfigVersion int `json:"config_version"` + Action string `json:"action"` + Actor string `json:"actor"` + TargetRevision string `json:"target_revision"` + ConfirmedRevision string `json:"confirmed_revision"` +} + +// SourceActionHandler executes a validated source action. +type SourceActionHandler struct{} + +// ValidatePayload rejects unknown keys and normalizes the internal contract. +func (h *SourceActionHandler) ValidatePayload(payload []byte) ([]byte, error) { + var input SourceActionPayload + decoder := json.NewDecoder(bytes.NewReader(payload)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(&input); err != nil { + return nil, errors.New(errPagesSourceActionInvalid) + } + if err := ensureJSONEOF(decoder); err != nil { + return nil, errors.New(errPagesSourceActionInvalid) + } + input.Action = strings.TrimSpace(input.Action) + input.Actor = strings.TrimSpace(input.Actor) + input.TargetRevision = strings.TrimSpace(input.TargetRevision) + input.ConfirmedRevision = strings.TrimSpace(input.ConfirmedRevision) + if input.SourceID == 0 || input.ConfigVersion <= 0 || + (input.Action != sourceActionCheck && input.Action != sourceActionSync) || + !validPagesSourceActor(input.Actor) || + !validOptionalSourceRevision(input.TargetRevision) || + !validOptionalSourceRevision(input.ConfirmedRevision) { + return nil, errors.New(errPagesSourceActionInvalid) + } + return json.Marshal(input) +} + +// Execute validates again inside the worker and performs the source action. +func (h *SourceActionHandler) Execute(ctx context.Context, payload []byte) (*task.TaskResult, error) { + normalized, err := h.ValidatePayload(payload) + if err != nil { + return nil, task.PermanentError(errPagesSourceActionInvalid) + } + var input SourceActionPayload + if err := json.Unmarshal(normalized, &input); err != nil { + return nil, task.PermanentError(errPagesSourceActionInvalid) + } + + var source model.PagesProjectSource + if err := db.DB(ctx).Where("id = ?", input.SourceID).First(&source).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + task.AppendLog(ctx, "[resolve] 来源已不存在，本次任务跳过") + return &task.TaskResult{Message: errPagesSourceActionStale}, nil + } + logger.ErrorF(ctx, "[PagesSource] load source failed: source_id=%d error=%v", input.SourceID, err) + return nil, errors.New(errPagesSourceSyncFailed) + } + if source.ConfigVersion != input.ConfigVersion { + task.AppendLog(ctx, "[resolve] 来源配置已变化，本次任务跳过") + return &task.TaskResult{Message: errPagesSourceActionStale}, nil + } + if input.Action == sourceActionCheck && source.SourceType == PagesSourceTypeRemoteURL { + return nil, task.PermanentError(errPagesSourceCheckUnsupported) + } + if source.SourceType != PagesSourceTypeRemoteURL { + return nil, task.PermanentError(errPagesSourceTypeUnsupported) + } + if input.TargetRevision != "" || input.ConfirmedRevision != "" { + return nil, task.PermanentError(errPagesSourceActionInvalid) + } + + task.AppendLog(ctx, "[resolve] 正在获取来源执行权") + snapshot, outcome, err := acquireSourceLease(ctx, input.SourceID, input.ConfigVersion, input.Action) + if err != nil { + logger.ErrorF(ctx, "[PagesSource] acquire lease failed: source_id=%d error=%v", input.SourceID, err) + return nil, errors.New(errPagesSourceSyncFailed) + } + switch outcome { + case sourceLeaseBusy: + task.AppendLog(ctx, "[resolve] 已有来源任务正在执行，本次任务跳过") + return &task.TaskResult{Message: errPagesSourceActionBusy}, nil + case sourceLeaseStale: + task.AppendLog(ctx, "[resolve] 来源配置或执行权已变化，本次任务跳过") + return &task.TaskResult{Message: errPagesSourceActionStale}, nil + } + + result, err := syncRemoteSource(ctx, snapshot, input.Actor) + if err != nil { + logger.ErrorF(ctx, "[PagesSource] sync failed: project_id=%d source_id=%d error=%v", snapshot.ProjectID, snapshot.SourceID, err) + if isPermanentSourceSyncError(err) { + return nil, task.PermanentError(errPagesSourceSyncFailed) + } + return nil, errors.New(errPagesSourceSyncFailed) + } + if result == nil || result.Stale { + task.AppendLog(ctx, "[activate] 来源配置或执行权已变化，本次任务未切换部署") + return &task.TaskResult{Message: errPagesSourceActionStale}, nil + } + message := "Pages 部署源同步并发布成功" + if result.Reused { + message = "Pages 部署源内容未变化，已重新激活现有部署" + } + return &task.TaskResult{Message: message, Detail: sourceSyncResultDetail(result)}, nil +} + +func ensureJSONEOF(decoder *json.Decoder) error { + var trailing any + err := decoder.Decode(&trailing) + if errors.Is(err, io.EOF) { + return nil + } + if err == nil { + return errUnexpectedJSONTrailingValue + } + return err +} + +func validPagesSourceActor(actor string) bool { + if actor == pagesSourceCreatedBySystem { + return true + } + if !strings.HasPrefix(actor, "user:") { + return false + } + id, err := strconv.ParseUint(strings.TrimPrefix(actor, "user:"), 10, 64) + return err == nil && id > 0 +} + +func validOptionalSourceRevision(value string) bool { + if value == "" { + return true + } + if len(value) != sourceRevisionHexLength { + return false + } + decoded, err := hex.DecodeString(value) + return err == nil && len(decoded) == 32 +} + +func isPermanentSourceSyncError(err error) bool { + if err == nil { + return false + } + message := err.Error() + return strings.Contains(message, errPagesPackageUnsupported) || + strings.Contains(message, errPagesPackageInvalid) || + strings.Contains(message, errPagesPackageEmpty) || + strings.Contains(message, errPagesPackageExtractedTooLarge) || + strings.Contains(message, errPagesPackageFileTooLarge) || + strings.Contains(message, errPagesEntryFileMissing) || + strings.Contains(message, errPagesSourceRemoteURLInvalid) || + strings.Contains(message, errPagesSourceNetworkPolicy) +} + +// DispatchSourceAction performs API preflight and enqueues a credential-free action. +func DispatchSourceAction( + ctx context.Context, + projectID uint, + action string, + actor string, + confirmedRevision string, +) (*SourceActionReceipt, error) { + action = strings.TrimSpace(action) + confirmedRevision = strings.TrimSpace(confirmedRevision) + if action != sourceActionCheck && action != sourceActionSync { + return nil, errors.New(errPagesSourceActionInvalid) + } + if !validPagesSourceActor(actor) { + return nil, errors.New(errPagesSourceActionInvalid) + } + + var source model.PagesProjectSource + if err := db.DB(ctx).Where("project_id = ?", projectID).First(&source).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil, errors.New(errPagesSourceNotFound) + } + return nil, err + } + if source.SourceType != PagesSourceTypeRemoteURL { + return nil, errors.New(errPagesSourceTypeUnsupported) + } + if action == sourceActionCheck { + return nil, errors.New(errPagesSourceCheckUnsupported) + } + if confirmedRevision != "" { + return nil, errors.New(errPagesSourceActionInvalid) + } + busy, err := sourceLeaseIsBusy(ctx, source.ID) + if err != nil { + return nil, err + } + if busy { + return nil, errors.New(errPagesSourceActionBusy) + } + + handler := &SourceActionHandler{} + rawPayload, err := json.Marshal(SourceActionPayload{ + SourceID: source.ID, + ConfigVersion: source.ConfigVersion, + Action: action, + Actor: actor, + TargetRevision: "", + ConfirmedRevision: confirmedRevision, + }) + if err != nil { + return nil, errors.New(errPagesSourceActionInvalid) + } + payload, err := handler.ValidatePayload(rawPayload) + if err != nil { + return nil, err + } + taskID, err := task.DispatchTask(ctx, TaskTypePagesSourceAction, payload, "manual") + if err != nil { + logger.ErrorF(ctx, "[PagesSource] dispatch action failed: project_id=%d source_id=%d action=%s error=%v", projectID, source.ID, action, err) + return nil, errors.New(errPagesSourceTaskDispatchFailed) + } + execution, err := model.GetTaskExecutionByTaskID(ctx, taskID) + if err != nil { + logger.ErrorF(ctx, "[PagesSource] load dispatched execution failed: source_id=%d task_id=%s error=%v", source.ID, taskID, err) + return nil, errors.New(errPagesSourceTaskDispatchFailed) + } + return &SourceActionReceipt{ + TaskID: taskID, + ExecutionID: strconv.FormatUint(execution.ID, 10), + Action: action, + }, nil +} diff --git a/internal/apps/openflare/pages/source_tasks_test.go b/internal/apps/openflare/pages/source_tasks_test.go new file mode 100644 index 00000000..92458dc6 --- /dev/null +++ b/internal/apps/openflare/pages/source_tasks_test.go @@ -0,0 +1,166 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "encoding/json" + "errors" + "strings" + "testing" + + "github.com/hibiken/asynq" +) + +func TestSourceActionPayloadValidationIsStrictAndCredentialFree(t *testing.T) { + handler := &SourceActionHandler{} + valid := SourceActionPayload{ + SourceID: 7, + ConfigVersion: 3, + Action: sourceActionSync, + Actor: "user:42", + } + raw, err := json.Marshal(valid) + if err != nil { + t.Fatalf("json.Marshal(valid payload) error = %v, want nil", err) + } + normalized, err := handler.ValidatePayload(raw) + if err != nil { + t.Fatalf("ValidatePayload(valid) error = %v, want nil", err) + } + var got SourceActionPayload + if err := json.Unmarshal(normalized, &got); err != nil { + t.Fatalf("json.Unmarshal(normalized payload) error = %v, want nil", err) + } + if got != valid { + t.Errorf("ValidatePayload(valid) = %+v, want %+v", got, valid) + } + for _, forbidden := range []string{"remote_url", "content_config_version", "expected_revision", "lease_token", "etag"} { + if strings.Contains(string(normalized), forbidden) { + t.Errorf("normalized payload = %s, want no forbidden field %q", normalized, forbidden) + } + } + + invalidPayloads := []struct { + name string + raw string + }{ + { + name: "unknown remote URL field", + raw: `{"source_id":7,"config_version":3,"action":"sync","actor":"user:42","remote_url":"https://example.com/site.zip?token=secret"}`, + }, + { + name: "unknown content version field", + raw: `{"source_id":7,"config_version":3,"action":"sync","actor":"user:42","content_config_version":9}`, + }, + { + name: "empty actor", + raw: `{"source_id":7,"config_version":3,"action":"sync","actor":""}`, + }, + { + name: "untrusted system actor", + raw: `{"source_id":7,"config_version":3,"action":"sync","actor":"system"}`, + }, + { + name: "zero user actor", + raw: `{"source_id":7,"config_version":3,"action":"sync","actor":"user:0"}`, + }, + { + name: "invalid action", + raw: `{"source_id":7,"config_version":3,"action":"activate","actor":"user:42"}`, + }, + { + name: "multiple JSON values", + raw: `{"source_id":7,"config_version":3,"action":"sync","actor":"user:42"} {}`, + }, + } + for _, test := range invalidPayloads { + t.Run(test.name, func(t *testing.T) { + normalized, err := handler.ValidatePayload([]byte(test.raw)) + if err == nil { + t.Errorf("ValidatePayload(%s) = %s, nil; want non-nil error", test.raw, normalized) + } + if err != nil && strings.Contains(err.Error(), "secret") { + t.Errorf("ValidatePayload(%s) error = %q, want credential-free error", test.name, err) + } + }) + } +} + +func TestSourceActionPayloadAcceptsOnlyRealActors(t *testing.T) { + tests := []struct { + actor string + want bool + }{ + {actor: "user:1", want: true}, + {actor: "user:18446744073709551615", want: true}, + {actor: pagesSourceCreatedBySystem, want: true}, + {actor: "", want: false}, + {actor: "user:0", want: false}, + {actor: "user:-1", want: false}, + {actor: "user:not-a-number", want: false}, + {actor: "system", want: false}, + } + for _, test := range tests { + if got := validPagesSourceActor(test.actor); got != test.want { + t.Errorf("validPagesSourceActor(%q) = %t, want %t", test.actor, got, test.want) + } + } +} + +func TestRemoteCheckActionIsPermanentWithoutExposingURL(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "task-remote-check") + secret := "task-query-secret" + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip?token="+secret, + RemoteNetworkPolicyPublic, + ) + raw, err := json.Marshal(SourceActionPayload{ + SourceID: source.ID, + ConfigVersion: source.ConfigVersion, + Action: sourceActionCheck, + Actor: "user:9", + }) + if err != nil { + t.Fatalf("json.Marshal(check payload) error = %v, want nil", err) + } + + result, err := (&SourceActionHandler{}).Execute(ctx, raw) + if result != nil { + t.Errorf("SourceActionHandler.Execute(remote check) result = %+v, want nil", result) + } + if err == nil { + t.Fatal("SourceActionHandler.Execute(remote check) error = nil, want permanent error") + } + if !errors.Is(err, asynq.SkipRetry) { + t.Errorf("SourceActionHandler.Execute(remote check) error = %v, want errors.Is(asynq.SkipRetry)", err) + } + if got, want := err.Error(), errPagesSourceCheckUnsupported; got != want { + t.Errorf("SourceActionHandler.Execute(remote check) error = %q, want %q", got, want) + } + if strings.Contains(err.Error(), secret) || strings.Contains(string(raw), secret) { + t.Errorf("remote check result error/payload = %q / %s, want no URL secret", err, raw) + } +} + +func TestPagesSourceActionMetaIsInternalOnly(t *testing.T) { + if !PagesSourceActionMeta.InternalOnly { + t.Error("PagesSourceActionMeta.InternalOnly = false, want true") + } + if PagesSourceActionMeta.Type != TaskTypePagesSourceAction { + t.Errorf("PagesSourceActionMeta.Type = %q, want %q", PagesSourceActionMeta.Type, TaskTypePagesSourceAction) + } + if PagesSourceActionMeta.AsynqTask != PagesSourceActionTask { + t.Errorf("PagesSourceActionMeta.AsynqTask = %q, want %q", PagesSourceActionMeta.AsynqTask, PagesSourceActionTask) + } + if PagesSourceActionMeta.Retryable { + t.Error("PagesSourceActionMeta.Retryable = true, want false for manual retry API") + } + if PagesSourceActionMeta.MaxRetry <= 0 { + t.Errorf("PagesSourceActionMeta.MaxRetry = %d, want bounded transient retries", PagesSourceActionMeta.MaxRetry) + } +} diff --git a/internal/apps/openflare/pages/source_test.go b/internal/apps/openflare/pages/source_test.go new file mode 100644 index 00000000..61f9dabe --- /dev/null +++ b/internal/apps/openflare/pages/source_test.go @@ -0,0 +1,386 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" +) + +func setupPagesSourceTest(t *testing.T) context.Context { + t.Helper() + cleanup := setupPagesTestDB(t) + t.Cleanup(cleanup) + sqlDB, err := db.DB(t.Context()).DB() + if err != nil { + t.Fatalf("db.DB().DB() error = %v, want nil", err) + } + // SQLite :memory: is scoped to one connection. Keeping one connection also + // makes lease tests exercise the production CAS without creating empty + // per-connection databases. + sqlDB.SetMaxOpenConns(1) + return t.Context() +} + +func mustCreatePagesSourceProject(t *testing.T, ctx context.Context, slug string) *model.PagesProject { + t.Helper() + view, err := CreateProject(ctx, Input{ + Name: "Source " + slug, + Slug: slug, + Enabled: true, + EntryFile: "index.html", + }) + if err != nil { + t.Fatalf("CreateProject(%q) error = %v, want nil", slug, err) + } + project, err := model.GetPagesProjectByID(ctx, view.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", view.ID, err) + } + return project +} + +func mustConfigureRemoteSource( + t *testing.T, + ctx context.Context, + projectID uint, + remoteURL string, + policy string, +) (*model.PagesProjectSource, *model.PagesProjectSourceRuntime) { + t.Helper() + _, err := UpdateSource(ctx, projectID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: remoteURL, + RemoteNetworkPolicy: policy, + }) + if err != nil { + t.Fatalf("UpdateSource(%d, %q) error = %v, want nil", projectID, remoteURL, err) + } + var source model.PagesProjectSource + if err := db.DB(ctx).Where("project_id = ?", projectID).First(&source).Error; err != nil { + t.Fatalf("load source for project %d error = %v, want nil", projectID, err) + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&runtime).Error; err != nil { + t.Fatalf("load runtime for source %d error = %v, want nil", source.ID, err) + } + return &source, &runtime +} + +func TestValidateRemoteSourceInputRejectsModeIncompatibleFields(t *testing.T) { + tests := []struct { + name string + input SourceUpdateInput + }{ + { + name: "missing source type", + input: SourceUpdateInput{ + RemoteURLSet: true, + RemoteURL: "https://example.com/site.zip", + }, + }, + { + name: "github type reserved for phase two", + input: SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/example/site", + }, + }, + { + name: "remote rejects repository field", + input: SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: "https://example.com/site.zip", + RepositoryURL: "https://github.com/example/site", + }, + }, + { + name: "remote rejects automatic updates", + input: SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: "https://example.com/site.zip", + AutoUpdateEnabled: true, + }, + }, + { + name: "url value requires replacement flag", + input: SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURL: "https://example.com/site.zip", + }, + }, + { + name: "invalid network policy", + input: SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: "https://example.com/site.zip", + RemoteNetworkPolicy: "private", + }, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + if err := validateRemoteSourceInput(test.input); err == nil { + t.Errorf("validateRemoteSourceInput(%+v) error = nil, want non-nil", test.input) + } + }) + } +} + +func TestUpdateSourceNewRemoteRequiresExplicitURL(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "remote-requires-url") + _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + }) + if err == nil { + t.Fatal("UpdateSource(new remote without URL) error = nil, want non-nil") + } + if got, want := err.Error(), errPagesSourceRemoteURLRequired; got != want { + t.Errorf("UpdateSource(new remote without URL) error = %q, want %q", got, want) + } +} + +func TestRemoteSourceCRUDPreservesSecretAndResetsRuntimeByIdentity(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "remote-crud") + firstURL := "https://Artifacts.Example.com:443/dist/site.zip?token=first-secret&expires=1" + source, runtime := mustConfigureRemoteSource(t, ctx, project.ID, firstURL, RemoteNetworkPolicyPublic) + + if got, want := source.ConfigVersion, 1; got != want { + t.Errorf("new source ConfigVersion = %d, want %d", got, want) + } + if got, want := runtime.SyncStatus, pagesSourceStatusIdle; got != want { + t.Errorf("new runtime SyncStatus = %q, want %q", got, want) + } + view, err := GetSource(ctx, project.ID) + if err != nil { + t.Fatalf("GetSource(%d) error = %v, want nil", project.ID, err) + } + if got, want := view.DisplayURL, "https://Artifacts.Example.com:443/dist/site.zip?***"; got != want { + t.Errorf("GetSource(%d).DisplayURL = %q, want %q", project.ID, got, want) + } + encodedView, err := json.Marshal(view) + if err != nil { + t.Fatalf("json.Marshal(GetSource(%d)) error = %v, want nil", project.ID, err) + } + if strings.Contains(string(encodedView), "first-secret") || strings.Contains(string(encodedView), "expires=1") { + t.Errorf("GetSource(%d) JSON = %s, want credential-free view", project.ID, encodedView) + } + if _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: firstURL, + RemoteNetworkPolicy: RemoteNetworkPolicyPublic, + }); err != nil { + t.Fatalf("UpdateSource(%d, no-op) error = %v, want nil", project.ID, err) + } + var unchangedSource model.PagesProjectSource + if err := db.DB(ctx).Where("id = ?", source.ID).First(&unchangedSource).Error; err != nil { + t.Fatalf("load no-op source error = %v, want nil", err) + } + if got, want := unchangedSource.ConfigVersion, source.ConfigVersion; got != want { + t.Errorf("no-op source ConfigVersion = %d, want unchanged %d", got, want) + } + + seenRevision := strings.Repeat("a", 64) + appliedRevision := strings.Repeat("b", 64) + future := time.Now().Add(time.Hour) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Updates(map[string]any{ + "last_seen_revision": seenRevision, + "last_seen_detail": `{"provider":"remote_url","label":"new.zip"}`, + "last_applied_revision": appliedRevision, + "last_applied_detail": `{"provider":"remote_url","label":"old.zip"}`, + "sync_status": pagesSourceStatusSyncing, + "lease_token": "in-flight", + "lease_expires_at": &future, + }).Error; err != nil { + t.Fatalf("seed source runtime error = %v, want nil", err) + } + + // Omit the secret URL while changing policy. The stored URL and cursor must + // survive, while the in-flight lease is fenced. + if _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: false, + RemoteNetworkPolicy: RemoteNetworkPolicyTrustedInternal, + }); err != nil { + t.Fatalf("UpdateSource(%d, preserve URL) error = %v, want nil", project.ID, err) + } + preservedSource, preservedRuntime, err := loadSourceByProject(ctx, project.ID) + if err != nil { + t.Fatalf("loadSourceByProject(%d) error = %v, want nil", project.ID, err) + } + if got, want := preservedSource.RemoteURL, firstURL; got != want { + t.Errorf("preserved RemoteURL = %q, want %q", got, want) + } + if got, want := preservedSource.ConfigVersion, 2; got != want { + t.Errorf("preserved source ConfigVersion = %d, want %d", got, want) + } + if got, want := preservedSource.SourceIdentity, source.SourceIdentity; got != want { + t.Errorf("preserved source identity = %q, want %q", got, want) + } + if got, want := preservedRuntime.LastSeenRevision, seenRevision; got != want { + t.Errorf("preserved LastSeenRevision = %q, want %q", got, want) + } + if got, want := preservedRuntime.SyncStatus, pagesSourceStatusUpdateAvailable; got != want { + t.Errorf("preserved runtime SyncStatus = %q, want %q", got, want) + } + if preservedRuntime.LeaseToken != "" || preservedRuntime.LeaseExpiresAt != nil { + t.Errorf("preserved runtime lease = (%q, %v), want cleared", preservedRuntime.LeaseToken, preservedRuntime.LeaseExpiresAt) + } + + // Replacing only the query secret keeps the canonical identity and cursors. + queryReplacementURL := "https://artifacts.example.com/dist/site.zip?token=second-secret" + if _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: queryReplacementURL, + RemoteNetworkPolicy: RemoteNetworkPolicyTrustedInternal, + }); err != nil { + t.Fatalf("UpdateSource(%d, query replacement) error = %v, want nil", project.ID, err) + } + querySource, queryRuntime, err := loadSourceByProject(ctx, project.ID) + if err != nil { + t.Fatalf("loadSourceByProject(%d) after query replacement error = %v, want nil", project.ID, err) + } + if got, want := querySource.SourceIdentity, source.SourceIdentity; got != want { + t.Errorf("query replacement identity = %q, want %q", got, want) + } + if got, want := queryRuntime.LastSeenRevision, seenRevision; got != want { + t.Errorf("query replacement LastSeenRevision = %q, want %q", got, want) + } + + // Replacing the path changes identity and clears all remote cursors. + pathReplacementURL := "https://artifacts.example.com/dist/other.zip?token=third-secret" + if _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: pathReplacementURL, + RemoteNetworkPolicy: RemoteNetworkPolicyTrustedInternal, + }); err != nil { + t.Fatalf("UpdateSource(%d, path replacement) error = %v, want nil", project.ID, err) + } + pathSource, pathRuntime, err := loadSourceByProject(ctx, project.ID) + if err != nil { + t.Fatalf("loadSourceByProject(%d) after path replacement error = %v, want nil", project.ID, err) + } + if pathSource.SourceIdentity == source.SourceIdentity { + t.Errorf("path replacement identity = %q, want a new identity", pathSource.SourceIdentity) + } + if pathRuntime.LastSeenRevision != "" || pathRuntime.LastAppliedRevision != "" { + t.Errorf("path replacement cursors = (%q, %q), want empty", pathRuntime.LastSeenRevision, pathRuntime.LastAppliedRevision) + } + if got, want := pathRuntime.SyncStatus, pagesSourceStatusIdle; got != want { + t.Errorf("path replacement SyncStatus = %q, want %q", got, want) + } + pathView, err := GetSource(ctx, project.ID) + if err != nil { + t.Fatalf("GetSource(%d) after path replacement error = %v, want nil", project.ID, err) + } + pathJSON, err := json.Marshal(pathView) + if err != nil { + t.Fatalf("json.Marshal(path view) error = %v, want nil", err) + } + for _, secret := range []string{"first-secret", "second-secret", "third-secret"} { + if strings.Contains(string(pathJSON), secret) { + t.Errorf("path view JSON = %s, want no secret %q", pathJSON, secret) + } + } +} + +func TestRemoteSourceIdentityIgnoresQueryAndNormalizesDefaultPort(t *testing.T) { + first, err := parseRemoteSourceURL("HTTPS://Artifacts.Example.com:443/dist/../dist/site.zip?token=one") + if err != nil { + t.Fatalf("parseRemoteSourceURL(first) error = %v, want nil", err) + } + second, err := parseRemoteSourceURL("https://artifacts.example.com/dist/site.zip?token=two") + if err != nil { + t.Fatalf("parseRemoteSourceURL(second) error = %v, want nil", err) + } + if got, want := remoteSourceIdentity(first), remoteSourceIdentity(second); got != want { + t.Errorf("remoteSourceIdentity(first) = %q, want %q", got, want) + } +} + +func TestDeleteSourceIsIdempotentAndKeepsDeploymentState(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "source-delete") + source, _ := mustConfigureRemoteSource( + t, + ctx, + project.ID, + "https://example.com/site.zip?token=delete-secret", + RemoteNetworkPolicyPublic, + ) + deployment := &model.PagesDeployment{ + ProjectID: project.ID, + DeploymentNumber: 1, + Checksum: strings.Repeat("c", 64), + Status: model.PagesDeploymentStatusActive, + CreatedBy: "user:1", + SourceType: "manual_upload", + TriggerType: "manual_upload", + } + if err := db.DB(ctx).Create(deployment).Error; err != nil { + t.Fatalf("create deployment error = %v, want nil", err) + } + if err := db.DB(ctx).Model(&model.PagesProject{}). + Where("id = ?", project.ID). + Update("active_deployment_id", deployment.ID).Error; err != nil { + t.Fatalf("set active deployment error = %v, want nil", err) + } + + for attempt := 1; attempt <= 2; attempt++ { + view, err := DeleteSource(ctx, project.ID) + if err != nil { + t.Fatalf("DeleteSource(%d), attempt %d error = %v, want nil", project.ID, attempt, err) + } + if got, want := view.SourceType, PagesSourceTypeManual; got != want { + t.Errorf("DeleteSource(%d), attempt %d SourceType = %q, want %q", project.ID, attempt, got, want) + } + } + var sourceCount, runtimeCount, deploymentCount int64 + if err := db.DB(ctx).Model(&model.PagesProjectSource{}).Where("id = ?", source.ID).Count(&sourceCount).Error; err != nil { + t.Fatalf("count source error = %v, want nil", err) + } + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}).Where("source_id = ?", source.ID).Count(&runtimeCount).Error; err != nil { + t.Fatalf("count runtime error = %v, want nil", err) + } + if err := db.DB(ctx).Model(&model.PagesDeployment{}).Where("id = ?", deployment.ID).Count(&deploymentCount).Error; err != nil { + t.Fatalf("count deployment error = %v, want nil", err) + } + if sourceCount != 0 || runtimeCount != 0 || deploymentCount != 1 { + t.Errorf("DeleteSource counts = source:%d runtime:%d deployment:%d, want 0, 0, 1", sourceCount, runtimeCount, deploymentCount) + } + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + if err != nil { + t.Fatalf("GetPagesProjectByID(%d) error = %v, want nil", project.ID, err) + } + if storedProject.ActiveDeploymentID == nil || *storedProject.ActiveDeploymentID != deployment.ID { + t.Errorf("active deployment = %v, want %d", storedProject.ActiveDeploymentID, deployment.ID) + } + + manual, err := GetSource(ctx, project.ID) + if err != nil { + t.Fatalf("GetSource(%d) after delete error = %v, want nil", project.ID, err) + } + if got, want := fmt.Sprint(manual.SourceType), PagesSourceTypeManual; got != want { + t.Errorf("GetSource(%d).SourceType = %q, want %q", project.ID, got, want) + } +} diff --git a/internal/db/migrator/goose/postgres/202607190001_add_pages_source_runtime.sql b/internal/db/migrator/goose/postgres/202607190001_add_pages_source_runtime.sql new file mode 100644 index 00000000..70540ab4 --- /dev/null +++ b/internal/db/migrator/goose/postgres/202607190001_add_pages_source_runtime.sql @@ -0,0 +1,82 @@ +-- +goose Up +ALTER TABLE of_pages_projects + ADD COLUMN IF NOT EXISTS content_config_version INTEGER NOT NULL DEFAULT 0; + +ALTER TABLE of_pages_deployments + ADD COLUMN IF NOT EXISTS source_type VARCHAR(32) NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS source_identity CHAR(64), + ADD COLUMN IF NOT EXISTS source_revision CHAR(64), + ADD COLUMN IF NOT EXISTS source_label VARCHAR(255) NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS source_meta TEXT NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS trigger_type VARCHAR(32) NOT NULL DEFAULT ''; + +UPDATE of_pages_deployments +SET source_type = 'manual_upload', + trigger_type = 'manual_upload'; + +CREATE TABLE IF NOT EXISTS of_pages_project_sources ( + id BIGSERIAL PRIMARY KEY, + project_id BIGINT NOT NULL, + source_type VARCHAR(32) NOT NULL DEFAULT '', + remote_url TEXT NOT NULL DEFAULT '', + remote_network_policy VARCHAR(32) NOT NULL DEFAULT '', + github_repository VARCHAR(255) NOT NULL DEFAULT '', + release_selector VARCHAR(16) NOT NULL DEFAULT '', + release_tag VARCHAR(255) NOT NULL DEFAULT '', + asset_name VARCHAR(255) NOT NULL DEFAULT '', + auto_update_enabled BOOLEAN NOT NULL DEFAULT FALSE, + check_interval_minutes INTEGER NOT NULL DEFAULT 0, + config_version INTEGER NOT NULL DEFAULT 0, + source_identity CHAR(64) NOT NULL DEFAULT '', + created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_project_sources_project_id + ON of_pages_project_sources (project_id); + +CREATE TABLE IF NOT EXISTS of_pages_project_source_runtime ( + source_id BIGINT PRIMARY KEY, + etag VARCHAR(512) NOT NULL DEFAULT '', + last_seen_revision CHAR(64) NOT NULL DEFAULT '', + last_seen_detail TEXT NOT NULL DEFAULT '', + last_applied_revision CHAR(64) NOT NULL DEFAULT '', + last_applied_detail TEXT NOT NULL DEFAULT '', + sync_status VARCHAR(32) NOT NULL DEFAULT '', + last_error TEXT NOT NULL DEFAULT '', + last_checked_at TIMESTAMPTZ, + last_synced_at TIMESTAMPTZ, + next_check_at TIMESTAMPTZ, + lease_expires_at TIMESTAMPTZ, + lease_token VARCHAR(64) NOT NULL DEFAULT '', + updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX IF NOT EXISTS idx_of_pages_project_source_runtime_next_check_at + ON of_pages_project_source_runtime (next_check_at); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_deployments_project_number + ON of_pages_deployments (project_id, deployment_number); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_deployments_source_revision + ON of_pages_deployments (project_id, source_identity, source_revision) + WHERE source_identity IS NOT NULL AND source_revision IS NOT NULL; + +-- +goose Down +DROP INDEX IF EXISTS idx_of_pages_deployments_source_revision; +DROP INDEX IF EXISTS idx_of_pages_deployments_project_number; +DROP INDEX IF EXISTS idx_of_pages_project_source_runtime_next_check_at; +DROP TABLE IF EXISTS of_pages_project_source_runtime; +DROP INDEX IF EXISTS idx_of_pages_project_sources_project_id; +DROP TABLE IF EXISTS of_pages_project_sources; + +ALTER TABLE of_pages_deployments + DROP COLUMN IF EXISTS trigger_type, + DROP COLUMN IF EXISTS source_meta, + DROP COLUMN IF EXISTS source_label, + DROP COLUMN IF EXISTS source_revision, + DROP COLUMN IF EXISTS source_identity, + DROP COLUMN IF EXISTS source_type; + +ALTER TABLE of_pages_projects + DROP COLUMN IF EXISTS content_config_version; diff --git a/internal/db/migrator/goose/sqlite/202607190001_add_pages_source_runtime.sql b/internal/db/migrator/goose/sqlite/202607190001_add_pages_source_runtime.sql new file mode 100644 index 00000000..da8012ef --- /dev/null +++ b/internal/db/migrator/goose/sqlite/202607190001_add_pages_source_runtime.sql @@ -0,0 +1,151 @@ +-- +goose Up +ALTER TABLE of_pages_projects + ADD COLUMN content_config_version INTEGER NOT NULL DEFAULT 0; + +ALTER TABLE of_pages_deployments + ADD COLUMN source_type TEXT NOT NULL DEFAULT ''; +ALTER TABLE of_pages_deployments + ADD COLUMN source_identity TEXT; +ALTER TABLE of_pages_deployments + ADD COLUMN source_revision TEXT; +ALTER TABLE of_pages_deployments + ADD COLUMN source_label TEXT NOT NULL DEFAULT ''; +ALTER TABLE of_pages_deployments + ADD COLUMN source_meta TEXT NOT NULL DEFAULT ''; +ALTER TABLE of_pages_deployments + ADD COLUMN trigger_type TEXT NOT NULL DEFAULT ''; + +UPDATE of_pages_deployments +SET source_type = 'manual_upload', + trigger_type = 'manual_upload'; + +CREATE TABLE IF NOT EXISTS of_pages_project_sources ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + project_id INTEGER NOT NULL, + source_type TEXT NOT NULL DEFAULT '', + remote_url TEXT NOT NULL DEFAULT '', + remote_network_policy TEXT NOT NULL DEFAULT '', + github_repository TEXT NOT NULL DEFAULT '', + release_selector TEXT NOT NULL DEFAULT '', + release_tag TEXT NOT NULL DEFAULT '', + asset_name TEXT NOT NULL DEFAULT '', + auto_update_enabled INTEGER NOT NULL DEFAULT 0, + check_interval_minutes INTEGER NOT NULL DEFAULT 0, + config_version INTEGER NOT NULL DEFAULT 0, + source_identity TEXT NOT NULL DEFAULT '', + created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_project_sources_project_id + ON of_pages_project_sources (project_id); + +CREATE TABLE IF NOT EXISTS of_pages_project_source_runtime ( + source_id INTEGER PRIMARY KEY, + etag TEXT NOT NULL DEFAULT '', + last_seen_revision TEXT NOT NULL DEFAULT '', + last_seen_detail TEXT NOT NULL DEFAULT '', + last_applied_revision TEXT NOT NULL DEFAULT '', + last_applied_detail TEXT NOT NULL DEFAULT '', + sync_status TEXT NOT NULL DEFAULT '', + last_error TEXT NOT NULL DEFAULT '', + last_checked_at DATETIME, + last_synced_at DATETIME, + next_check_at DATETIME, + lease_expires_at DATETIME, + lease_token TEXT NOT NULL DEFAULT '', + updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX IF NOT EXISTS idx_of_pages_project_source_runtime_next_check_at + ON of_pages_project_source_runtime (next_check_at); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_deployments_project_number + ON of_pages_deployments (project_id, deployment_number); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_deployments_source_revision + ON of_pages_deployments (project_id, source_identity, source_revision) + WHERE source_identity IS NOT NULL AND source_revision IS NOT NULL; + +-- +goose Down +DROP INDEX IF EXISTS idx_of_pages_deployments_source_revision; +DROP INDEX IF EXISTS idx_of_pages_deployments_project_number; +DROP INDEX IF EXISTS idx_of_pages_project_source_runtime_next_check_at; +DROP TABLE IF EXISTS of_pages_project_source_runtime; +DROP INDEX IF EXISTS idx_of_pages_project_sources_project_id; +DROP TABLE IF EXISTS of_pages_project_sources; + +-- SQLite 的 Down 必须重建受影响表，完整移除新增列并保留原有数据与索引。 +CREATE TABLE of_pages_deployments_before_source ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + project_id INTEGER NOT NULL, + deployment_number INTEGER NOT NULL, + checksum TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'uploaded', + upload_id INTEGER NOT NULL DEFAULT 0, + artifact_path TEXT NOT NULL, + file_count INTEGER NOT NULL DEFAULT 0, + total_size INTEGER NOT NULL DEFAULT 0, + created_by TEXT NOT NULL DEFAULT '', + created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, + activated_at DATETIME +); + +INSERT INTO of_pages_deployments_before_source ( + id, project_id, deployment_number, checksum, status, upload_id, artifact_path, + file_count, total_size, created_by, created_at, activated_at +) +SELECT + id, project_id, deployment_number, checksum, status, upload_id, artifact_path, + file_count, total_size, created_by, created_at, activated_at +FROM of_pages_deployments; + +DROP TABLE of_pages_deployments; +ALTER TABLE of_pages_deployments_before_source RENAME TO of_pages_deployments; + +CREATE INDEX IF NOT EXISTS idx_of_pages_deployments_project_id + ON of_pages_deployments (project_id); +CREATE INDEX IF NOT EXISTS idx_of_pages_deployments_checksum + ON of_pages_deployments (checksum); +CREATE INDEX IF NOT EXISTS idx_of_pages_deployments_status + ON of_pages_deployments (status); +CREATE INDEX IF NOT EXISTS idx_of_pages_deployments_upload_id + ON of_pages_deployments (upload_id); + +CREATE TABLE of_pages_projects_before_source ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + slug TEXT NOT NULL, + description TEXT NOT NULL DEFAULT '', + enabled INTEGER NOT NULL DEFAULT 1, + spa_fallback_enabled INTEGER NOT NULL DEFAULT 0, + spa_fallback_path TEXT NOT NULL DEFAULT '/index.html', + api_proxy_enabled INTEGER NOT NULL DEFAULT 0, + api_proxy_path TEXT NOT NULL DEFAULT '', + api_proxy_pass TEXT NOT NULL DEFAULT '', + api_proxy_rewrite TEXT NOT NULL DEFAULT '', + active_deployment_id INTEGER, + root_dir TEXT NOT NULL DEFAULT '', + entry_file TEXT NOT NULL DEFAULT 'index.html', + created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +INSERT INTO of_pages_projects_before_source ( + id, name, slug, description, enabled, spa_fallback_enabled, spa_fallback_path, + api_proxy_enabled, api_proxy_path, api_proxy_pass, api_proxy_rewrite, + active_deployment_id, root_dir, entry_file, created_at, updated_at +) +SELECT + id, name, slug, description, enabled, spa_fallback_enabled, spa_fallback_path, + api_proxy_enabled, api_proxy_path, api_proxy_pass, api_proxy_rewrite, + active_deployment_id, root_dir, entry_file, created_at, updated_at +FROM of_pages_projects; + +DROP TABLE of_pages_projects; +ALTER TABLE of_pages_projects_before_source RENAME TO of_pages_projects; + +CREATE UNIQUE INDEX IF NOT EXISTS idx_of_pages_projects_slug + ON of_pages_projects (slug); +CREATE INDEX IF NOT EXISTS idx_of_pages_projects_active_deployment_id + ON of_pages_projects (active_deployment_id); diff --git a/internal/db/migrator/migrator_test.go b/internal/db/migrator/migrator_test.go index ae9a0b88..780bdcb7 100644 --- a/internal/db/migrator/migrator_test.go +++ b/internal/db/migrator/migrator_test.go @@ -18,9 +18,9 @@ import ( "gorm.io/gorm" ) -// expectedMigratedSystemConfigCount 包含初始 32 项系统配置，以及 202606220004 -// 从 of_options 迁移过来的 48 项业务配置（OpenFlare/UptimeKuma/OpenResty）。 -const expectedMigratedSystemConfigCount = 80 +// expectedMigratedSystemConfigCount 包含初始 32 项系统配置、202606220004 +// 从 of_options 迁移过来的 48 项业务配置，以及 Pages 的 2 项业务配置。 +const expectedMigratedSystemConfigCount = 82 func TestMigrateInitializesSQLiteDatabase(t *testing.T) { sqliteDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{ diff --git a/internal/db/migrator/pages_source_migration_test.go b/internal/db/migrator/pages_source_migration_test.go new file mode 100644 index 00000000..be0534a8 --- /dev/null +++ b/internal/db/migrator/pages_source_migration_test.go @@ -0,0 +1,322 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package migrator + +import ( + "database/sql" + "fmt" + "os" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/glebarez/sqlite" + "github.com/pressly/goose/v3" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gorm.io/driver/postgres" + "gorm.io/gorm" +) + +const ( + pagesSourcePreviousMigration = int64(202607180001) + pagesSourceMigration = int64(202607190001) + pagesMigrationProjectID = uint(900001) + pagesMigrationDeploymentID = uint(900001) +) + +func TestPagesSourceMigrationSQLiteUpDownUp(t *testing.T) { + dbPath := t.TempDir() + "/pages-source-migration.db" + gormDB, err := gorm.Open(sqlite.Open(dbPath), &gorm.Config{ + DisableForeignKeyConstraintWhenMigrating: true, + }) + require.NoError(t, err) + + sqlDB, err := gormDB.DB() + require.NoError(t, err) + sqlDB.SetMaxOpenConns(1) + t.Cleanup(func() { require.NoError(t, sqlDB.Close()) }) + + runPagesSourceMigrationUpDownUp(t, gormDB, sqlDB, dialectSqlite, "goose/sqlite") + + var indexSQL string + require.NoError(t, gormDB.Raw( + "SELECT sql FROM sqlite_master WHERE type = 'index' AND name = ?", + "idx_of_pages_deployments_source_revision", + ).Scan(&indexSQL).Error) + assert.Contains(t, strings.ToUpper(indexSQL), "WHERE SOURCE_IDENTITY IS NOT NULL AND SOURCE_REVISION IS NOT NULL") +} + +func TestPagesSourceMigrationPostgresUpDownUp(t *testing.T) { + dsn := strings.TrimSpace(os.Getenv("OPENFLARE_TEST_POSTGRES_DSN")) + if dsn == "" { + t.Skip("OPENFLARE_TEST_POSTGRES_DSN is not set") + } + + gormDB, err := gorm.Open(postgres.Open(dsn), &gorm.Config{ + DisableForeignKeyConstraintWhenMigrating: true, + }) + require.NoError(t, err) + sqlDB, err := gormDB.DB() + require.NoError(t, err) + sqlDB.SetMaxOpenConns(1) + + schema := fmt.Sprintf("pages_source_migration_%d", time.Now().UnixNano()) + require.Regexp(t, `^[a-z0-9_]+$`, schema) + require.NoError(t, gormDB.Exec(`CREATE SCHEMA "`+schema+`"`).Error) + require.NoError(t, gormDB.Exec(`SET search_path TO "`+schema+`"`).Error) + t.Cleanup(func() { + assert.NoError(t, gormDB.Exec("SET search_path TO public").Error) + assert.NoError(t, gormDB.Exec(`DROP SCHEMA IF EXISTS "`+schema+`" CASCADE`).Error) + assert.NoError(t, sqlDB.Close()) + }) + + runPagesSourceMigrationUpDownUp(t, gormDB, sqlDB, dialectPostgres, "goose/postgres") +} + +func runPagesSourceMigrationUpDownUp( + t *testing.T, + gormDB *gorm.DB, + sqlDB *sql.DB, + dialect string, + dir string, +) { + t.Helper() + + goose.SetBaseFS(migrationFS) + require.NoError(t, goose.SetDialect(dialect)) + require.NoError(t, goose.UpTo(sqlDB, dir, pagesSourcePreviousMigration)) + seedPrePagesSourceMigrationData(t, gormDB) + + require.NoError(t, goose.UpTo(sqlDB, dir, pagesSourceMigration)) + assertPagesSourceMigrationUp(t, gormDB) + + require.NoError(t, goose.DownTo(sqlDB, dir, pagesSourcePreviousMigration)) + assertPagesSourceMigrationDown(t, gormDB) + + require.NoError(t, goose.UpTo(sqlDB, dir, pagesSourceMigration)) + assertPagesSourceMigrationUpAgain(t, gormDB) +} + +func seedPrePagesSourceMigrationData(t *testing.T, gormDB *gorm.DB) { + t.Helper() + + require.NoError(t, gormDB.Exec(` + INSERT INTO of_pages_projects ( + id, name, slug, description, enabled, active_deployment_id, root_dir, entry_file + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + `, + pagesMigrationProjectID, + "Migration Site", + "migration-site", + "keep-project-data", + true, + pagesMigrationDeploymentID, + "public", + "home.html", + ).Error) + require.NoError(t, gormDB.Exec(` + INSERT INTO of_pages_deployments ( + id, project_id, deployment_number, checksum, status, upload_id, artifact_path, + file_count, total_size, created_by + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `, + pagesMigrationDeploymentID, + pagesMigrationProjectID, + 1, + strings.Repeat("a", 64), + model.PagesDeploymentStatusActive, + uint64(700001), + "legacy/package.zip", + 2, + int64(128), + "user:1", + ).Error) +} + +func assertPagesSourceMigrationUp(t *testing.T, gormDB *gorm.DB) { + t.Helper() + migrator := gormDB.Migrator() + assert.True(t, migrator.HasTable(&model.PagesProjectSource{})) + assert.True(t, migrator.HasTable(&model.PagesProjectSourceRuntime{})) + assert.True(t, migrator.HasColumn(&model.PagesProject{}, "ContentConfigVersion")) + assert.True(t, migrator.HasColumn(&model.PagesDeployment{}, "SourceType")) + assert.True(t, migrator.HasColumn(&model.PagesDeployment{}, "SourceIdentity")) + assert.True(t, migrator.HasColumn(&model.PagesDeployment{}, "SourceRevision")) + assert.True(t, migrator.HasIndex(&model.PagesProjectSource{}, "idx_of_pages_project_sources_project_id")) + assert.True(t, migrator.HasIndex(&model.PagesProjectSourceRuntime{}, "idx_of_pages_project_source_runtime_next_check_at")) + assert.True(t, migrator.HasIndex(&model.PagesDeployment{}, "idx_of_pages_deployments_project_number")) + assert.True(t, migrator.HasIndex(&model.PagesDeployment{}, "idx_of_pages_deployments_source_revision")) + + var project model.PagesProject + require.NoError(t, gormDB.First(&project, pagesMigrationProjectID).Error) + assert.Equal(t, 0, project.ContentConfigVersion) + assert.Equal(t, "keep-project-data", project.Description) + assert.Equal(t, "public", project.RootDir) + assert.Equal(t, "home.html", project.EntryFile) + + var deployment model.PagesDeployment + require.NoError(t, gormDB.First(&deployment, pagesMigrationDeploymentID).Error) + assert.Equal(t, "manual_upload", deployment.SourceType) + assert.Equal(t, "manual_upload", deployment.TriggerType) + assert.Nil(t, deployment.SourceIdentity) + assert.Nil(t, deployment.SourceRevision) + assert.Equal(t, uint64(700001), deployment.UploadID) + + sourceID := createMigrationSourceRuntime(t, gormDB) + assertPagesSourceConstraints(t, gormDB, sourceID) +} + +func createMigrationSourceRuntime(t *testing.T, gormDB *gorm.DB) uint { + t.Helper() + source := model.PagesProjectSource{ + ProjectID: pagesMigrationProjectID, + SourceType: "remote_url", + RemoteURL: "https://example.com/site.zip?token=secret", + RemoteNetworkPolicy: "public", + CheckIntervalMinutes: 0, + ConfigVersion: 1, + SourceIdentity: strings.Repeat("b", 64), + } + require.NoError(t, gormDB.Create(&source).Error) + require.NotZero(t, source.ID) + require.NoError(t, gormDB.Create(&model.PagesProjectSourceRuntime{ + SourceID: source.ID, + SyncStatus: "idle", + }).Error) + return source.ID +} + +func assertPagesSourceConstraints(t *testing.T, gormDB *gorm.DB, sourceID uint) { + t.Helper() + + duplicateSource := model.PagesProjectSource{ + ProjectID: pagesMigrationProjectID, + SourceType: "remote_url", + ConfigVersion: 1, + SourceIdentity: strings.Repeat("c", 64), + } + assert.Error(t, gormDB.Create(&duplicateSource).Error) + + for number := 2; number <= 3; number++ { + require.NoError(t, createMigrationDeployment( + gormDB, + number, + strings.Repeat(string(rune('a'+number)), 64), + nil, + nil, + )) + } + + identity := strings.Repeat("d", 64) + revision := strings.Repeat("e", 64) + require.NoError(t, createMigrationDeployment( + gormDB, + 4, + strings.Repeat("f", 64), + &identity, + &revision, + )) + assert.Error(t, createMigrationDeployment( + gormDB, + 5, + strings.Repeat("0", 64), + &identity, + &revision, + )) + assert.Error(t, createMigrationDeployment( + gormDB, + 1, + strings.Repeat("1", 64), + nil, + nil, + )) + + var runtime model.PagesProjectSourceRuntime + require.NoError(t, gormDB.First(&runtime, sourceID).Error) + assert.Equal(t, "idle", runtime.SyncStatus) +} + +func createMigrationDeployment( + gormDB *gorm.DB, + deploymentNumber int, + checksum string, + identity *string, + revision *string, +) error { + return gormDB.Create(&model.PagesDeployment{ + ProjectID: pagesMigrationProjectID, + DeploymentNumber: deploymentNumber, + Checksum: checksum, + Status: model.PagesDeploymentStatusUploaded, + UploadID: uint64(710000 + deploymentNumber), + ArtifactPath: fmt.Sprintf("legacy/%d.zip", deploymentNumber), + SourceType: "manual_upload", + SourceIdentity: identity, + SourceRevision: revision, + TriggerType: "manual_upload", + }).Error +} + +func assertPagesSourceMigrationDown(t *testing.T, gormDB *gorm.DB) { + t.Helper() + migrator := gormDB.Migrator() + assert.False(t, migrator.HasTable(&model.PagesProjectSource{})) + assert.False(t, migrator.HasTable(&model.PagesProjectSourceRuntime{})) + assert.False(t, migrator.HasColumn(&model.PagesProject{}, "ContentConfigVersion")) + assert.False(t, migrator.HasColumn(&model.PagesDeployment{}, "SourceType")) + assert.False(t, migrator.HasColumn(&model.PagesDeployment{}, "SourceIdentity")) + assert.False(t, migrator.HasColumn(&model.PagesDeployment{}, "SourceRevision")) + assert.False(t, migrator.HasIndex(&model.PagesDeployment{}, "idx_of_pages_deployments_project_number")) + assert.False(t, migrator.HasIndex(&model.PagesDeployment{}, "idx_of_pages_deployments_source_revision")) + assert.True(t, migrator.HasIndex(&model.PagesDeployment{}, "idx_of_pages_deployments_project_id")) + assert.True(t, migrator.HasIndex(&model.PagesDeployment{}, "idx_of_pages_deployments_upload_id")) + + var project struct { + Description string + RootDir string + EntryFile string + ActiveDeploymentID *uint + } + require.NoError(t, gormDB.Table("of_pages_projects").Where("id = ?", pagesMigrationProjectID).Take(&project).Error) + assert.Equal(t, "keep-project-data", project.Description) + assert.Equal(t, "public", project.RootDir) + assert.Equal(t, "home.html", project.EntryFile) + require.NotNil(t, project.ActiveDeploymentID) + assert.Equal(t, pagesMigrationDeploymentID, *project.ActiveDeploymentID) + + var deployment struct { + UploadID uint64 + ArtifactPath string + FileCount int + TotalSize int64 + } + require.NoError(t, gormDB.Table("of_pages_deployments").Where("id = ?", pagesMigrationDeploymentID).Take(&deployment).Error) + assert.Equal(t, uint64(700001), deployment.UploadID) + assert.Equal(t, "legacy/package.zip", deployment.ArtifactPath) + assert.Equal(t, 2, deployment.FileCount) + assert.Equal(t, int64(128), deployment.TotalSize) + + var count int64 + require.NoError(t, gormDB.Table("of_pages_deployments").Where("project_id = ?", pagesMigrationProjectID).Count(&count).Error) + assert.Equal(t, int64(4), count) +} + +func assertPagesSourceMigrationUpAgain(t *testing.T, gormDB *gorm.DB) { + t.Helper() + assert.True(t, gormDB.Migrator().HasTable(&model.PagesProjectSource{})) + assert.True(t, gormDB.Migrator().HasTable(&model.PagesProjectSourceRuntime{})) + assert.True(t, gormDB.Migrator().HasColumn(&model.PagesProject{}, "ContentConfigVersion")) + assert.True(t, gormDB.Migrator().HasColumn(&model.PagesDeployment{}, "SourceRevision")) + + var count int64 + require.NoError(t, gormDB.Table("of_pages_deployments"). + Where("project_id = ? AND source_type = ? AND trigger_type = ?", pagesMigrationProjectID, "manual_upload", "manual_upload"). + Count(&count).Error) + assert.Equal(t, int64(4), count) + + require.NoError(t, gormDB.Table("of_pages_project_sources").Count(&count).Error) + assert.Zero(t, count, "source config is intentionally removed by Down and is not reconstructable") +} diff --git a/internal/db/postgres.go b/internal/db/postgres.go index 7aa72320..c16cd806 100644 --- a/internal/db/postgres.go +++ b/internal/db/postgres.go @@ -57,13 +57,10 @@ func initSQLite() { // Trace 注入 if err = db.Use( - tracing.NewPlugin( - tracing.WithoutMetrics(), - tracing.WithAttributes( - attribute.String("db.instance", sqlitePath), - attribute.String("db.system", "SQLite"), - ), - ), + newGORMTracingPlugin([]attribute.KeyValue{ + attribute.String("db.instance", sqlitePath), + attribute.String("db.system", "SQLite"), + }), ); err != nil { log.Fatalf("[SQLite] init trace failed: %v\n", err) } @@ -98,15 +95,12 @@ func initPostgres() { // Trace 注入 if err = db.Use( - tracing.NewPlugin( - tracing.WithoutMetrics(), - tracing.WithAttributes( - attribute.String("db.instance", dbConfig.Database), - attribute.String("db.ip", dbConfig.Host), - attribute.String("server.address", net.JoinHostPort(dbConfig.Host, strconv.Itoa(dbConfig.Port))), - attribute.String("db.system", "PostgreSQL"), - ), - ), + newGORMTracingPlugin([]attribute.KeyValue{ + attribute.String("db.instance", dbConfig.Database), + attribute.String("db.ip", dbConfig.Host), + attribute.String("server.address", net.JoinHostPort(dbConfig.Host, strconv.Itoa(dbConfig.Port))), + attribute.String("db.system", "PostgreSQL"), + }), ); err != nil { log.Fatalf("[PostgreSQL] init trace failed: %v\n", err) } @@ -160,6 +154,17 @@ func initPostgres() { } +// newGORMTracingPlugin 构造数据库链路追踪插件。查询参数只保留占位符，避免凭据等绑定值进入 Span。 +func newGORMTracingPlugin(attrs []attribute.KeyValue, extraOptions ...tracing.Option) gorm.Plugin { + options := []tracing.Option{ + tracing.WithoutMetrics(), + tracing.WithoutQueryVariables(), + tracing.WithAttributes(attrs...), + } + options = append(options, extraOptions...) + return tracing.NewPlugin(options...) +} + // buildDSN 构建 PostgreSQL DSN func buildDSN(host string, port int, username, password string) string { cfg := config.Config.Database diff --git a/internal/db/postgres_logger.go b/internal/db/postgres_logger.go index 0bfd0f7a..2ba1433c 100644 --- a/internal/db/postgres_logger.go +++ b/internal/db/postgres_logger.go @@ -49,6 +49,11 @@ func (l *gormZapLogger) Error(ctx context.Context, fmt string, args ...interface } } +// ParamsFilter 让 GORM 的 Trace 回调只接收参数化 SQL，避免绑定值被 Dialector.Explain 展开到日志。 +func (l *gormZapLogger) ParamsFilter(_ context.Context, sql string, _ ...interface{}) (string, []interface{}) { + return sql, nil +} + func (l *gormZapLogger) Trace(ctx context.Context, begin time.Time, fc func() (sql string, rowsAffected int64), err error) { elapsed := time.Since(begin) switch { diff --git a/internal/db/postgres_logger_test.go b/internal/db/postgres_logger_test.go index 20539a5b..0d16dc4a 100644 --- a/internal/db/postgres_logger_test.go +++ b/internal/db/postgres_logger_test.go @@ -4,11 +4,40 @@ package db import ( + "context" + "strings" "testing" + "time" + "github.com/glebarez/sqlite" + "gorm.io/gorm" gormLogger "gorm.io/gorm/logger" ) +type paramsFilterCaptureLogger struct { + filter *gormZapLogger + traces []string +} + +func (l *paramsFilterCaptureLogger) LogMode(gormLogger.LogLevel) gormLogger.Interface { + return l +} + +func (l *paramsFilterCaptureLogger) Info(context.Context, string, ...interface{}) {} + +func (l *paramsFilterCaptureLogger) Warn(context.Context, string, ...interface{}) {} + +func (l *paramsFilterCaptureLogger) Error(context.Context, string, ...interface{}) {} + +func (l *paramsFilterCaptureLogger) ParamsFilter(ctx context.Context, sql string, params ...interface{}) (string, []interface{}) { + return l.filter.ParamsFilter(ctx, sql, params...) +} + +func (l *paramsFilterCaptureLogger) Trace(_ context.Context, _ time.Time, fc func() (string, int64), _ error) { + sql, _ := fc() + l.traces = append(l.traces, sql) +} + func TestParseLogLevel(t *testing.T) { t.Parallel() @@ -37,3 +66,50 @@ func TestParseLogLevel(t *testing.T) { }) } } + +func TestGormZapLoggerParamsFilterDropsBoundValues(t *testing.T) { + t.Parallel() + + const ( + query = "UPDATE openflare_pages_sources SET remote_url = ? WHERE id = ?" + secret = "https://example.test/release.zip?token=super-secret" + ) + + filteredSQL, filteredParams := (&gormZapLogger{}).ParamsFilter(t.Context(), query, secret, int64(42)) + if filteredSQL != query { + t.Fatalf("ParamsFilter() sql = %q, want %q", filteredSQL, query) + } + if filteredParams != nil { + t.Fatalf("ParamsFilter() params = %#v, want nil", filteredParams) + } +} + +func TestGormZapLoggerKeepsParameterizedSQLInTrace(t *testing.T) { + t.Parallel() + + const secret = "https://example.test/release.zip?token=trace-secret" + capture := ¶msFilterCaptureLogger{filter: &gormZapLogger{}} + testDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{Logger: capture}) + if err != nil { + t.Fatalf("open sqlite: %v", err) + } + if err := testDB.Exec("CREATE TABLE source_secrets (remote_url TEXT NOT NULL)").Error; err != nil { + t.Fatalf("create table: %v", err) + } + + capture.traces = nil + if err := testDB.Exec("INSERT INTO source_secrets (remote_url) VALUES (?)", secret).Error; err != nil { + t.Fatalf("insert source secret: %v", err) + } + if len(capture.traces) != 1 { + t.Fatalf("trace count = %d, want 1", len(capture.traces)) + } + + traceSQL := capture.traces[0] + if strings.Contains(traceSQL, secret) || strings.Contains(traceSQL, "trace-secret") { + t.Fatalf("trace SQL leaked bound value: %q", traceSQL) + } + if !strings.Contains(traceSQL, "VALUES (?)") { + t.Fatalf("trace SQL = %q, want parameter placeholder", traceSQL) + } +} diff --git a/internal/db/postgres_tracing_test.go b/internal/db/postgres_tracing_test.go new file mode 100644 index 00000000..6958521f --- /dev/null +++ b/internal/db/postgres_tracing_test.go @@ -0,0 +1,69 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package db + +import ( + "context" + "strings" + "testing" + + "github.com/glebarez/sqlite" + "go.opentelemetry.io/otel/attribute" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + semconv "go.opentelemetry.io/otel/semconv/v1.30.0" + "gorm.io/gorm" + gormLogger "gorm.io/gorm/logger" + "gorm.io/plugin/opentelemetry/tracing" +) + +func TestGORMTracingPluginDoesNotRecordQueryVariables(t *testing.T) { + t.Parallel() + + const secret = "https://example.test/release.zip?token=otel-secret" + spanRecorder := tracetest.NewSpanRecorder() + tracerProvider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(spanRecorder)) + t.Cleanup(func() { + if err := tracerProvider.Shutdown(context.Background()); err != nil { + t.Errorf("shutdown tracer provider: %v", err) + } + }) + + testDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{ + Logger: gormLogger.Default.LogMode(gormLogger.Silent), + }) + if err != nil { + t.Fatalf("open sqlite: %v", err) + } + if err := testDB.Use(newGORMTracingPlugin( + []attribute.KeyValue{attribute.String("db.instance", "trace-test")}, + tracing.WithTracerProvider(tracerProvider), + )); err != nil { + t.Fatalf("register tracing plugin: %v", err) + } + if err := testDB.Exec("CREATE TABLE source_secrets (remote_url TEXT NOT NULL)").Error; err != nil { + t.Fatalf("create table: %v", err) + } + if err := testDB.Exec("INSERT INTO source_secrets (remote_url) VALUES (?)", secret).Error; err != nil { + t.Fatalf("insert source secret: %v", err) + } + + var queryText string + for _, span := range spanRecorder.Ended() { + for _, attr := range span.Attributes() { + if attr.Key == semconv.DBQueryTextKey && strings.Contains(attr.Value.AsString(), "INSERT INTO source_secrets") { + queryText = attr.Value.AsString() + } + } + } + if queryText == "" { + t.Fatal("database query text attribute not found") + } + if strings.Contains(queryText, secret) || strings.Contains(queryText, "otel-secret") { + t.Fatalf("db.query.text leaked bound value: %q", queryText) + } + if !strings.Contains(queryText, "VALUES (?)") { + t.Fatalf("db.query.text = %q, want parameter placeholder", queryText) + } +} diff --git a/internal/model/openflare_pages.go b/internal/model/openflare_pages.go index e35006be..31ffce9d 100644 --- a/internal/model/openflare_pages.go +++ b/internal/model/openflare_pages.go @@ -18,22 +18,23 @@ const ( // PagesProject OpenFlare Pages 静态托管项目。 type PagesProject struct { - ID uint `json:"id" gorm:"primaryKey;autoIncrement"` - Name string `json:"name" gorm:"size:255;not null"` - Slug string `json:"slug" gorm:"uniqueIndex;size:128;not null"` - Description string `json:"description" gorm:"type:text;not null;default:''"` - Enabled bool `json:"enabled" gorm:"not null;default:true"` - SPAFallbackEnabled bool `json:"spa_fallback_enabled" gorm:"not null;default:false"` - SPAFallbackPath string `json:"spa_fallback_path" gorm:"size:512;not null;default:'/index.html'"` - APIProxyEnabled bool `json:"api_proxy_enabled" gorm:"not null;default:false"` - APIProxyPath string `json:"api_proxy_path" gorm:"size:255;not null;default:''"` - APIProxyPass string `json:"api_proxy_pass" gorm:"size:2048;not null;default:''"` - APIProxyRewrite string `json:"api_proxy_rewrite" gorm:"size:255;not null;default:''"` - ActiveDeploymentID *uint `json:"active_deployment_id" gorm:"index"` - RootDir string `json:"root_dir" gorm:"size:512;not null;default:''"` - EntryFile string `json:"entry_file" gorm:"size:512;not null;default:'index.html'"` - CreatedAt time.Time `json:"created_at" gorm:"autoCreateTime"` - UpdatedAt time.Time `json:"updated_at" gorm:"autoUpdateTime"` + ID uint `json:"id" gorm:"primaryKey;autoIncrement"` + Name string `json:"name" gorm:"size:255;not null"` + Slug string `json:"slug" gorm:"uniqueIndex;size:128;not null"` + Description string `json:"description" gorm:"type:text;not null;default:''"` + Enabled bool `json:"enabled" gorm:"not null;default:true"` + SPAFallbackEnabled bool `json:"spa_fallback_enabled" gorm:"not null;default:false"` + SPAFallbackPath string `json:"spa_fallback_path" gorm:"size:512;not null;default:'/index.html'"` + APIProxyEnabled bool `json:"api_proxy_enabled" gorm:"not null;default:false"` + APIProxyPath string `json:"api_proxy_path" gorm:"size:255;not null;default:''"` + APIProxyPass string `json:"api_proxy_pass" gorm:"size:2048;not null;default:''"` + APIProxyRewrite string `json:"api_proxy_rewrite" gorm:"size:255;not null;default:''"` + ActiveDeploymentID *uint `json:"active_deployment_id" gorm:"index"` + RootDir string `json:"root_dir" gorm:"size:512;not null;default:''"` + EntryFile string `json:"entry_file" gorm:"size:512;not null;default:'index.html'"` + ContentConfigVersion int `json:"-" gorm:"not null;default:0"` + CreatedAt time.Time `json:"created_at" gorm:"autoCreateTime"` + UpdatedAt time.Time `json:"updated_at" gorm:"autoUpdateTime"` } // TableName 表名。 @@ -44,8 +45,8 @@ func (PagesProject) TableName() string { // PagesDeployment OpenFlare Pages 不可变部署记录。 type PagesDeployment struct { ID uint `json:"id" gorm:"primaryKey;autoIncrement"` - ProjectID uint `json:"project_id" gorm:"not null;index"` - DeploymentNumber int `json:"deployment_number" gorm:"not null"` + ProjectID uint `json:"project_id" gorm:"not null;index;uniqueIndex:idx_of_pages_deployments_project_number,priority:1;uniqueIndex:idx_of_pages_deployments_source_revision,priority:1,where:source_identity IS NOT NULL AND source_revision IS NOT NULL"` + DeploymentNumber int `json:"deployment_number" gorm:"not null;uniqueIndex:idx_of_pages_deployments_project_number,priority:2"` Checksum string `json:"checksum" gorm:"size:64;not null;index"` Status string `json:"status" gorm:"size:32;not null;default:'uploaded';index"` UploadID uint64 `json:"upload_id,string" gorm:"not null;default:0;index"` @@ -53,6 +54,12 @@ type PagesDeployment struct { FileCount int `json:"file_count" gorm:"not null;default:0"` TotalSize int64 `json:"total_size" gorm:"not null;default:0"` CreatedBy string `json:"created_by" gorm:"size:64;not null;default:''"` + SourceType string `json:"source_type" gorm:"size:32;not null;default:''"` + SourceIdentity *string `json:"-" gorm:"type:char(64);uniqueIndex:idx_of_pages_deployments_source_revision,priority:2,where:source_identity IS NOT NULL AND source_revision IS NOT NULL"` + SourceRevision *string `json:"-" gorm:"type:char(64);uniqueIndex:idx_of_pages_deployments_source_revision,priority:3,where:source_identity IS NOT NULL AND source_revision IS NOT NULL"` + SourceLabel string `json:"source_label" gorm:"size:255;not null;default:''"` + SourceMeta string `json:"-" gorm:"type:text;not null;default:''"` + TriggerType string `json:"trigger_type" gorm:"size:32;not null;default:''"` CreatedAt time.Time `json:"created_at" gorm:"autoCreateTime"` ActivatedAt *time.Time `json:"activated_at"` } diff --git a/internal/model/openflare_pages_source.go b/internal/model/openflare_pages_source.go new file mode 100644 index 00000000..6f110978 --- /dev/null +++ b/internal/model/openflare_pages_source.go @@ -0,0 +1,59 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package model + +import "time" + +// PagesProjectSource 保存 Pages 项目的持久部署源配置。 +// +// RemoteURL 可能包含签名参数，禁止直接序列化 model；对外接口必须映射到 +// pages 包内的脱敏 source view。 +type PagesProjectSource struct { + ID uint `json:"-" gorm:"primaryKey;autoIncrement"` + ProjectID uint `json:"-" gorm:"not null;uniqueIndex:idx_of_pages_project_sources_project_id"` + SourceType string `json:"-" gorm:"size:32;not null;default:''"` + RemoteURL string `json:"-" gorm:"type:text;not null;default:''"` + RemoteNetworkPolicy string `json:"-" gorm:"size:32;not null;default:''"` + GitHubRepository string `json:"-" gorm:"column:github_repository;size:255;not null;default:''"` + ReleaseSelector string `json:"-" gorm:"size:16;not null;default:''"` + ReleaseTag string `json:"-" gorm:"size:255;not null;default:''"` + AssetName string `json:"-" gorm:"size:255;not null;default:''"` + AutoUpdateEnabled bool `json:"-" gorm:"not null;default:false"` + CheckIntervalMinutes int `json:"-" gorm:"not null;default:0"` + ConfigVersion int `json:"-" gorm:"not null;default:0"` + SourceIdentity string `json:"-" gorm:"type:char(64);not null;default:''"` + CreatedAt time.Time `json:"-" gorm:"autoCreateTime"` + UpdatedAt time.Time `json:"-" gorm:"autoUpdateTime"` +} + +// TableName 返回 Pages 项目部署源配置表名。 +func (PagesProjectSource) TableName() string { + return "of_pages_project_sources" +} + +// PagesProjectSourceRuntime 保存 Pages 项目部署源的可变运行态。 +// +// Runtime 不冗余 project_id；调用方通过 SourceID 关联配置，并在最终提交时 +// 同时校验 source config version 与 project content config version。 +type PagesProjectSourceRuntime struct { + SourceID uint `json:"-" gorm:"primaryKey;autoIncrement:false"` + ETag string `json:"-" gorm:"column:etag;size:512;not null;default:''"` + LastSeenRevision string `json:"-" gorm:"type:char(64);not null;default:''"` + LastSeenDetail string `json:"-" gorm:"type:text;not null;default:''"` + LastAppliedRevision string `json:"-" gorm:"type:char(64);not null;default:''"` + LastAppliedDetail string `json:"-" gorm:"type:text;not null;default:''"` + SyncStatus string `json:"-" gorm:"size:32;not null;default:''"` + LastError string `json:"-" gorm:"type:text;not null;default:''"` + LastCheckedAt *time.Time `json:"-"` + LastSyncedAt *time.Time `json:"-"` + NextCheckAt *time.Time `json:"-" gorm:"index:idx_of_pages_project_source_runtime_next_check_at"` + LeaseExpiresAt *time.Time `json:"-"` + LeaseToken string `json:"-" gorm:"size:64;not null;default:''"` + UpdatedAt time.Time `json:"-" gorm:"autoUpdateTime"` +} + +// TableName 返回 Pages 项目部署源运行态表名。 +func (PagesProjectSourceRuntime) TableName() string { + return "of_pages_project_source_runtime" +} diff --git a/internal/model/openflare_pages_source_test.go b/internal/model/openflare_pages_source_test.go new file mode 100644 index 00000000..16b2b637 --- /dev/null +++ b/internal/model/openflare_pages_source_test.go @@ -0,0 +1,80 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package model + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/glebarez/sqlite" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gorm.io/gorm" +) + +func TestPagesSourceModelsMatchMigrationSchema(t *testing.T) { + gormDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{ + DisableForeignKeyConstraintWhenMigrating: true, + }) + require.NoError(t, err) + require.NoError(t, gormDB.AutoMigrate( + &PagesProject{}, + &PagesDeployment{}, + &PagesProjectSource{}, + &PagesProjectSourceRuntime{}, + )) + + assert.Equal(t, "of_pages_project_sources", (PagesProjectSource{}).TableName()) + assert.Equal(t, "of_pages_project_source_runtime", (PagesProjectSourceRuntime{}).TableName()) + assert.True(t, gormDB.Migrator().HasColumn(&PagesProjectSource{}, "github_repository")) + assert.False(t, gormDB.Migrator().HasColumn(&PagesProjectSource{}, "git_hub_repository")) + assert.True(t, gormDB.Migrator().HasColumn(&PagesProjectSourceRuntime{}, "etag")) + assert.False(t, gormDB.Migrator().HasColumn(&PagesProjectSourceRuntime{}, "e_tag")) + + var indexSQL string + require.NoError(t, gormDB.Raw( + "SELECT sql FROM sqlite_master WHERE type = 'index' AND name = ?", + "idx_of_pages_deployments_source_revision", + ).Scan(&indexSQL).Error) + assert.Contains(t, strings.ToUpper(indexSQL), "WHERE SOURCE_IDENTITY IS NOT NULL AND SOURCE_REVISION IS NOT NULL") +} + +func TestPagesSourceModelsDoNotSerializeSecretsOrFencingState(t *testing.T) { + sourceJSON, err := json.Marshal(PagesProjectSource{ + ID: 1, + ProjectID: 2, + RemoteURL: "https://example.com/site.zip?token=secret", + ConfigVersion: 3, + SourceIdentity: strings.Repeat("a", 64), + }) + require.NoError(t, err) + assert.JSONEq(t, `{}`, string(sourceJSON)) + + runtimeJSON, err := json.Marshal(PagesProjectSourceRuntime{ + SourceID: 1, + ETag: `"secret-etag"`, + LeaseToken: "secret-lease", + }) + require.NoError(t, err) + assert.JSONEq(t, `{}`, string(runtimeJSON)) + + identity := strings.Repeat("b", 64) + revision := strings.Repeat("c", 64) + deploymentJSON, err := json.Marshal(PagesDeployment{ + SourceType: "remote_url", + SourceIdentity: &identity, + SourceRevision: &revision, + SourceLabel: "site.zip", + SourceMeta: `{"provider":"remote_url","private":"secret"}`, + TriggerType: "manual_sync", + }) + require.NoError(t, err) + assert.NotContains(t, string(deploymentJSON), identity) + assert.NotContains(t, string(deploymentJSON), revision) + assert.NotContains(t, string(deploymentJSON), "private") + assert.Contains(t, string(deploymentJSON), `"source_type":"remote_url"`) + assert.Contains(t, string(deploymentJSON), `"source_label":"site.zip"`) + assert.Contains(t, string(deploymentJSON), `"trigger_type":"manual_sync"`) +} diff --git a/internal/router/v1/openflare/register_pages.go b/internal/router/v1/openflare/register_pages.go index 9fa2fb9c..89ba3b7c 100644 --- a/internal/router/v1/openflare/register_pages.go +++ b/internal/router/v1/openflare/register_pages.go @@ -18,6 +18,11 @@ func registerPagesRoutes(apiGroup *gin.RouterGroup) { apiutil.RegisterCollection(pagesRoute, "POST", pages.CreateProjectHandler) pagesRoute.POST("/:id/update", pages.UpdateProjectHandler) pagesRoute.POST("/:id/delete", pages.DeleteProjectHandler) + pagesRoute.GET("/:id/source", pages.GetSourceHandler) + pagesRoute.POST("/:id/source/update", pages.UpdateSourceHandler) + pagesRoute.POST("/:id/source/delete", pages.DeleteSourceHandler) + pagesRoute.POST("/:id/source/check", pages.CheckSourceHandler) + pagesRoute.POST("/:id/source/sync", pages.SyncSourceHandler) pagesRoute.GET("/:id/deployments", pages.ListDeploymentsHandler) pagesRoute.POST("/:id/deployments/upload", pages.UploadDeploymentHandler) pagesRoute.POST("/:id/deployments/upload-from-url", pages.UploadDeploymentFromURLHandler) diff --git a/internal/task/executor.go b/internal/task/executor.go index e41ae65e..c41ca9d8 100644 --- a/internal/task/executor.go +++ b/internal/task/executor.go @@ -428,7 +428,7 @@ func notifyTaskCompleted(ctx context.Context, execution *model.TaskExecution, re } func shouldFlushTaskExecutionLog(ctx context.Context, execErr error) bool { - if execErr == nil { + if isTerminalTaskExecutionError(execErr) { return true } @@ -440,6 +440,10 @@ func shouldFlushTaskExecutionLog(ctx context.Context, execErr error) bool { return retryCount >= maxRetry } +func isTerminalTaskExecutionError(execErr error) bool { + return execErr == nil || errors.Is(execErr, asynq.SkipRetry) +} + func handleFailedTask(ctx context.Context, execution *model.TaskExecution, t *asynq.Task, duration time.Duration, execErr error, span trace.Span) { execution.Status = model.TaskExecutionStatusFailed execution.ErrorMessage = execErr.Error() diff --git a/internal/task/executor_test.go b/internal/task/executor_test.go index fc53b770..b0bb2f9f 100644 --- a/internal/task/executor_test.go +++ b/internal/task/executor_test.go @@ -6,10 +6,12 @@ package task import ( "context" + "errors" "fmt" "testing" "time" + "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" "github.com/Rain-kl/Wavelet/internal/testhelper" "github.com/hibiken/asynq" @@ -284,6 +286,55 @@ func TestCompleteTaskExecutionFlushesLog(t *testing.T) { assert.Contains(t, found.Log, "任务执行成功") } +func TestCompleteTaskExecutionFlushesPermanentFailureLog(t *testing.T) { + cleanup := setupTest(t) + defer cleanup() + ctx := context.Background() + + execution := &model.TaskExecution{ + TaskID: "complete_permanent_flush_001", + TaskType: testTaskType, + TaskName: "测试任务", + Status: model.TaskExecutionStatusRunning, + Retryable: true, + MaxRetry: 3, + TriggeredBy: "manual", + } + err := model.CreateTaskExecution(ctx, execution) + require.NoError(t, err) + + ctx = withTaskID(ctx, execution.TaskID) + AppendLog(ctx, "永久失败前的日志") + execErr := PermanentError("来源配置无效") + + finishTime := time.Now() + completeTaskExecution( + ctx, + execution, + asynq.NewTask(testTaskType, nil), + 100*time.Millisecond, + finishTime, + nil, + execErr, + trace.SpanFromContext(ctx), + ) + + found, err := model.GetTaskExecutionByTaskID(ctx, execution.TaskID) + require.NoError(t, err) + assert.Equal(t, model.TaskExecutionStatusFailed, found.Status) + assert.Equal(t, "来源配置无效", found.ErrorMessage) + assert.Contains(t, found.Log, "永久失败前的日志") + assert.Contains(t, found.Log, "任务执行失败") + keys, err := db.Redis.Keys(ctx, "*"+execution.TaskID+"*").Result() + require.NoError(t, err) + assert.Empty(t, keys) +} + +func TestPermanentErrorIsTerminalForLogFlush(t *testing.T) { + assert.True(t, isTerminalTaskExecutionError(PermanentError("配置无效"))) + assert.False(t, isTerminalTaskExecutionError(errors.New("temporary failure"))) +} + func TestRetryTask(t *testing.T) { cleanup := setupTest(t) defer cleanup() diff --git a/internal/task/handlers/register.go b/internal/task/handlers/register.go index ade65f8c..641d9dca 100644 --- a/internal/task/handlers/register.go +++ b/internal/task/handlers/register.go @@ -8,6 +8,7 @@ package handlers import ( "github.com/Rain-kl/Wavelet/internal/apps/admin/push" "github.com/Rain-kl/Wavelet/internal/apps/openflare" + "github.com/Rain-kl/Wavelet/internal/apps/openflare/pages" "github.com/Rain-kl/Wavelet/internal/apps/openflare/tls" "github.com/Rain-kl/Wavelet/internal/apps/upload" "github.com/Rain-kl/Wavelet/internal/apps/user" @@ -51,6 +52,10 @@ func Register() { task.RegisterHandler(openflare.UptimeKumaSyncTask, &openflare.UptimeKumaSyncHandler{}) task.RegisterTaskMeta(openflare.UptimeKumaSyncMeta) + // pages source actions are only dispatched by the Pages domain API/scanner. + task.RegisterHandler(pages.PagesSourceActionTask, &pages.SourceActionHandler{}) + task.RegisterTaskMeta(pages.PagesSourceActionMeta) + // tls single renew task.RegisterHandler(tls.SSLSingleRenewTask, &tls.SSLSingleRenewHandler{}) task.RegisterTaskMeta(tls.SSLSingleRenewMeta) diff --git a/internal/task/meta.go b/internal/task/meta.go index 019fc637..888ed80c 100644 --- a/internal/task/meta.go +++ b/internal/task/meta.go @@ -31,6 +31,7 @@ type TaskMeta struct { MaxRetry int `json:"max_retry"` Queue string `json:"queue"` Retryable bool `json:"retryable"` // 是否支持手动重试 + InternalOnly bool `json:"-"` // 是否仅允许内部业务入口调度 Params []TaskParam `json:"params,omitempty"` } @@ -51,13 +52,18 @@ func RegisterTaskMeta(meta TaskMeta) { dispatchableTasks = append(dispatchableTasks, meta) } -// GetDispatchableTasks 获取所有已注册的元数据列表（返回副本以避免并发并发读写冲突） +// GetDispatchableTasks 获取允许通过通用 Admin 入口调度的元数据列表。 func GetDispatchableTasks() []TaskMeta { dispatchableTasksMutex.RLock() defer dispatchableTasksMutex.RUnlock() - metas := make([]TaskMeta, len(dispatchableTasks)) - copy(metas, dispatchableTasks) + metas := make([]TaskMeta, 0, len(dispatchableTasks)) + for _, meta := range dispatchableTasks { + if meta.InternalOnly { + continue + } + metas = append(metas, meta) + } return metas } diff --git a/internal/task/meta_test.go b/internal/task/meta_test.go index e91c1b6b..a22e05eb 100644 --- a/internal/task/meta_test.go +++ b/internal/task/meta_test.go @@ -29,3 +29,28 @@ func TestDuplicateTaskMeta(t *testing.T) { } } } + +func TestInternalOnlyTaskMetaIsHiddenFromDispatchableTasks(t *testing.T) { + const taskType = "test_internal_only_meta" + meta := task.TaskMeta{ + Type: taskType, + AsynqTask: "test:internal_only_meta", + Name: "内部测试任务", + InternalOnly: true, + } + task.RegisterTaskMeta(meta) + + registered := task.GetTaskMeta(taskType) + if registered == nil { + t.Fatal("GetTaskMeta() did not return internal-only metadata") + } + if !registered.InternalOnly { + t.Fatal("GetTaskMeta() lost InternalOnly flag") + } + + for _, dispatchable := range task.GetDispatchableTasks() { + if dispatchable.Type == taskType { + t.Fatalf("GetDispatchableTasks() exposed internal-only task %q", taskType) + } + } +} diff --git a/internal/task/permanent_error.go b/internal/task/permanent_error.go new file mode 100644 index 00000000..a0dad4f3 --- /dev/null +++ b/internal/task/permanent_error.go @@ -0,0 +1,35 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package task + +import ( + "strings" + + "github.com/hibiken/asynq" +) + +const defaultPermanentErrorMessage = "任务无法继续执行" + +type permanentTaskError struct { + message string +} + +// PermanentError marks a safe domain message as a non-retryable task failure. +// It intentionally accepts no underlying error so Error never exposes provider, +// URL, header, response-body, or other sensitive implementation details. +func PermanentError(message string) error { + message = strings.TrimSpace(message) + if message == "" { + message = defaultPermanentErrorMessage + } + return &permanentTaskError{message: message} +} + +func (e *permanentTaskError) Error() string { + return e.message +} + +func (e *permanentTaskError) Unwrap() error { + return asynq.SkipRetry +} diff --git a/internal/task/permanent_error_test.go b/internal/task/permanent_error_test.go new file mode 100644 index 00000000..f8ae9300 --- /dev/null +++ b/internal/task/permanent_error_test.go @@ -0,0 +1,27 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package task + +import ( + "errors" + "testing" + + "github.com/hibiken/asynq" + "github.com/stretchr/testify/assert" +) + +func TestPermanentErrorSkipsRetryWithoutExposingAsynqMessage(t *testing.T) { + err := PermanentError(" 来源配置无效 ") + + assert.True(t, errors.Is(err, asynq.SkipRetry)) + assert.Equal(t, "来源配置无效", err.Error()) + assert.NotContains(t, err.Error(), asynq.SkipRetry.Error()) +} + +func TestPermanentErrorUsesSafeFallbackForBlankMessage(t *testing.T) { + err := PermanentError(" ") + + assert.True(t, errors.Is(err, asynq.SkipRetry)) + assert.Equal(t, defaultPermanentErrorMessage, err.Error()) +} diff --git a/pkg/httppool/httppool.go b/pkg/httppool/httppool.go index 2297dc67..ed98a9b5 100644 --- a/pkg/httppool/httppool.go +++ b/pkg/httppool/httppool.go @@ -5,9 +5,11 @@ package httppool import ( + "context" "crypto/tls" "net" "net/http" + "net/url" "sync" "time" @@ -30,28 +32,66 @@ var ( once sync.Once ) +// TransportOptions configures the request-specific parts of a pooled HTTP +// transport. Pool sizes and timeout defaults remain managed by this package. +// A nil Proxy explicitly disables proxy use. +type TransportOptions struct { + Proxy func(*http.Request) (*url.URL, error) + DialContext func(context.Context, string, string) (net.Conn, error) + TLSClientConfig *tls.Config + ResponseHeaderTimeout time.Duration + TraceFilter func(*http.Request) bool +} + +// NewTransport returns an independently configurable pooled transport wrapped +// with OTel instrumentation. The supplied TLS configuration is cloned before +// use so later caller mutations cannot change an active transport. +func NewTransport(options TransportOptions) http.RoundTripper { + dialContext := options.DialContext + if dialContext == nil { + dialContext = (&net.Dialer{ + Timeout: dialTimeout, + KeepAlive: dialKeepAlive, + }).DialContext + } + + tlsConfig := options.TLSClientConfig + if tlsConfig == nil { + tlsConfig = &tls.Config{} + } else { + tlsConfig = tlsConfig.Clone() + } + if tlsConfig.ClientSessionCache == nil { + tlsConfig.ClientSessionCache = tls.NewLRUClientSessionCache(tlsSessionCacheSize) + } + + transport := &http.Transport{ + Proxy: options.Proxy, + DialContext: dialContext, + ForceAttemptHTTP2: true, + MaxIdleConns: maxIdleConns, + MaxIdleConnsPerHost: maxIdleConnsPerHost, + IdleConnTimeout: idleConnTimeout, + TLSHandshakeTimeout: tlsHandshakeTimeout, + ResponseHeaderTimeout: options.ResponseHeaderTimeout, + ExpectContinueTimeout: expectContinueTimeout, + TLSClientConfig: tlsConfig, + } + otelOptions := make([]otelhttp.Option, 0, 1) + if options.TraceFilter != nil { + otelOptions = append(otelOptions, otelhttp.WithFilter(options.TraceFilter)) + } + return otelhttp.NewTransport(transport, otelOptions...) +} + // DefaultTransport returns a globally shared, optimized http.RoundTripper // with OTel instrumentation. It maintains a pool of idle TCP connections // across hosts. func DefaultTransport() http.RoundTripper { once.Do(func() { - transport := &http.Transport{ + defaultTransport = NewTransport(TransportOptions{ Proxy: http.ProxyFromEnvironment, - DialContext: (&net.Dialer{ - Timeout: dialTimeout, - KeepAlive: dialKeepAlive, - }).DialContext, - ForceAttemptHTTP2: true, - MaxIdleConns: maxIdleConns, - MaxIdleConnsPerHost: maxIdleConnsPerHost, - IdleConnTimeout: idleConnTimeout, - TLSHandshakeTimeout: tlsHandshakeTimeout, - ExpectContinueTimeout: expectContinueTimeout, - TLSClientConfig: &tls.Config{ - ClientSessionCache: tls.NewLRUClientSessionCache(tlsSessionCacheSize), - }, - } - defaultTransport = otelhttp.NewTransport(transport) + }) }) return defaultTransport } diff --git a/pkg/httppool/httppool_test.go b/pkg/httppool/httppool_test.go index efc71871..a753bda7 100644 --- a/pkg/httppool/httppool_test.go +++ b/pkg/httppool/httppool_test.go @@ -4,6 +4,12 @@ package httppool import ( + "context" + "crypto/tls" + "io" + "net" + "net/http" + "net/http/httptest" "testing" "time" ) @@ -35,3 +41,60 @@ func TestNewClient(t *testing.T) { t.Error("NewClient() is not configured with the default transport") } } + +func TestNewTransportUsesConfiguredDirectDialer(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write([]byte("ok")) + })) + t.Cleanup(server.Close) + + var dialedAddress string + dialer := &net.Dialer{} + transport := NewTransport(TransportOptions{ + Proxy: nil, + DialContext: func(ctx context.Context, network string, address string) (net.Conn, error) { + dialedAddress = address + return dialer.DialContext(ctx, network, server.Listener.Addr().String()) + }, + }) + client := &http.Client{Transport: transport} + t.Cleanup(client.CloseIdleConnections) + + request, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://artifact.example/site.zip", nil) + if err != nil { + t.Fatalf("NewRequestWithContext() error = %v", err) + } + response, err := client.Do(request) + if err != nil { + t.Fatalf("client.Do() error = %v", err) + } + defer func() { _ = response.Body.Close() }() + if _, err := io.ReadAll(response.Body); err != nil { + t.Fatalf("ReadAll() error = %v", err) + } + if dialedAddress != "artifact.example:80" { + t.Fatalf("DialContext address = %q, want direct target", dialedAddress) + } +} + +func TestNewTransportClonesTLSConfig(t *testing.T) { + server := httptest.NewTLSServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(server.Close) + + tlsConfig := &tls.Config{InsecureSkipVerify: true} //nolint:gosec // test-only self-signed server + client := &http.Client{Transport: NewTransport(TransportOptions{TLSClientConfig: tlsConfig})} + t.Cleanup(client.CloseIdleConnections) + tlsConfig.InsecureSkipVerify = false + + request, err := http.NewRequestWithContext(t.Context(), http.MethodGet, server.URL, nil) + if err != nil { + t.Fatalf("NewRequestWithContext() error = %v", err) + } + response, err := client.Do(request) + if err != nil { + t.Fatalf("client.Do() error = %v", err) + } + _ = response.Body.Close() +} From c39a3edcc35d4b040380dd808683ac01abb78da1 Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 18:31:42 +0800 Subject: [PATCH 4/8] =?UTF-8?q?feat(pages):=20=E6=94=AF=E6=8C=81=20GitHub?= =?UTF-8?q?=20Release=20=E9=83=A8=E7=BD=B2=E6=BA=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 增加 latest/tag 手动检查与同步、ETag 与限流退避、资源替换确认，以及对应的前端来源管理和部署来源展示。 --- docs/changelog/index.md | 1 + docs/docs.go | 4 +- docs/plan/20260719-pages-source-sync-v2.md | 9 +- docs/swagger.json | 4 +- docs/swagger.yaml | 4 +- .../detail/components/deployment-history.tsx | 4 +- .../detail/components/pages-source-card.tsx | 282 ++++-- .../components/pages-source-details.tsx | 137 +++ .../detail/components/pages-source-dialog.tsx | 273 +++++- .../components/pages-source-github-fields.tsx | 130 +++ .../components/pages-source-validation.ts | 68 ++ frontend/components/ui/alert.tsx | 66 ++ frontend/lib/services/openflare/index.ts | 5 + .../lib/services/openflare/pages.service.ts | 4 +- frontend/lib/services/openflare/types.ts | 41 +- .../tests/openflare/pages-service.test.ts | 68 ++ .../tests/openflare/pages-source-ui.test.tsx | 467 ++++++++- .../openflare/pages-source-validation.test.ts | 100 ++ internal/apps/openflare/pages/errs.go | 106 ++- .../apps/openflare/pages/github_source.go | 343 +++++++ .../openflare/pages/github_source_action.go | 724 ++++++++++++++ .../pages/github_source_identity_test.go | 111 +++ .../openflare/pages/github_source_test.go | 888 ++++++++++++++++++ internal/apps/openflare/pages/logics.go | 8 +- internal/apps/openflare/pages/routers.go | 21 +- internal/apps/openflare/pages/source.go | 141 ++- .../apps/openflare/pages/source_runtime.go | 24 + internal/apps/openflare/pages/source_sync.go | 38 +- .../apps/openflare/pages/source_sync_test.go | 12 +- internal/apps/openflare/pages/source_tasks.go | 130 ++- internal/apps/openflare/pages/source_test.go | 31 +- internal/integration/githubrelease/client.go | 725 ++++++++++++++ .../integration/githubrelease/client_test.go | 748 +++++++++++++++ .../integration/githubrelease/transport.go | 312 ++++++ 34 files changed, 5751 insertions(+), 278 deletions(-) create mode 100644 frontend/app/(main)/pages/detail/components/pages-source-details.tsx create mode 100644 frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx create mode 100644 frontend/app/(main)/pages/detail/components/pages-source-validation.ts create mode 100644 frontend/components/ui/alert.tsx create mode 100644 frontend/tests/openflare/pages-source-validation.test.ts create mode 100644 internal/apps/openflare/pages/github_source.go create mode 100644 internal/apps/openflare/pages/github_source_action.go create mode 100644 internal/apps/openflare/pages/github_source_identity_test.go create mode 100644 internal/apps/openflare/pages/github_source_test.go create mode 100644 internal/integration/githubrelease/client.go create mode 100644 internal/integration/githubrelease/client_test.go create mode 100644 internal/integration/githubrelease/transport.go diff --git a/docs/changelog/index.md b/docs/changelog/index.md index db529121..9dc94e9d 100644 --- a/docs/changelog/index.md +++ b/docs/changelog/index.md @@ -23,6 +23,7 @@ sidebar: false ### 新增 +- Pages 项目新增持久部署源，可配置 Remote URL 或公开 GitHub Release，并支持手动检查、同步发布、来源状态查看与同一 Release 资源替换确认；部署历史会保留安全的来源快照。 - WAF 规则编排新增「UA 检查」节点：可要求携带 User-Agent、按浏览器/操作系统白名单（且/或）匹配，并优先屏蔽常见爬虫、非正常 UA（不含爬虫）与自定义正则 UA。 ### 改进 diff --git a/docs/docs.go b/docs/docs.go index 6947b49a..2f5ca21a 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -8879,7 +8879,7 @@ const docTemplate = `{ "SessionCookie": [] } ], - "description": "Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约", + "description": "异步检查 GitHub Release 来源；Remote URL 来源不支持检查更新", "produces": [ "application/json" ], @@ -9112,7 +9112,7 @@ const docTemplate = `{ "SessionCookie": [] } ], - "description": "Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回", + "description": "支持 Remote URL 与公开 GitHub Release 来源；敏感地址仅写入，不会在响应中返回", "consumes": [ "application/json" ], diff --git a/docs/plan/20260719-pages-source-sync-v2.md b/docs/plan/20260719-pages-source-sync-v2.md index 47859802..03978901 100644 --- a/docs/plan/20260719-pages-source-sync-v2.md +++ b/docs/plan/20260719-pages-source-sync-v2.md @@ -211,9 +211,16 @@ of_pages_projects GitHub： ```text -SHA-256("github|" + owner_repo + "|" + selector + "|" + tag + "|" + asset_name) +LP(value) = uint64be(byte_length(UTF8(value))) || UTF8(value) + +SHA-256( + "openflare:pages:github-release:v2" || + LP(owner_repo) || LP(selector) || LP(tag) || LP(asset_name) +) ``` +GitHub identity 对每个 UTF-8 字段使用无歧义的长度前缀编码，不能使用分隔符直接拼接；自动更新开关和检查间隔不参与 identity。 + Remote： ```text diff --git a/docs/swagger.json b/docs/swagger.json index 5c81cc5e..cc0f02c5 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -8872,7 +8872,7 @@ "SessionCookie": [] } ], - "description": "Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约", + "description": "异步检查 GitHub Release 来源；Remote URL 来源不支持检查更新", "produces": [ "application/json" ], @@ -9105,7 +9105,7 @@ "SessionCookie": [] } ], - "description": "Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回", + "description": "支持 Remote URL 与公开 GitHub Release 来源；敏感地址仅写入，不会在响应中返回", "consumes": [ "application/json" ], diff --git a/docs/swagger.yaml b/docs/swagger.yaml index b0e074d9..edce34e0 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -9468,7 +9468,7 @@ paths: - openflare-pages /api/v1/d/pages/{id}/source/check: post: - description: Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约 + description: 异步检查 GitHub Release 来源；Remote URL 来源不支持检查更新 parameters: - description: 项目 ID in: path @@ -9611,7 +9611,7 @@ paths: post: consumes: - application/json - description: Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回 + description: 支持 Remote URL 与公开 GitHub Release 来源；敏感地址仅写入，不会在响应中返回 parameters: - description: 项目 ID in: path diff --git a/frontend/app/(main)/pages/detail/components/deployment-history.tsx b/frontend/app/(main)/pages/detail/components/deployment-history.tsx index bd046500..7fe01105 100644 --- a/frontend/app/(main)/pages/detail/components/deployment-history.tsx +++ b/frontend/app/(main)/pages/detail/components/deployment-history.tsx @@ -45,7 +45,7 @@ const SOURCE_LABELS: Record = { manual_upload: '本地上传', manual_url: 'URL 导入', remote_url: 'Remote URL', - github_release: 'GitHub Release', + github_release: 'GitHub', }; const TRIGGER_LABELS: Record = { @@ -172,7 +172,7 @@ export function DeploymentHistory({ ) : deployments.length === 0 ? ( ) : ( deployments.map((deployment) => { diff --git a/frontend/app/(main)/pages/detail/components/pages-source-card.tsx b/frontend/app/(main)/pages/detail/components/pages-source-card.tsx index eced9522..9e101436 100644 --- a/frontend/app/(main)/pages/detail/components/pages-source-card.tsx +++ b/frontend/app/(main)/pages/detail/components/pages-source-card.tsx @@ -1,11 +1,28 @@ 'use client'; -import { useEffect, useRef, useState } from 'react'; +import { useCallback, useEffect, useRef, useState } from 'react'; import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; -import { Download, Pencil, RefreshCw, RotateCcw } from 'lucide-react'; +import { + Download, + Github, + Pencil, + RefreshCw, + RotateCcw, + Search, +} from 'lucide-react'; import { toast } from 'sonner'; import { ErrorInline } from '@/components/layout/error'; +import { + AlertDialog, + AlertDialogAction, + AlertDialogCancel, + AlertDialogContent, + AlertDialogDescription, + AlertDialogFooter, + AlertDialogHeader, + AlertDialogTitle, +} from '@/components/ui/alert-dialog'; import { Badge } from '@/components/ui/badge'; import { Button } from '@/components/ui/button'; import { @@ -21,12 +38,11 @@ import { Skeleton } from '@/components/ui/skeleton'; import { Spinner } from '@/components/ui/spinner'; import { AdminTaskService } from '@/lib/services/admin'; import { - type PagesSource, + type PagesSourceActionPayload, type PagesSourceActionReceipt, type PagesSourceStatus, PagesService, } from '@/lib/services/openflare'; -import { formatDateTime } from '@/lib/utils'; import { deploymentsQueryKey, @@ -34,7 +50,11 @@ import { projectsQueryKey, sourceQueryKey, } from '../../components/pages-utils'; -import { PagesSourceDialog } from './pages-source-dialog'; +import { type PagesSourceMode, PagesSourceDialog } from './pages-source-dialog'; +import { + GitHubSourceDetails, + RemoteSourceDetails, +} from './pages-source-details'; const ACTION_POLL_INTERVAL = 2_000; const ACTION_MAX_WAIT = 16 * 60 * 1_000; @@ -59,10 +79,8 @@ interface ActiveAction { startedAt: number; } -function revisionSummary(source: PagesSource) { - if (source.source_type === 'manual' || !source.last_applied) - return '尚未应用'; - return `${source.last_applied.label} · ${source.last_applied.revision.slice(0, 12)}`; +function sourceActionLabel(action: PagesSourceActionReceipt['action']) { + return action === 'check' ? '检查' : '同步并发布'; } export function PagesSourceCard({ projectId }: { projectId: number }) { @@ -70,11 +88,10 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { const handledExecutionID = useRef(null); const sourcePollingStartedAt = useRef(null); const [dialogOpen, setDialogOpen] = useState(false); - const [dialogMode, setDialogMode] = useState<'manual' | 'remote_url'>( - 'manual', - ); + const [dialogMode, setDialogMode] = useState('manual'); const [activeAction, setActiveAction] = useState(null); const [actionTimedOut, setActionTimedOut] = useState(false); + const [attentionDialogOpen, setAttentionDialogOpen] = useState(false); const sourceQuery = useQuery({ queryKey: sourceQueryKey(projectId), @@ -102,8 +119,11 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { 'task-execution', activeAction?.receipt.execution_id ?? '', ], - queryFn: () => - AdminTaskService.getTaskExecution(activeAction!.receipt.execution_id), + queryFn: () => { + const executionID = activeAction?.receipt.execution_id; + if (!executionID) throw new Error('缺少任务执行 ID'); + return AdminTaskService.getTaskExecution(executionID); + }, enabled: Boolean(activeAction) && !actionTimedOut, refetchInterval: (query) => { if (actionTimedOut) return false; @@ -114,6 +134,15 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { }, }); + const beginActionPolling = useCallback( + (receipt: PagesSourceActionReceipt) => { + handledExecutionID.current = null; + setActiveAction({ receipt, startedAt: Date.now() }); + setActionTimedOut(false); + }, + [], + ); + useEffect(() => { if (!activeAction || actionTimedOut) return; const elapsed = Date.now() - activeAction.startedAt; @@ -124,7 +153,11 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { useEffect(() => { const execution = executionQuery.data; - if (!execution || !['succeeded', 'failed'].includes(execution.status)) { + if ( + !activeAction || + !execution || + !['succeeded', 'failed'].includes(execution.status) + ) { return; } if (handledExecutionID.current === execution.id) return; @@ -142,21 +175,36 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { queryClient.invalidateQueries({ queryKey: projectsQueryKey }), ]); + const actionLabel = sourceActionLabel(activeAction.receipt.action); if (execution.status === 'succeeded') { - toast.success('部署源同步并发布完成'); + toast.success(`部署源${actionLabel}完成`); } else { - toast.error(execution.error_message || '部署源同步失败'); + toast.error(execution.error_message || `部署源${actionLabel}失败`); } setActiveAction(null); setActionTimedOut(false); - }, [executionQuery.data, projectId, queryClient]); + }, [activeAction, executionQuery.data, projectId, queryClient]); + + const checkMutation = useMutation({ + mutationFn: () => PagesService.checkSource(projectId), + onSuccess: async (receipt) => { + beginActionPolling(receipt); + await queryClient.invalidateQueries({ + queryKey: sourceQueryKey(projectId), + }); + toast.success('检查任务已提交'); + }, + onError: (error) => { + toast.error(error instanceof Error ? error.message : '检查任务提交失败'); + }, + }); const syncMutation = useMutation({ - mutationFn: () => PagesService.syncSource(projectId, {}), + mutationFn: (payload: PagesSourceActionPayload) => + PagesService.syncSource(projectId, payload), onSuccess: async (receipt) => { - handledExecutionID.current = null; - setActiveAction({ receipt, startedAt: Date.now() }); - setActionTimedOut(false); + setAttentionDialogOpen(false); + beginActionPolling(receipt); await queryClient.invalidateQueries({ queryKey: sourceQueryKey(projectId), }); @@ -176,13 +224,38 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { const sourceBusy = source?.source_type !== 'manual' && (source?.sync_status === 'checking' || source?.sync_status === 'syncing'); - const actionsDisabled = syncMutation.isPending || executionBusy || sourceBusy; + const actionsDisabled = + checkMutation.isPending || + syncMutation.isPending || + executionBusy || + sourceBusy; + const checkBusy = + checkMutation.isPending || + (executionBusy && activeAction?.receipt.action === 'check') || + (source?.source_type !== 'manual' && source?.sync_status === 'checking'); + const syncBusy = + syncMutation.isPending || + (executionBusy && activeAction?.receipt.action === 'sync') || + (source?.source_type !== 'manual' && source?.sync_status === 'syncing'); + const dispatchError = checkMutation.error ?? syncMutation.error; - const openSourceDialog = (mode: 'manual' | 'remote_url') => { + const openSourceDialog = (mode: PagesSourceMode) => { setDialogMode(mode); setDialogOpen(true); }; + const dispatchSync = () => { + checkMutation.reset(); + if ( + source?.source_type === 'github_release' && + source.sync_status === 'attention' + ) { + setAttentionDialogOpen(true); + return; + } + syncMutation.mutate({}); + }; + if (sourceQuery.isLoading) { return ( @@ -219,10 +292,22 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { ); } + const effectiveSourceStatus = executionBusy + ? activeAction?.receipt.action === 'check' + ? 'checking' + : 'syncing' + : source.source_type === 'manual' + ? undefined + : (source.sync_status ?? 'idle'); const status = source.source_type === 'manual' ? null - : SOURCE_STATUS[source.sync_status ?? 'idle']; + : SOURCE_STATUS[effectiveSourceStatus ?? 'idle']; + const attentionRevision = + source.source_type === 'github_release' && + source.sync_status === 'attention' + ? source.last_seen + : undefined; return ( <> @@ -250,47 +335,20 @@ export function PagesSourceCard({ projectId }: { projectId: number }) {

) : source.source_type === 'remote_url' ? ( -
-
- 脱敏地址 - {source.display_url} -
-
- 网络策略 - - {source.remote_network_policy === 'trusted_internal' - ? '受信内网模式' - : '公网安全模式'} - -
-
- 最近同步 - - {source.last_synced_at - ? formatDateTime(source.last_synced_at) - : '尚未同步'} - -
-
- - 已应用 revision - - - {revisionSummary(source)} - -
- {source.last_error ? ( -
- -
- ) : null} -
+ ) : ( -
- 当前版本暂不提供该来源类型的编辑界面。 -
+ )} + {dispatchError ? ( + + ) : null} {executionQuery.isError ? ( {source.source_type === 'manual' ? ( - - ) : source.source_type === 'remote_url' ? ( + <> + + + + ) : ( <> + {source.source_type === 'github_release' ? ( + + ) : null} - ) : null} + )}
- ) : ( + ) : mode === 'remote_url' ? ( <> Remote URL @@ -240,24 +388,48 @@ export function PagesSourceDialog({ onClick={() => { setReplaceURL(true); setRemoteURL(''); + setRevealRemoteURL(false); }} > 更换地址 ) : ( - { - setRemoteURL(event.target.value); - setURLError(''); - }} - /> +
+ { + setRemoteURL(event.target.value); + setURLError(''); + }} + /> + +
)} {urlError || @@ -274,6 +446,7 @@ export function PagesSourceDialog({ onClick={() => { setReplaceURL(false); setRemoteURL(''); + setRevealRemoteURL(false); setURLError(''); }} > @@ -313,6 +486,14 @@ export function PagesSourceDialog({ + ) : ( + )} @@ -321,13 +502,17 @@ export function PagesSourceDialog({ type='button' variant='outline' disabled={isPending} - onClick={() => onOpenChange(false)} + onClick={() => { + setRemoteURL(''); + setRevealRemoteURL(false); + onOpenChange(false); + }} > 取消 @@ -361,7 +546,7 @@ export function PagesSourceDialog({ if (confirmation === 'manual') { deleteMutation.mutate(); } else if (confirmation === 'trusted_internal') { - updateMutation.mutate(); + updateMutation.mutate(remotePayload()); } }} > diff --git a/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx b/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx new file mode 100644 index 00000000..7a8d3422 --- /dev/null +++ b/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx @@ -0,0 +1,130 @@ +import { + Field, + FieldDescription, + FieldLabel, + FieldTitle, +} from '@/components/ui/field'; +import { Input } from '@/components/ui/input'; +import { ToggleGroup, ToggleGroupItem } from '@/components/ui/toggle-group'; +import type { PagesGitHubReleaseSelector } from '@/lib/services/openflare'; + +export interface PagesGitHubSourceFormValue { + repositoryURL: string; + releaseSelector: PagesGitHubReleaseSelector; + releaseTag: string; + assetName: string; +} + +export interface PagesGitHubSourceFormErrors { + repository: string; + releaseTag: string; + assetName: string; +} + +interface PagesSourceGitHubFieldsProps { + value: PagesGitHubSourceFormValue; + errors: PagesGitHubSourceFormErrors; + defaultAssetName: string; + onChange: (value: PagesGitHubSourceFormValue) => void; + onErrorsChange: (errors: PagesGitHubSourceFormErrors) => void; +} + +export function PagesSourceGitHubFields({ + value, + errors, + defaultAssetName, + onChange, + onErrorsChange, +}: PagesSourceGitHubFieldsProps) { + return ( + <> + + + GitHub 仓库 URL + + { + onChange({ ...value, repositoryURL: event.target.value }); + onErrorsChange({ ...errors, repository: '' }); + }} + /> + + {errors.repository || '仅支持公开 github.com 仓库。'} + + + + + Release 选择 + { + if (selector === 'latest' || selector === 'tag') { + onChange({ ...value, releaseSelector: selector }); + onErrorsChange({ ...errors, releaseTag: '' }); + } + }} + > + + 最新 Release + + + 固定 Tag + + + + 当前阶段由管理员手动检查并决定是否发布。 + + + + {value.releaseSelector === 'tag' ? ( + + Release tag + { + onChange({ ...value, releaseTag: event.target.value }); + onErrorsChange({ ...errors, releaseTag: '' }); + }} + /> + + {errors.releaseTag || '精确检查并同步指定 tag。'} + + + ) : null} + + + + Release Asset 文件名 + + { + onChange({ ...value, assetName: event.target.value }); + onErrorsChange({ ...errors, assetName: '' }); + }} + /> + + {errors.assetName || '按文件名精确匹配已上传的 Release Asset。'} + + + + ); +} diff --git a/frontend/app/(main)/pages/detail/components/pages-source-validation.ts b/frontend/app/(main)/pages/detail/components/pages-source-validation.ts new file mode 100644 index 00000000..b497b369 --- /dev/null +++ b/frontend/app/(main)/pages/detail/components/pages-source-validation.ts @@ -0,0 +1,68 @@ +function hasWellFormedUnicode(value: string) { + for (let index = 0; index < value.length; index += 1) { + const code = value.charCodeAt(index); + if (code >= 0xd800 && code <= 0xdbff) { + const next = value.charCodeAt(index + 1); + if (!(next >= 0xdc00 && next <= 0xdfff)) return false; + index += 1; + } else if (code >= 0xdc00 && code <= 0xdfff) { + return false; + } + } + return true; +} + +function validGitHubSafeText(value: string) { + return ( + value !== '' && + new TextEncoder().encode(value).byteLength <= 255 && + !/[\u0000-\u001f\u007f-\u009f\u061c\u200e\u200f\u2028-\u202e\u2066-\u2069]/u.test( + value, + ) && + hasWellFormedUnicode(value) + ); +} + +export function validGitHubRepositoryURL(raw: string) { + const match = /^https:\/\/([^/]+)\/([^/]+)\/([^/]+)$/u.exec(raw); + if (!match) return false; + + const [, host, owner, rawRepository] = match; + const repository = rawRepository.replace(/\.git$/u, ''); + return ( + host.toLowerCase() === 'github.com' && + /^[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?$/iu.test(owner) && + /^[a-z0-9._-]+$/iu.test(repository) && + repository.length <= 100 && + !['.', '..'].includes(repository) + ); +} + +export function validGitHubAssetName(value: string) { + return ( + validGitHubSafeText(value) && + value !== '.' && + value !== '..' && + !value.includes('/') && + !value.includes('\\') + ); +} + +export function validGitHubReleaseTag(value: string) { + const components = value.split('/'); + return ( + validGitHubSafeText(value) && + !value.endsWith('.') && + !value.includes('..') && + !value.includes('@{') && + ![' ', '~', '^', ':', '?', '*', '[', '\\'].some((character) => + value.includes(character), + ) && + components.every( + (component) => + component !== '' && + !component.startsWith('.') && + !component.endsWith('.lock'), + ) + ); +} diff --git a/frontend/components/ui/alert.tsx b/frontend/components/ui/alert.tsx new file mode 100644 index 00000000..a7f3eac0 --- /dev/null +++ b/frontend/components/ui/alert.tsx @@ -0,0 +1,66 @@ +import * as React from 'react'; +import { cva, type VariantProps } from 'class-variance-authority'; + +import { cn } from '@/lib/utils'; + +const alertVariants = cva( + 'relative grid w-full grid-cols-[0_1fr] items-start gap-y-0.5 rounded-lg border px-4 py-3 text-sm has-[>svg]:grid-cols-[calc(var(--spacing)*4)_1fr] has-[>svg]:gap-x-3 [&>svg]:size-4 [&>svg]:translate-y-0.5 [&>svg]:text-current', + { + variants: { + variant: { + default: 'bg-card text-card-foreground', + destructive: + 'bg-card text-destructive *:data-[slot=alert-description]:text-destructive/90 [&>svg]:text-current', + }, + }, + defaultVariants: { + variant: 'default', + }, + }, +); + +function Alert({ + className, + variant, + ...props +}: React.ComponentProps<'div'> & VariantProps) { + return ( +
+ ); +} + +function AlertTitle({ className, ...props }: React.ComponentProps<'div'>) { + return ( +
+ ); +} + +function AlertDescription({ + className, + ...props +}: React.ComponentProps<'div'>) { + return ( +
+ ); +} + +export { Alert, AlertDescription, AlertTitle }; diff --git a/frontend/lib/services/openflare/index.ts b/frontend/lib/services/openflare/index.ts index 7bca6843..52ec0746 100644 --- a/frontend/lib/services/openflare/index.ts +++ b/frontend/lib/services/openflare/index.ts @@ -109,7 +109,11 @@ export type { OriginMutationPayload, PagesDeployment, PagesDeploymentFile, + PagesGitHubLatestSourceUpdatePayload, + PagesGitHubReleaseSelector, PagesGitHubReleaseSource, + PagesGitHubSourceUpdatePayload, + PagesGitHubTagSourceUpdatePayload, PagesManualSource, PagesProject, PagesProjectPayload, @@ -121,6 +125,7 @@ export type { PagesSourceActionReceipt, PagesSourceRevision, PagesSourceStatus, + PagesSourceUpdatePayload, PagesSourceUpdateResult, AcmeAccountItem, DnsAccountItem, diff --git a/frontend/lib/services/openflare/pages.service.ts b/frontend/lib/services/openflare/pages.service.ts index e45b0543..a666ef6b 100644 --- a/frontend/lib/services/openflare/pages.service.ts +++ b/frontend/lib/services/openflare/pages.service.ts @@ -12,10 +12,10 @@ import type { PagesDeploymentUploadPayload, PagesProject, PagesProjectPayload, - PagesRemoteSourceUpdatePayload, PagesSource, PagesSourceActionPayload, PagesSourceActionReceipt, + PagesSourceUpdatePayload, PagesSourceUpdateResult, } from './types'; @@ -51,7 +51,7 @@ export class PagesService extends OpenFlareBaseService { static updateSource( projectId: number, - payload: PagesRemoteSourceUpdatePayload, + payload: PagesSourceUpdatePayload, ): Promise { return this.post( `/${projectId}/source/update`, diff --git a/frontend/lib/services/openflare/types.ts b/frontend/lib/services/openflare/types.ts index 0b9be41c..66eace1c 100644 --- a/frontend/lib/services/openflare/types.ts +++ b/frontend/lib/services/openflare/types.ts @@ -462,6 +462,8 @@ export type PagesSourceStatus = export type PagesRemoteNetworkPolicy = 'public' | 'trusted_internal'; +export type PagesGitHubReleaseSelector = 'latest' | 'tag'; + export interface PagesSourceRevision { revision: string; label: string; @@ -493,15 +495,15 @@ export interface PagesRemoteURLSource extends PagesSourceRuntimeView { export interface PagesGitHubReleaseSource extends PagesSourceRuntimeView { source_type: 'github_release'; github_repository: string; - release_selector: 'latest' | 'tag'; - release_tag: string; + release_selector: PagesGitHubReleaseSelector; + release_tag?: string; asset_name: string; - auto_update_enabled: boolean; - check_interval_minutes: number; + auto_update_enabled?: false; + check_interval_minutes?: number; } /** - * 部署源使用判别联合，后续仓库构建来源只需增加独立 variant， + * 部署源使用判别联合，后续仓库构建来源只需增加独立 git_repository variant， * 不需要向 Remote 或 GitHub Release 填入构建字段。 */ export type PagesSource = @@ -514,6 +516,35 @@ export interface PagesRemoteSourceUpdatePayload { remote_network_policy: PagesRemoteNetworkPolicy; } +interface PagesGitHubSourceUpdateBase { + source_type: 'github_release'; + repository_url: string; + asset_name: string; + auto_update_enabled: false; +} + +export interface PagesGitHubLatestSourceUpdatePayload extends PagesGitHubSourceUpdateBase { + release_selector: 'latest'; + release_tag: ''; + check_interval_minutes: number; +} + +export interface PagesGitHubTagSourceUpdatePayload extends PagesGitHubSourceUpdateBase { + release_selector: 'tag'; + release_tag: string; + check_interval_minutes: 0; +} + +export type PagesGitHubSourceUpdatePayload = + PagesGitHubLatestSourceUpdatePayload | PagesGitHubTagSourceUpdatePayload; + +/** + * Source 更新请求保持 Provider 判别联合；未来仓库拉取构建使用独立 git_repository variant， + * 不向 Remote URL 或 GitHub Release payload 混入构建字段。 + */ +export type PagesSourceUpdatePayload = + PagesRemoteSourceUpdatePayload | PagesGitHubSourceUpdatePayload; + export interface PagesSourceActionPayload { confirmed_revision?: string; } diff --git a/frontend/tests/openflare/pages-service.test.ts b/frontend/tests/openflare/pages-service.test.ts index 516e5596..3aafe780 100644 --- a/frontend/tests/openflare/pages-service.test.ts +++ b/frontend/tests/openflare/pages-service.test.ts @@ -91,6 +91,74 @@ describe('PagesService', () => { ); }); + it('sends the complete GitHub latest discriminator payload', async () => { + vi.mocked(apiClient.post).mockResolvedValue( + response({ + source: { source_type: 'manual' }, + check_task: null, + warning: '', + }), + ); + + await PagesService.updateSource(12, { + source_type: 'github_release', + repository_url: 'https://github.com/openflare/site', + release_selector: 'latest', + release_tag: '', + asset_name: 'dist.zip', + auto_update_enabled: false, + check_interval_minutes: 60, + }); + + expect(apiClient.post).toHaveBeenCalledWith( + '/api/v1/d/pages/12/source/update', + { + source_type: 'github_release', + repository_url: 'https://github.com/openflare/site', + release_selector: 'latest', + release_tag: '', + asset_name: 'dist.zip', + auto_update_enabled: false, + check_interval_minutes: 60, + }, + undefined, + ); + }); + + it('sends safe disabled defaults with the GitHub tag discriminator', async () => { + vi.mocked(apiClient.post).mockResolvedValue( + response({ + source: { source_type: 'manual' }, + check_task: null, + warning: '', + }), + ); + + await PagesService.updateSource(12, { + source_type: 'github_release', + repository_url: 'https://github.com/openflare/site', + release_selector: 'tag', + release_tag: 'v1.2.3', + asset_name: 'site.tar.gz', + auto_update_enabled: false, + check_interval_minutes: 0, + }); + + expect(apiClient.post).toHaveBeenCalledWith( + '/api/v1/d/pages/12/source/update', + { + source_type: 'github_release', + repository_url: 'https://github.com/openflare/site', + release_selector: 'tag', + release_tag: 'v1.2.3', + asset_name: 'site.tar.gz', + auto_update_enabled: false, + check_interval_minutes: 0, + }, + undefined, + ); + }); + it('uploads only the package multipart field', async () => { vi.mocked(apiClient.post).mockResolvedValue(response({})); const file = new File(['site'], 'site.zip', { diff --git a/frontend/tests/openflare/pages-source-ui.test.tsx b/frontend/tests/openflare/pages-source-ui.test.tsx index b9773638..bdf61cc8 100644 --- a/frontend/tests/openflare/pages-source-ui.test.tsx +++ b/frontend/tests/openflare/pages-source-ui.test.tsx @@ -1,13 +1,19 @@ import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; -import { render, screen, waitFor } from '@testing-library/react'; +import { render, screen, waitFor, within } from '@testing-library/react'; import userEvent from '@testing-library/user-event'; import { beforeEach, describe, expect, it, vi } from 'vitest'; import { DeploymentUploadDialog } from '@/app/(main)/pages/components/deployment-upload-dialog'; import { DeploymentHistory } from '@/app/(main)/pages/detail/components/deployment-history'; import { PagesSourceCard } from '@/app/(main)/pages/detail/components/pages-source-card'; -import { AdminTaskService } from '@/lib/services/admin'; import { + AdminTaskService, + type TaskExecution, + type TaskExecutionStatus, +} from '@/lib/services/admin'; +import { + type PagesDeployment, + type PagesGitHubReleaseSource, type PagesRemoteURLSource, PagesService, } from '@/lib/services/openflare'; @@ -21,6 +27,7 @@ vi.mock('@/lib/services/openflare', async (importOriginal) => { getSource: vi.fn(), updateSource: vi.fn(), deleteSource: vi.fn(), + checkSource: vi.fn(), syncSource: vi.fn(), listDeployments: vi.fn(), listDeploymentFiles: vi.fn(), @@ -67,11 +74,73 @@ const remoteSource: PagesRemoteURLSource = { last_error: '', }; +const githubLatestSource: PagesGitHubReleaseSource = { + source_type: 'github_release', + github_repository: 'openflare/site', + release_selector: 'latest', + asset_name: 'dist.zip', + auto_update_enabled: false, + check_interval_minutes: 60, + sync_status: 'update_available', + update_available: true, + last_seen: { + revision: 'b'.repeat(64), + label: 'v1.2.3', + asset_name: 'dist.zip', + }, + last_applied: { + revision: 'a'.repeat(64), + label: 'v1.2.2', + asset_name: 'dist.zip', + }, + last_checked_at: '2026-07-19T10:00:00Z', + last_synced_at: '2026-07-18T10:00:00Z', + next_check_at: '2026-07-19T11:00:00Z', + last_error: '', +}; + +const githubAttentionRevision = 'c'.repeat(64); +const githubAttentionSource: PagesGitHubReleaseSource = { + ...githubLatestSource, + sync_status: 'attention', + update_available: true, + last_seen: { + revision: githubAttentionRevision, + label: 'v1.2.3', + asset_name: 'dist.zip', + }, +}; + +function taskExecution( + status: TaskExecutionStatus, + errorMessage = '', +): TaskExecution { + return { + id: '42', + task_id: 'manual_of_pages_source_action_1', + task_type: 'of_pages_source_action', + task_name: 'Pages 来源动作', + status, + retryable: false, + max_retry: 0, + retry_count: 0, + log: '', + error_message: errorMessage, + result: '', + duration: 1, + payload: '', + triggered_by: 'admin:1', + created_at: '2026-07-19T10:00:00Z', + updated_at: '2026-07-19T10:00:01Z', + }; +} + describe('Pages source UI', () => { beforeEach(() => { vi.mocked(PagesService.getSource).mockReset(); vi.mocked(PagesService.updateSource).mockReset(); vi.mocked(PagesService.deleteSource).mockReset(); + vi.mocked(PagesService.checkSource).mockReset(); vi.mocked(PagesService.syncSource).mockReset(); vi.mocked(PagesService.listDeployments).mockReset(); vi.mocked(PagesService.listDeploymentFiles).mockReset(); @@ -81,7 +150,8 @@ describe('Pages source UI', () => { vi.mocked(AdminTaskService.getTaskExecution).mockReset(); }); - it('keeps Phase 1 manual source focused on upload and Remote URL', async () => { + it('offers the three Phase 2 source types without future repository build controls', async () => { + const user = userEvent.setup(); vi.mocked(PagesService.getSource).mockResolvedValue({ source_type: 'manual', }); @@ -92,8 +162,327 @@ describe('Pages source UI', () => { expect( screen.getByRole('button', { name: '配置 Remote URL' }), ).toBeVisible(); + expect( + screen.getByRole('button', { name: '配置 GitHub Release' }), + ).toBeVisible(); expect(screen.queryByText('检查更新')).not.toBeInTheDocument(); expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); + + await user.click( + screen.getByRole('button', { name: '配置 GitHub Release' }), + ); + expect(screen.getByRole('radio', { name: '手动部署' })).toBeVisible(); + expect(screen.getByRole('radio', { name: 'Remote URL' })).toBeVisible(); + expect(screen.getByRole('radio', { name: 'GitHub Release' })).toBeVisible(); + expect( + screen.getByText('仓库源码构建将在后续作为独立来源类型提供。'), + ).toBeVisible(); + expect(screen.queryByText('构建命令')).not.toBeInTheDocument(); + expect(screen.queryByText('输出目录')).not.toBeInTheDocument(); + }); + + it('submits the GitHub latest payload while keeping automatic controls hidden', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue({ + source_type: 'manual', + }); + vi.mocked(PagesService.updateSource).mockResolvedValue({ + source: githubLatestSource, + check_task: null, + warning: '', + }); + + renderWithQuery(); + + await user.click( + await screen.findByRole('button', { name: '配置 GitHub Release' }), + ); + await user.type( + screen.getByLabelText('GitHub 仓库 URL'), + 'https://github.com/openflare/site', + ); + expect(screen.getByLabelText('Release Asset 文件名')).toHaveValue( + 'dist.zip', + ); + expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); + expect(screen.queryByText('检查间隔')).not.toBeInTheDocument(); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + + await waitFor(() => { + expect(PagesService.updateSource).toHaveBeenCalledWith(9, { + source_type: 'github_release', + repository_url: 'https://github.com/openflare/site', + release_selector: 'latest', + release_tag: '', + asset_name: 'dist.zip', + auto_update_enabled: false, + check_interval_minutes: 60, + }); + }); + }); + + it('rejects non-canonical GitHub repository URL paths', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue({ + source_type: 'manual', + }); + + renderWithQuery(); + + await user.click( + await screen.findByRole('button', { name: '配置 GitHub Release' }), + ); + const repositoryInput = screen.getByLabelText('GitHub 仓库 URL'); + await user.type(repositoryInput, 'https://github.com//openflare/site'); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + expect( + screen.getByText( + '请输入 https://github.com/{owner}/{repo} 格式的公开仓库地址', + ), + ).toBeVisible(); + + await user.clear(repositoryInput); + await user.type(repositoryInput, 'https://github.com/openflare/site/'); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + expect(PagesService.updateSource).not.toHaveBeenCalled(); + }); + + it('submits the GitHub tag discriminator with safe disabled defaults', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue({ + source_type: 'manual', + }); + vi.mocked(PagesService.updateSource).mockResolvedValue({ + source: { + ...githubLatestSource, + release_selector: 'tag', + release_tag: 'v1.2.3', + check_interval_minutes: undefined, + next_check_at: null, + }, + check_task: null, + warning: '', + }); + + renderWithQuery(); + + await user.click( + await screen.findByRole('button', { name: '配置 GitHub Release' }), + ); + await user.type( + screen.getByLabelText('GitHub 仓库 URL'), + 'https://github.com/openflare/site', + ); + await user.click(screen.getByRole('radio', { name: '固定 Tag' })); + await user.type( + screen.getByLabelText('Release tag'), + 'release/candidate#1&channel=stable', + ); + await user.clear(screen.getByLabelText('Release Asset 文件名')); + await user.type( + screen.getByLabelText('Release Asset 文件名'), + ' site?arch=amd64#stable.zip ', + ); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + + await waitFor(() => { + expect(PagesService.updateSource).toHaveBeenCalledWith(9, { + source_type: 'github_release', + repository_url: 'https://github.com/openflare/site', + release_selector: 'tag', + release_tag: 'release/candidate#1&channel=stable', + asset_name: ' site?arch=amd64#stable.zip ', + auto_update_enabled: false, + check_interval_minutes: 0, + }); + }); + }); + + it('rejects unsafe GitHub tag and asset values before saving', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue({ + source_type: 'manual', + }); + + renderWithQuery(); + + await user.click( + await screen.findByRole('button', { name: '配置 GitHub Release' }), + ); + await user.type( + screen.getByLabelText('GitHub 仓库 URL'), + 'https://github.com/openflare/site', + ); + await user.click(screen.getByRole('radio', { name: '固定 Tag' })); + await user.type(screen.getByLabelText('Release tag'), 'release//candidate'); + await user.clear(screen.getByLabelText('Release Asset 文件名')); + await user.type( + screen.getByLabelText('Release Asset 文件名'), + '../dist.zip', + ); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + + expect( + screen.getByText( + 'Release tag 须为有效 Git ref（1–255 字节，可使用 /、#、&、=）', + ), + ).toBeVisible(); + expect( + screen.getByText( + 'Asset 文件名须为 1–255 字节，且不能是路径或包含控制、换行、双向文本字符', + ), + ).toBeVisible(); + expect(PagesService.updateSource).not.toHaveBeenCalled(); + }); + + it('polls the initial check receipt and represents the queued window locally', async () => { + const user = userEvent.setup(); + const idleGitHubSource: PagesGitHubReleaseSource = { + ...githubLatestSource, + sync_status: 'idle', + update_available: false, + }; + vi.mocked(PagesService.getSource) + .mockResolvedValueOnce({ source_type: 'manual' }) + .mockResolvedValue(idleGitHubSource); + vi.mocked(PagesService.updateSource).mockResolvedValue({ + source: idleGitHubSource, + check_task: { + task_id: 'manual_of_pages_source_action_1', + execution_id: '42', + action: 'check', + }, + warning: '', + }); + vi.mocked(AdminTaskService.getTaskExecution).mockResolvedValue( + taskExecution('pending'), + ); + + renderWithQuery(); + + await user.click( + await screen.findByRole('button', { name: '配置 GitHub Release' }), + ); + await user.type( + screen.getByLabelText('GitHub 仓库 URL'), + 'https://github.com/openflare/site', + ); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + + await waitFor(() => { + expect(AdminTaskService.getTaskExecution).toHaveBeenCalledWith('42'); + expect(screen.getByText('检查中')).toBeVisible(); + }); + const checkButton = screen.getByRole('button', { name: /检查更新/ }); + expect(checkButton).toBeDisabled(); + expect(within(checkButton).getByRole('status')).toBeVisible(); + expect(PagesService.checkSource).not.toHaveBeenCalled(); + expect(PagesService.syncSource).not.toHaveBeenCalled(); + }); + + it('shows GitHub release state and hides Phase 3 controls', async () => { + vi.mocked(PagesService.getSource).mockResolvedValue(githubLatestSource); + + renderWithQuery(); + + expect(await screen.findByText('openflare/site')).toBeVisible(); + expect(screen.getByText('v1.2.3 · bbbbbbbbbbbb')).toBeVisible(); + expect(screen.getByText('v1.2.2 · aaaaaaaaaaaa')).toBeVisible(); + expect(screen.getByText('有可用更新')).toBeVisible(); + expect(screen.queryByText('下次检查时间')).not.toBeInTheDocument(); + expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); + expect(screen.queryByText('检查间隔')).not.toBeInTheDocument(); + }); + + it('dispatches a GitHub check and starts TaskExecution polling', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(githubLatestSource); + vi.mocked(PagesService.checkSource).mockResolvedValue({ + task_id: 'manual_of_pages_source_action_1', + execution_id: '42', + action: 'check', + }); + vi.mocked(AdminTaskService.getTaskExecution).mockResolvedValue( + taskExecution('succeeded'), + ); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '检查更新' })); + + await waitFor(() => { + expect(PagesService.checkSource).toHaveBeenCalledWith(9); + expect(AdminTaskService.getTaskExecution).toHaveBeenCalledWith('42'); + expect(PagesService.syncSource).not.toHaveBeenCalled(); + }); + }); + + it('renders a GitHub check dispatch error', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(githubLatestSource); + vi.mocked(PagesService.checkSource).mockRejectedValue( + new Error('GitHub API 暂不可用'), + ); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '检查更新' })); + + expect(await screen.findByText('GitHub API 暂不可用')).toBeVisible(); + }); + + it('renders a TaskExecution polling error with an explicit retry', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(githubLatestSource); + vi.mocked(PagesService.checkSource).mockResolvedValue({ + task_id: 'manual_of_pages_source_action_1', + execution_id: '42', + action: 'check', + }); + vi.mocked(AdminTaskService.getTaskExecution).mockRejectedValue( + new Error('任务状态暂时不可读取'), + ); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '检查更新' })); + + expect(await screen.findByText('任务状态暂时不可读取')).toBeVisible(); + expect(screen.getByRole('button', { name: '重试' })).toBeVisible(); + }); + + it('requires the exact currently displayed revision for attention sync', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(githubAttentionSource); + vi.mocked(PagesService.syncSource).mockResolvedValue({ + task_id: 'manual_of_pages_source_action_1', + execution_id: '42', + action: 'sync', + }); + vi.mocked(AdminTaskService.getTaskExecution).mockResolvedValue( + taskExecution('succeeded'), + ); + + renderWithQuery(); + + const alert = await screen.findByRole('alert'); + expect( + within(alert).getByText('Release Asset 发生变化，需要显式确认'), + ).toBeVisible(); + expect(within(alert).getByText(githubAttentionRevision)).toBeVisible(); + + await user.click(screen.getByRole('button', { name: '同步并发布' })); + const dialog = screen.getByRole('alertdialog'); + expect(within(dialog).getByText(githubAttentionRevision)).toBeVisible(); + await user.click( + within(dialog).getByRole('button', { name: '确认并发布' }), + ); + + await waitFor(() => { + expect(PagesService.syncSource).toHaveBeenCalledWith(9, { + confirmed_revision: githubAttentionRevision, + }); + }); }); it('never reuses the masked URL as an editable value', async () => { @@ -119,7 +508,13 @@ describe('Pages source UI', () => { 'https://artifacts.example.com/site.zip?token=...', ); expect(input).toHaveValue(''); + expect(input).toHaveAttribute('type', 'password'); await user.type(input, 'https://new.example.com/site.zip?token=new'); + expect( + screen.queryByText('https://new.example.com/site.zip?token=new'), + ).not.toBeInTheDocument(); + await user.click(screen.getByRole('button', { name: '显示 Remote URL' })); + expect(input).toHaveAttribute('type', 'url'); await user.click(screen.getByRole('button', { name: '保存 Remote 来源' })); await waitFor(() => { @@ -130,6 +525,17 @@ describe('Pages source UI', () => { remote_network_policy: 'public', }); }); + + await waitFor(() => { + expect(screen.queryByRole('dialog')).not.toBeInTheDocument(); + }); + await user.click(screen.getByRole('button', { name: '编辑来源' })); + await user.click(screen.getByRole('button', { name: '更换地址' })); + const reopenedInput = screen.getByPlaceholderText( + 'https://artifacts.example.com/site.zip?token=...', + ); + expect(reopenedInput).toHaveValue(''); + expect(reopenedInput).toHaveAttribute('type', 'password'); }); it('requires a second confirmation for trusted internal networking', async () => { @@ -169,24 +575,9 @@ describe('Pages source UI', () => { execution_id: '42', action: 'sync', }); - vi.mocked(AdminTaskService.getTaskExecution).mockResolvedValue({ - id: '42', - task_id: 'manual_of_pages_source_action_1', - task_type: 'of_pages_source_action', - task_name: 'Pages 来源动作', - status: 'succeeded', - retryable: false, - max_retry: 0, - retry_count: 0, - log: '', - error_message: '', - result: '', - duration: 1, - payload: '', - triggered_by: 'admin:1', - created_at: '2026-07-19T10:00:00Z', - updated_at: '2026-07-19T10:00:01Z', - }); + vi.mocked(AdminTaskService.getTaskExecution) + .mockResolvedValueOnce(taskExecution('pending')) + .mockResolvedValue(taskExecution('succeeded')); renderWithQuery(); @@ -196,6 +587,14 @@ describe('Pages source UI', () => { expect(PagesService.syncSource).toHaveBeenCalledWith(9, {}); expect(AdminTaskService.getTaskExecution).toHaveBeenCalledWith('42'); }); + expect(screen.getByRole('button', { name: /同步并发布/ })).toBeDisabled(); + + await waitFor( + () => { + expect(AdminTaskService.getTaskExecution).toHaveBeenCalledTimes(2); + }, + { timeout: 3_500 }, + ); }); it('shows the actual project entry and no one-off URL upload tab', () => { @@ -224,4 +623,30 @@ describe('Pages source UI', () => { expect(await screen.findByText('部署历史暂时不可用')).toBeVisible(); expect(screen.queryByText('暂无部署')).not.toBeInTheDocument(); }); + + it('renders the immutable GitHub deployment provenance in Chinese', async () => { + const deployment: PagesDeployment = { + id: 51, + project_id: 9, + deployment_number: 7, + checksum: 'd'.repeat(64), + status: 'active', + file_count: 12, + total_size: 4_096, + created_by: 'user:1', + source_type: 'github_release', + source_label: 'v1.2.3', + trigger_type: 'manual_sync', + created_at: '2026-07-19T10:00:00Z', + activated_at: '2026-07-19T10:00:01Z', + }; + vi.mocked(PagesService.listDeployments).mockResolvedValue([deployment]); + + renderWithQuery( + , + ); + + expect(await screen.findByText('GitHub · v1.2.3 · 手动同步')).toBeVisible(); + expect(screen.getByText('当前生产部署')).toBeVisible(); + }); }); diff --git a/frontend/tests/openflare/pages-source-validation.test.ts b/frontend/tests/openflare/pages-source-validation.test.ts new file mode 100644 index 00000000..e19d3ece --- /dev/null +++ b/frontend/tests/openflare/pages-source-validation.test.ts @@ -0,0 +1,100 @@ +import { describe, expect, it } from 'vitest'; + +import { + validGitHubAssetName, + validGitHubReleaseTag, + validGitHubRepositoryURL, +} from '@/app/(main)/pages/detail/components/pages-source-validation'; + +describe('Pages GitHub source validation', () => { + it('accepts only canonical public GitHub repository URLs', () => { + expect(validGitHubRepositoryURL('https://github.com/acme/site')).toBe(true); + expect(validGitHubRepositoryURL('https://GitHub.com/acme/site.git')).toBe( + true, + ); + + const invalid = [ + 'http://github.com/acme/site', + 'https://github.com//acme/site', + 'https://github.com/acme/site/', + 'https://github.com/acme/site/extra', + 'https://github.com/acme/./site', + 'https://github.com/acme/../site', + 'https://github.com/acme/%73ite', + 'https://github.com/acme/site?token=secret', + 'https://github.com:443/acme/site', + String.raw`https://github.com/acme\site`, + ]; + for (const value of invalid) { + expect(validGitHubRepositoryURL(value), value).toBe(false); + } + }); + + it('mirrors Git ref rules while preserving legal release tag characters', () => { + const valid = [ + '@', + 'release/v1#stable&channel=prod', + 'foo.LOCK', + '中文/发布=稳定', + ]; + for (const value of valid) { + expect(validGitHubReleaseTag(value), value).toBe(true); + } + + const invalid = [ + '', + 'release v1', + 'release~v1', + 'release^v1', + 'release:v1', + 'release?v1', + 'release*v1', + 'release[v1', + String.raw`release\v1`, + 'release..v1', + 'release@{v1', + 'release//v1', + '/release', + 'release/', + 'release.', + '.release', + 'release/.candidate', + 'release/v1.lock', + 'release\nsecret', + 'release\u2028secret', + 'release\u202esecret', + 'a'.repeat(256), + ]; + for (const value of invalid) { + expect(validGitHubReleaseTag(value), value).toBe(false); + } + }); + + it('preserves exact legal asset names and rejects path or display controls', () => { + const valid = [ + 'dist.zip', + 'dist?channel=stable&part#1.zip', + ' dist.zip ', + 'build=production.zip', + ]; + for (const value of valid) { + expect(validGitHubAssetName(value), value).toBe(true); + } + + const invalid = [ + '', + '.', + '..', + '../dist.zip', + String.raw`dir\dist.zip`, + 'dist\n.zip', + 'dist\u2028.zip', + 'dist\u202e.zip', + 'a'.repeat(256), + '\ud800', + ]; + for (const value of invalid) { + expect(validGitHubAssetName(value), value).toBe(false); + } + }); +}); diff --git a/internal/apps/openflare/pages/errs.go b/internal/apps/openflare/pages/errs.go index 15b2c270..64fc9a25 100644 --- a/internal/apps/openflare/pages/errs.go +++ b/internal/apps/openflare/pages/errs.go @@ -5,51 +5,63 @@ package pages const ( - errPagesProjectNotFound = "pages 项目不存在" - errPagesSlugExists = "pages 项目标识已存在" - errPagesNameRequired = "pages 项目名称不能为空" - errPagesSlugInvalid = "pages 项目标识只能包含小写字母、数字和连字符" - errPagesDeleteReferenced = "pages 项目已被规则引用，不能删除" - errPagesDeploymentNotFound = "pages 部署不存在" - errPagesDeploymentMismatch = "pages 部署不属于该项目" - errPagesDeleteActiveDeploy = "不能删除当前激活的 Pages 部署" - errPagesPackageMissing = "缺少 Pages 部署包" - errPagesPackageURLRequired = "请填写部署包下载链接" - errPagesPackageURLInvalid = "部署包下载链接无效，仅支持 http/https" - errPagesPackageURLDownloadFailed = "从链接下载部署包失败" - errPagesPackageURLTooLarge = "链接指向的部署包超过大小限制" - errPagesPackageNotZip = "pages 部署包必须是 .zip 文件" // legacy alias kept for tests - errPagesPackageUnsupported = "pages 部署包仅支持 zip、tar.gz、tar.xz、tar.bz2、tar、7z 格式" - errPagesPackageInvalidZip = "pages 部署包不是有效 zip 文件" // legacy alias - errPagesPackageInvalid = "pages 部署包不是有效的压缩文件" - errPagesPackageEmpty = "pages 部署包不能为空" - errPagesPackageExtractedTooLarge = "pages 部署包展开后体积超过限制" - errPagesPackageFileTooLarge = "pages 部署包内文件过大" - errPagesAPIProxyPathRequired = "启用 API 反代时，匹配路径不能为空" - errPagesAPIProxyPathPrefix = "API 反代匹配路径必须以 '/' 开头" - errPagesAPIProxyPassRequired = "启用 API 反代时，后端服务地址不能为空" //nolint:gosec // false positive: this is an error message, not hardcoded credentials - errPagesAPIProxyPassInvalid = "API 反代后端服务地址必须是有效的 HTTP/HTTPS URL" //nolint:gosec // false positive: this is an error message, not hardcoded credentials - errPagesPackagePathEmpty = "pages 部署包路径为空" - errPagesPackageUploadMissing = "pages 部署包上传记录不存在" - errPagesPackageNotInActiveConfig = "pages 部署尚未进入激活配置" - errPagesDeploymentHashMissing = "pages 部署包哈希缺失" - errPagesInvalidSnapshotFormat = "配置快照格式无效" - errPagesActorMissing = "无法识别当前用户" - errPagesEntryFileMissing = "当前激活部署中不存在指定入口文件" - errPagesSourceNotFound = "pages 部署源不存在" - errPagesSourceTypeRequired = "请选择 pages 部署源类型" - errPagesSourceTypeUnsupported = "当前阶段仅支持远程地址部署源" - errPagesSourceRemoteFields = "远程地址来源不能包含 GitHub 或自动更新配置" - errPagesSourceRemoteURLRequired = "请提供远程部署包地址" - errPagesSourceRemoteURLMode = "remote_url_set 与 remote_url 参数不匹配" - errPagesSourceRemoteURLInvalid = "远程部署包地址无效，仅支持不含用户信息和片段的 http/https 地址" - errPagesSourceNetworkPolicy = "远程地址网络策略仅支持 public 或 trusted_internal" - errPagesSourceCheckUnsupported = "远程地址来源不支持检查更新，请使用立即同步" - errPagesSourceActionBusy = "pages 部署源任务正在执行" - errPagesSourceActionInvalid = "pages 部署源任务参数无效" - errPagesSourceActionStale = "pages 部署源配置已变化，本次任务已跳过" - errPagesSourceLeaseLost = "pages 部署源任务执行权已失效" - errPagesSourceSyncFailed = "pages 部署源同步失败" - errPagesSourceTaskDispatchFailed = "pages 部署源任务入队失败" - errPagesSourceInternal = "pages 部署源操作失败，请稍后重试" + errPagesProjectNotFound = "pages 项目不存在" + errPagesSlugExists = "pages 项目标识已存在" + errPagesNameRequired = "pages 项目名称不能为空" + errPagesSlugInvalid = "pages 项目标识只能包含小写字母、数字和连字符" + errPagesDeleteReferenced = "pages 项目已被规则引用，不能删除" + errPagesDeploymentNotFound = "pages 部署不存在" + errPagesDeploymentMismatch = "pages 部署不属于该项目" + errPagesDeleteActiveDeploy = "不能删除当前激活的 Pages 部署" + errPagesPackageMissing = "缺少 Pages 部署包" + errPagesPackageURLRequired = "请填写部署包下载链接" + errPagesPackageURLInvalid = "部署包下载链接无效，仅支持 http/https" + errPagesPackageURLDownloadFailed = "从链接下载部署包失败" + errPagesPackageURLTooLarge = "链接指向的部署包超过大小限制" + errPagesPackageNotZip = "pages 部署包必须是 .zip 文件" // legacy alias kept for tests + errPagesPackageUnsupported = "pages 部署包仅支持 zip、tar.gz、tar.xz、tar.bz2、tar、7z 格式" + errPagesPackageInvalidZip = "pages 部署包不是有效 zip 文件" // legacy alias + errPagesPackageInvalid = "pages 部署包不是有效的压缩文件" + errPagesPackageEmpty = "pages 部署包不能为空" + errPagesPackageExtractedTooLarge = "pages 部署包展开后体积超过限制" + errPagesPackageFileTooLarge = "pages 部署包内文件过大" + errPagesAPIProxyPathRequired = "启用 API 反代时，匹配路径不能为空" + errPagesAPIProxyPathPrefix = "API 反代匹配路径必须以 '/' 开头" + errPagesAPIProxyPassRequired = "启用 API 反代时，后端服务地址不能为空" //nolint:gosec // false positive: this is an error message, not hardcoded credentials + errPagesAPIProxyPassInvalid = "API 反代后端服务地址必须是有效的 HTTP/HTTPS URL" //nolint:gosec // false positive: this is an error message, not hardcoded credentials + errPagesPackagePathEmpty = "pages 部署包路径为空" + errPagesPackageUploadMissing = "pages 部署包上传记录不存在" + errPagesPackageNotInActiveConfig = "pages 部署尚未进入激活配置" + errPagesDeploymentHashMissing = "pages 部署包哈希缺失" + errPagesInvalidSnapshotFormat = "配置快照格式无效" + errPagesActorMissing = "无法识别当前用户" + errPagesEntryFileMissing = "当前激活部署中不存在指定入口文件" + errPagesSourceNotFound = "pages 部署源不存在" + errPagesSourceTypeRequired = "请选择 pages 部署源类型" + errPagesSourceTypeUnsupported = "pages 部署源类型不受支持" + errPagesSourceRemoteFields = "远程地址来源不能包含 GitHub 或自动更新配置" + errPagesSourceRemoteURLRequired = "请提供远程部署包地址" + errPagesSourceRemoteURLMode = "remote_url_set 与 remote_url 参数不匹配" + errPagesSourceRemoteURLInvalid = "远程部署包地址无效，仅支持不含用户信息和片段的 http/https 地址" + errPagesSourceNetworkPolicy = "远程地址网络策略仅支持 public 或 trusted_internal" + errPagesSourceGitHubFields = "GitHub Release 来源不能包含远程地址配置" + errPagesSourceRepositoryInvalid = "GitHub 仓库地址无效，仅支持 https://github.com/{owner}/{repo}" + errPagesSourceSelectorInvalid = "GitHub Release 选择方式无效" + errPagesSourceAssetNameInvalid = "GitHub Release 资源名称必须是安全的文件名" + errPagesSourceCheckInterval = "GitHub latest 检查间隔必须在 5 到 1440 分钟之间" + errPagesSourceAutoNotAvailable = "自动更新将在后续阶段开放，当前必须保持关闭" + errPagesSourceReleaseNotFound = "未找到符合配置的 GitHub Release 资源" + errPagesSourceDigestInvalid = "GitHub Release 资源摘要格式无效" + errPagesSourceDigestMismatch = "GitHub Release 资源摘要校验失败" + errPagesSourceConfirmationNeeded = "检测到同一 Release 的资源已被替换，请刷新并确认当前版本" + errPagesSourceConfirmationStale = "确认的版本已变化，请刷新后重新确认" + errPagesSourceInitialCheckWarning = "部署源已保存，但首次检查任务入队失败，请稍后手动检查" + errPagesSourceCheckUnsupported = "远程地址来源不支持检查更新，请使用立即同步" + errPagesSourceActionBusy = "pages 部署源任务正在执行" + errPagesSourceActionInvalid = "pages 部署源任务参数无效" + errPagesSourceActionStale = "pages 部署源配置已变化，本次任务已跳过" + errPagesSourceLeaseLost = "pages 部署源任务执行权已失效" + errPagesSourceSyncFailed = "pages 部署源同步失败" + errPagesSourceTaskDispatchFailed = "pages 部署源任务入队失败" + errPagesSourceInternal = "pages 部署源操作失败，请稍后重试" ) diff --git a/internal/apps/openflare/pages/github_source.go b/internal/apps/openflare/pages/github_source.go new file mode 100644 index 00000000..d2e8fa66 --- /dev/null +++ b/internal/apps/openflare/pages/github_source.go @@ -0,0 +1,343 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "errors" + "net/url" + "path" + "regexp" + "strings" + "time" + "unicode" + "unicode/utf8" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/pkg/logger" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +const ( + githubReleaseSelectorLatest = "latest" + githubReleaseSelectorTag = "tag" + githubSourceIdentityDomain = "openflare:pages:github-release:v2" + initialCheckRetryDelay = 5 * time.Minute + githubRepositoryPathParts = 2 + githubCheckJitterRange = 301 + githubCheckJitterCenter = 150 +) + +var ( + githubOwnerPattern = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]{0,37}[A-Za-z0-9])?$`) + githubRepoPattern = regexp.MustCompile(`^[A-Za-z0-9._-]+$`) +) + +type githubSourceConfig struct { + Repository string + Selector string + Tag string + AssetName string + CheckInterval int + SourceIdentity string +} + +func validateGitHubSourceInput(input SourceUpdateInput) error { + if strings.TrimSpace(input.SourceType) != PagesSourceTypeGitHubRelease { + return errors.New(errPagesSourceTypeUnsupported) + } + if input.RemoteURLSet || strings.TrimSpace(input.RemoteURL) != "" || + strings.TrimSpace(input.RemoteNetworkPolicy) != "" { + return errors.New(errPagesSourceGitHubFields) + } + if input.AutoUpdateEnabled { + return errors.New(errPagesSourceAutoNotAvailable) + } + if _, err := normalizeGitHubRepositoryURL(input.RepositoryURL); err != nil { + return err + } + selector := strings.TrimSpace(input.ReleaseSelector) + if selector == "" { + selector = githubReleaseSelectorLatest + } + assetName := input.AssetName + if assetName == "" { + assetName = defaultGitHubAssetName + } + if !validGitHubAssetName(assetName) { + return errors.New(errPagesSourceAssetNameInvalid) + } + switch selector { + case githubReleaseSelectorLatest: + if input.ReleaseTag != "" { + return errors.New(errPagesSourceSelectorInvalid) + } + interval := input.CheckIntervalMinutes + if interval != 0 && (interval < minimumCheckInterval || interval > maximumCheckInterval) { + return errors.New(errPagesSourceCheckInterval) + } + case githubReleaseSelectorTag: + if !validGitHubReleaseTagConfig(input.ReleaseTag) || input.CheckIntervalMinutes != 0 { + return errors.New(errPagesSourceSelectorInvalid) + } + default: + return errors.New(errPagesSourceSelectorInvalid) + } + return nil +} + +func buildGitHubSourceConfig(input SourceUpdateInput) (githubSourceConfig, error) { + repository, err := normalizeGitHubRepositoryURL(input.RepositoryURL) + if err != nil { + return githubSourceConfig{}, err + } + selector := strings.TrimSpace(input.ReleaseSelector) + if selector == "" { + selector = githubReleaseSelectorLatest + } + tag := input.ReleaseTag + assetName := input.AssetName + if assetName == "" { + assetName = defaultGitHubAssetName + } + interval := input.CheckIntervalMinutes + if selector == githubReleaseSelectorLatest && interval == 0 { + interval = defaultCheckInterval + } + return githubSourceConfig{ + Repository: repository, + Selector: selector, + Tag: tag, + AssetName: assetName, + CheckInterval: interval, + SourceIdentity: buildGitHubSourceIdentity(repository, selector, tag, assetName), + }, nil +} + +func buildGitHubSourceIdentity(repository, selector, tag, assetName string) string { + fields := [...]string{repository, selector, tag, assetName} + encoded := make([]byte, 0, len(githubSourceIdentityDomain)+len(fields)*8+ + len(repository)+len(selector)+len(tag)+len(assetName)) + encoded = append(encoded, githubSourceIdentityDomain...) + var fieldLength [8]byte + for _, field := range fields { + // Go strings hold the validated UTF-8 bytes used by GitHub. Prefixing each + // field with its byte length prevents delimiter characters from creating + // ambiguous identities across field boundaries. + binary.BigEndian.PutUint64(fieldLength[:], uint64(len(field))) + encoded = append(encoded, fieldLength[:]...) + encoded = append(encoded, field...) + } + identityHash := sha256.Sum256(encoded) + return hex.EncodeToString(identityHash[:]) +} + +func normalizeGitHubRepositoryURL(raw string) (string, error) { + parsed, err := url.Parse(raw) + if err != nil || parsed.Scheme != "https" || !strings.EqualFold(parsed.Host, "github.com") || + parsed.User != nil || parsed.RawQuery != "" || parsed.ForceQuery || parsed.Fragment != "" || + strings.Contains(raw, "#") || + parsed.EscapedPath() != parsed.Path || !strings.HasPrefix(parsed.Path, "/") || + strings.HasPrefix(parsed.Path, "//") || strings.HasSuffix(parsed.Path, "/") { + return "", errors.New(errPagesSourceRepositoryInvalid) + } + parts := strings.Split(strings.TrimPrefix(parsed.Path, "/"), "/") + if len(parts) != githubRepositoryPathParts { + return "", errors.New(errPagesSourceRepositoryInvalid) + } + owner := parts[0] + repository := parts[1] + repository = strings.TrimSuffix(repository, ".git") + if !githubOwnerPattern.MatchString(owner) || !githubRepoPattern.MatchString(repository) || + len(repository) > 100 || repository == "." || repository == ".." { + return "", errors.New(errPagesSourceRepositoryInvalid) + } + return owner + "/" + repository, nil +} + +func validGitHubReleaseTagConfig(value string) bool { + if !validGitHubReleaseDisplayTag(value) || + strings.ContainsAny(value, " ~^:?*[\\") || strings.HasPrefix(value, "/") || + strings.HasSuffix(value, "/") || strings.HasSuffix(value, ".") || + strings.Contains(value, "//") || strings.Contains(value, "..") || strings.Contains(value, "@{") { + return false + } + for component := range strings.SplitSeq(value, "/") { + if strings.HasPrefix(component, ".") || strings.HasSuffix(component, ".lock") { + return false + } + } + return true +} + +func validGitHubReleaseDisplayTag(value string) bool { + if value == "" || len(value) > 255 || !utf8.ValidString(value) { + return false + } + for _, character := range value { + if unsafeGitHubInputRune(character) { + return false + } + } + return true +} + +func validGitHubAssetName(value string) bool { + if value == "" || len(value) > 255 || !utf8.ValidString(value) || + path.Base(value) != value || strings.Contains(value, "\\") || + value == "." || value == ".." { + return false + } + for _, character := range value { + if unsafeGitHubInputRune(character) { + return false + } + } + return true +} + +func unsafeGitHubInputRune(character rune) bool { + return unicode.IsControl(character) || character == '\u2028' || character == '\u2029' || + character == '\u061c' || character == '\u200e' || character == '\u200f' || + (character >= '\u202a' && character <= '\u202e') || + (character >= '\u2066' && character <= '\u2069') +} + +func updateGitHubSourceTx(tx *gorm.DB, projectID uint, input SourceUpdateInput) (bool, error) { + var project model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { + return false, err + } + existing, hasExisting, err := loadProjectSourceForUpdate(tx, projectID) + if err != nil { + return false, err + } + config, err := buildGitHubSourceConfig(input) + if err != nil { + return false, err + } + if !hasExisting { + return true, createGitHubSourceTx(tx, projectID, config) + } + if !githubSourceConfigChanged(existing, config) { + return false, nil + } + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", existing.ID).First(&runtime).Error; err != nil { + return false, err + } + identityChanged := existing.SourceIdentity != config.SourceIdentity + if err := tx.Model(existing).Updates(githubSourceUpdates(config, existing.ConfigVersion+1)).Error; err != nil { + return false, err + } + if err := resetRuntimeAfterGitHubUpdate(tx, &runtime, config, identityChanged); err != nil { + return false, err + } + return true, nil +} + +func createGitHubSourceTx(tx *gorm.DB, projectID uint, config githubSourceConfig) error { + source := &model.PagesProjectSource{ + ProjectID: projectID, + SourceType: PagesSourceTypeGitHubRelease, + GitHubRepository: config.Repository, + ReleaseSelector: config.Selector, + ReleaseTag: config.Tag, + AssetName: config.AssetName, + AutoUpdateEnabled: false, + CheckIntervalMinutes: config.CheckInterval, + ConfigVersion: 1, + SourceIdentity: config.SourceIdentity, + } + if err := tx.Create(source).Error; err != nil { + return err + } + runtime := &model.PagesProjectSourceRuntime{SourceID: source.ID, SyncStatus: pagesSourceStatusIdle} + if config.Selector == githubReleaseSelectorLatest { + next := nextGitHubCheckAt(time.Now(), source.ID, config.CheckInterval) + runtime.NextCheckAt = &next + } + return tx.Create(runtime).Error +} + +func githubSourceUpdates(config githubSourceConfig, version int) map[string]any { + return map[string]any{ + "source_type": PagesSourceTypeGitHubRelease, + "remote_url": "", + "remote_network_policy": "", + "github_repository": config.Repository, + "release_selector": config.Selector, + "release_tag": config.Tag, + "asset_name": config.AssetName, + sourceColumnAutoUpdateEnabled: false, + "check_interval_minutes": config.CheckInterval, + sourceColumnConfigVersion: version, + "source_identity": config.SourceIdentity, + } +} + +func githubSourceConfigChanged(existing *model.PagesProjectSource, config githubSourceConfig) bool { + return existing.SourceType != PagesSourceTypeGitHubRelease || existing.RemoteURL != "" || + existing.RemoteNetworkPolicy != "" || existing.GitHubRepository != config.Repository || + existing.ReleaseSelector != config.Selector || existing.ReleaseTag != config.Tag || + existing.AssetName != config.AssetName || existing.AutoUpdateEnabled || + existing.CheckIntervalMinutes != config.CheckInterval +} + +func resetRuntimeAfterGitHubUpdate( + tx *gorm.DB, + runtime *model.PagesProjectSourceRuntime, + config githubSourceConfig, + identityChanged bool, +) error { + if err := resetRuntimeAfterSourceUpdate(tx, runtime, identityChanged); err != nil { + return err + } + var nextCheckAt any + if config.Selector == githubReleaseSelectorLatest { + next := nextGitHubCheckAt(time.Now(), runtime.SourceID, config.CheckInterval) + nextCheckAt = &next + } + return tx.Model(runtime).Update("next_check_at", nextCheckAt).Error +} + +func nextGitHubCheckAt(now time.Time, sourceID uint, intervalMinutes int) time.Time { + // A stable, bounded offset avoids a thundering herd without persisting + // another scheduling field. Scanner Phase 3 reuses this calculation. + jitterSeconds := int64(sourceID%githubCheckJitterRange) - githubCheckJitterCenter + return now.Add(time.Duration(intervalMinutes)*time.Minute + time.Duration(jitterSeconds)*time.Second) +} + +func markInitialCheckDispatchFailed(ctx context.Context, sourceID uint, configVersion int) { + updates := map[string]any{ + sourceRuntimeColumnSyncStatus: pagesSourceStatusFailed, + sourceRuntimeColumnLastError: errPagesSourceInitialCheckWarning, + } + var source model.PagesProjectSource + if err := db.DB(ctx).Where("id = ? AND config_version = ?", sourceID, configVersion).First(&source).Error; err != nil { + if !errors.Is(err, gorm.ErrRecordNotFound) { + logger.ErrorF(ctx, "[PagesSource] load initial check source snapshot failed: source_id=%d error=%v", sourceID, err) + } + return + } + if source.ReleaseSelector == githubReleaseSelectorLatest { + next := time.Now().Add(initialCheckRetryDelay) + updates["next_check_at"] = &next + } + now := time.Now() + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", sourceID). + Where("lease_expires_at IS NULL OR lease_expires_at <= ?", now). + Where("EXISTS (SELECT 1 FROM of_pages_project_sources source WHERE source.id = ? AND source.config_version = ?)", sourceID, configVersion). + Updates(updates) + if result.Error != nil { + logger.ErrorF(ctx, "[PagesSource] mark initial check dispatch failure: source_id=%d error=%v", sourceID, result.Error) + } +} diff --git a/internal/apps/openflare/pages/github_source_action.go b/internal/apps/openflare/pages/github_source_action.go new file mode 100644 index 00000000..d4fc1e89 --- /dev/null +++ b/internal/apps/openflare/pages/github_source_action.go @@ -0,0 +1,724 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "regexp" + "strings" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/integration/githubrelease" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/task" + "github.com/Rain-kl/Wavelet/pkg/logger" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +const githubSourceDetailProvider = "github" + +var githubDigestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +type githubSourceProviderDomainError struct { + message string + permanent bool + retryAt *time.Time +} + +func (domainError *githubSourceProviderDomainError) Error() string { + return domainError.message +} + +type githubReleaseAPI interface { + Resolve(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) + Download(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) +} + +var newGitHubReleaseClient = func() githubReleaseAPI { + return githubrelease.NewClient() +} + +type githubSourceTarget struct { + Revision string + Detail sourceDetail + DetailJSON string + Release githubrelease.Release + Asset githubrelease.Asset + RetryAt *time.Time +} + +type githubCheckTaskResult struct { + Message string + Detail string + Stale bool +} + +type preparedGitHubSource struct { + target *githubSourceTarget + download *githubrelease.DownloadResult + format pagesarchive.Format + manifest *deploymentManifest + ingestState *sourceIngestState + limits pagesLimits +} + +func checkGitHubSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, +) (*githubCheckTaskResult, error) { + if snapshot == nil || snapshot.SourceType != PagesSourceTypeGitHubRelease { + return nil, errors.New(errPagesSourceTypeUnsupported) + } + task.AppendLog(ctx, "[check] 正在检查 GitHub Release：repo=%s asset=%s", snapshot.GitHubRepository, snapshot.AssetName) + client := newGitHubReleaseClient() + result, err := client.Resolve(ctx, githubrelease.ResolveRequest{ + Repository: snapshot.GitHubRepository, + Selector: githubrelease.Selector(snapshot.ReleaseSelector), + Tag: snapshot.ReleaseTag, + AssetName: snapshot.AssetName, + ETag: snapshot.ETag, + }) + if err != nil { + logger.WarnF(ctx, "[PagesSource] GitHub resolve failed: source_id=%d repo=%s error=%v", snapshot.SourceID, snapshot.GitHubRepository, err) + retryAt, _ := githubrelease.RetryAt(err) + domainErr := githubSourceDomainError(err) + if failErr := failGitHubCheckLease(ctx, snapshot, domainErr.Error(), retryAt); failErr != nil { + if errors.Is(failErr, errSourceFinalFence) { + return &githubCheckTaskResult{Message: errPagesSourceActionStale, Stale: true}, nil + } + return nil, failErr + } + return nil, domainErr + } + if result.NotModified { + if err := finishGitHubCheckNotModified(ctx, snapshot, result); err != nil { + if errors.Is(err, errSourceFinalFence) { + return &githubCheckTaskResult{Message: errPagesSourceActionStale, Stale: true}, nil + } + return nil, err + } + return &githubCheckTaskResult{Message: "GitHub Release 检查完成，内容未变化"}, nil + } + target, err := buildGitHubSourceTarget(result.Release, result.Asset, result.RetryAt) + if err != nil { + retryAt := time.Time{} + if result.RetryAt != nil { + retryAt = result.RetryAt.UTC() + } + if failErr := failGitHubCheckLease(ctx, snapshot, err.Error(), retryAt); failErr != nil { + if errors.Is(failErr, errSourceFinalFence) { + return &githubCheckTaskResult{Message: errPagesSourceActionStale, Stale: true}, nil + } + return nil, failErr + } + return nil, err + } + status, err := finishGitHubCheckTarget(ctx, snapshot, result, target) + if err != nil { + if errors.Is(err, errSourceFinalFence) { + return &githubCheckTaskResult{Message: errPagesSourceActionStale, Stale: true}, nil + } + return nil, err + } + detail, _ := json.Marshal(map[string]string{"revision": target.Revision, pagesDeploymentColumnStatus: status}) + message := "GitHub Release 检查完成" + switch status { + case pagesSourceStatusUpdateAvailable: + message = "发现新的 GitHub Release 部署包" + case pagesSourceStatusAttention: + message = "检测到同一 Release 的资源被替换，需要确认" + } + return &githubCheckTaskResult{Message: message, Detail: string(detail)}, nil +} + +func buildGitHubSourceTarget( + release githubrelease.Release, + asset githubrelease.Asset, + retryAt *time.Time, +) (*githubSourceTarget, error) { + digest := strings.ToLower(strings.TrimSpace(asset.Digest)) + if digest != "" && !githubDigestPattern.MatchString(digest) { + return nil, errors.New(errPagesSourceDigestInvalid) + } + if strings.TrimSpace(release.ID) == "" || strings.TrimSpace(asset.ID) == "" || + !validGitHubReleaseDisplayTag(release.Tag) || !validGitHubAssetName(asset.Name) || + asset.State != "uploaded" || asset.UpdatedAt.IsZero() { + return nil, errors.New(errPagesSourceReleaseNotFound) + } + updatedAt := asset.UpdatedAt.UTC().Format(time.RFC3339Nano) + rawRevision := "github:" + release.ID + ":" + asset.ID + ":" + updatedAt + ":" + digest + sum := sha256.Sum256([]byte(rawRevision)) + detail := sourceDetail{ + Provider: githubSourceDetailProvider, + Tag: release.Tag, + AssetName: asset.Name, + ReleaseID: release.ID, + AssetID: asset.ID, + AssetUpdatedAt: updatedAt, + Digest: digest, + } + detailJSON, err := json.Marshal(detail) + if err != nil { + return nil, errors.New(errPagesSourceSyncFailed) + } + return &githubSourceTarget{ + Revision: hex.EncodeToString(sum[:]), + Detail: detail, + DetailJSON: string(detailJSON), + Release: release, + Asset: asset, + RetryAt: retryAt, + }, nil +} + +func finishGitHubCheckNotModified( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + result githubrelease.ResolveResult, +) error { + return db.DB(ctx).Transaction(func(tx *gorm.DB) error { + runtime, now, err := lockOwnedSourceRuntime(tx, snapshot) + if err != nil { + return err + } + updates := githubCheckTerminalUpdates(snapshot, now, result.RetryAt) + updates["etag"] = result.ETag + updates[sourceRuntimeColumnSyncStatus] = normalizedSourceRuntimeStatus(runtime) + return tx.Model(runtime).Updates(updates).Error + }) +} + +func finishGitHubCheckTarget( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + result githubrelease.ResolveResult, + target *githubSourceTarget, +) (string, error) { + status := pagesSourceStatusIdle + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + runtime, now, err := lockOwnedSourceRuntime(tx, snapshot) + if err != nil { + return err + } + status = targetRuntimeStatus(target, runtime.LastAppliedRevision, runtime.LastAppliedDetail) + updates := githubCheckTerminalUpdates(snapshot, now, result.RetryAt) + updates["etag"] = result.ETag + updates["last_seen_revision"] = target.Revision + updates["last_seen_detail"] = target.DetailJSON + updates[sourceRuntimeColumnSyncStatus] = status + return tx.Model(runtime).Updates(updates).Error + }) + return status, err +} + +func githubCheckTerminalUpdates( + snapshot *sourceExecutionSnapshot, + now time.Time, + retryAt *time.Time, +) map[string]any { + updates := map[string]any{ + sourceRuntimeColumnLastError: "", + sourceRuntimeColumnLastCheckedAt: &now, + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + } + updates["next_check_at"] = nextCheckAfterGitHubResponse(snapshot, now, retryAt) + return updates +} + +func nextCheckAfterGitHubResponse( + snapshot *sourceExecutionSnapshot, + now time.Time, + retryAt *time.Time, +) any { + if snapshot.ReleaseSelector != githubReleaseSelectorLatest { + return nil + } + next := nextGitHubCheckAt(now, snapshot.SourceID, snapshot.CheckIntervalMinutes) + if retryAt != nil && retryAt.After(next) { + next = retryAt.UTC() + } + return &next +} + +func lockOwnedSourceRuntime( + tx *gorm.DB, + snapshot *sourceExecutionSnapshot, +) (*model.PagesProjectSourceRuntime, time.Time, error) { + var runtime model.PagesProjectSourceRuntime + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where("source_id = ?", snapshot.SourceID).First(&runtime).Error; err != nil { + return nil, time.Time{}, err + } + now := time.Now() + if runtime.LeaseToken != snapshot.LeaseToken || runtime.LeaseExpiresAt == nil || + !runtime.LeaseExpiresAt.After(now) { + return nil, time.Time{}, errSourceFinalFence + } + return &runtime, now, nil +} + +func failGitHubCheckLease( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + message string, + retryAt time.Time, +) error { + now := time.Now() + next := now.Add(initialCheckRetryDelay) + if retryAt.After(next) { + next = retryAt.UTC() + } + updates := map[string]any{ + sourceRuntimeColumnSyncStatus: pagesSourceStatusFailed, + sourceRuntimeColumnLastError: safeSourceRuntimeError(message), + sourceRuntimeColumnLastCheckedAt: &now, + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + } + if snapshot.ReleaseSelector == githubReleaseSelectorLatest { + updates["next_check_at"] = &next + } else { + updates["next_check_at"] = nil + } + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", snapshot.SourceID, snapshot.LeaseToken, now). + Updates(updates) + if result.Error != nil { + return result.Error + } + if result.RowsAffected != 1 { + return errSourceFinalFence + } + return nil +} + +func targetRuntimeStatus( + target *githubSourceTarget, + appliedRevision string, + appliedDetail string, +) string { + if target == nil || target.Revision == appliedRevision { + return pagesSourceStatusIdle + } + applied := sourceDetail{} + if unmarshalSourceDetail(appliedDetail, &applied) == nil && target.Detail.ReleaseID != "" && + target.Detail.ReleaseID == applied.ReleaseID { + return pagesSourceStatusAttention + } + return pagesSourceStatusUpdateAvailable +} + +func preflightGitHubSyncConfirmation(ctx context.Context, sourceID uint, confirmedRevision string) error { + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", sourceID).First(&runtime).Error; err != nil { + return err + } + replacement := sourceHasSameReleaseReplacement(&runtime) + if replacement && confirmedRevision == "" { + return errors.New(errPagesSourceConfirmationNeeded) + } + if confirmedRevision != "" && (!replacement || confirmedRevision != runtime.LastSeenRevision) { + return errors.New(errPagesSourceConfirmationStale) + } + return nil +} + +func syncGitHubSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + actor string, + targetRevision string, + confirmedRevision string, +) (outcome *sourceSyncOutcome, resultErr error) { + if snapshot == nil || snapshot.SourceType != PagesSourceTypeGitHubRelease || !validPagesSourceActor(actor) { + return nil, errors.New(errPagesSourceActionInvalid) + } + defer func() { + resultErr = finalizeGitHubSyncFailure(ctx, snapshot, resultErr) + }() + workCtx, heartbeat, err := startSourceLeaseHeartbeat( + ctx, snapshot, pagesSourceSyncLeaseDuration, pagesSourceHeartbeatInterval, + ) + if err != nil { + return sourceHeartbeatOutcome(err) + } + defer func() { _ = heartbeat.stop() }() + + client := newGitHubReleaseClient() + target, guardedOutcome, err := resolveAndGuardGitHubSync( + workCtx, client, snapshot, targetRevision, confirmedRevision, + ) + if err != nil { + return nil, err + } + if guardedOutcome != nil { + return guardedOutcome, nil + } + prepared, err := prepareGitHubSyncPackage(workCtx, client, snapshot, target) + if err != nil { + return nil, err + } + defer func() { + if cleanupErr := prepared.download.Cleanup(); cleanupErr != nil { + logger.WarnF(ctx, "[PagesSource] cleanup GitHub package failed: source_id=%d error=%v", snapshot.SourceID, cleanupErr) + } + }() + defer compensateSourceIngest(ctx, snapshot, prepared.ingestState) + if heartbeatErr := heartbeat.stop(); heartbeatErr != nil { + return sourceHeartbeatOutcome(heartbeatErr) + } + renewed, err := renewSourceLease(ctx, snapshot, pagesSourceSyncLeaseDuration) + if err != nil { + return nil, err + } + if !renewed { + return &sourceSyncOutcome{Stale: true}, nil + } + return activatePreparedGitHubSource(ctx, snapshot, actor, prepared) +} + +func finalizeGitHubSyncFailure( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + resultErr error, +) error { + if resultErr == nil { + return nil + } + cleanupCtx, cancel := sourceCleanupContext(ctx) + defer cancel() + finalizerErr := persistGitHubSyncFailure(cleanupCtx, snapshot, resultErr) + if finalizerErr == nil { + return resultErr + } + logger.WarnF( + cleanupCtx, + "[PagesSource] finalize GitHub sync failure failed: source_id=%d source_error=%s error=%v", + snapshot.SourceID, safeGitHubSourceError(resultErr), finalizerErr, + ) + // final fence 丢失表示已有新任务接管 runtime，不应覆盖；数据库 + // finalizer 失败则保持可重试，避免继承永久错误或 provider deadline 分类。 + if errors.Is(finalizerErr, errSourceFinalFence) { + return resultErr + } + return errors.New(errPagesSourceSyncFailed) +} + +func persistGitHubSyncFailure( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + resultErr error, +) error { + var domainError *githubSourceProviderDomainError + if errors.As(resultErr, &domainError) && domainError.retryAt != nil { + return failGitHubCheckLease(ctx, snapshot, domainError.message, *domainError.retryAt) + } + return failSourceLease(ctx, snapshot, safeGitHubSourceError(resultErr)) +} + +func activatePreparedGitHubSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + actor string, + prepared *preparedGitHubSource, +) (*sourceSyncOutcome, error) { + task.AppendLog(ctx, "[activate] 正在原子切换 GitHub Release 部署") + deployment, reused, referenced, err := commitSourceDeployment( + ctx, snapshot, prepared.target.Revision, prepared.download.SHA256, + prepared.target.Detail, prepared.target.DetailJSON, actor, prepared.manifest, + prepared.ingestState.Result, prepared.ingestState.HasIngest, prepared.target.RetryAt, + ) + prepared.ingestState.Referenced = referenced + if errors.Is(err, errSourceFinalFence) { + return &sourceSyncOutcome{Stale: true}, nil + } + if err != nil { + return nil, err + } + prepared.ingestState.Referenced = prepared.ingestState.HasIngest && deployment.UploadID == prepared.ingestState.Result.Upload.ID + if pruneErr := pruneProjectDeploymentHistory(ctx, snapshot.ProjectID, prepared.limits.HistoryCount, 0); pruneErr != nil { + logger.ErrorF(ctx, "[PagesSource] strict prune failed after GitHub sync: project_id=%d source_id=%d error=%v", snapshot.ProjectID, snapshot.SourceID, pruneErr) + } + view := buildDeploymentView(deployment) + return &sourceSyncOutcome{Deployment: &view, Reused: reused}, nil +} + +func resolveAndGuardGitHubSync( + ctx context.Context, + client githubReleaseAPI, + snapshot *sourceExecutionSnapshot, + targetRevision string, + confirmedRevision string, +) (*githubSourceTarget, *sourceSyncOutcome, error) { + task.AppendLog(ctx, "[resolve] 正在解析 GitHub Release：repo=%s asset=%s", snapshot.GitHubRepository, snapshot.AssetName) + resolved, err := client.Resolve(ctx, githubrelease.ResolveRequest{ + Repository: snapshot.GitHubRepository, + Selector: githubrelease.Selector(snapshot.ReleaseSelector), + Tag: snapshot.ReleaseTag, + AssetName: snapshot.AssetName, + }) + if err != nil { + logger.WarnF(ctx, "[PagesSource] GitHub resolve failed: source_id=%d repo=%s error=%v", snapshot.SourceID, snapshot.GitHubRepository, err) + return nil, nil, githubSourceDomainError(err) + } + if resolved.NotModified { + return nil, nil, errors.New(errPagesSourceReleaseNotFound) + } + target, err := buildGitHubSourceTarget(resolved.Release, resolved.Asset, resolved.RetryAt) + if err != nil { + return nil, nil, &githubSourceProviderDomainError{ + message: safeGitHubSourceError(err), + permanent: isPermanentSourceSyncError(err), + retryAt: resolved.RetryAt, + } + } + guardedOutcome, err := guardGitHubSyncTarget(ctx, snapshot, target, targetRevision, confirmedRevision) + return target, guardedOutcome, err +} + +func guardGitHubSyncTarget( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + target *githubSourceTarget, + targetRevision string, + confirmedRevision string, +) (*sourceSyncOutcome, error) { + status := targetRuntimeStatus(target, snapshot.LastAppliedRevision, snapshot.LastAppliedDetail) + if targetRevision != "" && targetRevision != target.Revision { + return releaseGuardedGitHubTarget(ctx, snapshot, target, status, "", true, true) + } + if confirmedRevision != "" && (confirmedRevision != snapshot.LastSeenRevision || confirmedRevision != target.Revision) { + return releaseGuardedGitHubTarget(ctx, snapshot, target, status, errPagesSourceConfirmationStale, false, false) + } + if status == pagesSourceStatusAttention && confirmedRevision != target.Revision { + return releaseGuardedGitHubTarget(ctx, snapshot, target, status, errPagesSourceConfirmationNeeded, false, false) + } + if confirmedRevision != "" && status != pagesSourceStatusAttention { + return nil, errors.New(errPagesSourceConfirmationStale) + } + return nil, nil +} + +func releaseGuardedGitHubTarget( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + target *githubSourceTarget, + status string, + lastError string, + expedite bool, + staleSuccess bool, +) (*sourceSyncOutcome, error) { + err := releaseGitHubSyncWithoutActivation(ctx, snapshot, target, status, lastError, expedite, target.RetryAt) + if errors.Is(err, errSourceFinalFence) || (err == nil && staleSuccess) { + return &sourceSyncOutcome{Stale: true}, nil + } + if err != nil { + return nil, err + } + return nil, errors.New(lastError) +} + +func prepareGitHubSyncPackage( + ctx context.Context, + client githubReleaseAPI, + snapshot *sourceExecutionSnapshot, + target *githubSourceTarget, +) (*preparedGitHubSource, error) { + limits := resolvePagesLimits(ctx) + task.AppendLog(ctx, "[download] 正在下载 GitHub Release asset：repo=%s asset=%s", snapshot.GitHubRepository, snapshot.AssetName) + download, err := client.Download(ctx, githubrelease.DownloadRequest{ + Repository: snapshot.GitHubRepository, + Asset: target.Asset, + MaxBytes: limits.PackageBytes, + }) + if err != nil { + logger.WarnF(ctx, "[PagesSource] GitHub download failed: source_id=%d repo=%s asset=%s error=%v", snapshot.SourceID, snapshot.GitHubRepository, snapshot.AssetName, err) + return nil, githubSourceDomainError(err) + } + prepared, err := inspectAndIngestGitHubPackage(ctx, snapshot, target, download, limits) + if err != nil { + if cleanupErr := download.Cleanup(); cleanupErr != nil { + logger.WarnF(ctx, "[PagesSource] cleanup GitHub package after preparation failure failed: source_id=%d error=%v", snapshot.SourceID, cleanupErr) + } + return nil, err + } + return prepared, nil +} + +func inspectAndIngestGitHubPackage( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + target *githubSourceTarget, + download *githubrelease.DownloadResult, + limits pagesLimits, +) (*preparedGitHubSource, error) { + if download.SHA256 == "" || download.Path == "" { + return nil, errors.New(errPagesSourceSyncFailed) + } + if target.Detail.Digest != "" && "sha256:"+download.SHA256 != target.Detail.Digest { + return nil, errors.New(errPagesSourceDigestMismatch) + } + format, ok := pagesarchive.DetectFormatFromName(target.Asset.Name) + var err error + if !ok { + format, _, err = detectRemoteSourceFormat(download.Path, target.Asset.Name, "") + if err != nil { + return nil, err + } + } + rootDir, err := validateAndNormalizePagesRootDir(snapshot.RootDir) + if err != nil { + return nil, err + } + entryFile, err := validateAndNormalizePagesEntryFile(snapshot.EntryFile) + if err != nil { + return nil, err + } + task.AppendLog(ctx, "[verify] 正在校验 GitHub Release 归档与入口") + manifest, err := inspectPagesPackage(download.Path, format, rootDir, entryFile, limits) + if err != nil { + return nil, err + } + ingestState, err := resolveGitHubSourceIngest(ctx, snapshot, target, download, format) + if err != nil { + return nil, err + } + return &preparedGitHubSource{ + target: target, download: download, format: format, + manifest: manifest, ingestState: ingestState, limits: limits, + }, nil +} + +func resolveGitHubSourceIngest( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + target *githubSourceTarget, + download *githubrelease.DownloadResult, + format pagesarchive.Format, +) (*sourceIngestState, error) { + if _, err := findSourceDeployment(ctx, snapshot.ProjectID, snapshot.SourceIdentity, target.Revision); err == nil { + return &sourceIngestState{}, nil + } else if !errors.Is(err, gorm.ErrRecordNotFound) { + return nil, err + } + task.AppendLog(ctx, "[ingest] 正在保存 GitHub Release 部署包") + result, err := ingestPagesDeploymentPackageWithSource( + ctx, download.Path, download.SHA256, snapshot.ProjectID, snapshot.SourceID, target.Asset.Name, format, + ) + if err != nil { + return nil, err + } + return &sourceIngestState{Result: result, HasIngest: true}, nil +} + +func releaseGitHubSyncWithoutActivation( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + target *githubSourceTarget, + status string, + lastError string, + expedite bool, + retryAt *time.Time, +) error { + now := time.Now() + nextCheckAt := nextCheckAfterGitHubResponse(snapshot, now, retryAt) + if expedite && snapshot.ReleaseSelector == githubReleaseSelectorLatest { + next := now.Add(initialCheckRetryDelay) + if retryAt != nil && retryAt.After(next) { + next = retryAt.UTC() + } + nextCheckAt = &next + } + updates := map[string]any{ + "last_seen_revision": target.Revision, + "last_seen_detail": target.DetailJSON, + sourceRuntimeColumnSyncStatus: status, + sourceRuntimeColumnLastError: lastError, + sourceRuntimeColumnLastCheckedAt: &now, + "next_check_at": nextCheckAt, + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + } + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", snapshot.SourceID, snapshot.LeaseToken, now). + Updates(updates) + if result.Error != nil { + return result.Error + } + if result.RowsAffected != 1 { + return errSourceFinalFence + } + return nil +} + +func safeGitHubSourceError(err error) string { + if err == nil { + return errPagesSourceSyncFailed + } + message := strings.TrimSpace(err.Error()) + for _, safeMessage := range []string{ + errPagesSourceSyncFailed, + errPagesSourceReleaseNotFound, + errPagesSourceDigestInvalid, + errPagesSourceDigestMismatch, + errPagesSourceConfirmationNeeded, + errPagesSourceConfirmationStale, + errPagesPackageURLTooLarge, + errPagesPackageEmpty, + errPagesPackageUnsupported, + errPagesPackageInvalid, + errPagesPackageExtractedTooLarge, + errPagesPackageFileTooLarge, + errPagesEntryFileMissing, + } { + if message == safeMessage { + return safeMessage + } + } + return errPagesSourceSyncFailed +} + +func githubSourceDomainError(err error) error { + message := errPagesSourceSyncFailed + retryAt, hasRetryAt := githubrelease.RetryAt(err) + var retryDeadline *time.Time + if hasRetryAt { + retryDeadline = &retryAt + } + if err == nil { + return &githubSourceProviderDomainError{message: message, permanent: false} + } + if githubrelease.IsDigestError(err) { + message = errPagesSourceDigestMismatch + return &githubSourceProviderDomainError{message: message, permanent: true} + } + if githubrelease.IsNotFound(err) { + message = errPagesSourceReleaseNotFound + return &githubSourceProviderDomainError{message: message, permanent: true} + } + if errors.Is(err, githubrelease.ErrAssetTooLarge) { + message = errPagesPackageURLTooLarge + return &githubSourceProviderDomainError{message: message, permanent: true} + } + if errors.Is(err, githubrelease.ErrEmptyAsset) { + message = errPagesPackageEmpty + return &githubSourceProviderDomainError{message: message, permanent: true} + } + return &githubSourceProviderDomainError{ + message: message, permanent: !githubrelease.IsRetryable(err), retryAt: retryDeadline, + } +} + +func shouldSkipGitHubActionRetry(err error) bool { + var domainError *githubSourceProviderDomainError + return errors.As(err, &domainError) && (domainError.permanent || domainError.retryAt != nil) +} diff --git a/internal/apps/openflare/pages/github_source_identity_test.go b/internal/apps/openflare/pages/github_source_identity_test.go new file mode 100644 index 00000000..c577dc41 --- /dev/null +++ b/internal/apps/openflare/pages/github_source_identity_test.go @@ -0,0 +1,111 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" +) + +func TestGitHubSourceIdentityLengthPrefixesFieldsAndResetsRuntime(t *testing.T) { + firstInput := SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/OpenFlare/site", + ReleaseSelector: githubReleaseSelectorTag, + ReleaseTag: "release|foo", + AssetName: "bar.zip", + } + secondInput := SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/OpenFlare/site", + ReleaseSelector: githubReleaseSelectorTag, + ReleaseTag: "release", + AssetName: "foo|bar.zip", + } + firstConfig, err := buildGitHubSourceConfig(firstInput) + if err != nil { + t.Fatalf("buildGitHubSourceConfig(first) error = %v, want nil", err) + } + secondConfig, err := buildGitHubSourceConfig(secondInput) + if err != nil { + t.Fatalf("buildGitHubSourceConfig(second) error = %v, want nil", err) + } + legacyIdentityInput := func(config githubSourceConfig) string { + return "github|" + config.Repository + "|" + config.Selector + "|" + + config.Tag + "|" + config.AssetName + } + if firstLegacy, secondLegacy := legacyIdentityInput(firstConfig), legacyIdentityInput(secondConfig); firstLegacy != secondLegacy { + t.Fatalf("legacy identity inputs differ: %q != %q; collision fixture is invalid", firstLegacy, secondLegacy) + } + if firstConfig.SourceIdentity == secondConfig.SourceIdentity { + t.Fatalf("length-prefixed identities collide: %q", firstConfig.SourceIdentity) + } + + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-identity-collision") + firstSource, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, firstInput) + if got, want := firstSource.SourceIdentity, firstConfig.SourceIdentity; got != want { + t.Fatalf("first source identity = %q, want %q", got, want) + } + + checkedAt := time.Now().Add(-time.Minute) + syncedAt := time.Now().Add(-30 * time.Second) + nextCheckAt := time.Now().Add(time.Hour) + leaseExpiresAt := time.Now().Add(time.Minute) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", firstSource.ID). + Updates(map[string]any{ + "etag": `"old-etag"`, + "last_seen_revision": strings.Repeat("a", 64), + "last_seen_detail": `{"provider":"github_release","tag":"release|foo"}`, + "last_applied_revision": strings.Repeat("b", 64), + "last_applied_detail": `{"provider":"github_release","tag":"older"}`, + "sync_status": pagesSourceStatusSyncing, + "last_error": "old error", + "last_checked_at": &checkedAt, + "last_synced_at": &syncedAt, + "next_check_at": &nextCheckAt, + "lease_expires_at": &leaseExpiresAt, + "lease_token": "old-lease", + }).Error; err != nil { + t.Fatalf("seed runtime cursors error = %v, want nil", err) + } + + secondSource, runtime := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, secondInput) + if secondSource.ID != firstSource.ID { + t.Errorf("updated source ID = %d, want unchanged %d", secondSource.ID, firstSource.ID) + } + if got, want := secondSource.SourceIdentity, secondConfig.SourceIdentity; got != want { + t.Errorf("updated source identity = %q, want %q", got, want) + } + if got, want := secondSource.ConfigVersion, firstSource.ConfigVersion+1; got != want { + t.Errorf("updated source config version = %d, want %d", got, want) + } + if runtime.ETag != "" || runtime.LastSeenRevision != "" || runtime.LastSeenDetail != "" || + runtime.LastAppliedRevision != "" || runtime.LastAppliedDetail != "" { + t.Errorf("identity change retained runtime cursors: %+v", runtime) + } + if runtime.LastCheckedAt != nil || runtime.LastSyncedAt != nil || runtime.NextCheckAt != nil { + t.Errorf( + "identity change retained runtime timestamps: checked=%v synced=%v next=%v", + runtime.LastCheckedAt, + runtime.LastSyncedAt, + runtime.NextCheckAt, + ) + } + if runtime.SyncStatus != pagesSourceStatusIdle || runtime.LastError != "" || + runtime.LeaseToken != "" || runtime.LeaseExpiresAt != nil { + t.Errorf( + "identity change retained runtime state: status=%q error=%q lease=(%q, %v)", + runtime.SyncStatus, + runtime.LastError, + runtime.LeaseToken, + runtime.LeaseExpiresAt, + ) + } +} diff --git a/internal/apps/openflare/pages/github_source_test.go b/internal/apps/openflare/pages/github_source_test.go new file mode 100644 index 00000000..6d823319 --- /dev/null +++ b/internal/apps/openflare/pages/github_source_test.go @@ -0,0 +1,888 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/integration/githubrelease" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/hibiken/asynq" + "gorm.io/gorm" +) + +type fakeGitHubReleaseClient struct { + resolve func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) + download func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) +} + +func (client *fakeGitHubReleaseClient) Resolve( + ctx context.Context, + request githubrelease.ResolveRequest, +) (githubrelease.ResolveResult, error) { + return client.resolve(ctx, request) +} + +func (client *fakeGitHubReleaseClient) Download( + ctx context.Context, + request githubrelease.DownloadRequest, +) (*githubrelease.DownloadResult, error) { + return client.download(ctx, request) +} + +func useFakeGitHubReleaseClient(t *testing.T, client githubReleaseAPI) { + t.Helper() + previous := newGitHubReleaseClient + newGitHubReleaseClient = func() githubReleaseAPI { return client } + t.Cleanup(func() { newGitHubReleaseClient = previous }) +} + +func mustConfigureGitHubSourceWithoutDispatch( + t *testing.T, + ctx context.Context, + projectID uint, + input SourceUpdateInput, +) (*model.PagesProjectSource, *model.PagesProjectSourceRuntime) { + t.Helper() + if err := validateGitHubSourceInput(input); err != nil { + t.Fatalf("validateGitHubSourceInput(%+v) error = %v, want nil", input, err) + } + if err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + _, err := updateGitHubSourceTx(tx, projectID, input) + return err + }); err != nil { + t.Fatalf("updateGitHubSourceTx(project=%d) error = %v, want nil", projectID, err) + } + return mustLoadPagesSource(t, ctx, projectID) +} + +func mustLoadPagesSource( + t *testing.T, + ctx context.Context, + projectID uint, +) (*model.PagesProjectSource, *model.PagesProjectSourceRuntime) { + t.Helper() + var source model.PagesProjectSource + if err := db.DB(ctx).Where("project_id = ?", projectID).First(&source).Error; err != nil { + t.Fatalf("load source for project %d error = %v, want nil", projectID, err) + } + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx).Where("source_id = ?", source.ID).First(&runtime).Error; err != nil { + t.Fatalf("load runtime for source %d error = %v, want nil", source.ID, err) + } + return &source, &runtime +} + +func TestGitHubSourceValidationNormalizationAndProviderSwitch(t *testing.T) { + ctx := setupPagesSourceTest(t) + setupPagesSourceDispatchTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-config") + input := SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/OpenFlare/site.git", + } + result, err := UpdateSourceAs(ctx, project.ID, input, "user:42") + if err != nil { + t.Fatalf("UpdateSourceAs(GitHub) error = %v, want nil", err) + } + if result.CheckTask == nil || result.CheckTask.Action != sourceActionCheck || result.Warning != "" { + t.Errorf("UpdateSourceAs(GitHub) result = %+v, want initial check receipt without warning", result) + } + execution, err := model.GetTaskExecutionByTaskID(ctx, result.CheckTask.TaskID) + if err != nil { + t.Fatalf("GetTaskExecutionByTaskID(%q) error = %v, want nil", result.CheckTask.TaskID, err) + } + var actionPayload SourceActionPayload + if err := json.Unmarshal([]byte(execution.Payload), &actionPayload); err != nil { + t.Fatalf("json.Unmarshal(initial check payload) error = %v, want nil", err) + } + if actionPayload.Actor != "user:42" || actionPayload.Action != sourceActionCheck || + actionPayload.TargetRevision != "" || actionPayload.ConfirmedRevision != "" { + t.Errorf("initial check payload = %+v, want real actor and credential-free check", actionPayload) + } + source, runtime := mustLoadPagesSource(t, ctx, project.ID) + if got, want := source.GitHubRepository, "OpenFlare/site"; got != want { + t.Errorf("GitHubRepository = %q, want %q", got, want) + } + if got, want := source.ReleaseSelector, githubReleaseSelectorLatest; got != want { + t.Errorf("ReleaseSelector = %q, want %q", got, want) + } + if got, want := source.AssetName, defaultGitHubAssetName; got != want { + t.Errorf("AssetName = %q, want %q", got, want) + } + if got, want := source.CheckIntervalMinutes, defaultCheckInterval; got != want { + t.Errorf("CheckIntervalMinutes = %d, want %d", got, want) + } + if got, want := source.SourceIdentity, "dbbd25307aaa3b88bc25353476940a049428655bd8421ac63045fdcb5fb23c9d"; got != want { + t.Errorf("SourceIdentity = %q, want %q", got, want) + } + if runtime.NextCheckAt == nil { + t.Error("GitHub latest NextCheckAt = nil, want scheduled value") + } + + var taskCount int64 + if err := db.DB(ctx).Model(&model.TaskExecution{}).Count(&taskCount).Error; err != nil { + t.Fatalf("count initial checks error = %v, want nil", err) + } + if _, err := UpdateSourceAs(ctx, project.ID, input, "user:42"); err != nil { + t.Fatalf("UpdateSourceAs(GitHub no-op) error = %v, want nil", err) + } + var noOpTaskCount int64 + if err := db.DB(ctx).Model(&model.TaskExecution{}).Count(&noOpTaskCount).Error; err != nil { + t.Fatalf("count no-op checks error = %v, want nil", err) + } + if noOpTaskCount != taskCount { + t.Errorf("no-op initial check count = %d, want unchanged %d", noOpTaskCount, taskCount) + } + + secret := "provider-switch-secret" + if _, err := UpdateSource(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeRemoteURL, + RemoteURLSet: true, + RemoteURL: "https://artifacts.example.com/site.zip?token=" + secret, + RemoteNetworkPolicy: RemoteNetworkPolicyPublic, + }); err != nil { + t.Fatalf("UpdateSource(GitHub to Remote) error = %v, want nil", err) + } + remote, _ := mustLoadPagesSource(t, ctx, project.ID) + if remote.GitHubRepository != "" || remote.ReleaseSelector != "" || remote.AssetName != "" || + remote.AutoUpdateEnabled || remote.CheckIntervalMinutes != 0 { + t.Errorf("Remote switched source retained GitHub fields: %+v", remote) + } + if _, err := UpdateSourceAs(ctx, project.ID, input, "user:42"); err != nil { + t.Fatalf("UpdateSourceAs(Remote to GitHub) error = %v, want nil", err) + } + github, _ := mustLoadPagesSource(t, ctx, project.ID) + if github.RemoteURL != "" || github.RemoteNetworkPolicy != "" { + t.Errorf("GitHub switched source retained Remote fields: URL=%q policy=%q", github.RemoteURL, github.RemoteNetworkPolicy) + } +} + +func TestGitHubSourceSaveSurvivesInitialCheckDispatchFailure(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-dispatch-warning") + result, err := UpdateSourceAs(ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }, "user:9") + if err != nil { + t.Fatalf("UpdateSourceAs(dispatch failure) error = %v, want saved source with warning", err) + } + if result.CheckTask != nil || result.Warning != errPagesSourceInitialCheckWarning { + t.Errorf("UpdateSourceAs(dispatch failure) result = %+v, want warning and nil check task", result) + } + source, runtime := mustLoadPagesSource(t, ctx, project.ID) + if source.GitHubRepository != "a/b" || runtime.SyncStatus != pagesSourceStatusFailed || + runtime.LastError != errPagesSourceInitialCheckWarning { + t.Errorf("saved source/runtime = repo:%q status:%q error:%q", source.GitHubRepository, runtime.SyncStatus, runtime.LastError) + } +} + +func TestGitHubSourceRejectsUnsafeOrPhaseThreeFields(t *testing.T) { + tests := []SourceUpdateInput{ + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "http://github.com/a/b"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a%20b/repo"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b/extra"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com//a/b"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b/"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b?"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b#"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AssetName: "dist\n.zip"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AssetName: "dist\u202e.zip"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AssetName: "dir/dist.zip"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AutoUpdateEnabled: true}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1", CheckIntervalMinutes: 60}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: " v1"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1\n"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1\u2028draft"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: `v1\draft`}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "release//v1"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "release/.draft"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "release/v1.lock"}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "latest", ReleaseTag: "v1"}, + } + for _, input := range tests { + if err := validateGitHubSourceInput(input); err == nil { + t.Errorf("validateGitHubSourceInput(%+v) error = nil, want non-nil", input) + } + } +} + +func TestGitHubSourceAcceptsLegalAssetAndTagCharacters(t *testing.T) { + tests := []SourceUpdateInput{ + { + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + AssetName: "dist?channel=stable&part#1.zip", + }, + { + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + AssetName: " dist.zip ", + }, + { + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + ReleaseSelector: "tag", + ReleaseTag: "release/v1#stable&build=1", + AssetName: "dist.zip", + }, + { + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b.git", + ReleaseSelector: "tag", + ReleaseTag: "@", + AssetName: "dist.zip", + }, + { + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + ReleaseSelector: "tag", + ReleaseTag: "release/v1.LOCK", + AssetName: "dist.zip", + }, + } + for _, input := range tests { + if err := validateGitHubSourceInput(input); err != nil { + t.Errorf("validateGitHubSourceInput(%+v) error = %v, want nil", input, err) + } + } +} + +func TestInitialCheckFailureUsesExactConfigFence(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-initial-fence") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + staleVersion := source.ConfigVersion + if err := db.DB(ctx).Model(source).Update("config_version", staleVersion+1).Error; err != nil { + t.Fatalf("increment source config version error = %v, want nil", err) + } + markInitialCheckDispatchFailed(ctx, source.ID, staleVersion) + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusIdle || runtime.LastError != "" { + t.Errorf("stale initial failure runtime = status:%q error:%q, want unchanged idle", runtime.SyncStatus, runtime.LastError) + } +} + +func TestGitHubCheckUsesETagAndDetectsSameReleaseReplacement(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-check") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + appliedRevision := strings.Repeat("a", 64) + appliedDetail := `{"provider":"github","release_id":"100","asset_id":"1","tag":"release/v1","asset_name":"dist.zip"}` + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}).Where("source_id = ?", source.ID).Updates(map[string]any{ + "etag": `"old-etag"`, + "last_applied_revision": appliedRevision, + "last_applied_detail": appliedDetail, + }).Error; err != nil { + t.Fatalf("seed GitHub runtime error = %v, want nil", err) + } + updatedAt := time.Date(2026, 7, 19, 10, 0, 0, 0, time.UTC) + var gotETag string + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(_ context.Context, request githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + gotETag = request.ETag + return githubrelease.ResolveResult{ + ETag: `"new-etag"`, + Release: githubrelease.Release{ID: "100", Tag: "release/v1"}, + Asset: githubrelease.Asset{ID: "2", Name: "dist.zip", State: "uploaded", UpdatedAt: updatedAt}, + }, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called during check, want resolve only") + return nil, nil + }, + }) + snapshot, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionCheck) + if err != nil || outcome != sourceLeaseAcquired { + t.Fatalf("acquire check lease = (%+v, %q, %v), want acquired", snapshot, outcome, err) + } + result, err := checkGitHubSource(ctx, snapshot) + if err != nil { + t.Fatalf("checkGitHubSource() error = %v, want nil", err) + } + if result.Stale { + t.Error("checkGitHubSource() stale = true, want false") + } + if got, want := gotETag, `"old-etag"`; got != want { + t.Errorf("Resolve ETag = %q, want %q", got, want) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusAttention || runtime.LastSeenRevision == "" { + t.Errorf("replacement runtime = status:%q seen:%q, want attention with revision", runtime.SyncStatus, runtime.LastSeenRevision) + } + view, err := GetSource(ctx, project.ID) + if err != nil { + t.Fatalf("GetSource() error = %v, want nil", err) + } + if view.LastSeen == nil || view.LastSeen.Label != "release/v1" { + t.Errorf("LastSeen = %+v, want full tag with slash", view.LastSeen) + } + if err := preflightGitHubSyncConfirmation(ctx, source.ID, ""); err == nil || err.Error() != errPagesSourceConfirmationNeeded { + t.Errorf("preflight without confirmation error = %v, want %q", err, errPagesSourceConfirmationNeeded) + } + if err := preflightGitHubSyncConfirmation(ctx, source.ID, runtime.LastSeenRevision); err != nil { + t.Errorf("preflight exact confirmation error = %v, want nil", err) + } +} + +func TestGitHubCheckNotModifiedRefreshesRuntimeWithoutDeployment(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-304") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(_ context.Context, request githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{NotModified: true, ETag: `"same"`}, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called for 304 check") + return nil, nil + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionCheck) + if _, err := checkGitHubSource(ctx, snapshot); err != nil { + t.Fatalf("checkGitHubSource(304) error = %v, want nil", err) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.ETag != `"same"` || runtime.LastCheckedAt == nil || runtime.NextCheckAt == nil || runtime.LeaseToken != "" { + t.Errorf("304 runtime = %+v, want refreshed timestamps/etag and released lease", runtime) + } + var deployments int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}).Where("project_id = ?", project.ID).Count(&deployments).Error; err != nil { + t.Fatalf("count deployments error = %v, want nil", err) + } + if deployments != 0 { + t.Errorf("deployments after check = %d, want 0", deployments) + } +} + +func TestGitHubTargetMismatchPreservesAttentionAndExpeditesRecheck(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-target-mismatch") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + appliedRevision := strings.Repeat("a", 64) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}).Where("source_id = ?", source.ID).Updates(map[string]any{ + "last_applied_revision": appliedRevision, + "last_applied_detail": `{"provider":"github","release_id":"100","asset_id":"1","tag":"v1","asset_name":"dist.zip"}`, + }).Error; err != nil { + t.Fatalf("seed applied runtime error = %v, want nil", err) + } + retryAt := time.Now().Add(2 * time.Hour).UTC() + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{ + Release: githubrelease.Release{ID: "100", Tag: "v1"}, + Asset: githubrelease.Asset{ + ID: "2", Name: "dist.zip", State: "uploaded", + UpdatedAt: time.Date(2026, 7, 19, 12, 0, 0, 0, time.UTC), + }, + RetryAt: &retryAt, + }, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called after target mismatch") + return nil, nil + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + outcome, err := syncGitHubSource(ctx, snapshot, pagesSourceCreatedBySystem, strings.Repeat("b", 64), "") + if err != nil { + t.Fatalf("syncGitHubSource(target mismatch) error = %v, want nil stale outcome", err) + } + if outcome == nil || !outcome.Stale { + t.Errorf("syncGitHubSource(target mismatch) = %+v, want stale", outcome) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusAttention { + t.Errorf("target mismatch SyncStatus = %q, want %q", runtime.SyncStatus, pagesSourceStatusAttention) + } + if runtime.NextCheckAt == nil || runtime.NextCheckAt.Before(retryAt) { + t.Errorf("target mismatch NextCheckAt = %v, want server deadline >= %v", runtime.NextCheckAt, retryAt) + } + var deployments int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}).Where("project_id = ?", project.ID).Count(&deployments).Error; err != nil { + t.Fatalf("count mismatch deployments error = %v, want nil", err) + } + if deployments != 0 { + t.Errorf("target mismatch deployments = %d, want 0", deployments) + } +} + +func TestGitHubCheckLostLeaseReturnsStaleWithoutOverwritingRuntime(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-check-fence") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{}, errors.New("transient provider failure") + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + return nil, errors.New("unexpected") + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionCheck) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}).Where("source_id = ?", source.ID).Updates(map[string]any{ + "lease_token": "new-owner", + "lease_expires_at": time.Now().Add(time.Minute), + "sync_status": pagesSourceStatusSyncing, + "last_error": "new-owner-state", + }).Error; err != nil { + t.Fatalf("replace lease owner error = %v, want nil", err) + } + result, err := checkGitHubSource(ctx, snapshot) + if err != nil { + t.Fatalf("checkGitHubSource(lost lease) error = %v, want stale no-op", err) + } + if result == nil || !result.Stale { + t.Errorf("checkGitHubSource(lost lease) = %+v, want stale", result) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.LeaseToken != "new-owner" || runtime.LastError != "new-owner-state" || runtime.SyncStatus != pagesSourceStatusSyncing { + t.Errorf("lost lease runtime = token:%q error:%q status:%q, want new owner state", runtime.LeaseToken, runtime.LastError, runtime.SyncStatus) + } +} + +func TestGitHubCheckRateLimitUsesServerDeadlineAndSuppressesFastRetry(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-rate-limit") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + retryAt := time.Now().Add(2 * time.Hour).UTC() + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{}, &githubrelease.Error{ + Kind: githubrelease.ErrMetadata, StatusCode: 429, RetryAt: &retryAt, + } + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + return nil, errors.New("unexpected") + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionCheck) + _, err := checkGitHubSource(ctx, snapshot) + if err == nil || err.Error() != errPagesSourceSyncFailed { + t.Fatalf("checkGitHubSource(rate limit) error = %v, want safe sync failure", err) + } + if !shouldSkipGitHubActionRetry(err) { + t.Error("shouldSkipGitHubActionRetry(rate limit) = false, want true") + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.NextCheckAt == nil || runtime.NextCheckAt.Before(retryAt) || runtime.SyncStatus != pagesSourceStatusFailed { + t.Errorf("rate limit runtime = next:%v status:%q, want deadline >= %v and failed", runtime.NextCheckAt, runtime.SyncStatus, retryAt) + } +} + +func TestGitHubCheckInvalidResolvedTargetUsesServerDeadline(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-invalid-check-target") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + retryAt := time.Now().Add(2 * time.Hour).UTC() + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{ + Release: githubrelease.Release{ID: "1", Tag: "v1"}, + Asset: githubrelease.Asset{ + ID: "2", Name: "dist.zip", State: "uploaded", + UpdatedAt: time.Now().UTC(), Digest: "sha256:invalid", + }, + RetryAt: &retryAt, + }, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called after invalid check target") + return nil, nil + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionCheck) + _, err := checkGitHubSource(ctx, snapshot) + if err == nil || err.Error() != errPagesSourceDigestInvalid { + t.Fatalf("checkGitHubSource(invalid target) error = %v, want %q", err, errPagesSourceDigestInvalid) + } + if !isPermanentSourceSyncError(err) { + t.Error("invalid check target classification = retryable, want permanent") + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusFailed || runtime.LastError != errPagesSourceDigestInvalid || + runtime.NextCheckAt == nil || runtime.NextCheckAt.Before(retryAt) || runtime.LeaseToken != "" { + t.Errorf( + "invalid check target runtime = status:%q error:%q next:%v lease:%q, want failed/%q/deadline >= %v/cleared", + runtime.SyncStatus, runtime.LastError, runtime.NextCheckAt, runtime.LeaseToken, + errPagesSourceDigestInvalid, retryAt, + ) + } +} + +func TestGitHubSyncInvalidResolvedTargetUsesServerDeadline(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-invalid-sync-target") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + retryAt := time.Now().Add(2 * time.Hour).UTC() + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{ + Release: githubrelease.Release{ID: "1", Tag: "v1"}, + Asset: githubrelease.Asset{ + ID: "2", Name: "dist.zip", State: "uploaded", + UpdatedAt: time.Now().UTC(), Digest: "sha256:invalid", + }, + RetryAt: &retryAt, + }, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called after invalid sync target") + return nil, nil + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + _, err := syncGitHubSource(ctx, snapshot, "user:7", "", "") + if err == nil || err.Error() != errPagesSourceDigestInvalid { + t.Fatalf("syncGitHubSource(invalid target) error = %v, want %q", err, errPagesSourceDigestInvalid) + } + if !isPermanentSourceSyncError(err) { + t.Error("invalid sync target classification = retryable, want permanent") + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusFailed || runtime.LastError != errPagesSourceDigestInvalid || + runtime.NextCheckAt == nil || runtime.NextCheckAt.Before(retryAt) || runtime.LeaseToken != "" { + t.Errorf( + "invalid sync target runtime = status:%q error:%q next:%v lease:%q, want failed/%q/deadline >= %v/cleared", + runtime.SyncStatus, runtime.LastError, runtime.NextCheckAt, runtime.LeaseToken, + errPagesSourceDigestInvalid, retryAt, + ) + } +} + +func TestGitHubCheckHandlerSkipsProviderFastRetry(t *testing.T) { + tests := []struct { + name string + status int + retryDate bool + }{ + {name: "bad request", status: http.StatusBadRequest}, + {name: "rate limited forbidden", status: http.StatusForbidden, retryDate: true}, + {name: "too many requests", status: http.StatusTooManyRequests, retryDate: true}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-handler-"+strings.ReplaceAll(test.name, " ", "-")) + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + var retryAt *time.Time + if test.retryDate { + deadline := time.Now().Add(2 * time.Hour).UTC() + retryAt = &deadline + } + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{}, &githubrelease.Error{ + Kind: githubrelease.ErrMetadata, StatusCode: test.status, RetryAt: retryAt, + } + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called after provider check failure") + return nil, nil + }, + }) + raw, err := json.Marshal(SourceActionPayload{ + SourceID: source.ID, ConfigVersion: source.ConfigVersion, + Action: sourceActionCheck, Actor: "user:7", + }) + if err != nil { + t.Fatalf("json.Marshal(check payload) error = %v, want nil", err) + } + result, err := (&SourceActionHandler{}).Execute(ctx, raw) + if result != nil || err == nil || !errors.Is(err, asynq.SkipRetry) { + t.Fatalf("SourceActionHandler.Execute(status %d) = result:%+v error:%v, want SkipRetry", test.status, result, err) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusFailed || runtime.NextCheckAt == nil || runtime.LeaseToken != "" { + t.Errorf("provider failure runtime = status:%q next:%v lease:%q", runtime.SyncStatus, runtime.NextCheckAt, runtime.LeaseToken) + } + if retryAt != nil && (runtime.NextCheckAt == nil || runtime.NextCheckAt.Before(*retryAt)) { + t.Errorf("provider failure NextCheckAt = %v, want deadline >= %v", runtime.NextCheckAt, *retryAt) + } + }) + } +} + +func TestGitHubSyncActivatesWithMetadataRevisionAndPackageChecksum(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-sync") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + AssetName: "dist.zip", + }) + packageBytes := testPagesZip(t, map[string]string{"index.html": "github-v1"}) + packageHash := sha256.Sum256(packageBytes) + updatedAt := time.Date(2026, 7, 19, 11, 0, 0, 0, time.UTC) + retryAt := time.Now().Add(2 * time.Hour).UTC() + release := githubrelease.Release{ID: "200", Tag: "release/v2"} + asset := githubrelease.Asset{ID: "10", Name: "dist.zip", State: "uploaded", UpdatedAt: updatedAt} + client := &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{Release: release, Asset: asset, RetryAt: &retryAt}, nil + }, + download: func(_ context.Context, request githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + path := filepath.Join(t.TempDir(), "download") + if err := os.WriteFile(path, packageBytes, 0o600); err != nil { + t.Fatalf("os.WriteFile(download) error = %v, want nil", err) + } + return &githubrelease.DownloadResult{ + Path: path, Size: int64(len(packageBytes)), SHA256: hex.EncodeToString(packageHash[:]), + }, nil + }, + } + useFakeGitHubReleaseClient(t, client) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + outcome, err := syncGitHubSource(ctx, snapshot, "user:7", "", "") + if err != nil { + t.Fatalf("syncGitHubSource() error = %v, want nil", err) + } + if outcome == nil || outcome.Deployment == nil || outcome.Stale { + t.Fatalf("syncGitHubSource() = %+v, want active deployment", outcome) + } + deployment, err := model.GetPagesDeploymentByID(ctx, outcome.Deployment.ID) + if err != nil { + t.Fatalf("GetPagesDeploymentByID(%d) error = %v, want nil", outcome.Deployment.ID, err) + } + if got, want := deployment.Checksum, hex.EncodeToString(packageHash[:]); got != want { + t.Errorf("deployment Checksum = %q, want package hash %q", got, want) + } + if deployment.SourceRevision == nil || *deployment.SourceRevision == deployment.Checksum { + t.Errorf("deployment SourceRevision = %v, want metadata revision distinct from package checksum", deployment.SourceRevision) + } + if got, want := deployment.SourceLabel, "release/v2"; got != want { + t.Errorf("deployment SourceLabel = %q, want %q", got, want) + } + if deployment.SourceType != PagesSourceTypeGitHubRelease || deployment.TriggerType != pagesSourceTriggerManualSync || + deployment.CreatedBy != "user:7" { + t.Errorf("deployment provenance = type:%q trigger:%q actor:%q", deployment.SourceType, deployment.TriggerType, deployment.CreatedBy) + } + if strings.Contains(deployment.SourceMeta, "http") || strings.Contains(deployment.SourceMeta, "token") { + t.Errorf("deployment SourceMeta = %q, want no URL or token", deployment.SourceMeta) + } + if !strings.Contains(deployment.SourceMeta, `"tag":"release/v2"`) || strings.Contains(deployment.SourceMeta, `"label"`) { + t.Errorf("deployment SourceMeta = %q, want provider-specific tag field", deployment.SourceMeta) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.NextCheckAt == nil || runtime.NextCheckAt.Before(retryAt) { + t.Errorf("sync runtime NextCheckAt = %v, want server deadline >= %v", runtime.NextCheckAt, retryAt) + } + + secondSnapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + second, err := syncGitHubSource(ctx, secondSnapshot, "user:7", "", "") + if err != nil { + t.Fatalf("syncGitHubSource(idempotent) error = %v, want nil", err) + } + if second == nil || !second.Reused || second.Deployment == nil || second.Deployment.ID != outcome.Deployment.ID { + t.Errorf("syncGitHubSource(idempotent) = %+v, want reused deployment %d", second, outcome.Deployment.ID) + } +} + +func TestGitHubSyncActivatesExactConfirmedReplacement(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-confirm-replacement") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + AssetName: "dist.zip", + }) + release := githubrelease.Release{ID: "300", Tag: "v3"} + asset := githubrelease.Asset{ + ID: "12", Name: "dist.zip", State: "uploaded", + UpdatedAt: time.Date(2026, 7, 19, 13, 0, 0, 0, time.UTC), + } + target, err := buildGitHubSourceTarget(release, asset, nil) + if err != nil { + t.Fatalf("buildGitHubSourceTarget() error = %v, want nil", err) + } + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}).Where("source_id = ?", source.ID).Updates(map[string]any{ + "last_seen_revision": target.Revision, + "last_seen_detail": target.DetailJSON, + "last_applied_revision": strings.Repeat("a", 64), + "last_applied_detail": `{"provider":"github","release_id":"300","asset_id":"11","tag":"v3","asset_name":"dist.zip"}`, + "sync_status": pagesSourceStatusAttention, + }).Error; err != nil { + t.Fatalf("seed replacement cursor error = %v, want nil", err) + } + packageBytes := testPagesZip(t, map[string]string{"index.html": "confirmed-v3"}) + packageHash := sha256.Sum256(packageBytes) + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{Release: release, Asset: asset}, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + path := filepath.Join(t.TempDir(), "confirmed.zip") + if err := os.WriteFile(path, packageBytes, 0o600); err != nil { + t.Fatalf("os.WriteFile(confirmed package) error = %v, want nil", err) + } + return &githubrelease.DownloadResult{ + Path: path, Size: int64(len(packageBytes)), SHA256: hex.EncodeToString(packageHash[:]), + }, nil + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + outcome, err := syncGitHubSource(ctx, snapshot, "user:9", "", target.Revision) + if err != nil { + t.Fatalf("syncGitHubSource(confirmed replacement) error = %v, want nil", err) + } + if outcome == nil || outcome.Deployment == nil || outcome.Stale { + t.Fatalf("syncGitHubSource(confirmed replacement) = %+v, want active deployment", outcome) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + if runtime.SyncStatus != pagesSourceStatusIdle || runtime.LastAppliedRevision != target.Revision { + t.Errorf("confirmed replacement runtime = status:%q applied:%q, want idle/%q", runtime.SyncStatus, runtime.LastAppliedRevision, target.Revision) + } +} + +func TestSourceActionPayloadSeparatesSystemTargetAndUserConfirmation(t *testing.T) { + handler := &SourceActionHandler{} + revision := strings.Repeat("a", 64) + invalid := []SourceActionPayload{ + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: "user:1", TargetRevision: revision}, + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, ConfirmedRevision: revision}, + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, TargetRevision: revision, ConfirmedRevision: revision}, + } + for _, payload := range invalid { + raw, _ := json.Marshal(payload) + if normalized, err := handler.ValidatePayload(raw); err == nil { + t.Errorf("ValidatePayload(%+v) = %s, nil; want error", payload, normalized) + } + } + valid := []SourceActionPayload{ + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, TargetRevision: revision}, + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: "user:1", ConfirmedRevision: revision}, + } + for _, payload := range valid { + raw, _ := json.Marshal(payload) + if _, err := handler.ValidatePayload(raw); err != nil { + t.Errorf("ValidatePayload(%+v) error = %v, want nil", payload, err) + } + } +} + +func TestGitHubProviderErrorsMapToSafeRetryClassification(t *testing.T) { + tests := []struct { + name string + provider error + want string + permanent bool + skipRetry bool + }{ + { + name: "asset missing", provider: &githubrelease.Error{Kind: githubrelease.ErrAssetNotFound, StatusCode: 200}, + want: errPagesSourceReleaseNotFound, permanent: true, skipRetry: true, + }, + { + name: "digest mismatch", provider: &githubrelease.Error{Kind: githubrelease.ErrDigestMismatch, StatusCode: 200}, + want: errPagesSourceDigestMismatch, permanent: true, skipRetry: true, + }, + { + name: "rate limit", provider: &githubrelease.Error{ + Kind: githubrelease.ErrMetadata, StatusCode: 429, + RetryAt: func() *time.Time { value := time.Now().Add(time.Hour); return &value }(), + }, + want: errPagesSourceSyncFailed, permanent: false, skipRetry: true, + }, + { + name: "network", provider: &githubrelease.Error{Kind: githubrelease.ErrDownload}, + want: errPagesSourceSyncFailed, permanent: false, skipRetry: false, + }, + { + name: "forbidden without retry", provider: &githubrelease.Error{Kind: githubrelease.ErrMetadata, StatusCode: 403}, + want: errPagesSourceSyncFailed, permanent: true, skipRetry: true, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + domainErr := githubSourceDomainError(test.provider) + if got := domainErr.Error(); got != test.want { + t.Errorf("githubSourceDomainError() = %q, want %q", got, test.want) + } + var typedDomainError *githubSourceProviderDomainError + if !errors.As(domainErr, &typedDomainError) { + t.Fatalf("githubSourceDomainError() type = %T, want *githubSourceProviderDomainError", domainErr) + } + if got := typedDomainError.permanent; got != test.permanent { + t.Errorf("githubSourceProviderDomainError.permanent = %t, want %t", got, test.permanent) + } + if got := shouldSkipGitHubActionRetry(domainErr); got != test.skipRetry { + t.Errorf("shouldSkipGitHubActionRetry() = %t, want %t", got, test.skipRetry) + } + if strings.Contains(domainErr.Error(), "status=") || strings.Contains(domainErr.Error(), "repo=") { + t.Errorf("githubSourceDomainError() = %q, want stable Pages message", domainErr) + } + }) + } +} + +func TestGitHubSyncRejectsStaleConfirmationWithoutChangingActive(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "github-confirm-stale") + oldActive := mustCreateActiveManualDeployment(t, ctx, project.ID, "old") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/a/b", + }) + asset := githubrelease.Asset{ID: "2", Name: "dist.zip", State: "uploaded", UpdatedAt: time.Now().UTC()} + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{Release: githubrelease.Release{ID: "1", Tag: "v1"}, Asset: asset}, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("Download called for stale confirmation") + return nil, nil + }, + }) + snapshot, _, _ := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + _, err := syncGitHubSource(ctx, snapshot, "user:1", "", strings.Repeat("f", 64)) + if err == nil || err.Error() != errPagesSourceConfirmationStale { + t.Errorf("syncGitHubSource(stale confirmation) error = %v, want %q", err, errPagesSourceConfirmationStale) + } + storedProject, loadErr := model.GetPagesProjectByID(ctx, project.ID) + if loadErr != nil { + t.Fatalf("GetPagesProjectByID() error = %v, want nil", loadErr) + } + if storedProject.ActiveDeploymentID == nil || *storedProject.ActiveDeploymentID != oldActive.ID { + t.Errorf("ActiveDeploymentID = %v, want old active %d", storedProject.ActiveDeploymentID, oldActive.ID) + } +} diff --git a/internal/apps/openflare/pages/logics.go b/internal/apps/openflare/pages/logics.go index 3a0f0a49..b154456b 100644 --- a/internal/apps/openflare/pages/logics.go +++ b/internal/apps/openflare/pages/logics.go @@ -822,8 +822,8 @@ func fenceDeploymentActivationSource( audit.SourceIdentity = state.Source.SourceIdentity audit.AutoDisabled = state.Source.AutoUpdateEnabled if err := tx.Model(state.Source).Updates(map[string]any{ - "config_version": state.Source.ConfigVersion + 1, - "auto_update_enabled": false, + sourceColumnConfigVersion: state.Source.ConfigVersion + 1, + sourceColumnAutoUpdateEnabled: false, }).Error; err != nil { return err } @@ -856,8 +856,8 @@ func switchActiveDeploymentTx( return err } if err := tx.Model(deployment).Updates(map[string]any{ - "status": model.PagesDeploymentStatusActive, - "activated_at": &now, + pagesDeploymentColumnStatus: model.PagesDeploymentStatusActive, + "activated_at": &now, }).Error; err != nil { return err } diff --git a/internal/apps/openflare/pages/routers.go b/internal/apps/openflare/pages/routers.go index 4ad4c862..50f93e50 100644 --- a/internal/apps/openflare/pages/routers.go +++ b/internal/apps/openflare/pages/routers.go @@ -45,6 +45,17 @@ func handleSourceLogicError(c *gin.Context, err error) bool { errPagesSourceRemoteURLMode, errPagesSourceRemoteURLInvalid, errPagesSourceNetworkPolicy, + errPagesSourceGitHubFields, + errPagesSourceRepositoryInvalid, + errPagesSourceSelectorInvalid, + errPagesSourceAssetNameInvalid, + errPagesSourceCheckInterval, + errPagesSourceAutoNotAvailable, + errPagesSourceReleaseNotFound, + errPagesSourceDigestInvalid, + errPagesSourceDigestMismatch, + errPagesSourceConfirmationNeeded, + errPagesSourceConfirmationStale, errPagesSourceCheckUnsupported, errPagesSourceActionInvalid: response.AbortBadRequest(c, err.Error()) @@ -253,7 +264,7 @@ func GetSourceHandler(c *gin.Context) { // UpdateSourceHandler 创建或更新 Pages 项目部署源。 // @Summary 更新 Pages 部署源 -// @Description Phase 1 支持 Remote URL 来源；完整地址仅写入，不会在响应中返回 +// @Description 支持 Remote URL 与公开 GitHub Release 来源；敏感地址仅写入，不会在响应中返回 // @Tags openflare-pages // @Accept json // @Produce json @@ -275,7 +286,11 @@ func UpdateSourceHandler(c *gin.Context) { if !decodeStrictJSON(c, &input, false) { return } - result, err := UpdateSource(c.Request.Context(), projectID, input) + actor, ok := currentPagesActor(c) + if !ok { + return + } + result, err := UpdateSourceAs(c.Request.Context(), projectID, input, actor) if handleSourceLogicError(c, err) { return } @@ -309,7 +324,7 @@ func DeleteSourceHandler(c *gin.Context) { // CheckSourceHandler 请求检查 Pages 部署源。 // @Summary 检查 Pages 部署源 -// @Description Remote URL 来源不支持检查更新；该端点为后续 GitHub Release 来源保留统一动作契约 +// @Description 异步检查 GitHub Release 来源；Remote URL 来源不支持检查更新 // @Tags openflare-pages // @Produce json // @Security SessionCookie diff --git a/internal/apps/openflare/pages/source.go b/internal/apps/openflare/pages/source.go index 9bc46217..37312a09 100644 --- a/internal/apps/openflare/pages/source.go +++ b/internal/apps/openflare/pages/source.go @@ -26,7 +26,7 @@ const ( PagesSourceTypeManual = "manual" // PagesSourceTypeRemoteURL represents a persisted artifact URL. PagesSourceTypeRemoteURL = "remote_url" - // PagesSourceTypeGitHubRelease is reserved for Phase 2. + // PagesSourceTypeGitHubRelease represents a public GitHub Release asset. PagesSourceTypeGitHubRelease = "github_release" pagesSourceStatusIdle = "idle" @@ -37,6 +37,10 @@ const ( pagesSourceStatusAttention = "attention" defaultRemoteAssetLabel = "pages-package" + defaultGitHubAssetName = "dist.zip" + defaultCheckInterval = 60 + minimumCheckInterval = 5 + maximumCheckInterval = 1440 ) // SourceUpdateInput is the discriminated source configuration payload. @@ -72,7 +76,7 @@ type SourceView struct { ReleaseSelector string `json:"release_selector,omitempty"` ReleaseTag string `json:"release_tag,omitempty"` AssetName string `json:"asset_name,omitempty"` - AutoUpdateEnabled bool `json:"auto_update_enabled,omitempty"` + AutoUpdateEnabled *bool `json:"auto_update_enabled,omitempty"` CheckIntervalMinutes int `json:"check_interval_minutes,omitempty"` SyncStatus string `json:"sync_status,omitempty"` UpdateAvailable bool `json:"update_available,omitempty"` @@ -99,10 +103,15 @@ type SourceUpdateResult struct { } type sourceDetail struct { - Provider string `json:"provider"` - Label string `json:"label"` - AssetName string `json:"asset_name,omitempty"` - ReleaseID string `json:"release_id,omitempty"` + Provider string `json:"provider"` + DisplayName string `json:"display_name,omitempty"` + Tag string `json:"tag,omitempty"` + LegacyLabel string `json:"label,omitempty"` + AssetName string `json:"asset_name,omitempty"` + ReleaseID string `json:"release_id,omitempty"` + AssetID string `json:"asset_id,omitempty"` + AssetUpdatedAt string `json:"asset_updated_at,omitempty"` + Digest string `json:"digest,omitempty"` } type remoteSourceConfig struct { @@ -126,14 +135,43 @@ func GetSource(ctx context.Context, projectID uint) (*SourceView, error) { return buildSourceView(source, runtime) } -// UpdateSource creates or updates a Remote URL source and its 1:1 runtime row. +// UpdateSource creates or updates a source. Direct callers use the system actor; +// HTTP handlers should call UpdateSourceAs so the initial check is auditable. func UpdateSource(ctx context.Context, projectID uint, input SourceUpdateInput) (*SourceUpdateResult, error) { - if err := validateRemoteSourceInput(input); err != nil { + return UpdateSourceAs(ctx, projectID, input, pagesSourceCreatedBySystem) +} + +// UpdateSourceAs persists source configuration and queues the first GitHub check +// after commit when the GitHub configuration was materially changed. +func UpdateSourceAs( + ctx context.Context, + projectID uint, + input SourceUpdateInput, + actor string, +) (*SourceUpdateResult, error) { + if !validPagesSourceActor(actor) { + return nil, errors.New(errPagesSourceActionInvalid) + } + if err := validateSourceUpdateInput(input); err != nil { return nil, err } + changed := false + var persistedSource model.PagesProjectSource err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { - return updateRemoteSourceTx(tx, projectID, input) + var err error + switch strings.TrimSpace(input.SourceType) { + case PagesSourceTypeRemoteURL: + changed, err = updateRemoteSourceTx(tx, projectID, input) + case PagesSourceTypeGitHubRelease: + changed, err = updateGitHubSourceTx(tx, projectID, input) + default: + err = errors.New(errPagesSourceTypeUnsupported) + } + if err != nil || !changed || strings.TrimSpace(input.SourceType) != PagesSourceTypeGitHubRelease { + return err + } + return tx.Where("project_id = ?", projectID).First(&persistedSource).Error }) if err != nil { return nil, err @@ -143,24 +181,34 @@ func UpdateSource(ctx context.Context, projectID uint, input SourceUpdateInput) if err != nil { return nil, err } - return &SourceUpdateResult{Source: view, Warning: ""}, nil + result := &SourceUpdateResult{Source: view, Warning: ""} + if changed && strings.TrimSpace(input.SourceType) == PagesSourceTypeGitHubRelease { + receipt, dispatchErr := dispatchSourceActionSnapshot(ctx, persistedSource, sourceActionCheck, actor, "", "", "manual") + if dispatchErr != nil { + result.Warning = errPagesSourceInitialCheckWarning + markInitialCheckDispatchFailed(ctx, persistedSource.ID, persistedSource.ConfigVersion) + } else { + result.CheckTask = receipt + } + } + return result, nil } -func updateRemoteSourceTx(tx *gorm.DB, projectID uint, input SourceUpdateInput) error { +func updateRemoteSourceTx(tx *gorm.DB, projectID uint, input SourceUpdateInput) (bool, error) { var project model.PagesProject if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&project, projectID).Error; err != nil { - return err + return false, err } existing, hasExisting, err := loadProjectSourceForUpdate(tx, projectID) if err != nil { - return err + return false, err } config, err := buildRemoteSourceConfig(existing, hasExisting, input) if err != nil { - return err + return false, err } if !hasExisting { - return createRemoteSourceTx(tx, projectID, config) + return true, createRemoteSourceTx(tx, projectID, config) } return updateExistingRemoteSourceTx(tx, existing, config) } @@ -227,33 +275,33 @@ func updateExistingRemoteSourceTx( tx *gorm.DB, existing *model.PagesProjectSource, config remoteSourceConfig, -) error { +) (bool, error) { if !remoteSourceConfigChanged(existing, config) { - return nil + return false, nil } var runtime model.PagesProjectSourceRuntime if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). Where("source_id = ?", existing.ID). First(&runtime).Error; err != nil { - return err + return false, err } identityChanged := existing.SourceIdentity != config.Identity if err := tx.Model(existing).Updates(map[string]any{ - "source_type": PagesSourceTypeRemoteURL, - "remote_url": config.URL, - "remote_network_policy": config.Policy, - "github_repository": "", - "release_selector": "", - "release_tag": "", - "asset_name": "", - "auto_update_enabled": false, - "check_interval_minutes": 0, - "config_version": existing.ConfigVersion + 1, - "source_identity": config.Identity, + "source_type": PagesSourceTypeRemoteURL, + "remote_url": config.URL, + "remote_network_policy": config.Policy, + "github_repository": "", + "release_selector": "", + "release_tag": "", + "asset_name": "", + sourceColumnAutoUpdateEnabled: false, + "check_interval_minutes": 0, + sourceColumnConfigVersion: existing.ConfigVersion + 1, + "source_identity": config.Identity, }).Error; err != nil { - return err + return false, err } - return resetRuntimeAfterSourceUpdate(tx, &runtime, identityChanged) + return true, resetRuntimeAfterSourceUpdate(tx, &runtime, identityChanged) } func remoteSourceConfigChanged(existing *model.PagesProjectSource, config remoteSourceConfig) bool { @@ -328,6 +376,19 @@ func validateRemoteSourceInput(input SourceUpdateInput) error { return nil } +func validateSourceUpdateInput(input SourceUpdateInput) error { + switch strings.TrimSpace(input.SourceType) { + case PagesSourceTypeRemoteURL: + return validateRemoteSourceInput(input) + case PagesSourceTypeGitHubRelease: + return validateGitHubSourceInput(input) + case "": + return errors.New(errPagesSourceTypeRequired) + default: + return errors.New(errPagesSourceTypeUnsupported) + } +} + func resolveUpdatedRemoteURL(existing *model.PagesProjectSource, hasExisting bool, input SourceUpdateInput) (string, error) { if input.RemoteURLSet { return strings.TrimSpace(input.RemoteURL), nil @@ -430,7 +491,8 @@ func buildSourceView(source *model.PagesProjectSource, runtime *model.PagesProje view.ReleaseSelector = source.ReleaseSelector view.ReleaseTag = source.ReleaseTag view.AssetName = source.AssetName - view.AutoUpdateEnabled = source.AutoUpdateEnabled + autoUpdateEnabled := source.AutoUpdateEnabled + view.AutoUpdateEnabled = &autoUpdateEnabled view.CheckIntervalMinutes = source.CheckIntervalMinutes default: return nil, errors.New(errPagesSourceTypeUnsupported) @@ -441,7 +503,7 @@ func buildSourceView(source *model.PagesProjectSource, runtime *model.PagesProje func revisionView(revision string, detailJSON string) *SourceRevisionView { detail := sourceDetail{} _ = unmarshalSourceDetail(detailJSON, &detail) - label := strings.TrimSpace(detail.Label) + label := sourceDetailLabel(detail) if label == "" { label = defaultRemoteAssetLabel } @@ -452,6 +514,19 @@ func revisionView(revision string, detailJSON string) *SourceRevisionView { } } +func sourceDetailLabel(detail sourceDetail) string { + if detail.Provider == githubSourceDetailProvider { + if label := strings.TrimSpace(detail.Tag); label != "" { + return label + } + return strings.TrimSpace(detail.LegacyLabel) + } + if label := strings.TrimSpace(detail.DisplayName); label != "" { + return label + } + return strings.TrimSpace(detail.LegacyLabel) +} + func unmarshalSourceDetail(raw string, detail *sourceDetail) error { if detail == nil || strings.TrimSpace(raw) == "" { return nil diff --git a/internal/apps/openflare/pages/source_runtime.go b/internal/apps/openflare/pages/source_runtime.go index 0dd8fc0f..98801661 100644 --- a/internal/apps/openflare/pages/source_runtime.go +++ b/internal/apps/openflare/pages/source_runtime.go @@ -24,10 +24,14 @@ const ( sourceRuntimeErrorMaxBytes = 512 sourceRevisionHexLength = 64 + sourceColumnAutoUpdateEnabled = "auto_update_enabled" + sourceColumnConfigVersion = "config_version" sourceRuntimeColumnSyncStatus = "sync_status" sourceRuntimeColumnLastError = "last_error" + sourceRuntimeColumnLastCheckedAt = "last_checked_at" sourceRuntimeColumnLeaseToken = "lease_token" sourceRuntimeColumnLeaseExpiresAt = "lease_expires_at" + pagesDeploymentColumnStatus = "status" ) type sourceLeaseOutcome string @@ -50,6 +54,16 @@ type sourceExecutionSnapshot struct { SourceIdentity string RemoteURL string RemoteNetworkPolicy string + GitHubRepository string + ReleaseSelector string + ReleaseTag string + AssetName string + CheckIntervalMinutes int + ETag string + LastSeenRevision string + LastSeenDetail string + LastAppliedRevision string + LastAppliedDetail string RootDir string EntryFile string LeaseToken string @@ -161,6 +175,16 @@ func loadSourceExecutionSnapshot( SourceIdentity: source.SourceIdentity, RemoteURL: source.RemoteURL, RemoteNetworkPolicy: source.RemoteNetworkPolicy, + GitHubRepository: source.GitHubRepository, + ReleaseSelector: source.ReleaseSelector, + ReleaseTag: source.ReleaseTag, + AssetName: source.AssetName, + CheckIntervalMinutes: source.CheckIntervalMinutes, + ETag: runtime.ETag, + LastSeenRevision: runtime.LastSeenRevision, + LastSeenDetail: runtime.LastSeenDetail, + LastAppliedRevision: runtime.LastAppliedRevision, + LastAppliedDetail: runtime.LastAppliedDetail, RootDir: project.RootDir, EntryFile: project.EntryFile, LeaseToken: token, diff --git a/internal/apps/openflare/pages/source_sync.go b/internal/apps/openflare/pages/source_sync.go index 78430cb9..4ef52431 100644 --- a/internal/apps/openflare/pages/source_sync.go +++ b/internal/apps/openflare/pages/source_sync.go @@ -226,12 +226,14 @@ func syncRemoteSource( ctx, snapshot, prepared.Candidate.Checksum, + prepared.Candidate.Checksum, prepared.Detail, prepared.DetailJSON, actor, prepared.Manifest, ingestState.Result, ingestState.HasIngest, + nil, ) ingestState.Referenced = referenced if errors.Is(err, errSourceFinalFence) { @@ -288,7 +290,7 @@ func prepareRemoteSource( cleanupFailedRemoteCandidate(ctx, snapshot, candidate) return nil, err } - detail := sourceDetail{Provider: PagesSourceTypeRemoteURL, Label: safeRemoteSourceLabel(candidate.SafeLabel)} + detail := sourceDetail{Provider: PagesSourceTypeRemoteURL, DisplayName: safeRemoteSourceLabel(candidate.SafeLabel)} detailJSON, err := json.Marshal(detail) if err != nil { cleanupFailedRemoteCandidate(ctx, snapshot, candidate) @@ -336,7 +338,7 @@ func resolveSourceIngest( prepared.Candidate.Checksum, snapshot.ProjectID, snapshot.SourceID, - prepared.Detail.Label, + sourceDetailLabel(prepared.Detail), prepared.Candidate.Format, ) if err != nil { @@ -384,12 +386,14 @@ func commitSourceDeployment( ctx context.Context, snapshot *sourceExecutionSnapshot, revision string, + packageChecksum string, detail sourceDetail, detailJSON string, actor string, manifest *deploymentManifest, ingestResult upload.IngestResult, hasIngest bool, + nextCheckNotBefore *time.Time, ) (*model.PagesDeployment, bool, bool, error) { if snapshot == nil || manifest == nil { return nil, false, false, errors.New(errPagesSourceSyncFailed) @@ -403,7 +407,7 @@ func commitSourceDeployment( return err } target, targetReused, err := resolveSourceDeploymentTx( - tx, state, revision, detail, detailJSON, actor, manifest, ingestResult, hasIngest, + tx, state, revision, packageChecksum, detail, detailJSON, actor, manifest, ingestResult, hasIngest, ) if err != nil { return err @@ -417,7 +421,7 @@ func commitSourceDeployment( if err := refreshSourceCommitLease(state, snapshot); err != nil { return err } - if err := activateSourceDeploymentTx(tx, state, target, revision, detailJSON); err != nil { + if err := activateSourceDeploymentTx(tx, state, target, revision, detailJSON, nextCheckNotBefore); err != nil { return err } committed = *target @@ -491,6 +495,7 @@ func resolveSourceDeploymentTx( tx *gorm.DB, state *sourceCommitState, revision string, + packageChecksum string, detail sourceDetail, detailJSON string, actor string, @@ -515,7 +520,7 @@ func resolveSourceDeploymentTx( return nil, false, errSourceFinalFence } return createSourceDeploymentTx( - tx, state, revision, detail, detailJSON, actor, manifest, ingestResult, + tx, state, revision, packageChecksum, detail, detailJSON, actor, manifest, ingestResult, ) } @@ -523,6 +528,7 @@ func createSourceDeploymentTx( tx *gorm.DB, state *sourceCommitState, revision string, + packageChecksum string, detail sourceDetail, detailJSON string, actor string, @@ -541,7 +547,7 @@ func createSourceDeploymentTx( target := &model.PagesDeployment{ ProjectID: state.Project.ID, DeploymentNumber: maxNumber + 1, - Checksum: revision, + Checksum: packageChecksum, Status: model.PagesDeploymentStatusUploaded, UploadID: ingestResult.Upload.ID, FileCount: manifest.FileCount, @@ -550,7 +556,7 @@ func createSourceDeploymentTx( SourceType: state.Source.SourceType, SourceIdentity: &identity, SourceRevision: &revisionValue, - SourceLabel: detail.Label, + SourceLabel: sourceDetailLabel(detail), SourceMeta: detailJSON, TriggerType: pagesSourceTriggerManualSync, } @@ -629,6 +635,7 @@ func activateSourceDeploymentTx( target *model.PagesDeployment, revision string, detailJSON string, + nextCheckNotBefore *time.Time, ) error { if err := tx.Model(&model.PagesDeployment{}). Where("project_id = ?", state.Project.ID). @@ -636,8 +643,8 @@ func activateSourceDeploymentTx( return err } if err := tx.Model(target).Updates(map[string]any{ - "status": model.PagesDeploymentStatusActive, - "activated_at": &state.Now, + pagesDeploymentColumnStatus: model.PagesDeploymentStatusActive, + "activated_at": &state.Now, }).Error; err != nil { return err } @@ -645,6 +652,15 @@ func activateSourceDeploymentTx( return err } finishedAt := sourceCommitNow() + var nextCheckAt any + if state.Source.SourceType == PagesSourceTypeGitHubRelease && + state.Source.ReleaseSelector == githubReleaseSelectorLatest { + next := nextGitHubCheckAt(finishedAt, state.Source.ID, state.Source.CheckIntervalMinutes) + if nextCheckNotBefore != nil && nextCheckNotBefore.After(next) { + next = nextCheckNotBefore.UTC() + } + nextCheckAt = &next + } result := tx.Model(&model.PagesProjectSourceRuntime{}). Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", state.Runtime.SourceID, @@ -658,9 +674,9 @@ func activateSourceDeploymentTx( "last_applied_detail": detailJSON, sourceRuntimeColumnSyncStatus: pagesSourceStatusIdle, sourceRuntimeColumnLastError: "", - "last_checked_at": &finishedAt, + sourceRuntimeColumnLastCheckedAt: &finishedAt, "last_synced_at": &finishedAt, - "next_check_at": nil, + "next_check_at": nextCheckAt, sourceRuntimeColumnLeaseToken: "", sourceRuntimeColumnLeaseExpiresAt: nil, }) diff --git a/internal/apps/openflare/pages/source_sync_test.go b/internal/apps/openflare/pages/source_sync_test.go index 870446a8..0569e3d2 100644 --- a/internal/apps/openflare/pages/source_sync_test.go +++ b/internal/apps/openflare/pages/source_sync_test.go @@ -388,12 +388,14 @@ func TestCommitSourceDeploymentRechecksLeaseAfterUploadLocks(t *testing.T) { ctx, snapshot, deployment.Checksum, - sourceDetail{Provider: PagesSourceTypeRemoteURL, Label: deployment.SourceLabel}, + deployment.Checksum, + sourceDetail{Provider: PagesSourceTypeRemoteURL, DisplayName: deployment.SourceLabel}, deployment.SourceMeta, "user:5", &deploymentManifest{}, upload.IngestResult{}, false, + nil, ) if !errors.Is(err, errSourceFinalFence) { t.Fatalf("commitSourceDeployment(expired after upload lock) error = %v, want %v", err, errSourceFinalFence) @@ -552,7 +554,7 @@ func TestCommitSourceDeploymentRejectsDeletedTargetUpload(t *testing.T) { SourceIdentity: &identity, SourceRevision: &revision, SourceLabel: "deleted.zip", - SourceMeta: `{"provider":"remote_url","label":"deleted.zip"}`, + SourceMeta: `{"provider":"remote_url","display_name":"deleted.zip"}`, TriggerType: pagesSourceTriggerManualSync, } if err := db.DB(ctx).Create(deployment).Error; err != nil { @@ -575,12 +577,14 @@ func TestCommitSourceDeploymentRejectsDeletedTargetUpload(t *testing.T) { ctx, snapshot, revision, - sourceDetail{Provider: PagesSourceTypeRemoteURL, Label: "deleted.zip"}, - `{"provider":"remote_url","label":"deleted.zip"}`, + revision, + sourceDetail{Provider: PagesSourceTypeRemoteURL, DisplayName: "deleted.zip"}, + `{"provider":"remote_url","display_name":"deleted.zip"}`, "user:1", manifest, upload.IngestResult{}, false, + nil, ) if !errors.Is(err, errSourceFinalFence) { t.Errorf("commitSourceDeployment(deleted upload) error = %v, want %v", err, errSourceFinalFence) diff --git a/internal/apps/openflare/pages/source_tasks.go b/internal/apps/openflare/pages/source_tasks.go index 5fef7d5b..5e72b383 100644 --- a/internal/apps/openflare/pages/source_tasks.go +++ b/internal/apps/openflare/pages/source_tasks.go @@ -77,7 +77,11 @@ func (h *SourceActionHandler) ValidatePayload(payload []byte) ([]byte, error) { (input.Action != sourceActionCheck && input.Action != sourceActionSync) || !validPagesSourceActor(input.Actor) || !validOptionalSourceRevision(input.TargetRevision) || - !validOptionalSourceRevision(input.ConfirmedRevision) { + !validOptionalSourceRevision(input.ConfirmedRevision) || + (input.Action == sourceActionCheck && (input.TargetRevision != "" || input.ConfirmedRevision != "")) || + (input.TargetRevision != "" && input.ConfirmedRevision != "") || + (input.TargetRevision != "" && input.Actor != pagesSourceCreatedBySystem) || + (input.ConfirmedRevision != "" && !strings.HasPrefix(input.Actor, "user:")) { return nil, errors.New(errPagesSourceActionInvalid) } return json.Marshal(input) @@ -110,10 +114,13 @@ func (h *SourceActionHandler) Execute(ctx context.Context, payload []byte) (*tas if input.Action == sourceActionCheck && source.SourceType == PagesSourceTypeRemoteURL { return nil, task.PermanentError(errPagesSourceCheckUnsupported) } - if source.SourceType != PagesSourceTypeRemoteURL { + if source.SourceType != PagesSourceTypeRemoteURL && source.SourceType != PagesSourceTypeGitHubRelease { return nil, task.PermanentError(errPagesSourceTypeUnsupported) } - if input.TargetRevision != "" || input.ConfirmedRevision != "" { + if source.SourceType == PagesSourceTypeRemoteURL && (input.TargetRevision != "" || input.ConfirmedRevision != "") { + return nil, task.PermanentError(errPagesSourceActionInvalid) + } + if input.Action == sourceActionCheck && (input.TargetRevision != "" || input.ConfirmedRevision != "") { return nil, task.PermanentError(errPagesSourceActionInvalid) } @@ -132,10 +139,43 @@ func (h *SourceActionHandler) Execute(ctx context.Context, payload []byte) (*tas return &task.TaskResult{Message: errPagesSourceActionStale}, nil } - result, err := syncRemoteSource(ctx, snapshot, input.Actor) + if input.Action == sourceActionCheck { + return executeGitHubCheckAction(ctx, snapshot) + } + return executeSourceSyncAction(ctx, &source, snapshot, input) +} + +func executeGitHubCheckAction(ctx context.Context, snapshot *sourceExecutionSnapshot) (*task.TaskResult, error) { + checkResult, checkErr := checkGitHubSource(ctx, snapshot) + if checkErr != nil { + logger.ErrorF(ctx, "[PagesSource] check failed: project_id=%d source_id=%d error=%v", snapshot.ProjectID, snapshot.SourceID, checkErr) + if isPermanentSourceSyncError(checkErr) || shouldSkipGitHubActionRetry(checkErr) { + return nil, task.PermanentError(checkErr.Error()) + } + return nil, errors.New(errPagesSourceSyncFailed) + } + if checkResult == nil || checkResult.Stale { + return &task.TaskResult{Message: errPagesSourceActionStale}, nil + } + return &task.TaskResult{Message: checkResult.Message, Detail: checkResult.Detail}, nil +} + +func executeSourceSyncAction( + ctx context.Context, + source *model.PagesProjectSource, + snapshot *sourceExecutionSnapshot, + input SourceActionPayload, +) (*task.TaskResult, error) { + var result *sourceSyncOutcome + var err error + if source.SourceType == PagesSourceTypeGitHubRelease { + result, err = syncGitHubSource(ctx, snapshot, input.Actor, input.TargetRevision, input.ConfirmedRevision) + } else { + result, err = syncRemoteSource(ctx, snapshot, input.Actor) + } if err != nil { logger.ErrorF(ctx, "[PagesSource] sync failed: project_id=%d source_id=%d error=%v", snapshot.ProjectID, snapshot.SourceID, err) - if isPermanentSourceSyncError(err) { + if isPermanentSourceSyncError(err) || shouldSkipGitHubActionRetry(err) { return nil, task.PermanentError(errPagesSourceSyncFailed) } return nil, errors.New(errPagesSourceSyncFailed) @@ -191,13 +231,19 @@ func isPermanentSourceSyncError(err error) bool { } message := err.Error() return strings.Contains(message, errPagesPackageUnsupported) || + strings.Contains(message, errPagesPackageURLTooLarge) || strings.Contains(message, errPagesPackageInvalid) || strings.Contains(message, errPagesPackageEmpty) || strings.Contains(message, errPagesPackageExtractedTooLarge) || strings.Contains(message, errPagesPackageFileTooLarge) || strings.Contains(message, errPagesEntryFileMissing) || strings.Contains(message, errPagesSourceRemoteURLInvalid) || - strings.Contains(message, errPagesSourceNetworkPolicy) + strings.Contains(message, errPagesSourceNetworkPolicy) || + strings.Contains(message, errPagesSourceReleaseNotFound) || + strings.Contains(message, errPagesSourceDigestInvalid) || + strings.Contains(message, errPagesSourceDigestMismatch) || + strings.Contains(message, errPagesSourceConfirmationNeeded) || + strings.Contains(message, errPagesSourceConfirmationStale) } // DispatchSourceAction performs API preflight and enqueues a credential-free action. @@ -207,8 +253,20 @@ func DispatchSourceAction( action string, actor string, confirmedRevision string, +) (*SourceActionReceipt, error) { + return dispatchSourceActionByProject(ctx, projectID, action, actor, "", confirmedRevision) +} + +func dispatchSourceActionByProject( + ctx context.Context, + projectID uint, + action string, + actor string, + targetRevision string, + confirmedRevision string, ) (*SourceActionReceipt, error) { action = strings.TrimSpace(action) + targetRevision = strings.TrimSpace(targetRevision) confirmedRevision = strings.TrimSpace(confirmedRevision) if action != sourceActionCheck && action != sourceActionSync { return nil, errors.New(errPagesSourceActionInvalid) @@ -224,14 +282,8 @@ func DispatchSourceAction( } return nil, err } - if source.SourceType != PagesSourceTypeRemoteURL { - return nil, errors.New(errPagesSourceTypeUnsupported) - } - if action == sourceActionCheck { - return nil, errors.New(errPagesSourceCheckUnsupported) - } - if confirmedRevision != "" { - return nil, errors.New(errPagesSourceActionInvalid) + if err := validateSourceActionPreflight(ctx, &source, action, targetRevision, confirmedRevision); err != nil { + return nil, err } busy, err := sourceLeaseIsBusy(ctx, source.ID) if err != nil { @@ -240,14 +292,58 @@ func DispatchSourceAction( if busy { return nil, errors.New(errPagesSourceActionBusy) } + return dispatchSourceActionSnapshot(ctx, source, action, actor, targetRevision, confirmedRevision, "manual") +} +func validateSourceActionPreflight( + ctx context.Context, + source *model.PagesProjectSource, + action string, + targetRevision string, + confirmedRevision string, +) error { + if source == nil { + return errors.New(errPagesSourceNotFound) + } + if source.SourceType != PagesSourceTypeRemoteURL && source.SourceType != PagesSourceTypeGitHubRelease { + return errors.New(errPagesSourceTypeUnsupported) + } + if action == sourceActionCheck && source.SourceType == PagesSourceTypeRemoteURL { + return errors.New(errPagesSourceCheckUnsupported) + } + if source.SourceType == PagesSourceTypeRemoteURL && (targetRevision != "" || confirmedRevision != "") { + return errors.New(errPagesSourceActionInvalid) + } + if action == sourceActionCheck && (targetRevision != "" || confirmedRevision != "") { + return errors.New(errPagesSourceActionInvalid) + } + if source.SourceType == PagesSourceTypeGitHubRelease && action == sourceActionSync { + if err := preflightGitHubSyncConfirmation(ctx, source.ID, confirmedRevision); err != nil { + return err + } + } + return nil +} + +func dispatchSourceActionSnapshot( + ctx context.Context, + source model.PagesProjectSource, + action string, + actor string, + targetRevision string, + confirmedRevision string, + triggeredBy string, +) (*SourceActionReceipt, error) { + if task.AsynqClient == nil { + return nil, errors.New(errPagesSourceTaskDispatchFailed) + } handler := &SourceActionHandler{} rawPayload, err := json.Marshal(SourceActionPayload{ SourceID: source.ID, ConfigVersion: source.ConfigVersion, Action: action, Actor: actor, - TargetRevision: "", + TargetRevision: targetRevision, ConfirmedRevision: confirmedRevision, }) if err != nil { @@ -257,9 +353,9 @@ func DispatchSourceAction( if err != nil { return nil, err } - taskID, err := task.DispatchTask(ctx, TaskTypePagesSourceAction, payload, "manual") + taskID, err := task.DispatchTask(ctx, TaskTypePagesSourceAction, payload, triggeredBy) if err != nil { - logger.ErrorF(ctx, "[PagesSource] dispatch action failed: project_id=%d source_id=%d action=%s error=%v", projectID, source.ID, action, err) + logger.ErrorF(ctx, "[PagesSource] dispatch action failed: project_id=%d source_id=%d action=%s error=%v", source.ProjectID, source.ID, action, err) return nil, errors.New(errPagesSourceTaskDispatchFailed) } execution, err := model.GetTaskExecutionByTaskID(ctx, taskID) diff --git a/internal/apps/openflare/pages/source_test.go b/internal/apps/openflare/pages/source_test.go index 61f9dabe..8fd52aad 100644 --- a/internal/apps/openflare/pages/source_test.go +++ b/internal/apps/openflare/pages/source_test.go @@ -30,6 +30,33 @@ func setupPagesSourceTest(t *testing.T) context.Context { return t.Context() } +func TestRevisionViewReadsLegacySourceDetailLabel(t *testing.T) { + tests := []struct { + name string + detail string + want string + }{ + { + name: "remote", + detail: `{"provider":"remote_url","label":"legacy.zip"}`, + want: "legacy.zip", + }, + { + name: "github", + detail: `{"provider":"github","label":"v1.2.3","asset_name":"dist.zip"}`, + want: "v1.2.3", + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + view := revisionView(strings.Repeat("a", 64), test.detail) + if view.Label != test.want { + t.Errorf("revisionView(%s).Label = %q, want %q", test.name, view.Label, test.want) + } + }) + } +} + func mustCreatePagesSourceProject(t *testing.T, ctx context.Context, slug string) *model.PagesProject { t.Helper() view, err := CreateProject(ctx, Input{ @@ -203,9 +230,9 @@ func TestRemoteSourceCRUDPreservesSecretAndResetsRuntimeByIdentity(t *testing.T) Where("source_id = ?", source.ID). Updates(map[string]any{ "last_seen_revision": seenRevision, - "last_seen_detail": `{"provider":"remote_url","label":"new.zip"}`, + "last_seen_detail": `{"provider":"remote_url","display_name":"new.zip"}`, "last_applied_revision": appliedRevision, - "last_applied_detail": `{"provider":"remote_url","label":"old.zip"}`, + "last_applied_detail": `{"provider":"remote_url","display_name":"old.zip"}`, "sync_status": pagesSourceStatusSyncing, "lease_token": "in-flight", "lease_expires_at": &future, diff --git a/internal/integration/githubrelease/client.go b/internal/integration/githubrelease/client.go new file mode 100644 index 00000000..83373ea8 --- /dev/null +++ b/internal/integration/githubrelease/client.go @@ -0,0 +1,725 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +// Package githubrelease resolves and downloads public GitHub Release assets. +// It deliberately does not know about Pages projects, deployments or runtime +// state so other callers can reuse the same constrained HTTP contract. +package githubrelease + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "io" + "math" + "net/http" + "net/url" + "os" + "path" + "regexp" + "strconv" + "strings" + "time" + "unicode" + "unicode/utf8" +) + +const ( + // APIVersion is the GitHub REST API contract used by this package. + APIVersion = "2026-03-10" + + // SelectorLatest uses GitHub's repository latest-release endpoint. + SelectorLatest Selector = "latest" + // SelectorTag resolves one exact GitHub release tag. + SelectorTag Selector = "tag" + + defaultAPIBaseURL = "https://api.github.com" + defaultUserAgent = "OpenFlare-GitHubRelease/1.0" + metadataAccept = "application/vnd.github+json" + assetAccept = "application/octet-stream" + maxMetadataBytes = 4 << 20 + maxAssetErrorNames = 10 + maxSafeTextBytes = 255 + maxSafeAssetNameLen = 96 + maxDigestBytes = 96 + maxETagBytes = 512 + safePartsCapacity = 6 +) + +var ( + errInvalidRequest = errors.New("GitHub Release 请求参数无效") + errMetadata = errors.New("GitHub Release 元数据响应无效") + errAssetMissing = errors.New("GitHub Release 中未找到指定的已上传 asset") + errDownload = errors.New("GitHub Release asset 下载失败") + errTooLarge = errors.New("GitHub Release asset 超过大小限制") + errEmptyAsset = errors.New("GitHub Release asset 内容为空") + errDigest = errors.New("GitHub Release asset digest 无效或校验失败") + errCleanup = errors.New("GitHub Release 临时文件清理失败") + + ownerPattern = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]{0,37}[A-Za-z0-9])?$`) + repoPattern = regexp.MustCompile(`^[A-Za-z0-9._-]+$`) + hexPattern = regexp.MustCompile(`^[0-9a-fA-F]{64}$`) +) + +var ( + // ErrInvalidRequest identifies caller configuration errors. + ErrInvalidRequest = errInvalidRequest + // ErrMetadata identifies malformed, unavailable or failed Release metadata requests. + ErrMetadata = errMetadata + // ErrAssetNotFound identifies an otherwise valid Release without the exact uploaded asset. + ErrAssetNotFound = errAssetMissing + // ErrDownload identifies network or HTTP failures while downloading an asset. + ErrDownload = errDownload + // ErrAssetTooLarge identifies assets that exceed the caller's hard byte limit. + ErrAssetTooLarge = errTooLarge + // ErrEmptyAsset identifies an empty downloaded asset. + ErrEmptyAsset = errEmptyAsset + // ErrDigestMismatch identifies malformed or mismatched declared SHA-256 digests. + ErrDigestMismatch = errDigest +) + +// Selector identifies GitHub's own latest endpoint or one exact tag. +type Selector string + +// ResolveRequest describes one public repository release asset lookup. +type ResolveRequest struct { + Repository string + Selector Selector + Tag string + AssetName string + ETag string +} + +// Release contains only metadata safe and necessary for source resolution. +type Release struct { + ID string `json:"release_id"` + Tag string `json:"tag"` + Name string `json:"name,omitempty"` + Draft bool `json:"draft"` + Prerelease bool `json:"prerelease"` + PublishedAt time.Time `json:"published_at,omitempty"` +} + +// Asset contains the immutable target metadata returned by a resolve call. +type Asset struct { + ID string `json:"asset_id"` + Name string `json:"asset_name"` + State string `json:"state"` + Size int64 `json:"size"` + UpdatedAt time.Time `json:"updated_at,omitempty"` + Digest string `json:"digest,omitempty"` +} + +// ResolveResult is either a selected uploaded asset or a not-modified marker. +type ResolveResult struct { + NotModified bool `json:"not_modified"` + ETag string `json:"etag,omitempty"` + Release Release `json:"release,omitempty"` + Asset Asset `json:"asset,omitempty"` + RetryAt *time.Time `json:"retry_at,omitempty"` +} + +// DownloadRequest identifies an already resolved asset. Asset IDs never come +// from an untrusted URL and the download endpoint is built locally. +type DownloadRequest struct { + Repository string + Asset Asset + MaxBytes int64 +} + +// DownloadResult owns a temporary file. Call Cleanup after ingestion. +type DownloadResult struct { + Path string + Size int64 + SHA256 string + DeclaredDigest string +} + +// Cleanup removes the temporary file and is safe to call more than once. +func (result *DownloadResult) Cleanup() error { + if result == nil || result.Path == "" { + return nil + } + name := result.Path + err := os.Remove(name) + if err == nil || errors.Is(err, os.ErrNotExist) { + result.Path = "" + return nil + } + return errCleanup +} + +// Error is a safe provider error. It never retains a response body, request +// URL, redirect location or request headers. +type Error struct { + Kind error + StatusCode int + RequestID string + Repository string + Tag string + AssetName string + AvailableAssets []string + RetryAt *time.Time +} + +func (providerError *Error) Error() string { + if providerError == nil { + return "GitHub Release 请求失败" + } + message := "GitHub Release 请求失败" + if providerError.Kind != nil { + message = providerError.Kind.Error() + } + parts := make([]string, 0, safePartsCapacity) + if providerError.StatusCode != 0 { + parts = append(parts, "status="+strconv.Itoa(providerError.StatusCode)) + } + if providerError.RequestID != "" { + parts = append(parts, "request_id="+providerError.RequestID) + } + if providerError.Repository != "" { + parts = append(parts, "repo="+providerError.Repository) + } + if providerError.Tag != "" { + parts = append(parts, "tag="+providerError.Tag) + } + if providerError.AssetName != "" { + parts = append(parts, "asset="+providerError.AssetName) + } + if len(providerError.AvailableAssets) > 0 { + parts = append(parts, "available="+strings.Join(providerError.AvailableAssets, ",")) + } + if len(parts) == 0 { + return message + } + return message + " (" + strings.Join(parts, " ") + ")" +} + +func (providerError *Error) Unwrap() error { + if providerError == nil { + return nil + } + return providerError.Kind +} + +// RetryAt extracts the server-directed retry deadline from an error. +func RetryAt(err error) (time.Time, bool) { + var providerError *Error + if !errors.As(err, &providerError) || providerError.RetryAt == nil { + return time.Time{}, false + } + return *providerError.RetryAt, true +} + +// RetryTime is retained as a compatibility alias for early callers. +// +// Deprecated: use RetryAt. +func RetryTime(err error) (time.Time, bool) { + return RetryAt(err) +} + +// IsNotFound reports both a missing Release endpoint and a Release that lacks +// the exact uploaded asset requested by the caller. +func IsNotFound(err error) bool { + if errors.Is(err, ErrAssetNotFound) { + return true + } + var providerError *Error + return errors.As(err, &providerError) && providerError.StatusCode == http.StatusNotFound +} + +// IsDigestError reports malformed or mismatched declared asset digests. +func IsDigestError(err error) bool { + return errors.Is(err, ErrDigestMismatch) +} + +// IsRetryable classifies provider failures without relying on localized error +// strings. Configuration, not-found, size, empty-content and digest failures +// are permanent. Network failures, 408/425/429 and 5xx responses are retryable. +func IsRetryable(err error) bool { + if err == nil || errors.Is(err, ErrInvalidRequest) || IsNotFound(err) || + errors.Is(err, ErrAssetTooLarge) || errors.Is(err, ErrEmptyAsset) || IsDigestError(err) { + return false + } + var providerError *Error + if !errors.As(err, &providerError) { + return false + } + if providerError.StatusCode == 0 { + return errors.Is(err, ErrMetadata) || errors.Is(err, ErrDownload) || errors.Is(err, errCleanup) + } + if providerError.StatusCode < http.StatusBadRequest { + return errors.Is(err, ErrMetadata) || errors.Is(err, ErrDownload) || errors.Is(err, errCleanup) + } + if providerError.RetryAt != nil { + return true + } + return providerError.StatusCode == http.StatusRequestTimeout || + providerError.StatusCode == http.StatusTooEarly || + providerError.StatusCode == http.StatusTooManyRequests || + providerError.StatusCode >= http.StatusInternalServerError +} + +// Client accesses public GitHub Releases using a fixed, constrained transport. +type Client struct { + httpClient *http.Client + baseURL string + createTemp func(string, string) (*os.File, error) + now func() time.Time +} + +// NewClient constructs a production client for api.github.com. Public +// repositories do not require or send a token. +func NewClient() *Client { + return newClient(defaultClientOptions()) +} + +// Resolve calls GitHub's latest or exact-tag endpoint and selects one exact, +// case-sensitive uploaded asset. It never falls back to source archives. +func (client *Client) Resolve(ctx context.Context, request ResolveRequest) (ResolveResult, error) { + repository, tag, endpoint, err := normalizeResolveRequest(client.baseURL, request) + if err != nil { + return ResolveResult{}, safeError(errInvalidRequest, 0, "", repository, tag, validErrorAssetName(request.AssetName), nil, nil) + } + httpRequest, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil) + if err != nil { + return ResolveResult{}, safeError(errInvalidRequest, 0, "", repository, tag, validErrorAssetName(request.AssetName), nil, nil) + } + applyMetadataHeaders(httpRequest, request.ETag) + response, err := client.httpClient.Do(httpRequest) //nolint:gosec // endpoint and every dial target are constrained + if err != nil { + return ResolveResult{}, safeError(errMetadata, 0, "", repository, tag, request.AssetName, nil, nil) + } + defer func() { _ = response.Body.Close() }() + + retryAt := responseRetryAt(response, client.now()) + etag := safeETag(response.Header.Get("ETag")) + if response.StatusCode == http.StatusNotModified { + if etag == "" { + etag = safeETag(request.ETag) + } + return ResolveResult{NotModified: true, ETag: etag, RetryAt: retryAt}, nil + } + if response.StatusCode != http.StatusOK { + return ResolveResult{}, safeHTTPError(errMetadata, response, repository, tag, request.AssetName, retryAt) + } + + body, readErr := io.ReadAll(io.LimitReader(response.Body, maxMetadataBytes+1)) + if readErr != nil || len(body) > maxMetadataBytes || !utf8.Valid(body) { + return ResolveResult{}, safeHTTPError(errMetadata, response, repository, tag, request.AssetName, retryAt) + } + var payload releasePayload + decoder := json.NewDecoder(bytes.NewReader(body)) + decoder.UseNumber() + if err := decoder.Decode(&payload); err != nil { + return ResolveResult{}, safeHTTPError(errMetadata, response, repository, tag, request.AssetName, retryAt) + } + if err := ensureJSONEOF(decoder); err != nil { + return ResolveResult{}, safeHTTPError(errMetadata, response, repository, tag, request.AssetName, retryAt) + } + release, assets, err := convertRelease(payload) + if err != nil { + return ResolveResult{}, safeHTTPError(errMetadata, response, repository, tag, request.AssetName, retryAt) + } + for _, asset := range assets { + if asset.State == "uploaded" && asset.Name == request.AssetName { + return ResolveResult{ + ETag: etag, + Release: release, + Asset: asset, + RetryAt: retryAt, + }, nil + } + } + available := safeAssetNames(assets) + return ResolveResult{}, safeError( + errAssetMissing, + response.StatusCode, + response.Header.Get("X-GitHub-Request-Id"), + repository, + release.Tag, + request.AssetName, + available, + retryAt, + ) +} + +// Download streams an asset into a package-owned temporary file while +// enforcing a hard byte limit and verifying GitHub's declared sha256 digest. +func (client *Client) Download(ctx context.Context, request DownloadRequest) (*DownloadResult, error) { + repository, err := normalizeRepository(request.Repository) + if err != nil || request.MaxBytes <= 0 || !validPositiveID(request.Asset.ID) || + !validAssetName(request.Asset.Name) || request.Asset.Size < 0 { + return nil, safeError(errInvalidRequest, 0, "", repository, "", validErrorAssetName(request.Asset.Name), nil, nil) + } + if request.Asset.Size > request.MaxBytes { + return nil, safeError(errTooLarge, 0, "", repository, "", validErrorAssetName(request.Asset.Name), nil, nil) + } + endpoint := strings.TrimRight(client.baseURL, "/") + "/repos/" + repository + "/releases/assets/" + request.Asset.ID + httpRequest, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil) + if err != nil { + return nil, safeError(errInvalidRequest, 0, "", repository, "", request.Asset.Name, nil, nil) + } + applyAssetHeaders(httpRequest) + response, err := client.httpClient.Do(httpRequest) //nolint:gosec // endpoint and every dial target are constrained + if err != nil { + return nil, safeError(errDownload, 0, "", repository, "", request.Asset.Name, nil, nil) + } + defer func() { _ = response.Body.Close() }() + retryAt := responseRetryAt(response, client.now()) + if response.StatusCode != http.StatusOK { + return nil, safeHTTPError(errDownload, response, repository, "", request.Asset.Name, retryAt) + } + if response.ContentLength > request.MaxBytes { + return nil, safeHTTPError(errTooLarge, response, repository, "", request.Asset.Name, retryAt) + } + + result, err := client.streamAsset(response.Body, request.MaxBytes, request.Asset.Digest) + if err != nil { + return nil, safeError(err, response.StatusCode, response.Header.Get("X-GitHub-Request-Id"), repository, "", request.Asset.Name, nil, retryAt) + } + return result, nil +} + +func (client *Client) streamAsset(body io.Reader, maxBytes int64, declaredDigest string) (result *DownloadResult, resultErr error) { + tempFile, err := client.createTemp("", "openflare-github-release-*") + if err != nil { + return nil, errDownload + } + tempPath := tempFile.Name() + defer func() { + closeErr := tempFile.Close() + if resultErr == nil && closeErr != nil { + resultErr = errDownload + } + if resultErr != nil { + if removeErr := os.Remove(tempPath); removeErr != nil && !errors.Is(removeErr, os.ErrNotExist) { + resultErr = errCleanup + } + } + }() + + hasher := sha256.New() + readLimit := maxBytes + if readLimit < math.MaxInt64 { + readLimit++ + } + size, err := io.Copy(io.MultiWriter(tempFile, hasher), io.LimitReader(body, readLimit)) + if err != nil { + return nil, errDownload + } + if size > maxBytes { + return nil, errTooLarge + } + if size == 0 { + return nil, errEmptyAsset + } + checksum := hex.EncodeToString(hasher.Sum(nil)) + if err := verifyDeclaredDigest(declaredDigest, checksum); err != nil { + return nil, err + } + return &DownloadResult{ + Path: tempPath, + Size: size, + SHA256: checksum, + DeclaredDigest: strings.ToLower(strings.TrimSpace(declaredDigest)), + }, nil +} + +type releasePayload struct { + ID json.Number `json:"id"` + Tag string `json:"tag_name"` + Name string `json:"name"` + Draft bool `json:"draft"` + Prerelease bool `json:"prerelease"` + PublishedAt string `json:"published_at"` + Assets []assetPayload `json:"assets"` +} + +type assetPayload struct { + ID json.Number `json:"id"` + Name string `json:"name"` + State string `json:"state"` + Size int64 `json:"size"` + UpdatedAt string `json:"updated_at"` + Digest string `json:"digest"` +} + +func convertRelease(payload releasePayload) (Release, []Asset, error) { + releaseID, err := positiveJSONID(payload.ID) + if err != nil { + return Release{}, nil, err + } + if !validReleaseDisplayTag(payload.Tag) { + return Release{}, nil, errMetadata + } + publishedAt, err := parseOptionalTime(payload.PublishedAt) + if err != nil { + return Release{}, nil, err + } + release := Release{ + ID: releaseID, + Tag: payload.Tag, + Name: safeText(payload.Name, maxSafeTextBytes), + Draft: payload.Draft, + Prerelease: payload.Prerelease, + PublishedAt: publishedAt, + } + assets := make([]Asset, 0, len(payload.Assets)) + for _, rawAsset := range payload.Assets { + assetID, assetErr := positiveJSONID(rawAsset.ID) + if assetErr != nil || rawAsset.Size < 0 { + return Release{}, nil, errMetadata + } + updatedAt, assetErr := parseOptionalTime(rawAsset.UpdatedAt) + if assetErr != nil { + return Release{}, nil, errMetadata + } + assets = append(assets, Asset{ + ID: assetID, + Name: rawAsset.Name, + State: rawAsset.State, + Size: rawAsset.Size, + UpdatedAt: updatedAt, + Digest: safeText(rawAsset.Digest, maxDigestBytes), + }) + } + return release, assets, nil +} + +func normalizeResolveRequest(baseURL string, request ResolveRequest) (string, string, string, error) { + repository, err := normalizeRepository(request.Repository) + if err != nil || !validAssetName(request.AssetName) { + return repository, validErrorTag(request.Tag), "", errInvalidRequest + } + baseURL = strings.TrimRight(baseURL, "/") + switch request.Selector { + case SelectorLatest: + if strings.TrimSpace(request.Tag) != "" { + return repository, "", "", errInvalidRequest + } + return repository, "latest", baseURL + "/repos/" + repository + "/releases/latest", nil + case SelectorTag: + if !validTag(request.Tag) { + return repository, validErrorTag(request.Tag), "", errInvalidRequest + } + return repository, request.Tag, baseURL + "/repos/" + repository + "/releases/tags/" + url.PathEscape(request.Tag), nil + default: + return repository, validErrorTag(request.Tag), "", errInvalidRequest + } +} + +func normalizeRepository(repository string) (string, error) { + repository = strings.TrimSpace(repository) + parts := strings.Split(repository, "/") + if len(parts) != 2 || !ownerPattern.MatchString(parts[0]) || !repoPattern.MatchString(parts[1]) || + len(parts[1]) > 100 || parts[1] == "." || parts[1] == ".." { + return "", errInvalidRequest + } + return parts[0] + "/" + parts[1], nil +} + +func validAssetName(assetName string) bool { + return validLogText(assetName, maxSafeTextBytes, false) && path.Base(assetName) == assetName && + assetName != "." && assetName != ".." && !strings.ContainsAny(assetName, `/\`) +} + +func validTag(tag string) bool { + if !validLogText(tag, maxSafeTextBytes, false) || strings.ContainsAny(tag, " ~^:?*[\\") || + strings.Contains(tag, "..") || strings.Contains(tag, "@{") || strings.Contains(tag, "//") || + strings.HasPrefix(tag, "/") || strings.HasSuffix(tag, "/") || strings.HasSuffix(tag, ".") { + return false + } + for _, component := range strings.Split(tag, "/") { + if component == "" || strings.HasPrefix(component, ".") || strings.HasSuffix(component, ".lock") { + return false + } + } + return true +} + +func validReleaseDisplayTag(tag string) bool { + return validLogText(tag, maxSafeTextBytes, false) +} + +func validLogText(value string, maxBytes int, allowEmpty bool) bool { + if (!allowEmpty && value == "") || len(value) > maxBytes || !utf8.ValidString(value) { + return false + } + for _, character := range value { + if isLogControl(character) { + return false + } + } + return true +} + +func isLogControl(character rune) bool { + if unicode.IsControl(character) || character == '\u2028' || character == '\u2029' { + return true + } + switch character { + case '\u061c', '\u200e', '\u200f', + '\u202a', '\u202b', '\u202c', '\u202d', '\u202e', + '\u2066', '\u2067', '\u2068', '\u2069': + return true + default: + return false + } +} + +func validErrorTag(tag string) string { + if !validTag(tag) || containsSecretDelimiter(tag) { + return "" + } + return tag +} + +func validErrorAssetName(assetName string) string { + if !validAssetName(assetName) || containsSecretDelimiter(assetName) { + return "" + } + return assetName +} + +func containsSecretDelimiter(value string) bool { + return strings.ContainsAny(value, "?&=#") || strings.Contains(value, "://") +} + +func validPositiveID(id string) bool { + parsed, err := strconv.ParseInt(id, 10, 64) + return err == nil && parsed > 0 && strconv.FormatInt(parsed, 10) == id +} + +func positiveJSONID(id json.Number) (string, error) { + parsed, err := strconv.ParseInt(id.String(), 10, 64) + if err != nil || parsed <= 0 { + return "", errMetadata + } + return strconv.FormatInt(parsed, 10), nil +} + +func parseOptionalTime(value string) (time.Time, error) { + if value == "" { + return time.Time{}, nil + } + parsed, err := time.Parse(time.RFC3339, value) + if err != nil { + return time.Time{}, errMetadata + } + return parsed, nil +} + +func ensureJSONEOF(decoder *json.Decoder) error { + var trailing any + if err := decoder.Decode(&trailing); errors.Is(err, io.EOF) { + return nil + } + return errMetadata +} + +func verifyDeclaredDigest(declaredDigest string, checksum string) error { + declaredDigest = strings.TrimSpace(declaredDigest) + if declaredDigest == "" { + return nil + } + algorithm, digest, ok := strings.Cut(declaredDigest, ":") + if !ok || !strings.EqualFold(algorithm, "sha256") || !hexPattern.MatchString(digest) || + !strings.EqualFold(digest, checksum) { + return errDigest + } + return nil +} + +func safeAssetNames(assets []Asset) []string { + count := len(assets) + if count > maxAssetErrorNames { + count = maxAssetErrorNames + } + names := make([]string, 0, count) + for _, asset := range assets[:count] { + name := safeText(asset.Name, maxSafeAssetNameLen) + if containsSecretDelimiter(name) { + name = "" + } + names = append(names, name) + } + return names +} + +func safeText(value string, maxBytes int) string { + var builder strings.Builder + for _, character := range value { + if isLogControl(character) { + builder.WriteByte('?') + continue + } + builder.WriteRune(character) + if builder.Len() >= maxBytes { + break + } + } + result := builder.String() + for len(result) > maxBytes { + _, size := utf8.DecodeLastRuneInString(result) + result = result[:len(result)-size] + } + return result +} + +func safeETag(value string) string { + value = strings.TrimSpace(value) + if len(value) > maxETagBytes || safeText(value, maxETagBytes) != value { + return "" + } + return value +} + +func safeHTTPError(kind error, response *http.Response, repository string, tag string, assetName string, retryAt *time.Time) error { + return safeError( + kind, + response.StatusCode, + response.Header.Get("X-GitHub-Request-Id"), + repository, + tag, + assetName, + nil, + retryAt, + ) +} + +func safeError( + kind error, + statusCode int, + requestID string, + repository string, + tag string, + assetName string, + availableAssets []string, + retryAt *time.Time, +) error { + return &Error{ + Kind: kind, + StatusCode: statusCode, + RequestID: safeErrorToken(requestID, maxSafeTextBytes), + Repository: safeErrorToken(repository, maxSafeTextBytes), + Tag: safeErrorToken(tag, maxSafeTextBytes), + AssetName: safeErrorToken(assetName, maxSafeAssetNameLen), + AvailableAssets: availableAssets, + RetryAt: retryAt, + } +} + +func safeErrorToken(value string, maxBytes int) string { + if !validLogText(value, maxBytes, true) { + return "" + } + value = safeText(value, maxBytes) + if containsSecretDelimiter(value) { + return "" + } + return value +} diff --git a/internal/integration/githubrelease/client_test.go b/internal/integration/githubrelease/client_test.go new file mode 100644 index 00000000..04a93b7a --- /dev/null +++ b/internal/integration/githubrelease/client_test.go @@ -0,0 +1,748 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package githubrelease + +import ( + "context" + "crypto/sha256" + "crypto/tls" + "encoding/hex" + "errors" + "fmt" + "net" + "net/http" + "net/http/httptest" + "net/netip" + "net/url" + "os" + "path/filepath" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +type resolverFunc func(context.Context, string, string) ([]netip.Addr, error) + +func (resolve resolverFunc) LookupNetIP(ctx context.Context, network string, host string) ([]netip.Addr, error) { + return resolve(ctx, network, host) +} + +func TestResolveLatestUsesGitHubContractAndSelectsUploadedAsset(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.URL.Path != "/repos/acme/site/releases/latest" { + t.Errorf("path = %q", request.URL.Path) + } + assertHeader(t, request, "Accept", metadataAccept) + assertHeader(t, request, "User-Agent", defaultUserAgent) + assertHeader(t, request, "X-GitHub-Api-Version", APIVersion) + assertHeader(t, request, "If-None-Match", `W/"old"`) + writer.Header().Set("ETag", `W/"new"`) + writer.Header().Set("X-RateLimit-Remaining", "0") + writer.Header().Set("X-RateLimit-Reset", "1800000000") + _, _ = writer.Write([]byte(`{ + "id": 9007199254740991, + "tag_name": "v1.2.3", + "name": "Stable", + "published_at": "2026-07-18T12:00:00Z", + "assets": [ + {"id": 11, "name": "dist.zip", "state": "new", "size": 1}, + {"id": 9007199254740990, "name": "dist.zip", "state": "uploaded", "size": 42, + "updated_at": "2026-07-18T12:10:00Z", "digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"} + ] + }`)) + })) + defer server.Close() + + client := newTestClient(t, server.URL, nil) + result, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", + Selector: SelectorLatest, + AssetName: "dist.zip", + ETag: `W/"old"`, + }) + if err != nil { + t.Fatalf("Resolve() error = %v", err) + } + if result.Release.ID != "9007199254740991" || result.Asset.ID != "9007199254740990" { + t.Fatalf("IDs lost precision: release=%q asset=%q", result.Release.ID, result.Asset.ID) + } + if result.ETag != `W/"new"` || result.Asset.Name != "dist.zip" || result.Asset.State != "uploaded" { + t.Fatalf("Resolve() = %+v", result) + } + if result.RetryAt == nil || result.RetryAt.Unix() != 1800000000 { + t.Fatalf("RetryAt = %v", result.RetryAt) + } +} + +func TestResolveTagEscapesPathAndHandlesNotModified(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.RequestURI != "/repos/acme/site/releases/tags/release%2Fcandidate" { + t.Errorf("RequestURI = %q", request.RequestURI) + } + writer.WriteHeader(http.StatusNotModified) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + result, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", + Selector: SelectorTag, + Tag: "release/candidate", + AssetName: "dist.zip", + ETag: `"cached"`, + }) + if err != nil { + t.Fatalf("Resolve() error = %v", err) + } + if !result.NotModified || result.ETag != `"cached"` { + t.Fatalf("Resolve() = %+v", result) + } +} + +func TestResolveAssetMissingTruncatesSafeNamesAndNeverIncludesBody(t *testing.T) { + t.Parallel() + assets := make([]string, 0, 12) + for index := 0; index < 12; index++ { + assets = append(assets, fmt.Sprintf(`{"id":%d,"name":"asset-%02d.zip","state":"uploaded","size":1}`, index+1, index)) + } + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write([]byte(`{"id":1,"tag_name":"v1","message":"body-token","assets":[` + strings.Join(assets, ",") + `]}`)) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + _, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist.zip", + }) + if !errors.Is(err, errAssetMissing) { + t.Fatalf("Resolve() error = %v", err) + } + message := err.Error() + if !strings.Contains(message, "asset-00.zip") || !strings.Contains(message, "asset-09.zip") { + t.Fatalf("error misses safe truncated names: %s", message) + } + if strings.Contains(message, "asset-10.zip") || strings.Contains(message, "asset-11.zip") || strings.Contains(message, "body-token") { + t.Fatalf("error leaked/truncation failed: %s", message) + } +} + +func TestResolveHTTPErrorParsesRateLimitWithoutBodyLeak(t *testing.T) { + t.Parallel() + now := time.Date(2026, time.July, 19, 10, 0, 0, 0, time.UTC) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("Retry-After", "90") + writer.Header().Set("X-GitHub-Request-Id", "request-123") + writer.WriteHeader(http.StatusTooManyRequests) + _, _ = writer.Write([]byte(`{"message":"signed_url=https://secret.example/a?token=hidden"}`)) + })) + defer server.Close() + client := newTestClient(t, server.URL, func(options *clientOptions) { options.now = func() time.Time { return now } }) + _, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist.zip", + }) + if err == nil || !strings.Contains(err.Error(), "status=429") || !strings.Contains(err.Error(), "request_id=request-123") { + t.Fatalf("Resolve() error = %v", err) + } + if strings.Contains(err.Error(), "secret.example") || strings.Contains(err.Error(), "hidden") { + t.Fatalf("error leaked body: %s", err) + } + retryAt, ok := RetryTime(err) + if !ok || !retryAt.Equal(now.Add(90*time.Second)) { + t.Fatalf("RetryTime() = %v, %v", retryAt, ok) + } +} + +func TestDownloadStreamsVerifiesDigestAndCleansUp(t *testing.T) { + t.Parallel() + payload := []byte("package bytes") + digest := sha256.Sum256(payload) + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + if request.URL.Path != "/repos/acme/site/releases/assets/42" { + t.Errorf("path = %q", request.URL.Path) + } + assertHeader(t, request, "Accept", assetAccept) + assertHeader(t, request, "Accept-Encoding", "identity") + _, _ = writer.Write(payload) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + result, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", + Asset: Asset{ + ID: "42", Name: "dist.zip", Digest: "sha256:" + hex.EncodeToString(digest[:]), + }, + MaxBytes: 1024, + }) + if err != nil { + t.Fatalf("Download() error = %v", err) + } + if result.Size != int64(len(payload)) || result.SHA256 != hex.EncodeToString(digest[:]) { + t.Fatalf("Download() = %+v", result) + } + if _, err := os.Stat(result.Path); err != nil { + t.Fatalf("temp file stat: %v", err) + } + if err := result.Cleanup(); err != nil { + t.Fatalf("Cleanup() error = %v", err) + } + if err := result.Cleanup(); err != nil { + t.Fatalf("second Cleanup() error = %v", err) + } +} + +func TestDownloadFollows302AndStripsCrossHostSensitiveHeaders(t *testing.T) { + t.Parallel() + var targetHost string + target := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + for _, header := range []string{"Authorization", "Cookie", "Proxy-Authorization", "Referer", "If-None-Match", "If-Modified-Since", "X-GitHub-Api-Version"} { + if value := request.Header.Get(header); value != "" { + t.Errorf("redirect leaked %s=%q", header, value) + } + } + _, _ = writer.Write([]byte("redirected package")) + })) + defer target.Close() + targetURL, _ := url.Parse(target.URL) + targetHost = "asset.example.test:" + targetURL.Port() + + api := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("Location", "http://"+targetHost+"/signed/package.zip?token=must-not-leak") + writer.WriteHeader(http.StatusFound) + })) + defer api.Close() + apiURL, _ := url.Parse(api.URL) + baseURL := "http://api.example.test:" + apiURL.Port() + client := newMappedTestClient(t, baseURL, nil) + result, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: Asset{ID: "42", Name: "dist.zip"}, MaxBytes: 1024, + }) + if err != nil { + t.Fatalf("Download() redirect error = %v", err) + } + if cleanupErr := result.Cleanup(); cleanupErr != nil { + t.Fatalf("Cleanup() error = %v", cleanupErr) + } + + request, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, baseURL+"/repos/acme/site/releases/assets/42", nil) + applyAssetHeaders(request) + request.Header.Set("Authorization", "Bearer secret") + request.Header.Set("Cookie", "session=secret") + request.Header.Set("Proxy-Authorization", "proxy-secret") + request.Header.Set("Referer", "https://secret.example/path?token=x") + request.Header.Set("If-None-Match", `"secret-etag"`) + request.Header.Set("If-Modified-Since", time.Now().Format(http.TimeFormat)) + response, err := client.httpClient.Do(request) + if err != nil { + t.Fatalf("Do() error = %v", err) + } + _ = response.Body.Close() +} + +func TestRedirectSSRFAndDNSRebindingAreRejectedWithoutURLLeak(t *testing.T) { + t.Parallel() + t.Run("literal private redirect", func(t *testing.T) { + api := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("Location", "http://127.0.0.1/private?token=secret-query") + writer.WriteHeader(http.StatusFound) + })) + defer api.Close() + client := newTestClient(t, api.URL, nil) + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: Asset{ID: "1", Name: "dist.zip"}, MaxBytes: 100, + }) + if err == nil || strings.Contains(err.Error(), "secret-query") || strings.Contains(err.Error(), "127.0.0.1") { + t.Fatalf("Download() error = %v", err) + } + }) + + t.Run("DNS rebind between redirect and dial", func(t *testing.T) { + api := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("Location", "http://rebind.example.test/package.zip") + writer.WriteHeader(http.StatusFound) + })) + defer api.Close() + var lock sync.Mutex + calls := map[string]int{} + resolve := resolverFunc(func(_ context.Context, _ string, host string) ([]netip.Addr, error) { + lock.Lock() + defer lock.Unlock() + calls[host]++ + if host == "rebind.example.test" && calls[host] > 1 { + return []netip.Addr{netip.MustParseAddr("127.0.0.1")}, nil + } + return []netip.Addr{netip.MustParseAddr("8.8.8.8")}, nil + }) + client := newTestClient(t, api.URL, func(options *clientOptions) { options.resolver = resolve }) + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: Asset{ID: "1", Name: "dist.zip"}, MaxBytes: 100, + }) + if err == nil { + t.Fatal("Download() error = nil") + } + lock.Lock() + defer lock.Unlock() + if calls["rebind.example.test"] != 2 { + t.Fatalf("rebind lookup calls = %d", calls["rebind.example.test"]) + } + }) +} + +func TestDownloadFailureRemovesTemporaryFile(t *testing.T) { + t.Parallel() + payload := []byte("package bytes") + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write(payload) + })) + defer server.Close() + tempDir := t.TempDir() + client := newTestClient(t, server.URL, func(options *clientOptions) { + options.createTemp = func(_ string, pattern string) (*os.File, error) { + return os.CreateTemp(tempDir, pattern) + } + }) + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", + Asset: Asset{ + ID: "42", Name: "dist.zip", Digest: "sha256:" + strings.Repeat("0", 64), + }, + MaxBytes: 1024, + }) + if !errors.Is(err, errDigest) { + t.Fatalf("Download() error = %v", err) + } + files, readErr := filepath.Glob(filepath.Join(tempDir, "*")) + if readErr != nil || len(files) != 0 { + t.Fatalf("temporary files after failure = %v, err=%v", files, readErr) + } +} + +func TestResolveRejectsInvalidRepositoryAndAssetWithoutRequest(t *testing.T) { + t.Parallel() + client := NewClient() + for _, request := range []ResolveRequest{ + {Repository: "https://github.com/acme/site", Selector: SelectorLatest, AssetName: "dist.zip"}, + {Repository: "acme/site/extra", Selector: SelectorLatest, AssetName: "dist.zip"}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: "../dist.zip"}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: `dir\dist.zip`}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: string([]byte{'d', 'i', 's', 't', 0xff})}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist\nsecret.zip"}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist\u2028secret.zip"}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist\u202esecret.zip"}, + {Repository: "acme/site", Selector: SelectorTag, AssetName: "dist.zip"}, + } { + _, err := client.Resolve(context.Background(), request) + if !errors.Is(err, errInvalidRequest) { + t.Errorf("Resolve(%+v) error = %v", request, err) + } + } +} + +func TestResolveAndDownloadAssetNameWithDelimiters(t *testing.T) { + t.Parallel() + assetName := "dist?channel=stable#1&x.zip" + payload := []byte("package with delimiter name") + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + switch request.URL.Path { + case "/repos/acme/site/releases/latest": + if request.Header.Get("If-None-Match") == "missing" { + _, _ = writer.Write([]byte(`{"id":1,"tag_name":"v1","assets":[]}`)) + return + } + _, _ = fmt.Fprintf(writer, `{"id":1,"tag_name":"release/v1","assets":[{"id":42,"name":%q,"state":"uploaded","size":%d}]}`, + assetName, len(payload)) + case "/repos/acme/site/releases/assets/42": + _, _ = writer.Write(payload) + default: + http.NotFound(writer, request) + } + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + + resolved, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: assetName, + }) + if err != nil { + t.Fatalf("Resolve() error = %v", err) + } + if resolved.Asset.Name != assetName || resolved.Release.Tag != "release/v1" { + t.Fatalf("Resolve() = %+v", resolved) + } + download, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: resolved.Asset, MaxBytes: 1024, + }) + if err != nil { + t.Fatalf("Download() error = %v", err) + } + if cleanupErr := download.Cleanup(); cleanupErr != nil { + t.Fatalf("Cleanup() error = %v", cleanupErr) + } + + _, err = client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: assetName, ETag: "missing", + }) + if !errors.Is(err, ErrAssetNotFound) { + t.Fatalf("missing Resolve() error = %v", err) + } + if strings.Contains(err.Error(), assetName) || strings.Contains(err.Error(), "channel=stable") { + t.Fatalf("missing error leaked delimiter-bearing name: %v", err) + } +} + +func TestFixedTagGitRefRulesAndEscaping(t *testing.T) { + t.Parallel() + valid := []string{"@", "release/v1#stable&channel=prod", "foo.LOCK", "中文/发布=稳定"} + for _, tag := range valid { + if !validTag(tag) { + t.Errorf("validTag(%q) = false", tag) + } + } + invalid := []string{ + "", "release v1", "release~v1", "release^v1", "release:v1", "release?v1", "release*v1", + "release[v1", `release\v1`, "release..v1", "release@{v1", "release//v1", "/release", "release/", + "release.", ".release", "release/.candidate", "release.lock", "release/v1.lock", "release\nsecret", + "release\u2028secret", "release\u202esecret", string([]byte{'v', '1', 0xff}), strings.Repeat("a", 256), + } + for _, tag := range invalid { + if validTag(tag) { + t.Errorf("validTag(%q) = true", tag) + } + } + + tag := "release/v1#stable&channel=prod" + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + wantURI := "/repos/acme/site/releases/tags/" + url.PathEscape(tag) + if request.RequestURI != wantURI || request.URL.RawQuery != "" || request.URL.Fragment != "" { + t.Errorf("tag request = %q query=%q fragment=%q, want %q", request.RequestURI, request.URL.RawQuery, request.URL.Fragment, wantURI) + } + _, _ = fmt.Fprintf(writer, `{"id":1,"tag_name":%q,"assets":[{"id":2,"name":"dist.zip","state":"uploaded","size":1}]}`, tag) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + result, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorTag, Tag: tag, AssetName: "dist.zip", + }) + if err != nil { + t.Fatalf("Resolve() error = %v", err) + } + if result.Release.Tag != tag { + t.Fatalf("Release.Tag = %q", result.Release.Tag) + } +} + +func TestResolveDoesNotMatchSanitizedRemoteAssetName(t *testing.T) { + t.Parallel() + tests := []struct { + name string + remote string + requested string + }{ + {name: "unicode line separator", remote: "dist\u2028.zip", requested: "dist?.zip"}, + {name: "overlong", remote: strings.Repeat("a", 256), requested: strings.Repeat("a", 255)}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = fmt.Fprintf(writer, `{"id":1,"tag_name":"release/v1","assets":[{"id":2,"name":%q,"state":"uploaded","size":1}]}`, test.remote) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + _, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: test.requested, + }) + if !errors.Is(err, ErrAssetNotFound) { + t.Fatalf("Resolve() error = %v", err) + } + }) + } +} + +func TestReleaseDisplayTagValidation(t *testing.T) { + t.Parallel() + for _, tag := range []string{"release/v1", "release v1", "release/v1#stable&channel=prod"} { + if !validReleaseDisplayTag(tag) { + t.Errorf("validReleaseDisplayTag(%q) = false", tag) + } + } + for _, tag := range []string{"", strings.Repeat("a", 256), "release\nsecret", "release\u2028secret", "release\u202esecret"} { + if validReleaseDisplayTag(tag) { + t.Errorf("validReleaseDisplayTag(%q) = true", tag) + } + } +} + +func TestResolveRejectsInvalidUTF8Metadata(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write(append([]byte(`{"id":1,"tag_name":"v1","assets":[{"id":2,"name":"dist`), + append([]byte{0xff}, []byte(`.zip","state":"uploaded","size":1}]}`)...)...)) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + _, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist�.zip", + }) + if !errors.Is(err, ErrMetadata) { + t.Fatalf("Resolve() error = %v", err) + } +} + +func TestDownloadRejectsImpossibleMetadataBeforeNetwork(t *testing.T) { + t.Parallel() + var requests atomic.Int64 + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + requests.Add(1) + writer.WriteHeader(http.StatusInternalServerError) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + tests := []struct { + name string + size int64 + kind error + limit int64 + }{ + {name: "negative", size: -1, kind: ErrInvalidRequest, limit: 100}, + {name: "declared too large", size: 101, kind: ErrAssetTooLarge, limit: 100}, + } + for _, test := range tests { + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", + Asset: Asset{ID: "1", Name: "dist?token=hidden#asset.zip", Size: test.size}, + MaxBytes: test.limit, + }) + if !errors.Is(err, test.kind) { + t.Errorf("%s Download() error = %v", test.name, err) + } + if strings.Contains(err.Error(), "token=hidden") { + t.Errorf("%s error leaked asset name: %v", test.name, err) + } + } + if got := requests.Load(); got != 0 { + t.Fatalf("HTTP requests = %d, want 0", got) + } +} + +func TestLogControlCharactersNeverEnterSafeErrors(t *testing.T) { + t.Parallel() + controls := []string{"\u2028", "\u2029", "\u061c", "\u200e", "\u200f", "\u202e", "\u2066", "\u2069"} + for _, control := range controls { + secret := "before" + control + "after" + err := safeError(errInvalidRequest, 0, secret, secret, secret, secret, nil, nil) + message := err.Error() + if strings.Contains(message, secret) || strings.Contains(message, control) || strings.Contains(message, "before") { + t.Errorf("safe error retained control %U: %q", []rune(control)[0], message) + } + } +} + +func TestResolveRejectsMetadataOverHardLimit(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write([]byte(`{"id":1,"tag_name":"v1","assets":[]}` + strings.Repeat(" ", maxMetadataBytes))) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + _, err := client.Resolve(context.Background(), ResolveRequest{ + Repository: "acme/site", Selector: SelectorLatest, AssetName: "dist.zip", + }) + if !errors.Is(err, ErrMetadata) { + t.Fatalf("Resolve() error = %v", err) + } +} + +func TestProductionTransportRejectsSelfSignedTLS(t *testing.T) { + t.Parallel() + server := httptest.NewTLSServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + _, _ = writer.Write([]byte("package")) + })) + defer server.Close() + parsed, _ := url.Parse(server.URL) + dialer := &net.Dialer{Timeout: time.Second} + client := newClient(clientOptions{ + baseURL: "https://api.example.test:" + parsed.Port(), + resolver: resolverFunc(func(_ context.Context, _ string, _ string) ([]netip.Addr, error) { + return []netip.Addr{netip.MustParseAddr("8.8.8.8")}, nil + }), + dialContext: func(ctx context.Context, network string, _ string) (net.Conn, error) { + return dialer.DialContext(ctx, network, server.Listener.Addr().String()) + }, + tlsConfig: &tls.Config{MinVersion: tls.VersionTLS12}, + createTemp: os.CreateTemp, + now: time.Now, + clientTimeout: 5 * time.Second, + }) + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: Asset{ID: "1", Name: "dist.zip"}, MaxBytes: 100, + }) + if !errors.Is(err, ErrDownload) { + t.Fatalf("Download() error = %v", err) + } + if strings.Contains(err.Error(), "api.example.test") || strings.Contains(err.Error(), server.URL) { + t.Fatalf("TLS error leaked URL: %v", err) + } +} + +func TestStableErrorClassification(t *testing.T) { + t.Parallel() + now := time.Now() + assetMissing := safeError(errAssetMissing, http.StatusOK, "", "acme/site", "v1", "dist.zip", nil, nil) + if !IsNotFound(assetMissing) || IsRetryable(assetMissing) { + t.Fatalf("asset missing classification failed: %v", assetMissing) + } + metadata404 := safeError(errMetadata, http.StatusNotFound, "", "acme/site", "v1", "dist.zip", nil, nil) + if !IsNotFound(metadata404) || IsRetryable(metadata404) { + t.Fatalf("metadata 404 classification failed: %v", metadata404) + } + digest := safeError(errDigest, http.StatusOK, "", "acme/site", "", "dist.zip", nil, nil) + if !IsDigestError(digest) || IsRetryable(digest) { + t.Fatalf("digest classification failed: %v", digest) + } + for _, retryable := range []error{ + safeError(errMetadata, 0, "", "acme/site", "", "dist.zip", nil, nil), + safeError(errMetadata, http.StatusOK, "", "acme/site", "", "dist.zip", nil, nil), + safeError(errDownload, http.StatusOK, "", "acme/site", "", "dist.zip", nil, nil), + safeError(errMetadata, http.StatusInternalServerError, "", "acme/site", "", "dist.zip", nil, nil), + safeError(errMetadata, http.StatusForbidden, "", "acme/site", "", "dist.zip", nil, &now), + } { + if !IsRetryable(retryable) { + t.Errorf("IsRetryable(%v) = false", retryable) + } + } +} + +func TestDownloadRedirectLimitIsSafe(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { + step, _ := strconv.Atoi(request.URL.Query().Get("step")) + writer.Header().Set("Location", fmt.Sprintf("/repos/acme/site/releases/assets/1?step=%d&token=redirect-secret", step+1)) + writer.WriteHeader(http.StatusFound) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: Asset{ID: "1", Name: "dist.zip"}, MaxBytes: 100, + }) + if err == nil { + t.Fatal("Download() error = nil") + } + if strings.Contains(err.Error(), "redirect-secret") || strings.Contains(err.Error(), "step=") { + t.Fatalf("redirect error leaked Location: %v", err) + } +} + +func TestInvalidRequestDoesNotEchoURLQueryTagOrAsset(t *testing.T) { + t.Parallel() + client := NewClient() + requests := []ResolveRequest{ + {Repository: "https://github.com/acme/site?token=repo-secret", Selector: SelectorLatest, AssetName: "dist.zip"}, + {Repository: "acme/site", Selector: SelectorTag, Tag: "?token=tag-secret", AssetName: "dist.zip"}, + {Repository: "acme/site", Selector: SelectorLatest, AssetName: "../dist.zip?token=asset-secret"}, + } + for _, request := range requests { + _, err := client.Resolve(context.Background(), request) + if err == nil { + t.Fatalf("Resolve(%+v) error = nil", request) + } + for _, secret := range []string{"repo-secret", "tag-secret", "asset-secret", "https://github.com"} { + if strings.Contains(err.Error(), secret) { + t.Fatalf("Resolve(%+v) leaked %q: %v", request, secret, err) + } + } + } +} + +func newTestClient(t *testing.T, rawBaseURL string, customize func(*clientOptions)) *Client { + t.Helper() + parsed, err := url.Parse(rawBaseURL) + if err != nil { + t.Fatal(err) + } + baseURL := "http://api.example.test:" + parsed.Port() + return newMappedTestClient(t, baseURL, customize) +} + +func newMappedTestClient(t *testing.T, baseURL string, customize func(*clientOptions)) *Client { + t.Helper() + resolve := resolverFunc(func(_ context.Context, _ string, _ string) ([]netip.Addr, error) { + return []netip.Addr{netip.MustParseAddr("8.8.8.8")}, nil + }) + dialer := &net.Dialer{Timeout: time.Second} + options := clientOptions{ + baseURL: baseURL, + resolver: resolve, + allowHTTP: true, + tlsConfig: &tls.Config{MinVersion: tls.VersionTLS12}, + dialContext: func(ctx context.Context, network string, address string) (net.Conn, error) { + _, port, splitErr := net.SplitHostPort(address) + if splitErr != nil { + return nil, splitErr + } + return dialer.DialContext(ctx, network, net.JoinHostPort("127.0.0.1", port)) + }, + createTemp: os.CreateTemp, + now: time.Now, + clientTimeout: 5 * time.Second, + } + if customize != nil { + customize(&options) + } + return newClient(options) +} + +func assertHeader(t *testing.T, request *http.Request, name string, expected string) { + t.Helper() + if actual := request.Header.Get(name); actual != expected { + t.Errorf("%s = %q, want %q", name, actual, expected) + } +} + +func TestResponseRetryAtHTTPDate(t *testing.T) { + t.Parallel() + want := time.Date(2026, time.July, 19, 12, 30, 0, 0, time.UTC) + response := &http.Response{Header: make(http.Header)} + response.Header.Set("Retry-After", want.Format(http.TimeFormat)) + if got := responseRetryAt(response, time.Time{}); got == nil || !got.Equal(want) { + t.Fatalf("responseRetryAt() = %v", got) + } +} + +func TestResponseRetryAtRejectsDurationOverflow(t *testing.T) { + t.Parallel() + response := &http.Response{Header: make(http.Header)} + response.Header.Set("Retry-After", strconv.FormatInt(maxRetryAfterSeconds+1, 10)) + if got := responseRetryAt(response, time.Now()); got != nil { + t.Fatalf("responseRetryAt(overflow) = %v", got) + } +} + +func TestSafeETagDropsOversizedOrControlValue(t *testing.T) { + t.Parallel() + if got := safeETag(strings.Repeat("x", 513)); got != "" { + t.Fatalf("safeETag(overlong) = %q", got) + } + if got := safeETag("ok\nsecret"); got != "" { + t.Fatalf("safeETag(control) = %q", got) + } +} + +func TestDownloadSizeLimit(t *testing.T) { + t.Parallel() + server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) { + writer.Header().Set("Content-Length", strconv.Itoa(20)) + _, _ = writer.Write([]byte(strings.Repeat("x", 20))) + })) + defer server.Close() + client := newTestClient(t, server.URL, nil) + _, err := client.Download(context.Background(), DownloadRequest{ + Repository: "acme/site", Asset: Asset{ID: "1", Name: "dist.zip"}, MaxBytes: 10, + }) + if !errors.Is(err, errTooLarge) { + t.Fatalf("Download() error = %v", err) + } +} diff --git a/internal/integration/githubrelease/transport.go b/internal/integration/githubrelease/transport.go new file mode 100644 index 00000000..16a82471 --- /dev/null +++ b/internal/integration/githubrelease/transport.go @@ -0,0 +1,312 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package githubrelease + +import ( + "context" + "crypto/tls" + "errors" + "math" + "net" + "net/http" + "net/netip" + "net/url" + "os" + "strconv" + "strings" + "time" + + "github.com/Rain-kl/Wavelet/pkg/httppool" +) + +const ( + clientTimeout = 10 * time.Minute + dialTimeout = 30 * time.Second + dialKeepAlive = 30 * time.Second + responseHeaderTimeout = 30 * time.Second + maxRedirects = 5 + maxRetryAfterSeconds = math.MaxInt64 / int64(time.Second) +) + +var ( + errBlockedTarget = errors.New("GitHub Release 请求目标不是公网地址") + errResolveTarget = errors.New("GitHub Release 请求目标解析失败") + errRedirectLimit = errors.New("GitHub Release asset 重定向次数过多") + + publicIPv6Prefix = netip.MustParsePrefix("2000::/3") + nonPublicPrefixes = []netip.Prefix{ + netip.MustParsePrefix("0.0.0.0/8"), + netip.MustParsePrefix("10.0.0.0/8"), + netip.MustParsePrefix("100.64.0.0/10"), + netip.MustParsePrefix("127.0.0.0/8"), + netip.MustParsePrefix("169.254.0.0/16"), + netip.MustParsePrefix("172.16.0.0/12"), + netip.MustParsePrefix("192.0.0.0/24"), + netip.MustParsePrefix("192.0.2.0/24"), + netip.MustParsePrefix("192.168.0.0/16"), + netip.MustParsePrefix("198.18.0.0/15"), + netip.MustParsePrefix("198.51.100.0/24"), + netip.MustParsePrefix("203.0.113.0/24"), + netip.MustParsePrefix("224.0.0.0/4"), + netip.MustParsePrefix("240.0.0.0/4"), + netip.MustParsePrefix("::/128"), + netip.MustParsePrefix("::1/128"), + netip.MustParsePrefix("::ffff:0:0/96"), + netip.MustParsePrefix("64:ff9b::/96"), + netip.MustParsePrefix("100::/64"), + netip.MustParsePrefix("2001:db8::/32"), + netip.MustParsePrefix("fc00::/7"), + netip.MustParsePrefix("fe80::/10"), + netip.MustParsePrefix("ff00::/8"), + } +) + +type resolver interface { + LookupNetIP(context.Context, string, string) ([]netip.Addr, error) +} + +type clientOptions struct { + baseURL string + resolver resolver + dialContext func(context.Context, string, string) (net.Conn, error) + tlsConfig *tls.Config + allowHTTP bool + createTemp func(string, string) (*os.File, error) + now func() time.Time + clientTimeout time.Duration +} + +func defaultClientOptions() clientOptions { + dialer := &net.Dialer{Timeout: dialTimeout, KeepAlive: dialKeepAlive} + return clientOptions{ + baseURL: defaultAPIBaseURL, + resolver: net.DefaultResolver, + dialContext: dialer.DialContext, + tlsConfig: &tls.Config{MinVersion: tls.VersionTLS12}, + createTemp: os.CreateTemp, + now: time.Now, + clientTimeout: clientTimeout, + } +} + +func newClient(options clientOptions) *Client { + if options.baseURL == "" { + options.baseURL = defaultAPIBaseURL + } + if options.resolver == nil { + options.resolver = net.DefaultResolver + } + if options.dialContext == nil { + dialer := &net.Dialer{Timeout: dialTimeout, KeepAlive: dialKeepAlive} + options.dialContext = dialer.DialContext + } + if options.tlsConfig == nil { + options.tlsConfig = &tls.Config{MinVersion: tls.VersionTLS12} + } + if options.createTemp == nil { + options.createTemp = os.CreateTemp + } + if options.now == nil { + options.now = time.Now + } + if options.clientTimeout <= 0 { + options.clientTimeout = clientTimeout + } + + secureDial := publicDialer(options.resolver, options.dialContext) + transport := httppool.NewTransport(httppool.TransportOptions{ + Proxy: nil, + DialContext: secureDial, + TLSClientConfig: options.tlsConfig, + ResponseHeaderTimeout: responseHeaderTimeout, + TraceFilter: func(request *http.Request) bool { + return request.URL == nil || request.URL.RawQuery == "" + }, + }) + httpClient := &http.Client{Timeout: options.clientTimeout, Transport: transport} + httpClient.CheckRedirect = func(next *http.Request, previous []*http.Request) error { + if len(previous) > maxRedirects { + return errRedirectLimit + } + if err := validateTarget(next.Context(), next.URL, options.resolver, options.allowHTTP); err != nil { + return err + } + if len(previous) > 0 && !sameHost(previous[len(previous)-1].URL, next.URL) { + stripCrossHostHeaders(next) + } + return nil + } + return &Client{ + httpClient: httpClient, + baseURL: strings.TrimRight(options.baseURL, "/"), + createTemp: options.createTemp, + now: options.now, + } +} + +func applyMetadataHeaders(request *http.Request, etag string) { + request.Header.Set("Accept", metadataAccept) + request.Header.Set("User-Agent", defaultUserAgent) + request.Header.Set("X-GitHub-Api-Version", APIVersion) + if etag = safeETag(etag); etag != "" { + request.Header.Set("If-None-Match", etag) + } +} + +func applyAssetHeaders(request *http.Request) { + request.Header.Set("Accept", assetAccept) + request.Header.Set("Accept-Encoding", "identity") + request.Header.Set("User-Agent", defaultUserAgent) + request.Header.Set("X-GitHub-Api-Version", APIVersion) +} + +func stripCrossHostHeaders(request *http.Request) { + for _, header := range []string{ + "Authorization", + "Cookie", + "Proxy-Authorization", + "Referer", + "If-None-Match", + "If-Modified-Since", + "X-GitHub-Api-Version", + } { + request.Header.Del(header) + } +} + +func sameHost(left *url.URL, right *url.URL) bool { + if left == nil || right == nil { + return false + } + return strings.EqualFold(left.Hostname(), right.Hostname()) && effectivePort(left) == effectivePort(right) +} + +func effectivePort(target *url.URL) string { + if port := target.Port(); port != "" { + return port + } + if strings.EqualFold(target.Scheme, "https") { + return "443" + } + return "80" +} + +func validateTarget(ctx context.Context, target *url.URL, targetResolver resolver, allowHTTP bool) error { + if target == nil || target.User != nil || target.Fragment != "" || target.Opaque != "" || target.Hostname() == "" { + return errBlockedTarget + } + isHTTPS := strings.EqualFold(target.Scheme, "https") + isAllowedHTTP := allowHTTP && strings.EqualFold(target.Scheme, "http") + if !isHTTPS && !isAllowedHTTP { + return errBlockedTarget + } + _, err := resolvePublicIPs(ctx, targetResolver, target.Hostname()) + return err +} + +func publicDialer( + targetResolver resolver, + directDial func(context.Context, string, string) (net.Conn, error), +) func(context.Context, string, string) (net.Conn, error) { + return func(ctx context.Context, network string, address string) (net.Conn, error) { + host, port, err := net.SplitHostPort(address) + if err != nil { + return nil, errResolveTarget + } + addresses, err := resolvePublicIPs(ctx, targetResolver, host) + if err != nil { + return nil, err + } + for _, resolved := range addresses { + if !ipMatchesNetwork(resolved, network) { + continue + } + connection, dialErr := directDial(ctx, network, net.JoinHostPort(resolved.String(), port)) + if dialErr == nil { + return connection, nil + } + } + return nil, errDownload + } +} + +func resolvePublicIPs(ctx context.Context, targetResolver resolver, host string) ([]netip.Addr, error) { + if strings.Contains(host, "%") { + return nil, errBlockedTarget + } + if literal, err := netip.ParseAddr(host); err == nil { + if !isPublicIP(literal) { + return nil, errBlockedTarget + } + return []netip.Addr{literal}, nil + } + if targetResolver == nil { + return nil, errResolveTarget + } + addresses, err := targetResolver.LookupNetIP(ctx, "ip", host) + if err != nil || len(addresses) == 0 { + return nil, errResolveTarget + } + for _, address := range addresses { + if !isPublicIP(address) { + return nil, errBlockedTarget + } + } + return addresses, nil +} + +func isPublicIP(address netip.Addr) bool { + if !address.IsValid() || address.Zone() != "" { + return false + } + address = address.Unmap() + if !address.IsGlobalUnicast() { + return false + } + if address.Is6() && !publicIPv6Prefix.Contains(address) { + return false + } + for _, prefix := range nonPublicPrefixes { + if prefix.Contains(address) { + return false + } + } + return true +} + +func ipMatchesNetwork(address netip.Addr, network string) bool { + switch network { + case "tcp4": + return address.Unmap().Is4() + case "tcp6": + return address.Unmap().Is6() + default: + return true + } +} + +func responseRetryAt(response *http.Response, now time.Time) *time.Time { + if response == nil { + return nil + } + if retryAfter := strings.TrimSpace(response.Header.Get("Retry-After")); retryAfter != "" { + if seconds, err := strconv.ParseInt(retryAfter, 10, 64); err == nil && seconds >= 0 && seconds <= maxRetryAfterSeconds { + retryAt := now.Add(time.Duration(seconds) * time.Second) + return &retryAt + } + if retryAt, err := http.ParseTime(retryAfter); err == nil { + retryAt = retryAt.UTC() + return &retryAt + } + } + if strings.TrimSpace(response.Header.Get("X-RateLimit-Remaining")) != "0" { + return nil + } + reset, err := strconv.ParseInt(strings.TrimSpace(response.Header.Get("X-RateLimit-Reset")), 10, 64) + if err != nil || reset <= 0 { + return nil + } + retryAt := time.Unix(reset, 0).UTC() + return &retryAt +} From 848884d8cdeb364b6501e08bffed7f7a33ce264c Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 19:08:43 +0800 Subject: [PATCH 5/8] =?UTF-8?q?fix(pages):=20=E5=A2=9E=E5=8A=A0=E9=83=A8?= =?UTF-8?q?=E7=BD=B2=E5=8C=85=E5=AD=A4=E5=84=BF=E8=A1=A5=E5=81=BF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按项目、来源、运行时与上传记录锁序补偿异常中断遗留的部署包。\n同时隐藏并保护系统内部排程，避免通用任务管理入口修改 scanner。 --- docs/docs.go | 8 +- docs/swagger.json | 8 +- docs/swagger.yaml | 6 +- internal/apps/admin/task/routers.go | 22 +- internal/apps/admin/task/routers_test.go | 103 +++++ .../openflare/pages/source_orphan_cleanup.go | 328 +++++++++++++++ .../pages/source_orphan_cleanup_test.go | 381 ++++++++++++++++++ internal/model/openflare_pages_cleanup.go | 75 ++++ .../model/openflare_pages_cleanup_test.go | 193 +++++++++ 9 files changed, 1119 insertions(+), 5 deletions(-) create mode 100644 internal/apps/openflare/pages/source_orphan_cleanup.go create mode 100644 internal/apps/openflare/pages/source_orphan_cleanup_test.go create mode 100644 internal/model/openflare_pages_cleanup.go create mode 100644 internal/model/openflare_pages_cleanup_test.go diff --git a/docs/docs.go b/docs/docs.go index 2f5ca21a..d37cf169 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -2595,7 +2595,7 @@ const docTemplate = `{ "SessionCookie": [] } ], - "description": "返回系统所有的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态，需要管理员权限", + "description": "返回管理员可管理的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态；系统内部排程不会暴露，需要管理员权限", "produces": [ "application/json" ], @@ -2860,6 +2860,12 @@ const docTemplate = `{ "$ref": "#/definitions/response.Any" } }, + "404": { + "description": "定时任务不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, "500": { "description": "删除定时任务失败", "schema": { diff --git a/docs/swagger.json b/docs/swagger.json index cc0f02c5..e02e9b98 100644 --- a/docs/swagger.json +++ b/docs/swagger.json @@ -2588,7 +2588,7 @@ "SessionCookie": [] } ], - "description": "返回系统所有的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态，需要管理员权限", + "description": "返回管理员可管理的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态；系统内部排程不会暴露，需要管理员权限", "produces": [ "application/json" ], @@ -2853,6 +2853,12 @@ "$ref": "#/definitions/response.Any" } }, + "404": { + "description": "定时任务不存在", + "schema": { + "$ref": "#/definitions/response.Any" + } + }, "500": { "description": "删除定时任务失败", "schema": { diff --git a/docs/swagger.yaml b/docs/swagger.yaml index edce34e0..3c9e9cdf 100644 --- a/docs/swagger.yaml +++ b/docs/swagger.yaml @@ -5717,7 +5717,7 @@ paths: - admin /api/v1/admin/tasks/schedules: get: - description: 返回系统所有的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态，需要管理员权限 + description: 返回管理员可管理的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态；系统内部排程不会暴露，需要管理员权限 produces: - application/json responses: @@ -5822,6 +5822,10 @@ paths: description: 无管理员权限 schema: $ref: '#/definitions/response.Any' + "404": + description: 定时任务不存在 + schema: + $ref: '#/definitions/response.Any' "500": description: 删除定时任务失败 schema: diff --git a/internal/apps/admin/task/routers.go b/internal/apps/admin/task/routers.go index 594f1eb2..78d391a5 100644 --- a/internal/apps/admin/task/routers.go +++ b/internal/apps/admin/task/routers.go @@ -202,7 +202,7 @@ func RetryTask(c *gin.Context) { // ListSchedules 获取定时任务列表 // @Summary 获取定时任务列表 -// @Description 返回系统所有的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态，需要管理员权限 +// @Description 返回管理员可管理的定时任务配置列表，包括名称、关联的异步任务类型、Cron 表达式和启用状态；系统内部排程不会暴露，需要管理员权限 // @Tags admin // @Produce json // @Security SessionCookie @@ -216,7 +216,15 @@ func ListSchedules(c *gin.Context) { response.AbortInternal(c, err.Error()) return } - c.JSON(http.StatusOK, response.OK(schedules)) + visible := make([]model.Schedule, 0, len(schedules)) + for _, schedule := range schedules { + meta := task.GetTaskMeta(schedule.TaskType) + if meta != nil && meta.InternalOnly { + continue + } + visible = append(visible, schedule) + } + c.JSON(http.StatusOK, response.OK(visible)) } // CreateScheduleRequest 创建定时任务请求 @@ -405,6 +413,7 @@ func getAdminTaskMeta(taskType string) *task.TaskMeta { // @Failure 400 {object} response.Any "参数错误" // @Failure 401 {object} response.Any "未登录" // @Failure 403 {object} response.Any "无管理员权限" +// @Failure 404 {object} response.Any "定时任务不存在" // @Failure 500 {object} response.Any "删除定时任务失败" // @Router /api/v1/admin/tasks/schedules/{id} [delete] func DeleteSchedule(c *gin.Context) { @@ -413,6 +422,15 @@ func DeleteSchedule(c *gin.Context) { response.AbortBadRequest(c, "无效的定时任务ID") return } + schedule, err := model.GetScheduleByID(c.Request.Context(), id) + if err != nil { + response.AbortNotFound(c, ScheduleNotFound) + return + } + if meta := task.GetTaskMeta(schedule.TaskType); meta != nil && meta.InternalOnly { + response.AbortBadRequest(c, InvalidTaskType) + return + } if err := model.DeleteSchedule(c.Request.Context(), id); err != nil { response.AbortInternal(c, fmt.Sprintf("%s: %v", ScheduleDeleteFailed, err)) diff --git a/internal/apps/admin/task/routers_test.go b/internal/apps/admin/task/routers_test.go index 694861e7..3cd63fb9 100644 --- a/internal/apps/admin/task/routers_test.go +++ b/internal/apps/admin/task/routers_test.go @@ -75,8 +75,10 @@ func setupTestRouter(authUser *model.User) *gin.Engine { adminGroup.GET("/tasks/executions", ListTaskExecutions) adminGroup.GET("/tasks/executions/:id", GetTaskExecution) adminGroup.POST("/tasks/executions/:id/retry", RetryTask) + adminGroup.GET("/tasks/schedules", ListSchedules) adminGroup.POST("/tasks/schedules", CreateSchedule) adminGroup.PUT("/tasks/schedules/:id", UpdateSchedule) + adminGroup.DELETE("/tasks/schedules/:id", DeleteSchedule) return r } @@ -137,6 +139,39 @@ func TestInternalOnlyTaskAdminBoundaries(t *testing.T) { router := setupTestRouter(adminUser) ctx := context.Background() + t.Run("list hides internal-only schedule", func(t *testing.T) { + internalSchedule := &model.Schedule{ + Name: "隐藏的系统内部排程", + TaskType: testInternalOnlyTaskType, + Cron: "*/5 * * * *", + Payload: "{}", + IsActive: true, + } + publicSchedule := &model.Schedule{ + Name: "可见的公开排程", + TaskType: uploadtask.TaskTypeSystemCleanup, + Cron: "0 * * * *", + Payload: "{}", + IsActive: true, + } + require.NoError(t, model.CreateSchedule(ctx, internalSchedule)) + require.NoError(t, model.CreateSchedule(ctx, publicSchedule)) + + req := httptest.NewRequest(http.MethodGet, "/api/v1/admin/tasks/schedules", nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + var resp response.Any + require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp)) + data, err := json.Marshal(resp.Data) + require.NoError(t, err) + var schedules []model.Schedule + require.NoError(t, json.Unmarshal(data, &schedules)) + assert.NotContains(t, scheduleIDs(schedules), internalSchedule.ID) + assert.Contains(t, scheduleIDs(schedules), publicSchedule.ID) + }) + t.Run("dispatch rejects internal-only task", func(t *testing.T) { body, err := json.Marshal(DispatchTaskRequest{TaskType: testInternalOnlyTaskType}) require.NoError(t, err) @@ -239,6 +274,74 @@ func TestInternalOnlyTaskAdminBoundaries(t *testing.T) { assert.Equal(t, "公开排程", unchanged.Name) assert.Equal(t, uploadtask.TaskTypeSystemCleanup, unchanged.TaskType) }) + + t.Run("delete rejects internal-only schedule", func(t *testing.T) { + schedule := &model.Schedule{ + Name: "不可删除的系统内部排程", + TaskType: testInternalOnlyTaskType, + Cron: "*/5 * * * *", + IsActive: true, + } + require.NoError(t, model.CreateSchedule(ctx, schedule)) + req := httptest.NewRequest( + http.MethodDelete, + fmt.Sprintf("/api/v1/admin/tasks/schedules/%d", schedule.ID), + nil, + ) + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusBadRequest, w.Code) + var resp response.Any + require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp)) + assert.Equal(t, InvalidTaskType, resp.ErrorMsg) + preserved, err := model.GetScheduleByID(ctx, schedule.ID) + require.NoError(t, err) + assert.Equal(t, testInternalOnlyTaskType, preserved.TaskType) + }) + + t.Run("delete missing schedule returns not found", func(t *testing.T) { + req := httptest.NewRequest(http.MethodDelete, "/api/v1/admin/tasks/schedules/999999", nil) + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusNotFound, w.Code) + var resp response.Any + require.NoError(t, json.Unmarshal(w.Body.Bytes(), &resp)) + assert.Equal(t, ScheduleNotFound, resp.ErrorMsg) + }) + + t.Run("delete public schedule remains allowed", func(t *testing.T) { + schedule := &model.Schedule{ + Name: "可删除的公开排程", + TaskType: uploadtask.TaskTypeSystemCleanup, + Cron: "0 * * * *", + IsActive: false, + } + require.NoError(t, model.CreateSchedule(ctx, schedule)) + req := httptest.NewRequest( + http.MethodDelete, + fmt.Sprintf("/api/v1/admin/tasks/schedules/%d", schedule.ID), + nil, + ) + w := httptest.NewRecorder() + + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + _, err := model.GetScheduleByID(ctx, schedule.ID) + assert.Error(t, err) + }) +} + +func scheduleIDs(schedules []model.Schedule) []uint64 { + ids := make([]uint64, 0, len(schedules)) + for _, schedule := range schedules { + ids = append(ids, schedule.ID) + } + return ids } func TestDispatchTask(t *testing.T) { diff --git a/internal/apps/openflare/pages/source_orphan_cleanup.go b/internal/apps/openflare/pages/source_orphan_cleanup.go new file mode 100644 index 00000000..6f80380f --- /dev/null +++ b/internal/apps/openflare/pages/source_orphan_cleanup.go @@ -0,0 +1,328 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "errors" + "fmt" + "strconv" + "time" + + "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/repository" + "github.com/Rain-kl/Wavelet/pkg/logger" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +const pagesOrphanUploadIsolation = 2 * time.Hour + +// PagesOrphanCleanupSummary describes one bounded delayed compensation pass. +// Every candidate is counted exactly once in one outcome field. +// +//nolint:revive // Keep the domain-qualified exported name for scanner/task result clarity. +type PagesOrphanCleanupSummary struct { + Candidates int `json:"candidates"` + Reconciled int `json:"reconciled"` + Referenced int `json:"referenced"` + LeaseBusy int `json:"lease_busy"` + InvalidMarker int `json:"invalid_marker"` + Skipped int `json:"skipped"` + Failed int `json:"failed"` +} + +type pagesOrphanMarker struct { + ProjectID uint + SourceID *uint +} + +type pagesOrphanCleanupOutcome uint8 + +const ( + pagesOrphanCleanupSkipped pagesOrphanCleanupOutcome = iota + pagesOrphanCleanupReconciled + pagesOrphanCleanupReferenced + pagesOrphanCleanupLeaseBusy + pagesOrphanCleanupInvalidMarker +) + +// ReconcilePagesOrphanUploads performs one bounded delayed compensation pass. +// Individual candidate failures are counted and logged so they do not prevent +// the scanner from continuing with source checks. +func ReconcilePagesOrphanUploads( + ctx context.Context, + now time.Time, +) (PagesOrphanCleanupSummary, error) { + if now.IsZero() { + now = time.Now() + } + cutoff := now.UTC().Add(-pagesOrphanUploadIsolation) + systemUser := repository.GetSystemUser(ctx) + candidates, err := model.ListPagesOrphanUploadCandidates(ctx, model.PagesOrphanUploadCandidateQuery{ + SystemUserID: systemUser.ID, + UploadType: upload.ReservedPagesDeploymentType, + Marker: pagesIngestMarkerV2, + CreatedBefore: cutoff, + }) + if err != nil { + return PagesOrphanCleanupSummary{}, err + } + + summary := PagesOrphanCleanupSummary{Candidates: len(candidates)} + for index := range candidates { + if err := ctx.Err(); err != nil { + return summary, err + } + candidate := &candidates[index] + marker, err := parsePagesOrphanMarker(candidate.Metadata) + if err != nil { + summary.InvalidMarker++ + logger.WarnF(ctx, "[PagesSource] orphan upload marker invalid: upload_id=%d error=%v", candidate.ID, err) + continue + } + + outcome, err := reconcilePagesOrphanUploadCandidate( + ctx, + candidate, + marker, + systemUser.ID, + cutoff, + ) + if err != nil { + summary.Failed++ + logger.WarnF(ctx, "[PagesSource] orphan upload reconciliation failed: upload_id=%d error=%v", candidate.ID, err) + continue + } + summary.add(outcome) + } + return summary, nil +} + +func (summary *PagesOrphanCleanupSummary) add(outcome pagesOrphanCleanupOutcome) { + switch outcome { + case pagesOrphanCleanupReconciled: + summary.Reconciled++ + case pagesOrphanCleanupReferenced: + summary.Referenced++ + case pagesOrphanCleanupLeaseBusy: + summary.LeaseBusy++ + case pagesOrphanCleanupInvalidMarker: + summary.InvalidMarker++ + default: + summary.Skipped++ + } +} + +func reconcilePagesOrphanUploadCandidate( + ctx context.Context, + candidate *model.Upload, + marker pagesOrphanMarker, + systemUserID uint64, + cutoff time.Time, +) (pagesOrphanCleanupOutcome, error) { + if candidate == nil || candidate.ID == 0 { + return pagesOrphanCleanupSkipped, nil + } + + outcome := pagesOrphanCleanupSkipped + uploadLocked := false + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + scopeOutcome, proceed, err := lockPagesOrphanCleanupScope(ctx, tx, candidate.ID, marker) + if err != nil { + return err + } + if !proceed { + outcome = scopeOutcome + return nil + } + lockedOutcome, locked, err := reconcileLockedPagesOrphanUpload( + ctx, + tx, + candidate.ID, + marker, + systemUserID, + cutoff, + ) + if err != nil { + return err + } + outcome = lockedOutcome + uploadLocked = locked + return nil + }) + if err != nil { + return pagesOrphanCleanupSkipped, err + } + if uploadLocked { + // Also heal a prior post-commit cache invalidation interruption when the + // status transition was an idempotent no-op. + upload.InvalidateUploadMetaCache(ctx, candidate.ID) + } + return outcome, nil +} + +func lockPagesOrphanCleanupScope( + ctx context.Context, + tx *gorm.DB, + uploadID uint64, + marker pagesOrphanMarker, +) (pagesOrphanCleanupOutcome, bool, error) { + var project model.PagesProject + if _, err := lockOptionalPagesCleanupRecord(tx, &project, "id = ?", marker.ProjectID); err != nil { + return pagesOrphanCleanupSkipped, false, err + } + if marker.SourceID == nil { + return pagesOrphanCleanupSkipped, true, nil + } + + var source model.PagesProjectSource + sourceExists, err := lockOptionalPagesCleanupRecord(tx, &source, "id = ?", *marker.SourceID) + if err != nil { + return pagesOrphanCleanupSkipped, false, err + } + if !sourceExists { + return pagesOrphanCleanupSkipped, true, nil + } + if source.ProjectID != marker.ProjectID { + logger.WarnF(ctx, + "[PagesSource] orphan upload source ownership mismatch: upload_id=%d project_id=%d source_id=%d source_project_id=%d", + uploadID, + marker.ProjectID, + *marker.SourceID, + source.ProjectID, + ) + return pagesOrphanCleanupInvalidMarker, false, nil + } + + var runtime model.PagesProjectSourceRuntime + runtimeExists, err := lockOptionalPagesCleanupRecord(tx, &runtime, "source_id = ?", source.ID) + if err != nil { + return pagesOrphanCleanupSkipped, false, err + } + // Read the real clock only after obtaining the runtime row lock. The scanner + // snapshot time is only an isolation cutoff and may be stale after lock wait. + leaseCheckedAt := time.Now() + if runtimeExists && runtime.LeaseExpiresAt != nil && runtime.LeaseExpiresAt.After(leaseCheckedAt) { + return pagesOrphanCleanupLeaseBusy, false, nil + } + return pagesOrphanCleanupSkipped, true, nil +} + +func reconcileLockedPagesOrphanUpload( + ctx context.Context, + tx *gorm.DB, + uploadID uint64, + marker pagesOrphanMarker, + systemUserID uint64, + cutoff time.Time, +) (pagesOrphanCleanupOutcome, bool, error) { + var lockedUpload model.Upload + found, err := lockOptionalPagesCleanupRecord(tx, &lockedUpload, "id = ?", uploadID) + if err != nil || !found { + return pagesOrphanCleanupSkipped, false, err + } + + lockedMarker, err := parsePagesOrphanMarker(lockedUpload.Metadata) + if err != nil { + logger.WarnF(ctx, "[PagesSource] orphan upload marker changed or invalid: upload_id=%d error=%v", uploadID, err) + return pagesOrphanCleanupInvalidMarker, true, nil + } + if lockedUpload.Status != model.UploadStatusUsed || + lockedUpload.UserID != systemUserID || + lockedUpload.Type != upload.ReservedPagesDeploymentType || + !lockedUpload.CreatedAt.Before(cutoff) { + return pagesOrphanCleanupSkipped, true, nil + } + if lockedMarker.ProjectID != marker.ProjectID || !sameOptionalPagesSourceID(lockedMarker.SourceID, marker.SourceID) { + logger.WarnF(ctx, "[PagesSource] orphan upload marker changed during reconciliation: upload_id=%d", uploadID) + return pagesOrphanCleanupInvalidMarker, true, nil + } + + var references int64 + if err := tx.Model(&model.PagesDeployment{}). + Where("upload_id = ?", lockedUpload.ID). + Count(&references).Error; err != nil { + return pagesOrphanCleanupSkipped, true, err + } + if references > 0 { + return pagesOrphanCleanupReferenced, true, nil + } + + transitioned, err := upload.RemoveLockedTx(tx, &lockedUpload) + if err != nil { + return pagesOrphanCleanupSkipped, true, err + } + if transitioned { + return pagesOrphanCleanupReconciled, true, nil + } + return pagesOrphanCleanupSkipped, true, nil +} + +func lockOptionalPagesCleanupRecord( + tx *gorm.DB, + value any, + query string, + args ...any, +) (bool, error) { + err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}). + Where(query, args...). + First(value).Error + if err == nil { + return true, nil + } + if errors.Is(err, gorm.ErrRecordNotFound) { + return false, nil + } + return false, err +} + +func parsePagesOrphanMarker(metadata model.UploadMetadata) (pagesOrphanMarker, error) { + if metadata.Extra == nil { + return pagesOrphanMarker{}, errors.New("pages marker metadata missing") + } + marker, ok := metadata.Extra[pagesIngestMarkerKey].(string) + if !ok || marker != pagesIngestMarkerV2 { + return pagesOrphanMarker{}, errors.New("pages marker version invalid") + } + projectID, err := parsePagesOrphanMetadataID(metadata.Extra, pagesProjectIDMetadataKey) + if err != nil { + return pagesOrphanMarker{}, err + } + result := pagesOrphanMarker{ProjectID: projectID} + if _, exists := metadata.Extra[pagesSourceIDMetadataKey]; exists { + sourceID, err := parsePagesOrphanMetadataID(metadata.Extra, pagesSourceIDMetadataKey) + if err != nil { + return pagesOrphanMarker{}, err + } + result.SourceID = &sourceID + } + return result, nil +} + +func parsePagesOrphanMetadataID(extra map[string]any, key string) (uint, error) { + raw, exists := extra[key] + if !exists { + return 0, fmt.Errorf("pages marker %s missing", key) + } + value, ok := raw.(string) + if !ok || value == "" { + return 0, fmt.Errorf("pages marker %s must be a decimal string", key) + } + parsed, err := strconv.ParseUint(value, 10, 64) + maxModelID := uint64(^uint(0) >> 1) + if err != nil || parsed == 0 || parsed > maxModelID || strconv.FormatUint(parsed, 10) != value { + return 0, fmt.Errorf("pages marker %s is not a canonical non-zero decimal ID", key) + } + return uint(parsed), nil +} + +func sameOptionalPagesSourceID(left, right *uint) bool { + if left == nil || right == nil { + return left == nil && right == nil + } + return *left == *right +} diff --git a/internal/apps/openflare/pages/source_orphan_cleanup_test.go b/internal/apps/openflare/pages/source_orphan_cleanup_test.go new file mode 100644 index 00000000..2919c299 --- /dev/null +++ b/internal/apps/openflare/pages/source_orphan_cleanup_test.go @@ -0,0 +1,381 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "errors" + "strconv" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +func TestParsePagesOrphanMarker(t *testing.T) { + tests := []struct { + name string + extra map[string]any + wantProject uint + wantSource uint + wantSourceOK bool + wantErr bool + }{ + { + name: "manual source marker", + extra: map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: "12", + }, + wantProject: 12, + }, + { + name: "persistent source marker", + extra: map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: "12", + pagesSourceIDMetadataKey: "34", + }, + wantProject: 12, + wantSource: 34, + wantSourceOK: true, + }, + { + name: "project ID must be canonical decimal", + extra: map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: "012", + }, + wantErr: true, + }, + { + name: "source ID must be a string", + extra: map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: "12", + pagesSourceIDMetadataKey: float64(34), + }, + wantErr: true, + }, + { + name: "zero ID rejected", + extra: map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: "0", + }, + wantErr: true, + }, + { + name: "wrong marker version rejected", + extra: map[string]any{ + pagesIngestMarkerKey: "pages_deployment_v1", + pagesProjectIDMetadataKey: "12", + }, + wantErr: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + got, err := parsePagesOrphanMarker(model.UploadMetadata{Extra: test.extra}) + if gotErr := err != nil; gotErr != test.wantErr { + t.Fatalf("parsePagesOrphanMarker(%v) error = %v, want error presence = %t", test.extra, err, test.wantErr) + } + if test.wantErr { + return + } + if got.ProjectID != test.wantProject { + t.Errorf("parsePagesOrphanMarker(%v).ProjectID = %d, want %d", test.extra, got.ProjectID, test.wantProject) + } + if gotSourceOK := got.SourceID != nil; gotSourceOK != test.wantSourceOK { + t.Fatalf("parsePagesOrphanMarker(%v).SourceID presence = %t, want %t", test.extra, gotSourceOK, test.wantSourceOK) + } + if got.SourceID != nil && *got.SourceID != test.wantSource { + t.Errorf("parsePagesOrphanMarker(%v).SourceID = %d, want %d", test.extra, *got.SourceID, test.wantSource) + } + }) + } +} + +func TestReconcilePagesOrphanUploadsDeletesEligibleUploadOnce(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + now := time.Now().UTC() + project := createPagesOrphanProject(t, ctx, "eligible-orphan") + candidate := createPagesOrphanUpload(t, ctx, now.Add(-3*time.Hour), project.ID, nil) + if err := upload.RebuildUploadStats(ctx); err != nil { + t.Fatalf("RebuildUploadStats() error = %v, want nil", err) + } + + summary, err := ReconcilePagesOrphanUploads(ctx, now) + if err != nil { + t.Fatalf("ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if summary.Candidates != 1 || summary.Reconciled != 1 || cleanupOutcomeTotal(summary) != 1 { + t.Errorf("ReconcilePagesOrphanUploads() summary = %+v, want one reconciled candidate", summary) + } + assertPagesCleanupUploadStatus(t, ctx, candidate.ID, model.UploadStatusDeleted) + assertPagesCleanupTotalStat(t, ctx, 0) + + second, err := ReconcilePagesOrphanUploads(ctx, now.Add(time.Minute)) + if err != nil { + t.Fatalf("second ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if second.Candidates != 0 || cleanupOutcomeTotal(second) != 0 { + t.Errorf("second ReconcilePagesOrphanUploads() summary = %+v, want empty", second) + } + assertPagesCleanupTotalStat(t, ctx, 0) +} + +func TestReconcilePagesOrphanUploadsAllowsDeletedProjectAndSource(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + now := time.Now().UTC() + missingSourceID := uint(9876) + candidate := createPagesOrphanUpload(t, ctx, now.Add(-3*time.Hour), 8765, &missingSourceID) + + summary, err := ReconcilePagesOrphanUploads(ctx, now) + if err != nil { + t.Fatalf("ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if summary.Candidates != 1 || summary.Reconciled != 1 || cleanupOutcomeTotal(summary) != 1 { + t.Errorf("ReconcilePagesOrphanUploads() summary = %+v, want deleted project/source treated as one orphan", summary) + } + assertPagesCleanupUploadStatus(t, ctx, candidate.ID, model.UploadStatusDeleted) +} + +func TestReconcilePagesOrphanUploadsSkipsBusyLeaseAndSourceMismatch(t *testing.T) { + t.Run("unexpired source lease", func(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + realNow := time.Now().UTC() + // A deliberately future scanner snapshot proves lease freshness uses the + // real clock after the runtime lock, not this isolation-cutoff input. + scannerNow := realNow.Add(24 * time.Hour) + project := createPagesOrphanProject(t, ctx, "busy-orphan") + source := createPagesOrphanSource(t, ctx, project.ID) + future := realNow.Add(time.Hour) + if err := db.DB(ctx).Create(&model.PagesProjectSourceRuntime{ + SourceID: source.ID, + LeaseToken: "busy-worker", + LeaseExpiresAt: &future, + }).Error; err != nil { + t.Fatalf("create busy source runtime error = %v, want nil", err) + } + candidate := createPagesOrphanUpload(t, ctx, realNow.Add(-3*time.Hour), project.ID, &source.ID) + + summary, err := ReconcilePagesOrphanUploads(ctx, scannerNow) + if err != nil { + t.Fatalf("ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if summary.LeaseBusy != 1 || cleanupOutcomeTotal(summary) != 1 { + t.Errorf("ReconcilePagesOrphanUploads() summary = %+v, want one lease-busy candidate", summary) + } + assertPagesCleanupUploadStatus(t, ctx, candidate.ID, model.UploadStatusUsed) + }) + + t.Run("source belongs to another project", func(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + now := time.Now().UTC() + markerProject := createPagesOrphanProject(t, ctx, "marker-project") + actualProject := createPagesOrphanProject(t, ctx, "actual-project") + source := createPagesOrphanSource(t, ctx, actualProject.ID) + candidate := createPagesOrphanUpload(t, ctx, now.Add(-3*time.Hour), markerProject.ID, &source.ID) + + summary, err := ReconcilePagesOrphanUploads(ctx, now) + if err != nil { + t.Fatalf("ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if summary.InvalidMarker != 1 || cleanupOutcomeTotal(summary) != 1 { + t.Errorf("ReconcilePagesOrphanUploads() summary = %+v, want one ownership mismatch", summary) + } + assertPagesCleanupUploadStatus(t, ctx, candidate.ID, model.UploadStatusUsed) + }) +} + +func TestReconcilePagesOrphanUploadsRejectsMalformedMarker(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + now := time.Now().UTC() + candidate := createPagesOrphanUpload(t, ctx, now.Add(-3*time.Hour), 1, nil) + metadata := candidate.Metadata + metadata.Extra[pagesProjectIDMetadataKey] = "01" + candidate.Metadata = metadata + if err := db.DB(ctx).Save(candidate).Error; err != nil { + t.Fatalf("seed malformed candidate marker error = %v, want nil", err) + } + + summary, err := ReconcilePagesOrphanUploads(ctx, now) + if err != nil { + t.Fatalf("ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if summary.InvalidMarker != 1 || cleanupOutcomeTotal(summary) != 1 { + t.Errorf("ReconcilePagesOrphanUploads() summary = %+v, want one invalid marker", summary) + } + assertPagesCleanupUploadStatus(t, ctx, candidate.ID, model.UploadStatusUsed) +} + +func TestPagesOrphanCleanupAndDeploymentCommitInterleavings(t *testing.T) { + t.Run("deployment reference commits first", func(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + now := time.Now().UTC() + project := createPagesOrphanProject(t, ctx, "deployment-first") + candidate := createPagesOrphanUpload(t, ctx, now.Add(-3*time.Hour), project.ID, nil) + marker, err := parsePagesOrphanMarker(candidate.Metadata) + if err != nil { + t.Fatalf("parsePagesOrphanMarker() error = %v, want nil", err) + } + if err := db.DB(ctx).Create(&model.PagesDeployment{ + ProjectID: project.ID, + DeploymentNumber: 1, + Checksum: "deployment-first", + Status: model.PagesDeploymentStatusUploaded, + UploadID: candidate.ID, + }).Error; err != nil { + t.Fatalf("create deployment reference error = %v, want nil", err) + } + + outcome, err := reconcilePagesOrphanUploadCandidate(ctx, candidate, marker, 999, now.Add(-2*time.Hour)) + if err != nil { + t.Fatalf("reconcilePagesOrphanUploadCandidate() error = %v, want nil", err) + } + if outcome != pagesOrphanCleanupReferenced { + t.Errorf("reconcilePagesOrphanUploadCandidate() outcome = %d, want %d", outcome, pagesOrphanCleanupReferenced) + } + assertPagesCleanupUploadStatus(t, ctx, candidate.ID, model.UploadStatusUsed) + }) + + t.Run("cleanup commits first", func(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + now := time.Now().UTC() + project := createPagesOrphanProject(t, ctx, "cleanup-first") + candidate := createPagesOrphanUpload(t, ctx, now.Add(-3*time.Hour), project.ID, nil) + + summary, err := ReconcilePagesOrphanUploads(ctx, now) + if err != nil { + t.Fatalf("ReconcilePagesOrphanUploads() error = %v, want nil", err) + } + if summary.Reconciled != 1 { + t.Fatalf("ReconcilePagesOrphanUploads() summary = %+v, want one reconciled candidate", summary) + } + + target := &model.PagesDeployment{ProjectID: project.ID, UploadID: candidate.ID} + err = db.DB(ctx).Transaction(func(tx *gorm.DB) error { + var lockedProject model.PagesProject + if err := tx.Clauses(clause.Locking{Strength: pagesRowLockStrength}).First(&lockedProject, project.ID).Error; err != nil { + return err + } + return lockSourceDeploymentUploadsTx(tx, target, upload.IngestResult{}, false) + }) + if !errors.Is(err, errSourceFinalFence) { + t.Errorf("final deployment upload lock after cleanup error = %v, want %v", err, errSourceFinalFence) + } + var references int64 + if err := db.DB(ctx).Model(&model.PagesDeployment{}).Where("upload_id = ?", candidate.ID).Count(&references).Error; err != nil { + t.Fatalf("count deployment references error = %v, want nil", err) + } + if references != 0 { + t.Errorf("deployment references after cleanup-first interleaving = %d, want 0", references) + } + }) +} + +func cleanupOutcomeTotal(summary PagesOrphanCleanupSummary) int { + return summary.Reconciled + summary.Referenced + summary.LeaseBusy + summary.InvalidMarker + summary.Skipped + summary.Failed +} + +func createPagesOrphanProject(t *testing.T, ctx context.Context, slug string) *model.PagesProject { + t.Helper() + project := &model.PagesProject{Name: slug, Slug: slug, Enabled: true} + if err := db.DB(ctx).Create(project).Error; err != nil { + t.Fatalf("create Pages orphan project %q error = %v, want nil", slug, err) + } + return project +} + +func createPagesOrphanSource(t *testing.T, ctx context.Context, projectID uint) *model.PagesProjectSource { + t.Helper() + source := &model.PagesProjectSource{ + ProjectID: projectID, + SourceType: PagesSourceTypeRemoteURL, + ConfigVersion: 1, + SourceIdentity: "orphan-source-identity", + } + if err := db.DB(ctx).Create(source).Error; err != nil { + t.Fatalf("create Pages orphan source for project %d error = %v, want nil", projectID, err) + } + return source +} + +func createPagesOrphanUpload( + t *testing.T, + ctx context.Context, + createdAt time.Time, + projectID uint, + sourceID *uint, +) *model.Upload { + t.Helper() + extra := map[string]any{ + pagesIngestMarkerKey: pagesIngestMarkerV2, + pagesProjectIDMetadataKey: strconv.FormatUint(uint64(projectID), 10), + } + if sourceID != nil { + extra[pagesSourceIDMetadataKey] = strconv.FormatUint(uint64(*sourceID), 10) + } + candidate := &model.Upload{ + UserID: 999, + FileName: "site.zip", + FilePath: "pages/orphan-site.zip", + FileSize: 64, + MimeType: "application/zip", + Extension: "zip", + Hash: "orphan-checksum", + Type: upload.ReservedPagesDeploymentType, + Status: model.UploadStatusUsed, + AccessMode: 0, + Metadata: model.UploadMetadata{Extra: extra}, + CreatedAt: createdAt, + UpdatedAt: createdAt, + } + if err := db.DB(ctx).Create(candidate).Error; err != nil { + t.Fatalf("create Pages orphan upload error = %v, want nil", err) + } + return candidate +} + +func assertPagesCleanupUploadStatus(t *testing.T, ctx context.Context, uploadID uint64, want model.UploadStatus) { + t.Helper() + var got model.Upload + if err := db.DB(ctx).First(&got, uploadID).Error; err != nil { + t.Fatalf("load upload %d error = %v, want nil", uploadID, err) + } + if got.Status != want { + t.Errorf("upload %d status = %q, want %q", uploadID, got.Status, want) + } +} + +func assertPagesCleanupTotalStat(t *testing.T, ctx context.Context, want int64) { + t.Helper() + var stat model.UploadStat + if err := db.DB(ctx).Where("dimension = ? AND stat_key = ?", model.UploadStatDimensionTotal, "").First(&stat).Error; err != nil { + t.Fatalf("load total upload stat error = %v, want nil", err) + } + if stat.FileCount != want { + t.Errorf("total upload stat FileCount = %d, want %d", stat.FileCount, want) + } +} diff --git a/internal/model/openflare_pages_cleanup.go b/internal/model/openflare_pages_cleanup.go new file mode 100644 index 00000000..a740a54e --- /dev/null +++ b/internal/model/openflare_pages_cleanup.go @@ -0,0 +1,75 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package model + +import ( + "context" + "errors" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" +) + +const ( + // PagesOrphanUploadCandidateLimit bounds one delayed Pages upload cleanup pass. + PagesOrphanUploadCandidateLimit = 100 + + pagesOrphanMarkerPredicatePostgres = "w_uploads.metadata #>> '{extra,pages_ingest_marker}' = ?" + pagesOrphanMarkerPredicateSQLite = "CASE WHEN json_valid(w_uploads.metadata) THEN json_extract(w_uploads.metadata, '$.extra.pages_ingest_marker') ELSE NULL END = ?" +) + +// PagesOrphanUploadCandidateQuery describes the fail-closed SQL candidate set +// for delayed Pages upload compensation. +type PagesOrphanUploadCandidateQuery struct { + SystemUserID uint64 + UploadType string + Marker string + CreatedBefore time.Time +} + +// ListPagesOrphanUploadCandidates returns at most 100 unreferenced, isolated +// Pages V2 upload records. Callers must still lock and recheck every condition +// before deleting a candidate. +func ListPagesOrphanUploadCandidates( + ctx context.Context, + input PagesOrphanUploadCandidateQuery, +) ([]Upload, error) { + if input.SystemUserID == 0 || input.UploadType == "" || input.Marker == "" || input.CreatedBefore.IsZero() { + return nil, errors.New("invalid pages orphan upload candidate query") + } + markerPredicate, err := pagesOrphanMarkerPredicate(db.DB(ctx).Name()) + if err != nil { + return nil, err + } + + deploymentTable := (PagesDeployment{}).TableName() + uploadTable := (Upload{}).TableName() + var candidates []Upload + err = db.DB(ctx). + Model(&Upload{}). + Where(uploadTable+".status = ?", UploadStatusUsed). + Where(uploadTable+".user_id = ?", input.SystemUserID). + Where(uploadTable+".type = ?", input.UploadType). + Where(uploadTable+".created_at < ?", input.CreatedBefore). + Where(markerPredicate, input.Marker). + Where("NOT EXISTS (SELECT 1 FROM " + deploymentTable + " WHERE " + deploymentTable + ".upload_id = " + uploadTable + ".id)"). + Order(uploadTable + ".id ASC"). + Limit(PagesOrphanUploadCandidateLimit). + Find(&candidates).Error + if err != nil { + return nil, err + } + return candidates, nil +} + +func pagesOrphanMarkerPredicate(dialect string) (string, error) { + switch dialect { + case "postgres": + return pagesOrphanMarkerPredicatePostgres, nil + case "sqlite": + return pagesOrphanMarkerPredicateSQLite, nil + default: + return "", errors.New("unsupported database dialect for Pages orphan cleanup") + } +} diff --git a/internal/model/openflare_pages_cleanup_test.go b/internal/model/openflare_pages_cleanup_test.go new file mode 100644 index 00000000..cc249d15 --- /dev/null +++ b/internal/model/openflare_pages_cleanup_test.go @@ -0,0 +1,193 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package model + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/glebarez/sqlite" + "gorm.io/gorm" +) + +func TestPagesOrphanMarkerPredicate(t *testing.T) { + tests := []struct { + name string + dialect string + want string + wantErr bool + }{ + { + name: "postgres jsonb path", + dialect: "postgres", + want: "metadata #>> '{extra,pages_ingest_marker}'", + }, + { + name: "sqlite guarded json extract", + dialect: "sqlite", + want: "CASE WHEN json_valid(w_uploads.metadata) THEN json_extract", + }, + { + name: "unknown dialect rejected", + dialect: "mysql", + wantErr: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + got, err := pagesOrphanMarkerPredicate(test.dialect) + if gotErr := err != nil; gotErr != test.wantErr { + t.Fatalf("pagesOrphanMarkerPredicate(%q) error = %v, want error presence = %t", test.dialect, err, test.wantErr) + } + if test.want != "" && !strings.Contains(got, test.want) { + t.Errorf("pagesOrphanMarkerPredicate(%q) = %q, want substring %q", test.dialect, got, test.want) + } + }) + } +} + +func TestListPagesOrphanUploadCandidatesFiltersAndLimits(t *testing.T) { + ctx := context.Background() + gormDB := setupPagesCleanupModelTestDB(t) + cutoff := time.Now().UTC().Add(-2 * time.Hour) + old := cutoff.Add(-time.Minute) + marker := UploadMetadata{Extra: map[string]any{ + "pages_ingest_marker": "pages_deployment_v2", + "pages_project_id": "1", + }} + + valid := make([]Upload, 0, PagesOrphanUploadCandidateLimit+1) + for index := 0; index < PagesOrphanUploadCandidateLimit+1; index++ { + valid = append(valid, pagesCleanupModelUpload(uint64(index+100), 999, "openflare_pages_deployment", UploadStatusUsed, old, marker)) + } + if err := gormDB.Create(&valid).Error; err != nil { + t.Fatalf("create valid candidates error = %v, want nil", err) + } + + referenced := pagesCleanupModelUpload(1, 999, "openflare_pages_deployment", UploadStatusUsed, old, marker) + wrongOwner := pagesCleanupModelUpload(2, 1000, "openflare_pages_deployment", UploadStatusUsed, old, marker) + wrongType := pagesCleanupModelUpload(3, 999, "generic", UploadStatusUsed, old, marker) + wrongStatus := pagesCleanupModelUpload(4, 999, "openflare_pages_deployment", UploadStatusPending, old, marker) + fresh := pagesCleanupModelUpload(5, 999, "openflare_pages_deployment", UploadStatusUsed, cutoff, marker) + wrongMarker := pagesCleanupModelUpload(6, 999, "openflare_pages_deployment", UploadStatusUsed, old, UploadMetadata{Extra: map[string]any{ + "pages_ingest_marker": "pages_deployment_v1", + "pages_project_id": "1", + }}) + for _, upload := range []Upload{referenced, wrongOwner, wrongType, wrongStatus, fresh, wrongMarker} { + if err := gormDB.Create(&upload).Error; err != nil { + t.Fatalf("create filtered upload %d error = %v, want nil", upload.ID, err) + } + } + if err := gormDB.Create(&PagesDeployment{ + ProjectID: 1, + DeploymentNumber: 1, + Checksum: "referenced", + Status: PagesDeploymentStatusUploaded, + UploadID: referenced.ID, + }).Error; err != nil { + t.Fatalf("create referenced deployment error = %v, want nil", err) + } + + invalidJSON := pagesCleanupModelUpload(7, 999, "openflare_pages_deployment", UploadStatusUsed, old, marker) + if err := gormDB.Create(&invalidJSON).Error; err != nil { + t.Fatalf("create invalid JSON upload error = %v, want nil", err) + } + if err := gormDB.Table((Upload{}).TableName()).Where("id = ?", invalidJSON.ID). + UpdateColumn("metadata", "{invalid").Error; err != nil { + t.Fatalf("corrupt upload metadata error = %v, want nil", err) + } + + got, err := ListPagesOrphanUploadCandidates(ctx, PagesOrphanUploadCandidateQuery{ + SystemUserID: 999, + UploadType: "openflare_pages_deployment", + Marker: "pages_deployment_v2", + CreatedBefore: cutoff, + }) + if err != nil { + t.Fatalf("ListPagesOrphanUploadCandidates() error = %v, want nil", err) + } + if len(got) != PagesOrphanUploadCandidateLimit { + t.Fatalf("ListPagesOrphanUploadCandidates() count = %d, want %d", len(got), PagesOrphanUploadCandidateLimit) + } + for index, candidate := range got { + wantID := uint64(index + 100) + if candidate.ID != wantID { + t.Errorf("ListPagesOrphanUploadCandidates()[%d].ID = %d, want %d", index, candidate.ID, wantID) + } + } +} + +func TestListPagesOrphanUploadCandidatesSkipsInvalidSQLiteJSON(t *testing.T) { + ctx := context.Background() + gormDB := setupPagesCleanupModelTestDB(t) + cutoff := time.Now().UTC().Add(-2 * time.Hour) + upload := pagesCleanupModelUpload(1, 999, "openflare_pages_deployment", UploadStatusUsed, cutoff.Add(-time.Minute), UploadMetadata{}) + if err := gormDB.Create(&upload).Error; err != nil { + t.Fatalf("create invalid JSON candidate error = %v, want nil", err) + } + if err := gormDB.Table((Upload{}).TableName()).Where("id = ?", upload.ID). + UpdateColumn("metadata", "{invalid").Error; err != nil { + t.Fatalf("corrupt upload metadata error = %v, want nil", err) + } + + got, err := ListPagesOrphanUploadCandidates(ctx, PagesOrphanUploadCandidateQuery{ + SystemUserID: 999, + UploadType: "openflare_pages_deployment", + Marker: "pages_deployment_v2", + CreatedBefore: cutoff, + }) + if err != nil { + t.Fatalf("ListPagesOrphanUploadCandidates(invalid JSON) error = %v, want nil", err) + } + if len(got) != 0 { + t.Errorf("ListPagesOrphanUploadCandidates(invalid JSON) count = %d, want 0", len(got)) + } +} + +func setupPagesCleanupModelTestDB(t *testing.T) *gorm.DB { + t.Helper() + + gormDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{ + DisableForeignKeyConstraintWhenMigrating: true, + }) + if err != nil { + t.Fatalf("open Pages cleanup model test database error = %v, want nil", err) + } + if err := gormDB.AutoMigrate(&Upload{}, &PagesDeployment{}); err != nil { + t.Fatalf("migrate Pages cleanup model test database error = %v, want nil", err) + } + db.SetDB(gormDB) + t.Cleanup(func() { db.SetDB(nil) }) + return gormDB +} + +func pagesCleanupModelUpload( + id uint64, + userID uint64, + uploadType string, + status UploadStatus, + createdAt time.Time, + metadata UploadMetadata, +) Upload { + return Upload{ + ID: id, + UserID: userID, + FileName: "site.zip", + FilePath: "pages/site.zip", + FileSize: 10, + MimeType: "application/zip", + Extension: "zip", + Hash: "checksum", + Type: uploadType, + Status: status, + AccessMode: 0, + Metadata: metadata, + CreatedAt: createdAt, + UpdatedAt: createdAt, + } +} From 999428cf9a0266256e0e6e5d38b14c2af8719ff6 Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 19:09:21 +0800 Subject: [PATCH 6/8] =?UTF-8?q?feat(pages):=20=E5=A2=9E=E5=8A=A0=E6=9D=A5?= =?UTF-8?q?=E6=BA=90=E6=89=AB=E6=8F=8F=E4=B8=8E=E8=87=AA=E5=8A=A8=E6=9B=B4?= =?UTF-8?q?=E6=96=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 GitHub latest 来源增加五分钟 scanner、按来源间隔检查、精确 revision 自动发布与租约恢复。\n记录退避和投递统计，并为 PostgreSQL 与 SQLite 幂等创建内部排程。 --- docs/changelog/index.md | 4 +- internal/apps/openflare/pages/errs.go | 1 + .../apps/openflare/pages/github_source.go | 18 +- .../openflare/pages/github_source_action.go | 88 ++- .../openflare/pages/github_source_test.go | 9 +- .../pages/source_manual_test_helpers_test.go | 51 ++ .../apps/openflare/pages/source_runtime.go | 38 ++ .../apps/openflare/pages/source_scanner.go | 478 ++++++++++++++++ .../openflare/pages/source_scanner_test.go | 511 ++++++++++++++++++ internal/apps/openflare/pages/source_sync.go | 38 +- internal/apps/openflare/pages/source_tasks.go | 82 ++- .../apps/openflare/pages/source_tasks_test.go | 18 + .../202607190002_seed_pages_source_scan.sql | 38 ++ .../202607190002_seed_pages_source_scan.sql | 21 + .../pages_source_scan_migration_test.go | 160 ++++++ internal/task/handlers/register.go | 3 + 16 files changed, 1493 insertions(+), 65 deletions(-) create mode 100644 internal/apps/openflare/pages/source_manual_test_helpers_test.go create mode 100644 internal/apps/openflare/pages/source_scanner.go create mode 100644 internal/apps/openflare/pages/source_scanner_test.go create mode 100644 internal/db/migrator/goose/postgres/202607190002_seed_pages_source_scan.sql create mode 100644 internal/db/migrator/goose/sqlite/202607190002_seed_pages_source_scan.sql create mode 100644 internal/db/migrator/pages_source_scan_migration_test.go diff --git a/docs/changelog/index.md b/docs/changelog/index.md index 9dc94e9d..7a83c8af 100644 --- a/docs/changelog/index.md +++ b/docs/changelog/index.md @@ -23,7 +23,7 @@ sidebar: false ### 新增 -- Pages 项目新增持久部署源，可配置 Remote URL 或公开 GitHub Release，并支持手动检查、同步发布、来源状态查看与同一 Release 资源替换确认；部署历史会保留安全的来源快照。 +- Pages 项目新增持久部署源，可配置 Remote URL 或公开 GitHub Release，并支持手动检查、同步发布、来源状态查看与同一 Release 资源替换确认；GitHub latest 来源可按设定间隔自动检查并发布更新，部署历史会保留安全的来源快照。 - WAF 规则编排新增「UA 检查」节点：可要求携带 User-Agent、按浏览器/操作系统白名单（且/或）匹配，并优先屏蔽常见爬虫、非正常 UA（不含爬虫）与自定义正则 UA。 ### 改进 @@ -33,7 +33,7 @@ sidebar: false ### 修复 -- 修复 Pages 部署包路径校验、归档展开限额、历史版本裁剪、代理路由绑定与 Agent 下载过程中的安全和一致性问题；大包改为流式处理，部署入口、旧版目录切换、保留版本及上传记录在并发场景下更加可靠。 +- 修复 Pages 部署包路径校验、归档展开限额、历史版本裁剪、代理路由绑定与 Agent 下载过程中的安全和一致性问题；大包改为流式处理，部署入口、旧版目录切换、保留版本及上传记录在并发场景下更加可靠，异常中断遗留的部署包也会被安全补偿清理。 ## [v3.4.0] - 2026-07-19 diff --git a/internal/apps/openflare/pages/errs.go b/internal/apps/openflare/pages/errs.go index 64fc9a25..0d2493ef 100644 --- a/internal/apps/openflare/pages/errs.go +++ b/internal/apps/openflare/pages/errs.go @@ -61,6 +61,7 @@ const ( errPagesSourceActionInvalid = "pages 部署源任务参数无效" errPagesSourceActionStale = "pages 部署源配置已变化，本次任务已跳过" errPagesSourceLeaseLost = "pages 部署源任务执行权已失效" + errPagesSourceLeaseExpired = "上次 pages 部署源任务租约已过期" errPagesSourceSyncFailed = "pages 部署源同步失败" errPagesSourceTaskDispatchFailed = "pages 部署源任务入队失败" errPagesSourceInternal = "pages 部署源操作失败，请稍后重试" diff --git a/internal/apps/openflare/pages/github_source.go b/internal/apps/openflare/pages/github_source.go index d2e8fa66..12db7291 100644 --- a/internal/apps/openflare/pages/github_source.go +++ b/internal/apps/openflare/pages/github_source.go @@ -44,6 +44,7 @@ type githubSourceConfig struct { Selector string Tag string AssetName string + AutoUpdate bool CheckInterval int SourceIdentity string } @@ -56,9 +57,6 @@ func validateGitHubSourceInput(input SourceUpdateInput) error { strings.TrimSpace(input.RemoteNetworkPolicy) != "" { return errors.New(errPagesSourceGitHubFields) } - if input.AutoUpdateEnabled { - return errors.New(errPagesSourceAutoNotAvailable) - } if _, err := normalizeGitHubRepositoryURL(input.RepositoryURL); err != nil { return err } @@ -83,7 +81,7 @@ func validateGitHubSourceInput(input SourceUpdateInput) error { return errors.New(errPagesSourceCheckInterval) } case githubReleaseSelectorTag: - if !validGitHubReleaseTagConfig(input.ReleaseTag) || input.CheckIntervalMinutes != 0 { + if !validGitHubReleaseTagConfig(input.ReleaseTag) || input.AutoUpdateEnabled || input.CheckIntervalMinutes != 0 { return errors.New(errPagesSourceSelectorInvalid) } default: @@ -110,11 +108,17 @@ func buildGitHubSourceConfig(input SourceUpdateInput) (githubSourceConfig, error if selector == githubReleaseSelectorLatest && interval == 0 { interval = defaultCheckInterval } + autoUpdate := input.AutoUpdateEnabled + if selector == githubReleaseSelectorTag { + autoUpdate = false + interval = 0 + } return githubSourceConfig{ Repository: repository, Selector: selector, Tag: tag, AssetName: assetName, + AutoUpdate: autoUpdate, CheckInterval: interval, SourceIdentity: buildGitHubSourceIdentity(repository, selector, tag, assetName), }, nil @@ -251,7 +255,7 @@ func createGitHubSourceTx(tx *gorm.DB, projectID uint, config githubSourceConfig ReleaseSelector: config.Selector, ReleaseTag: config.Tag, AssetName: config.AssetName, - AutoUpdateEnabled: false, + AutoUpdateEnabled: config.AutoUpdate, CheckIntervalMinutes: config.CheckInterval, ConfigVersion: 1, SourceIdentity: config.SourceIdentity, @@ -276,7 +280,7 @@ func githubSourceUpdates(config githubSourceConfig, version int) map[string]any "release_selector": config.Selector, "release_tag": config.Tag, "asset_name": config.AssetName, - sourceColumnAutoUpdateEnabled: false, + sourceColumnAutoUpdateEnabled: config.AutoUpdate, "check_interval_minutes": config.CheckInterval, sourceColumnConfigVersion: version, "source_identity": config.SourceIdentity, @@ -287,7 +291,7 @@ func githubSourceConfigChanged(existing *model.PagesProjectSource, config github return existing.SourceType != PagesSourceTypeGitHubRelease || existing.RemoteURL != "" || existing.RemoteNetworkPolicy != "" || existing.GitHubRepository != config.Repository || existing.ReleaseSelector != config.Selector || existing.ReleaseTag != config.Tag || - existing.AssetName != config.AssetName || existing.AutoUpdateEnabled || + existing.AssetName != config.AssetName || existing.AutoUpdateEnabled != config.AutoUpdate || existing.CheckIntervalMinutes != config.CheckInterval } diff --git a/internal/apps/openflare/pages/github_source_action.go b/internal/apps/openflare/pages/github_source_action.go index d4fc1e89..6aaea778 100644 --- a/internal/apps/openflare/pages/github_source_action.go +++ b/internal/apps/openflare/pages/github_source_action.go @@ -28,9 +28,10 @@ const githubSourceDetailProvider = "github" var githubDigestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) type githubSourceProviderDomainError struct { - message string - permanent bool - retryAt *time.Time + message string + permanent bool + retryAt *time.Time + statusCode int } func (domainError *githubSourceProviderDomainError) Error() string { @@ -56,9 +57,12 @@ type githubSourceTarget struct { } type githubCheckTaskResult struct { - Message string - Detail string - Stale bool + Message string + Detail string + Revision string + Status string + RetryAt *time.Time + Stale bool } type preparedGitHubSource struct { @@ -99,13 +103,21 @@ func checkGitHubSource( return nil, domainErr } if result.NotModified { - if err := finishGitHubCheckNotModified(ctx, snapshot, result); err != nil { + revision, status, err := finishGitHubCheckNotModified(ctx, snapshot, result) + if err != nil { if errors.Is(err, errSourceFinalFence) { return &githubCheckTaskResult{Message: errPagesSourceActionStale, Stale: true}, nil } return nil, err } - return &githubCheckTaskResult{Message: "GitHub Release 检查完成，内容未变化"}, nil + detail, _ := json.Marshal(map[string]string{"revision": revision, pagesDeploymentColumnStatus: status}) + return &githubCheckTaskResult{ + Message: "GitHub Release 检查完成，内容未变化", + Detail: string(detail), + Revision: revision, + Status: status, + RetryAt: result.RetryAt, + }, nil } target, err := buildGitHubSourceTarget(result.Release, result.Asset, result.RetryAt) if err != nil { @@ -136,7 +148,10 @@ func checkGitHubSource( case pagesSourceStatusAttention: message = "检测到同一 Release 的资源被替换，需要确认" } - return &githubCheckTaskResult{Message: message, Detail: string(detail)}, nil + return &githubCheckTaskResult{ + Message: message, Detail: string(detail), Revision: target.Revision, + Status: status, RetryAt: result.RetryAt, + }, nil } func buildGitHubSourceTarget( @@ -183,17 +198,22 @@ func finishGitHubCheckNotModified( ctx context.Context, snapshot *sourceExecutionSnapshot, result githubrelease.ResolveResult, -) error { - return db.DB(ctx).Transaction(func(tx *gorm.DB) error { +) (string, string, error) { + var revision string + var status string + err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { runtime, now, err := lockOwnedSourceRuntime(tx, snapshot) if err != nil { return err } + revision = runtime.LastSeenRevision + status = normalizedSourceRuntimeStatus(runtime) updates := githubCheckTerminalUpdates(snapshot, now, result.RetryAt) updates["etag"] = result.ETag - updates[sourceRuntimeColumnSyncStatus] = normalizedSourceRuntimeStatus(runtime) + updates[sourceRuntimeColumnSyncStatus] = status return tx.Model(runtime).Updates(updates).Error }) + return revision, status, err } func finishGitHubCheckTarget( @@ -230,7 +250,7 @@ func githubCheckTerminalUpdates( sourceRuntimeColumnLeaseToken: "", sourceRuntimeColumnLeaseExpiresAt: nil, } - updates["next_check_at"] = nextCheckAfterGitHubResponse(snapshot, now, retryAt) + updates[sourceRuntimeColumnNextCheckAt] = nextCheckAfterGitHubResponse(snapshot, now, retryAt) return updates } @@ -244,7 +264,7 @@ func nextCheckAfterGitHubResponse( } next := nextGitHubCheckAt(now, snapshot.SourceID, snapshot.CheckIntervalMinutes) if retryAt != nil && retryAt.After(next) { - next = retryAt.UTC() + next = retryAt.In(now.Location()) } return &next } @@ -275,7 +295,7 @@ func failGitHubCheckLease( now := time.Now() next := now.Add(initialCheckRetryDelay) if retryAt.After(next) { - next = retryAt.UTC() + next = retryAt.In(now.Location()) } updates := map[string]any{ sourceRuntimeColumnSyncStatus: pagesSourceStatusFailed, @@ -285,9 +305,9 @@ func failGitHubCheckLease( sourceRuntimeColumnLeaseExpiresAt: nil, } if snapshot.ReleaseSelector == githubReleaseSelectorLatest { - updates["next_check_at"] = &next + updates[sourceRuntimeColumnNextCheckAt] = &next } else { - updates["next_check_at"] = nil + updates[sourceRuntimeColumnNextCheckAt] = nil } result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). Where("source_id = ? AND lease_token = ? AND lease_expires_at > ?", snapshot.SourceID, snapshot.LeaseToken, now). @@ -332,16 +352,20 @@ func preflightGitHubSyncConfirmation(ctx context.Context, sourceID uint, confirm return nil } -func syncGitHubSource( +func syncGitHubSourceWithTrigger( ctx context.Context, snapshot *sourceExecutionSnapshot, actor string, targetRevision string, confirmedRevision string, + triggerType string, ) (outcome *sourceSyncOutcome, resultErr error) { if snapshot == nil || snapshot.SourceType != PagesSourceTypeGitHubRelease || !validPagesSourceActor(actor) { return nil, errors.New(errPagesSourceActionInvalid) } + if !validSourceDeploymentTrigger(triggerType) { + return nil, errors.New(errPagesSourceActionInvalid) + } defer func() { resultErr = finalizeGitHubSyncFailure(ctx, snapshot, resultErr) }() @@ -383,7 +407,7 @@ func syncGitHubSource( if !renewed { return &sourceSyncOutcome{Stale: true}, nil } - return activatePreparedGitHubSource(ctx, snapshot, actor, prepared) + return activatePreparedGitHubSource(ctx, snapshot, actor, triggerType, prepared) } func finalizeGitHubSyncFailure( @@ -429,12 +453,13 @@ func activatePreparedGitHubSource( ctx context.Context, snapshot *sourceExecutionSnapshot, actor string, + triggerType string, prepared *preparedGitHubSource, ) (*sourceSyncOutcome, error) { task.AppendLog(ctx, "[activate] 正在原子切换 GitHub Release 部署") - deployment, reused, referenced, err := commitSourceDeployment( + deployment, reused, referenced, err := commitSourceDeploymentWithTrigger( ctx, snapshot, prepared.target.Revision, prepared.download.SHA256, - prepared.target.Detail, prepared.target.DetailJSON, actor, prepared.manifest, + prepared.target.Detail, prepared.target.DetailJSON, actor, triggerType, prepared.manifest, prepared.ingestState.Result, prepared.ingestState.HasIngest, prepared.target.RetryAt, ) prepared.ingestState.Referenced = referenced @@ -634,7 +659,7 @@ func releaseGitHubSyncWithoutActivation( if expedite && snapshot.ReleaseSelector == githubReleaseSelectorLatest { next := now.Add(initialCheckRetryDelay) if retryAt != nil && retryAt.After(next) { - next = retryAt.UTC() + next = retryAt.In(now.Location()) } nextCheckAt = &next } @@ -644,7 +669,7 @@ func releaseGitHubSyncWithoutActivation( sourceRuntimeColumnSyncStatus: status, sourceRuntimeColumnLastError: lastError, sourceRuntimeColumnLastCheckedAt: &now, - "next_check_at": nextCheckAt, + sourceRuntimeColumnNextCheckAt: nextCheckAt, sourceRuntimeColumnLeaseToken: "", sourceRuntimeColumnLeaseExpiresAt: nil, } @@ -689,32 +714,37 @@ func safeGitHubSourceError(err error) string { func githubSourceDomainError(err error) error { message := errPagesSourceSyncFailed + statusCode := 0 + var providerError *githubrelease.Error + if errors.As(err, &providerError) { + statusCode = providerError.StatusCode + } retryAt, hasRetryAt := githubrelease.RetryAt(err) var retryDeadline *time.Time if hasRetryAt { retryDeadline = &retryAt } if err == nil { - return &githubSourceProviderDomainError{message: message, permanent: false} + return &githubSourceProviderDomainError{message: message, permanent: false, statusCode: statusCode} } if githubrelease.IsDigestError(err) { message = errPagesSourceDigestMismatch - return &githubSourceProviderDomainError{message: message, permanent: true} + return &githubSourceProviderDomainError{message: message, permanent: true, statusCode: statusCode} } if githubrelease.IsNotFound(err) { message = errPagesSourceReleaseNotFound - return &githubSourceProviderDomainError{message: message, permanent: true} + return &githubSourceProviderDomainError{message: message, permanent: true, statusCode: statusCode} } if errors.Is(err, githubrelease.ErrAssetTooLarge) { message = errPagesPackageURLTooLarge - return &githubSourceProviderDomainError{message: message, permanent: true} + return &githubSourceProviderDomainError{message: message, permanent: true, statusCode: statusCode} } if errors.Is(err, githubrelease.ErrEmptyAsset) { message = errPagesPackageEmpty - return &githubSourceProviderDomainError{message: message, permanent: true} + return &githubSourceProviderDomainError{message: message, permanent: true, statusCode: statusCode} } return &githubSourceProviderDomainError{ - message: message, permanent: !githubrelease.IsRetryable(err), retryAt: retryDeadline, + message: message, permanent: !githubrelease.IsRetryable(err), retryAt: retryDeadline, statusCode: statusCode, } } diff --git a/internal/apps/openflare/pages/github_source_test.go b/internal/apps/openflare/pages/github_source_test.go index 6d823319..61b5e340 100644 --- a/internal/apps/openflare/pages/github_source_test.go +++ b/internal/apps/openflare/pages/github_source_test.go @@ -190,7 +190,7 @@ func TestGitHubSourceSaveSurvivesInitialCheckDispatchFailure(t *testing.T) { } } -func TestGitHubSourceRejectsUnsafeOrPhaseThreeFields(t *testing.T) { +func TestGitHubSourceRejectsUnsafeOrModeIncompatibleFields(t *testing.T) { tests := []SourceUpdateInput{ {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "http://github.com/a/b"}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a%20b/repo"}, @@ -202,8 +202,8 @@ func TestGitHubSourceRejectsUnsafeOrPhaseThreeFields(t *testing.T) { {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AssetName: "dist\n.zip"}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AssetName: "dist\u202e.zip"}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AssetName: "dir/dist.zip"}, - {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", AutoUpdateEnabled: true}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1", CheckIntervalMinutes: 60}, + {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1", AutoUpdateEnabled: true}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: " v1"}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1\n"}, {SourceType: PagesSourceTypeGitHubRelease, RepositoryURL: "https://github.com/a/b", ReleaseSelector: "tag", ReleaseTag: "v1\u2028draft"}, @@ -779,6 +779,7 @@ func TestSourceActionPayloadSeparatesSystemTargetAndUserConfirmation(t *testing. revision := strings.Repeat("a", 64) invalid := []SourceActionPayload{ {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: "user:1", TargetRevision: revision}, + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, TriggerType: pagesSourceTriggerManualSync}, {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, ConfirmedRevision: revision}, {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, TargetRevision: revision, ConfirmedRevision: revision}, } @@ -789,8 +790,8 @@ func TestSourceActionPayloadSeparatesSystemTargetAndUserConfirmation(t *testing. } } valid := []SourceActionPayload{ - {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, TargetRevision: revision}, - {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: "user:1", ConfirmedRevision: revision}, + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: pagesSourceCreatedBySystem, TriggerType: pagesSourceTriggerScheduledAutoUpdate, TargetRevision: revision}, + {SourceID: 1, ConfigVersion: 1, Action: sourceActionSync, Actor: "user:1", TriggerType: pagesSourceTriggerManualSync, ConfirmedRevision: revision}, } for _, payload := range valid { raw, _ := json.Marshal(payload) diff --git a/internal/apps/openflare/pages/source_manual_test_helpers_test.go b/internal/apps/openflare/pages/source_manual_test_helpers_test.go new file mode 100644 index 00000000..d54fc991 --- /dev/null +++ b/internal/apps/openflare/pages/source_manual_test_helpers_test.go @@ -0,0 +1,51 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "time" + + "github.com/Rain-kl/Wavelet/internal/apps/upload" + "github.com/Rain-kl/Wavelet/internal/model" +) + +func syncRemoteSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + actor string, +) (*sourceSyncOutcome, error) { + return syncRemoteSourceWithTrigger(ctx, snapshot, actor, pagesSourceTriggerManualSync) +} + +func syncGitHubSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + actor string, + targetRevision string, + confirmedRevision string, +) (*sourceSyncOutcome, error) { + return syncGitHubSourceWithTrigger( + ctx, snapshot, actor, targetRevision, confirmedRevision, pagesSourceTriggerManualSync, + ) +} + +func commitSourceDeployment( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + revision string, + packageChecksum string, + detail sourceDetail, + detailJSON string, + actor string, + manifest *deploymentManifest, + ingestResult upload.IngestResult, + hasIngest bool, + nextCheckNotBefore *time.Time, +) (*model.PagesDeployment, bool, bool, error) { + return commitSourceDeploymentWithTrigger( + ctx, snapshot, revision, packageChecksum, detail, detailJSON, actor, + pagesSourceTriggerManualSync, manifest, ingestResult, hasIngest, nextCheckNotBefore, + ) +} diff --git a/internal/apps/openflare/pages/source_runtime.go b/internal/apps/openflare/pages/source_runtime.go index 98801661..fdefb536 100644 --- a/internal/apps/openflare/pages/source_runtime.go +++ b/internal/apps/openflare/pages/source_runtime.go @@ -29,6 +29,7 @@ const ( sourceRuntimeColumnSyncStatus = "sync_status" sourceRuntimeColumnLastError = "last_error" sourceRuntimeColumnLastCheckedAt = "last_checked_at" + sourceRuntimeColumnNextCheckAt = "next_check_at" sourceRuntimeColumnLeaseToken = "lease_token" sourceRuntimeColumnLeaseExpiresAt = "lease_expires_at" pagesDeploymentColumnStatus = "status" @@ -58,6 +59,7 @@ type sourceExecutionSnapshot struct { ReleaseSelector string ReleaseTag string AssetName string + AutoUpdateEnabled bool CheckIntervalMinutes int ETag string LastSeenRevision string @@ -179,6 +181,7 @@ func loadSourceExecutionSnapshot( ReleaseSelector: source.ReleaseSelector, ReleaseTag: source.ReleaseTag, AssetName: source.AssetName, + AutoUpdateEnabled: source.AutoUpdateEnabled, CheckIntervalMinutes: source.CheckIntervalMinutes, ETag: runtime.ETag, LastSeenRevision: runtime.LastSeenRevision, @@ -300,6 +303,41 @@ func sourceLeaseIsBusy(ctx context.Context, sourceID uint) (bool, error) { return runtime.LeaseExpiresAt != nil && runtime.LeaseExpiresAt.After(time.Now()), nil } +// recoverExpiredSourceLease clears one exact expired lease owner. Matching the +// token, observed expiry and status prevents a scanner from overwriting a +// worker that renewed or was replaced after the candidate query. +func recoverExpiredSourceLease( + ctx context.Context, + sourceID uint, + token string, + expiresAt time.Time, + status string, + now time.Time, + nextCheckAt *time.Time, +) (bool, error) { + if sourceID == 0 || token == "" || + (status != pagesSourceStatusChecking && status != pagesSourceStatusSyncing) { + return false, nil + } + updates := map[string]any{ + sourceRuntimeColumnSyncStatus: pagesSourceStatusFailed, + sourceRuntimeColumnLastError: errPagesSourceLeaseExpired, + sourceRuntimeColumnLeaseToken: "", + sourceRuntimeColumnLeaseExpiresAt: nil, + sourceRuntimeColumnNextCheckAt: nextCheckAt, + } + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", sourceID). + Where("lease_token = ?", token). + Where("lease_expires_at = ? AND lease_expires_at <= ?", expiresAt, now). + Where("sync_status = ?", status). + Updates(updates) + if result.Error != nil { + return false, result.Error + } + return result.RowsAffected == 1, nil +} + // fenceAndNormalizeRuntime invalidates in-flight work while preserving safe // seen/applied cursors. The caller must already hold the source row lock. func fenceAndNormalizeRuntime(tx *gorm.DB, sourceID uint) error { diff --git a/internal/apps/openflare/pages/source_scanner.go b/internal/apps/openflare/pages/source_scanner.go new file mode 100644 index 00000000..29fc0dc8 --- /dev/null +++ b/internal/apps/openflare/pages/source_scanner.go @@ -0,0 +1,478 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/task" + "github.com/Rain-kl/Wavelet/pkg/logger" + "gorm.io/gorm" +) + +const ( + // PagesSourceScanTask is the private Asynq task type for the periodic scanner. + PagesSourceScanTask = "openflare:pages_source_scan" + // TaskTypePagesSourceScan is the internal task meta type seeded in w_schedules. + TaskTypePagesSourceScan = "of_pages_source_scan" + + pagesSourceScanBatchSize = 20 +) + +// PagesSourceScanMeta is available to the scheduler registry but hidden from +// generic Admin task dispatch and schedule mutation APIs. +var PagesSourceScanMeta = task.TaskMeta{ + Type: TaskTypePagesSourceScan, + AsynqTask: PagesSourceScanTask, + Name: "OpenFlare Pages 部署源扫描", + Description: "补偿孤儿部署包、恢复过期执行权并串行检查到期的 GitHub latest 部署源", + SupportsTime: false, + MaxRetry: 0, + Queue: task.QueueDefault, + Retryable: false, + InternalOnly: true, +} + +type pagesSourceScanPayload struct{} + +type pagesSourceScanSummary struct { + ExpiredCandidates int `json:"expired_candidates"` + RecoveredLeases int `json:"recovered_leases"` + OrphanCleanup PagesOrphanCleanupSummary `json:"orphan_cleanup"` + DueSources int `json:"due_sources"` + SelectedSources int `json:"selected_sources"` + CheckedSources int `json:"checked_sources"` + UpdatesFound int `json:"updates_found"` + AttentionSources int `json:"attention_sources"` + DispatchedSyncs int `json:"dispatched_syncs"` + FailedDispatches int `json:"failed_dispatches"` + BusySources int `json:"busy_sources"` + StaleSources int `json:"stale_sources"` + FailedSources int `json:"failed_sources"` + Backlog int `json:"backlog"` + ProviderBackoffs []pagesSourceProviderBackoff `json:"provider_backoffs,omitempty"` +} + +type pagesSourceProviderBackoff struct { + SourceID uint `json:"source_id"` + StatusCode int `json:"status_code"` + RetryAt string `json:"retry_at"` +} + +type expiredSourceLeaseCandidate struct { + SourceID uint + LeaseToken string + LeaseExpiresAt time.Time + SyncStatus string + SourceType string + ReleaseSelector string +} + +type dueGitHubSourceCandidate struct { + SourceID uint + ConfigVersion int +} + +var ( + pagesSourceScanNow = time.Now + reconcilePagesSourceOrphans = ReconcilePagesOrphanUploads + dispatchPagesSourceAutoSync = func( + ctx context.Context, + source model.PagesProjectSource, + targetRevision string, + ) (*SourceActionReceipt, error) { + return dispatchSourceActionSnapshotWithTrigger( + ctx, + source, + sourceActionSync, + pagesSourceCreatedBySystem, + pagesSourceTriggerScheduledAutoUpdate, + targetRevision, + "", + "system", + ) + } +) + +// SourceScanHandler serializes provider checks inside one scheduled task. A +// source-level lease still permits overlapping scanner executions safely. +type SourceScanHandler struct{} + +// ValidatePayload accepts only an empty object; the scanner has no user input. +func (handler *SourceScanHandler) ValidatePayload(payload []byte) ([]byte, error) { + if len(bytes.TrimSpace(payload)) == 0 { + payload = []byte("{}") + } + var input pagesSourceScanPayload + decoder := json.NewDecoder(bytes.NewReader(payload)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(&input); err != nil { + return nil, errors.New(errPagesSourceActionInvalid) + } + if err := ensureJSONEOF(decoder); err != nil { + return nil, errors.New(errPagesSourceActionInvalid) + } + return []byte("{}"), nil +} + +// Execute recovers expired leases and checks at most 20 due latest sources in +// stable order. Provider and dispatch failures are isolated per source. +func (handler *SourceScanHandler) Execute(ctx context.Context, payload []byte) (*task.TaskResult, error) { + if _, err := handler.ValidatePayload(payload); err != nil { + return nil, task.PermanentError(errPagesSourceActionInvalid) + } + + now := pagesSourceScanNow() + summary := pagesSourceScanSummary{} + if err := recoverExpiredPagesSourceLeases(ctx, now, &summary); err != nil { + return nil, err + } + orphanSummary, err := reconcilePagesSourceOrphans(ctx, now) + if err != nil { + return nil, err + } + summary.OrphanCleanup = orphanSummary + task.AppendLog( + ctx, + "[cleanup] orphan 候选=%d，已补偿=%d，仍被引用=%d，lease busy=%d，非法 marker=%d，跳过=%d，失败=%d", + orphanSummary.Candidates, + orphanSummary.Reconciled, + orphanSummary.Referenced, + orphanSummary.LeaseBusy, + orphanSummary.InvalidMarker, + orphanSummary.Skipped, + orphanSummary.Failed, + ) + if err := scanDueGitHubSources(ctx, now, &summary); err != nil { + return nil, err + } + + detail, err := json.Marshal(summary) + if err != nil { + return nil, err + } + message := fmt.Sprintf( + "Pages 部署源扫描完成：恢复 %d 个租约，补偿 %d 个孤儿记录，检查 %d 个来源，投递 %d 个自动更新，积压 %d 个", + summary.RecoveredLeases, + summary.OrphanCleanup.Reconciled, + summary.CheckedSources, + summary.DispatchedSyncs, + summary.Backlog, + ) + return &task.TaskResult{Message: message, Detail: string(detail)}, nil +} + +func recoverExpiredPagesSourceLeases( + ctx context.Context, + now time.Time, + summary *pagesSourceScanSummary, +) error { + var candidates []expiredSourceLeaseCandidate + err := db.DB(ctx). + Table("of_pages_project_source_runtime AS runtime"). + Select(`runtime.source_id, runtime.lease_token, runtime.lease_expires_at, + runtime.sync_status, source.source_type, source.release_selector`). + Joins("JOIN of_pages_project_sources AS source ON source.id = runtime.source_id"). + Where("runtime.lease_token <> ''"). + Where("runtime.lease_expires_at IS NOT NULL AND runtime.lease_expires_at <= ?", now). + Where("runtime.sync_status IN ?", []string{pagesSourceStatusChecking, pagesSourceStatusSyncing}). + Order("runtime.source_id ASC"). + Scan(&candidates).Error + if err != nil { + return err + } + summary.ExpiredCandidates = len(candidates) + for _, candidate := range candidates { + var nextCheckAt *time.Time + if candidate.SourceType == PagesSourceTypeGitHubRelease && + candidate.ReleaseSelector == githubReleaseSelectorLatest { + next := nextGitHubCheckAt(now, candidate.SourceID, minimumCheckInterval) + nextCheckAt = &next + } + recovered, recoverErr := recoverExpiredSourceLease( + ctx, + candidate.SourceID, + candidate.LeaseToken, + candidate.LeaseExpiresAt, + candidate.SyncStatus, + now, + nextCheckAt, + ) + if recoverErr != nil { + summary.FailedSources++ + logger.WarnF( + ctx, + "[PagesSourceScan] recover expired lease failed: source_id=%d error=%v", + candidate.SourceID, + recoverErr, + ) + continue + } + if recovered { + summary.RecoveredLeases++ + task.AppendLog(ctx, "[recover] 已恢复过期来源租约：source_id=%d", candidate.SourceID) + } + } + return nil +} + +func scanDueGitHubSources( + ctx context.Context, + now time.Time, + summary *pagesSourceScanSummary, +) error { + dueQuery := func() *gorm.DB { + return db.DB(ctx). + Table("of_pages_project_source_runtime AS runtime"). + Joins("JOIN of_pages_project_sources AS source ON source.id = runtime.source_id"). + Where("source.source_type = ?", PagesSourceTypeGitHubRelease). + Where("source.release_selector = ?", githubReleaseSelectorLatest). + Where("runtime.next_check_at IS NOT NULL AND runtime.next_check_at <= ?", now) + } + var dueCount int64 + if err := dueQuery().Count(&dueCount).Error; err != nil { + return err + } + summary.DueSources = int(dueCount) + + var candidates []dueGitHubSourceCandidate + if err := dueQuery(). + Select("source.id AS source_id, source.config_version"). + Order("runtime.next_check_at ASC"). + Order("source.id ASC"). + Limit(pagesSourceScanBatchSize). + Scan(&candidates).Error; err != nil { + return err + } + summary.SelectedSources = len(candidates) + task.AppendLog( + ctx, + "[scan] 到期来源=%d，本批=%d", + summary.DueSources, + summary.SelectedSources, + ) + + for _, candidate := range candidates { + scanOneDueGitHubSource(ctx, candidate, summary) + } + var remainingDue int64 + if err := dueQuery().Count(&remainingDue).Error; err != nil { + return err + } + summary.Backlog = int(remainingDue) + task.AppendLog(ctx, "[scan] 本批处理后仍到期来源=%d", summary.Backlog) + return nil +} + +func scanOneDueGitHubSource( + ctx context.Context, + candidate dueGitHubSourceCandidate, + summary *pagesSourceScanSummary, +) { + snapshot, outcome, err := acquireSourceLease( + ctx, + candidate.SourceID, + candidate.ConfigVersion, + sourceActionCheck, + ) + if err != nil { + summary.FailedSources++ + logger.WarnF(ctx, "[PagesSourceScan] acquire check lease failed: source_id=%d error=%v", candidate.SourceID, err) + return + } + switch outcome { + case sourceLeaseBusy: + summary.BusySources++ + task.AppendLog(ctx, "[check] 来源正在执行其它任务，跳过：source_id=%d", candidate.SourceID) + return + case sourceLeaseStale: + summary.StaleSources++ + return + } + if snapshot == nil || snapshot.SourceType != PagesSourceTypeGitHubRelease || + snapshot.ReleaseSelector != githubReleaseSelectorLatest { + summary.StaleSources++ + if snapshot != nil { + if finalizeErr := failSourceLease(ctx, snapshot, errPagesSourceActionStale); finalizeErr != nil { + logger.WarnF( + ctx, + "[PagesSourceScan] finalize stale source failed: source_id=%d error=%v", + snapshot.SourceID, + finalizeErr, + ) + } + } + return + } + + checkResult, checkErr := checkGitHubSource(ctx, snapshot) + if checkErr != nil { + summary.FailedSources++ + recordPagesSourceProviderBackoff(ctx, candidate.SourceID, checkErr, summary) + logger.WarnF( + ctx, + "[PagesSourceScan] source check failed: source_id=%d error=%s", + candidate.SourceID, + safeGitHubSourceError(checkErr), + ) + return + } + if checkResult == nil || checkResult.Stale { + summary.StaleSources++ + return + } + handleCheckedGitHubSource(ctx, snapshot, checkResult, summary) +} + +func recordPagesSourceProviderBackoff( + ctx context.Context, + sourceID uint, + checkErr error, + summary *pagesSourceScanSummary, +) { + var domainError *githubSourceProviderDomainError + if !errors.As(checkErr, &domainError) || + (domainError.statusCode != 403 && domainError.statusCode != 429) { + return + } + + retryAt := domainError.retryAt + var runtime model.PagesProjectSourceRuntime + if err := db.DB(ctx). + Select("next_check_at"). + Where("source_id = ?", sourceID). + First(&runtime).Error; err != nil { + logger.WarnF(ctx, "[PagesSourceScan] load provider backoff deadline failed: source_id=%d error=%v", sourceID, err) + } else if runtime.NextCheckAt != nil { + retryAt = runtime.NextCheckAt + } + + retryAtText := "unknown" + if retryAt != nil { + retryAtText = retryAt.UTC().Format(time.RFC3339) + } + summary.ProviderBackoffs = append(summary.ProviderBackoffs, pagesSourceProviderBackoff{ + SourceID: sourceID, StatusCode: domainError.statusCode, RetryAt: retryAtText, + }) + task.AppendLog( + ctx, + "[check] GitHub provider 退避：source_id=%d status=%d retry_at=%s", + sourceID, + domainError.statusCode, + retryAtText, + ) +} + +func handleCheckedGitHubSource( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + checkResult *githubCheckTaskResult, + summary *pagesSourceScanSummary, +) { + summary.CheckedSources++ + switch checkResult.Status { + case pagesSourceStatusUpdateAvailable: + summary.UpdatesFound++ + case pagesSourceStatusAttention: + summary.AttentionSources++ + } + if !snapshot.AutoUpdateEnabled || checkResult.Status != pagesSourceStatusUpdateAvailable || + !validOptionalSourceRevision(checkResult.Revision) || checkResult.Revision == "" { + return + } + + source := model.PagesProjectSource{ + ID: snapshot.SourceID, + ProjectID: snapshot.ProjectID, + ConfigVersion: snapshot.SourceConfigVersion, + } + receipt, dispatchErr := dispatchPagesSourceAutoSync(ctx, source, checkResult.Revision) + if dispatchErr == nil { + summary.DispatchedSyncs++ + if receipt != nil { + task.AppendLog( + ctx, + "[dispatch] 已投递自动更新：source_id=%d execution_id=%s revision=%s", + snapshot.SourceID, + receipt.ExecutionID, + checkResult.Revision, + ) + } + return + } + + summary.FailedSources++ + summary.FailedDispatches++ + logger.WarnF( + ctx, + "[PagesSourceScan] dispatch auto sync failed: source_id=%d revision=%s error=%v", + snapshot.SourceID, + checkResult.Revision, + dispatchErr, + ) + updated, recordErr := recordPagesSourceAutoDispatchFailure( + ctx, + snapshot, + checkResult.Revision, + checkResult.RetryAt, + ) + if recordErr != nil { + logger.WarnF( + ctx, + "[PagesSourceScan] record auto sync dispatch failure failed: source_id=%d error=%v", + snapshot.SourceID, + recordErr, + ) + } else if !updated { + summary.StaleSources++ + } +} + +func recordPagesSourceAutoDispatchFailure( + ctx context.Context, + snapshot *sourceExecutionSnapshot, + revision string, + retryAt *time.Time, +) (bool, error) { + if snapshot == nil || revision == "" { + return false, nil + } + now := pagesSourceScanNow() + next := nextGitHubCheckAt(now, snapshot.SourceID, minimumCheckInterval) + if retryAt != nil && retryAt.After(next) { + next = retryAt.In(now.Location()) + } + result := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", snapshot.SourceID). + Where("sync_status = ? AND last_seen_revision = ?", pagesSourceStatusUpdateAvailable, revision). + Where("lease_expires_at IS NULL OR lease_expires_at <= ?", now). + Where(`EXISTS ( + SELECT 1 FROM of_pages_project_sources AS source + WHERE source.id = ? AND source.config_version = ? + AND source.source_type = ? AND source.release_selector = ? + AND source.auto_update_enabled = ? + )`, + snapshot.SourceID, + snapshot.SourceConfigVersion, + PagesSourceTypeGitHubRelease, + githubReleaseSelectorLatest, + true, + ). + Updates(map[string]any{ + sourceRuntimeColumnSyncStatus: pagesSourceStatusUpdateAvailable, + sourceRuntimeColumnLastError: errPagesSourceTaskDispatchFailed, + sourceRuntimeColumnNextCheckAt: &next, + }) + if result.Error != nil { + return false, result.Error + } + return result.RowsAffected == 1, nil +} diff --git a/internal/apps/openflare/pages/source_scanner_test.go b/internal/apps/openflare/pages/source_scanner_test.go new file mode 100644 index 00000000..2d7c2394 --- /dev/null +++ b/internal/apps/openflare/pages/source_scanner_test.go @@ -0,0 +1,511 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/integration/githubrelease" + "github.com/Rain-kl/Wavelet/internal/model" + "gorm.io/gorm" +) + +type scannerDispatchedSync struct { + SourceID uint + Revision string +} + +func TestGitHubLatestAutoConfigPreservesIdentityAndRuntimeCursor(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "scanner-auto-config") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/scanner/auto-config", + AutoUpdateEnabled: false, + CheckIntervalMinutes: 60, + }) + identity := source.SourceIdentity + seenRevision := strings.Repeat("a", sourceRevisionHexLength) + appliedRevision := strings.Repeat("b", sourceRevisionHexLength) + future := time.Now().Add(time.Hour) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Updates(map[string]any{ + "etag": `"cursor-etag"`, + "last_seen_revision": seenRevision, + "last_seen_detail": `{"provider":"github","release_id":"2","asset_id":"2","tag":"v2","asset_name":"dist.zip"}`, + "last_applied_revision": appliedRevision, + "last_applied_detail": `{"provider":"github","release_id":"1","asset_id":"1","tag":"v1","asset_name":"dist.zip"}`, + "sync_status": pagesSourceStatusSyncing, + "last_error": "old error", + "lease_token": "in-flight", + "lease_expires_at": &future, + }).Error; err != nil { + t.Fatalf("seed runtime error = %v, want nil", err) + } + + input := SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/scanner/auto-config", + AutoUpdateEnabled: true, + CheckIntervalMinutes: 15, + } + if err := validateGitHubSourceInput(input); err != nil { + t.Fatalf("validateGitHubSourceInput(auto latest) error = %v, want nil", err) + } + if err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + changed, err := updateGitHubSourceTx(tx, project.ID, input) + if err == nil && !changed { + return errors.New("auto config update was treated as no-op") + } + return err + }); err != nil { + t.Fatalf("updateGitHubSourceTx(auto latest) error = %v, want nil", err) + } + + updated, runtime := mustLoadPagesSource(t, ctx, project.ID) + if updated.SourceIdentity != identity || updated.ConfigVersion != source.ConfigVersion+1 { + t.Errorf( + "updated source = identity:%q version:%d, want identity:%q version:%d", + updated.SourceIdentity, + updated.ConfigVersion, + identity, + source.ConfigVersion+1, + ) + } + if !updated.AutoUpdateEnabled || updated.CheckIntervalMinutes != 15 { + t.Errorf("updated auto config = enabled:%t interval:%d, want true/15", updated.AutoUpdateEnabled, updated.CheckIntervalMinutes) + } + if runtime.ETag != `"cursor-etag"` || runtime.LastSeenRevision != seenRevision || + runtime.LastAppliedRevision != appliedRevision { + t.Errorf("runtime cursor changed after auto-only update: %+v", runtime) + } + if runtime.LeaseToken != "" || runtime.LeaseExpiresAt != nil || runtime.LastError != "" || + runtime.SyncStatus != pagesSourceStatusUpdateAvailable || runtime.NextCheckAt == nil { + t.Errorf( + "runtime fence = token:%q expiry:%v error:%q status:%q next:%v", + runtime.LeaseToken, + runtime.LeaseExpiresAt, + runtime.LastError, + runtime.SyncStatus, + runtime.NextCheckAt, + ) + } + + tagConfig, err := buildGitHubSourceConfig(SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/scanner/auto-config", + ReleaseSelector: githubReleaseSelectorTag, + ReleaseTag: "v1", + AutoUpdateEnabled: true, + CheckIntervalMinutes: 60, + }) + if err != nil { + t.Fatalf("buildGitHubSourceConfig(tag) error = %v, want nil", err) + } + if tagConfig.AutoUpdate || tagConfig.CheckInterval != 0 { + t.Errorf("tag config auto/interval = %t/%d, want false/0", tagConfig.AutoUpdate, tagConfig.CheckInterval) + } +} + +func TestRecoverExpiredPagesSourceLeaseUsesExactCASAndStableJitter(t *testing.T) { + ctx := setupPagesSourceTest(t) + project := mustCreatePagesSourceProject(t, ctx, "scanner-expired-lease") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/scanner/expired-lease", + }) + now := time.Date(2026, 7, 19, 12, 0, 0, 0, time.UTC) + usePagesSourceScannerClock(t, now) + expiredAt := now.Add(-time.Minute) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Updates(map[string]any{ + "sync_status": pagesSourceStatusChecking, + "lease_token": "expired-owner", + "lease_expires_at": &expiredAt, + "next_check_at": &expiredAt, + }).Error; err != nil { + t.Fatalf("seed expired lease error = %v, want nil", err) + } + + summary := pagesSourceScanSummary{} + if err := recoverExpiredPagesSourceLeases(ctx, now, &summary); err != nil { + t.Fatalf("recoverExpiredPagesSourceLeases() error = %v, want nil", err) + } + if summary.ExpiredCandidates != 1 || summary.RecoveredLeases != 1 || summary.FailedSources != 0 { + t.Errorf("recovery summary = %+v, want one recovered lease", summary) + } + _, runtime := mustLoadPagesSource(t, ctx, project.ID) + wantNext := nextGitHubCheckAt(now, source.ID, minimumCheckInterval) + if runtime.SyncStatus != pagesSourceStatusFailed || runtime.LastError != errPagesSourceLeaseExpired || + runtime.LeaseToken != "" || runtime.LeaseExpiresAt != nil || runtime.NextCheckAt == nil || + runtime.NextCheckAt.Sub(wantNext) != 0 { + t.Errorf( + "recovered runtime = status:%q error:%q token:%q expiry:%v next:%v, want next %v", + runtime.SyncStatus, + runtime.LastError, + runtime.LeaseToken, + runtime.LeaseExpiresAt, + runtime.NextCheckAt, + wantNext, + ) + } + + renewedExpiry := now.Add(time.Minute) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Updates(map[string]any{ + "sync_status": pagesSourceStatusSyncing, + "lease_token": "renewed-owner", + "lease_expires_at": &renewedExpiry, + }).Error; err != nil { + t.Fatalf("seed renewed lease error = %v, want nil", err) + } + recovered, err := recoverExpiredSourceLease( + ctx, + source.ID, + "renewed-owner", + expiredAt, + pagesSourceStatusSyncing, + now, + &wantNext, + ) + if err != nil || recovered { + t.Fatalf("recoverExpiredSourceLease(stale expiry) = %t, %v; want false, nil", recovered, err) + } + _, runtime = mustLoadPagesSource(t, ctx, project.ID) + if runtime.LeaseToken != "renewed-owner" || runtime.LeaseExpiresAt == nil || + runtime.LeaseExpiresAt.Sub(renewedExpiry) != 0 || runtime.SyncStatus != pagesSourceStatusSyncing { + t.Errorf("stale recovery overwrote renewed lease: %+v", runtime) + } +} + +func TestScheduledAutoSyncPersistsExplicitDeploymentTrigger(t *testing.T) { + ctx := setupPagesSourceSyncTest(t) + project := mustCreatePagesSourceProject(t, ctx, "scanner-scheduled-trigger") + source, _ := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/scanner/scheduled-trigger", + AutoUpdateEnabled: true, + }) + packageBytes := testPagesZip(t, map[string]string{"index.html": "scheduled-v1"}) + packageHash := sha256.Sum256(packageBytes) + release := githubrelease.Release{ID: "scheduled-release", Tag: "v1"} + asset := githubrelease.Asset{ + ID: "scheduled-asset", Name: defaultGitHubAssetName, State: "uploaded", + UpdatedAt: time.Date(2026, 7, 19, 12, 30, 0, 0, time.UTC), + } + target, err := buildGitHubSourceTarget(release, asset, nil) + if err != nil { + t.Fatalf("buildGitHubSourceTarget() error = %v, want nil", err) + } + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(context.Context, githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + return githubrelease.ResolveResult{Release: release, Asset: asset}, nil + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + path := filepath.Join(t.TempDir(), "scheduled.zip") + if err := os.WriteFile(path, packageBytes, 0o600); err != nil { + t.Fatalf("os.WriteFile(scheduled package) error = %v", err) + } + return &githubrelease.DownloadResult{ + Path: path, Size: int64(len(packageBytes)), SHA256: hex.EncodeToString(packageHash[:]), + }, nil + }, + }) + snapshot, outcome, err := acquireSourceLease(ctx, source.ID, source.ConfigVersion, sourceActionSync) + if err != nil || outcome != sourceLeaseAcquired { + t.Fatalf("acquireSourceLease() = %+v, %q, %v; want acquired", snapshot, outcome, err) + } + synced, err := syncGitHubSourceWithTrigger( + ctx, + snapshot, + pagesSourceCreatedBySystem, + target.Revision, + "", + pagesSourceTriggerScheduledAutoUpdate, + ) + if err != nil || synced == nil || synced.Deployment == nil || synced.Stale { + t.Fatalf("syncGitHubSourceWithTrigger() = %+v, %v; want active deployment", synced, err) + } + deployment, err := model.GetPagesDeploymentByID(ctx, synced.Deployment.ID) + if err != nil { + t.Fatalf("GetPagesDeploymentByID(%d) error = %v", synced.Deployment.ID, err) + } + if deployment.TriggerType != pagesSourceTriggerScheduledAutoUpdate || + deployment.CreatedBy != pagesSourceCreatedBySystem { + t.Errorf( + "scheduled provenance = trigger:%q actor:%q, want %q/%q", + deployment.TriggerType, + deployment.CreatedBy, + pagesSourceTriggerScheduledAutoUpdate, + pagesSourceCreatedBySystem, + ) + } +} + +func TestPagesSourceScannerSerialBatchIsolation304AndActualBacklog(t *testing.T) { + ctx := setupPagesSourceTest(t) + now := time.Now().Truncate(time.Second) + usePagesSourceScannerClock(t, now) + dueAt := now.Add(-time.Hour) + + type fixture struct { + source *model.PagesProjectSource + runtime *model.PagesProjectSourceRuntime + repository string + } + fixtures := make([]fixture, 0, 22) + byRepository := make(map[string]int, 22) + for index := 1; index <= 22; index++ { + project := mustCreatePagesSourceProject(t, ctx, fmt.Sprintf("scanner-batch-%02d", index)) + repository := fmt.Sprintf("scanner/source-%02d", index) + source, runtime := mustConfigureGitHubSourceWithoutDispatch(t, ctx, project.ID, SourceUpdateInput{ + SourceType: PagesSourceTypeGitHubRelease, + RepositoryURL: "https://github.com/" + repository, + AutoUpdateEnabled: index != 4, + CheckIntervalMinutes: 60, + }) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", source.ID). + Update("next_check_at", &dueAt).Error; err != nil { + t.Fatalf("mark source %d due error = %v, want nil", source.ID, err) + } + fixtures = append(fixtures, fixture{source: source, runtime: runtime, repository: repository}) + byRepository[repository] = index + } + + busyUntil := now.Add(time.Hour) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", fixtures[0].source.ID). + Updates(map[string]any{ + "sync_status": pagesSourceStatusChecking, + "lease_token": "busy-owner", + "lease_expires_at": &busyUntil, + }).Error; err != nil { + t.Fatalf("seed busy source error = %v, want nil", err) + } + + stored304Revision := strings.Repeat("3", sourceRevisionHexLength) + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", fixtures[2].source.ID). + Updates(map[string]any{ + "etag": `"stored-etag"`, + "last_seen_revision": stored304Revision, + "last_seen_detail": `{"provider":"github","release_id":"release-3","asset_id":"3","tag":"v3","asset_name":"dist.zip"}`, + "sync_status": pagesSourceStatusUpdateAvailable, + }).Error; err != nil { + t.Fatalf("seed 304 cursor error = %v, want nil", err) + } + + if err := db.DB(ctx).Model(&model.PagesProjectSourceRuntime{}). + Where("source_id = ?", fixtures[4].source.ID). + Updates(map[string]any{ + "last_applied_revision": strings.Repeat("a", sourceRevisionHexLength), + "last_applied_detail": `{"provider":"github","release_id":"shared-release","asset_id":"old","tag":"v5","asset_name":"dist.zip"}`, + }).Error; err != nil { + t.Fatalf("seed replacement cursor error = %v, want nil", err) + } + + retryAt := now.Add(2 * time.Hour) + calledRepositories := make([]string, 0, pagesSourceScanBatchSize) + useFakeGitHubReleaseClient(t, &fakeGitHubReleaseClient{ + resolve: func(_ context.Context, request githubrelease.ResolveRequest) (githubrelease.ResolveResult, error) { + index := byRepository[request.Repository] + calledRepositories = append(calledRepositories, request.Repository) + switch index { + case 2: + return githubrelease.ResolveResult{}, &githubrelease.Error{ + Kind: githubrelease.ErrMetadata, StatusCode: 429, RetryAt: &retryAt, + } + case 3: + if request.ETag != `"stored-etag"` { + t.Errorf("304 source ETag = %q, want stored ETag", request.ETag) + } + return githubrelease.ResolveResult{NotModified: true, ETag: request.ETag}, nil + default: + releaseID := fmt.Sprintf("release-%d", index) + if index == 5 { + releaseID = "shared-release" + } + result := githubrelease.ResolveResult{ + ETag: fmt.Sprintf(`"etag-%d"`, index), + Release: githubrelease.Release{ID: releaseID, Tag: fmt.Sprintf("v%d", index)}, + Asset: githubrelease.Asset{ + ID: fmt.Sprintf("asset-%d", index), + Name: defaultGitHubAssetName, + State: "uploaded", + UpdatedAt: now.Add(time.Duration(index) * time.Minute), + }, + } + if index == 6 { + result.RetryAt = &retryAt + } + return result, nil + } + }, + download: func(context.Context, githubrelease.DownloadRequest) (*githubrelease.DownloadResult, error) { + t.Fatal("scanner downloaded an asset; want check-only behavior") + return nil, nil + }, + }) + + dispatched := make([]scannerDispatchedSync, 0, pagesSourceScanBatchSize) + previousDispatch := dispatchPagesSourceAutoSync + dispatchPagesSourceAutoSync = func( + _ context.Context, + source model.PagesProjectSource, + revision string, + ) (*SourceActionReceipt, error) { + if source.ID == fixtures[5].source.ID { + return nil, errors.New("injected dispatch failure") + } + dispatched = append(dispatched, scannerDispatchedSync{SourceID: source.ID, Revision: revision}) + return &SourceActionReceipt{ExecutionID: fmt.Sprintf("%d", source.ID), Action: sourceActionSync}, nil + } + t.Cleanup(func() { dispatchPagesSourceAutoSync = previousDispatch }) + + result, err := (&SourceScanHandler{}).Execute(ctx, []byte("{}")) + if err != nil { + t.Fatalf("SourceScanHandler.Execute() error = %v, want nil", err) + } + var summary pagesSourceScanSummary + if err := json.Unmarshal([]byte(result.Detail), &summary); err != nil { + t.Fatalf("json.Unmarshal(scan detail) error = %v, want nil", err) + } + if summary.DueSources != 22 || summary.SelectedSources != pagesSourceScanBatchSize || + summary.CheckedSources != 18 || summary.UpdatesFound != 17 || summary.AttentionSources != 1 || + summary.DispatchedSyncs != 15 || summary.FailedDispatches != 1 || + summary.BusySources != 1 || summary.FailedSources != 2 || + summary.Backlog != 3 { + t.Errorf("scan summary = %+v, want due=22 selected=20 checked=18 updates=17 attention=1 dispatched=15 dispatch_failed=1 busy=1 failed=2 backlog=3", summary) + } + if len(summary.ProviderBackoffs) != 1 || + summary.ProviderBackoffs[0].SourceID != fixtures[1].source.ID || + summary.ProviderBackoffs[0].StatusCode != 429 || + summary.ProviderBackoffs[0].RetryAt != retryAt.UTC().Format(time.RFC3339) { + t.Errorf("scan provider backoffs = %+v, want source=%d status=429 retry_at=%s", summary.ProviderBackoffs, fixtures[1].source.ID, retryAt.UTC().Format(time.RFC3339)) + } + + if len(calledRepositories) != 19 { + t.Fatalf("Resolve calls = %d, want 19 (one busy source in selected batch)", len(calledRepositories)) + } + for index, repository := range calledRepositories { + want := fixtures[index+1].repository + if repository != want { + t.Fatalf("Resolve order[%d] = %q, want %q", index, repository, want) + } + } + + if !containsDispatchedSource(dispatched, fixtures[2].source.ID, stored304Revision) { + t.Errorf("304 stored revision was not dispatched: %+v", dispatched) + } + if containsDispatchedSource(dispatched, fixtures[3].source.ID, "") { + t.Errorf("auto=false source was dispatched: %+v", dispatched) + } + if containsDispatchedSource(dispatched, fixtures[4].source.ID, "") { + t.Errorf("attention source was dispatched: %+v", dispatched) + } + + _, dispatchFailedRuntime := mustLoadPagesSource(t, ctx, fixtures[5].source.ProjectID) + if dispatchFailedRuntime.SyncStatus != pagesSourceStatusUpdateAvailable || + dispatchFailedRuntime.LastError != errPagesSourceTaskDispatchFailed || + dispatchFailedRuntime.NextCheckAt == nil || dispatchFailedRuntime.NextCheckAt.Before(retryAt) || + dispatchFailedRuntime.LastSeenRevision == "" { + t.Errorf( + "dispatch failure runtime = status:%q error:%q next:%v seen:%q, want preserved update and provider deadline >= %v", + dispatchFailedRuntime.SyncStatus, + dispatchFailedRuntime.LastError, + dispatchFailedRuntime.NextCheckAt, + dispatchFailedRuntime.LastSeenRevision, + retryAt, + ) + } +} + +func TestPagesSourceScannerIncludesOrphanCleanupSummary(t *testing.T) { + ctx := setupPagesSourceTest(t) + now := time.Now().Truncate(time.Second) + usePagesSourceScannerClock(t, now) + + previousReconcile := reconcilePagesSourceOrphans + reconcilePagesSourceOrphans = func( + _ context.Context, + gotNow time.Time, + ) (PagesOrphanCleanupSummary, error) { + if !gotNow.Equal(now) { + t.Errorf("orphan cleanup now = %v, want %v", gotNow, now) + } + return PagesOrphanCleanupSummary{ + Candidates: 7, + Reconciled: 1, + Referenced: 2, + LeaseBusy: 1, + InvalidMarker: 1, + Skipped: 1, + Failed: 1, + }, nil + } + t.Cleanup(func() { reconcilePagesSourceOrphans = previousReconcile }) + + result, err := (&SourceScanHandler{}).Execute(ctx, []byte("{}")) + if err != nil { + t.Fatalf("SourceScanHandler.Execute() error = %v, want nil", err) + } + var summary pagesSourceScanSummary + if err := json.Unmarshal([]byte(result.Detail), &summary); err != nil { + t.Fatalf("json.Unmarshal(scan detail) error = %v, want nil", err) + } + if summary.OrphanCleanup.Candidates != 7 || summary.OrphanCleanup.Reconciled != 1 || + summary.OrphanCleanup.Referenced != 2 || summary.OrphanCleanup.LeaseBusy != 1 || + summary.OrphanCleanup.InvalidMarker != 1 || summary.OrphanCleanup.Skipped != 1 || + summary.OrphanCleanup.Failed != 1 { + t.Errorf("orphan cleanup summary = %+v, want injected result", summary.OrphanCleanup) + } +} + +func TestPagesSourceScanPayloadAndMetaAreInternalOnly(t *testing.T) { + handler := &SourceScanHandler{} + if normalized, err := handler.ValidatePayload(nil); err != nil || string(normalized) != "{}" { + t.Errorf("ValidatePayload(nil) = %s, %v; want {}, nil", normalized, err) + } + if _, err := handler.ValidatePayload([]byte(`{"unexpected":true}`)); err == nil { + t.Error("ValidatePayload(unknown field) error = nil, want non-nil") + } + if !PagesSourceScanMeta.InternalOnly || PagesSourceScanMeta.Type != TaskTypePagesSourceScan || + PagesSourceScanMeta.AsynqTask != PagesSourceScanTask || PagesSourceScanMeta.MaxRetry != 0 { + t.Errorf("PagesSourceScanMeta = %+v, want internal bounded scheduled scanner", PagesSourceScanMeta) + } + if PagesSourceScanMeta.SupportsTime { + t.Error("PagesSourceScanMeta.SupportsTime = true, want empty scanner payload") + } +} + +func usePagesSourceScannerClock(t *testing.T, now time.Time) { + t.Helper() + previous := pagesSourceScanNow + pagesSourceScanNow = func() time.Time { return now } + t.Cleanup(func() { pagesSourceScanNow = previous }) +} + +func containsDispatchedSource(dispatched []scannerDispatchedSync, sourceID uint, revision string) bool { + for _, item := range dispatched { + if item.SourceID == sourceID && (revision == "" || item.Revision == revision) { + return true + } + } + return false +} diff --git a/internal/apps/openflare/pages/source_sync.go b/internal/apps/openflare/pages/source_sync.go index 4ef52431..972305dd 100644 --- a/internal/apps/openflare/pages/source_sync.go +++ b/internal/apps/openflare/pages/source_sync.go @@ -25,10 +25,11 @@ import ( ) const ( - pagesSourceTriggerManualSync = "manual_sync" - pagesSourceCreatedBySystem = "system:pages-source-sync" - pagesSourceHeartbeatInterval = pagesSourceSyncLeaseDuration / 3 - pagesSourceCleanupTimeout = 15 * time.Second + pagesSourceTriggerManualSync = "manual_sync" + pagesSourceTriggerScheduledAutoUpdate = "scheduled_auto_update" + pagesSourceCreatedBySystem = "system:pages-source-sync" + pagesSourceHeartbeatInterval = pagesSourceSyncLeaseDuration / 3 + pagesSourceCleanupTimeout = 15 * time.Second ) var ( @@ -154,16 +155,17 @@ func sourceCleanupContext(ctx context.Context) (context.Context, context.CancelF return context.WithTimeout(context.WithoutCancel(ctx), pagesSourceCleanupTimeout) } -func syncRemoteSource( +func syncRemoteSourceWithTrigger( ctx context.Context, snapshot *sourceExecutionSnapshot, actor string, + triggerType string, ) (outcome *sourceSyncOutcome, resultErr error) { if snapshot == nil || snapshot.SourceType != PagesSourceTypeRemoteURL { return nil, errors.New(errPagesSourceTypeUnsupported) } actor = strings.TrimSpace(actor) - if actor == "" { + if actor == "" || !validSourceDeploymentTrigger(triggerType) { return nil, errors.New(errPagesSourceActionInvalid) } defer func() { @@ -222,7 +224,7 @@ func syncRemoteSource( } task.AppendLog(ctx, "[activate] 正在原子切换生产部署") - deployment, reused, referenced, err := commitSourceDeployment( + deployment, reused, referenced, err := commitSourceDeploymentWithTrigger( ctx, snapshot, prepared.Candidate.Checksum, @@ -230,6 +232,7 @@ func syncRemoteSource( prepared.Detail, prepared.DetailJSON, actor, + triggerType, prepared.Manifest, ingestState.Result, ingestState.HasIngest, @@ -382,7 +385,7 @@ func findSourceDeployment( return &deployment, nil } -func commitSourceDeployment( +func commitSourceDeploymentWithTrigger( ctx context.Context, snapshot *sourceExecutionSnapshot, revision string, @@ -390,6 +393,7 @@ func commitSourceDeployment( detail sourceDetail, detailJSON string, actor string, + triggerType string, manifest *deploymentManifest, ingestResult upload.IngestResult, hasIngest bool, @@ -398,6 +402,9 @@ func commitSourceDeployment( if snapshot == nil || manifest == nil { return nil, false, false, errors.New(errPagesSourceSyncFailed) } + if !validSourceDeploymentTrigger(triggerType) { + return nil, false, false, errors.New(errPagesSourceActionInvalid) + } var committed model.PagesDeployment reused := false ingestReferenced := false @@ -407,7 +414,8 @@ func commitSourceDeployment( return err } target, targetReused, err := resolveSourceDeploymentTx( - tx, state, revision, packageChecksum, detail, detailJSON, actor, manifest, ingestResult, hasIngest, + tx, state, revision, packageChecksum, detail, detailJSON, actor, triggerType, + manifest, ingestResult, hasIngest, ) if err != nil { return err @@ -499,6 +507,7 @@ func resolveSourceDeploymentTx( detail sourceDetail, detailJSON string, actor string, + triggerType string, manifest *deploymentManifest, ingestResult upload.IngestResult, hasIngest bool, @@ -520,7 +529,7 @@ func resolveSourceDeploymentTx( return nil, false, errSourceFinalFence } return createSourceDeploymentTx( - tx, state, revision, packageChecksum, detail, detailJSON, actor, manifest, ingestResult, + tx, state, revision, packageChecksum, detail, detailJSON, actor, triggerType, manifest, ingestResult, ) } @@ -532,6 +541,7 @@ func createSourceDeploymentTx( detail sourceDetail, detailJSON string, actor string, + triggerType string, manifest *deploymentManifest, ingestResult upload.IngestResult, ) (*model.PagesDeployment, bool, error) { @@ -558,7 +568,7 @@ func createSourceDeploymentTx( SourceRevision: &revisionValue, SourceLabel: sourceDetailLabel(detail), SourceMeta: detailJSON, - TriggerType: pagesSourceTriggerManualSync, + TriggerType: triggerType, } result := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(target) if result.Error != nil { @@ -573,6 +583,10 @@ func createSourceDeploymentTx( return target, false, nil } +func validSourceDeploymentTrigger(triggerType string) bool { + return triggerType == pagesSourceTriggerManualSync || triggerType == pagesSourceTriggerScheduledAutoUpdate +} + func reloadSourceDeploymentTx( tx *gorm.DB, projectID uint, @@ -657,7 +671,7 @@ func activateSourceDeploymentTx( state.Source.ReleaseSelector == githubReleaseSelectorLatest { next := nextGitHubCheckAt(finishedAt, state.Source.ID, state.Source.CheckIntervalMinutes) if nextCheckNotBefore != nil && nextCheckNotBefore.After(next) { - next = nextCheckNotBefore.UTC() + next = nextCheckNotBefore.In(finishedAt.Location()) } nextCheckAt = &next } diff --git a/internal/apps/openflare/pages/source_tasks.go b/internal/apps/openflare/pages/source_tasks.go index 5e72b383..9d8fdf65 100644 --- a/internal/apps/openflare/pages/source_tasks.go +++ b/internal/apps/openflare/pages/source_tasks.go @@ -51,6 +51,7 @@ type SourceActionPayload struct { ConfigVersion int `json:"config_version"` Action string `json:"action"` Actor string `json:"actor"` + TriggerType string `json:"trigger_type"` TargetRevision string `json:"target_revision"` ConfirmedRevision string `json:"confirmed_revision"` } @@ -71,22 +72,59 @@ func (h *SourceActionHandler) ValidatePayload(payload []byte) ([]byte, error) { } input.Action = strings.TrimSpace(input.Action) input.Actor = strings.TrimSpace(input.Actor) + input.TriggerType = strings.TrimSpace(input.TriggerType) input.TargetRevision = strings.TrimSpace(input.TargetRevision) input.ConfirmedRevision = strings.TrimSpace(input.ConfirmedRevision) - if input.SourceID == 0 || input.ConfigVersion <= 0 || - (input.Action != sourceActionCheck && input.Action != sourceActionSync) || - !validPagesSourceActor(input.Actor) || - !validOptionalSourceRevision(input.TargetRevision) || - !validOptionalSourceRevision(input.ConfirmedRevision) || - (input.Action == sourceActionCheck && (input.TargetRevision != "" || input.ConfirmedRevision != "")) || - (input.TargetRevision != "" && input.ConfirmedRevision != "") || - (input.TargetRevision != "" && input.Actor != pagesSourceCreatedBySystem) || - (input.ConfirmedRevision != "" && !strings.HasPrefix(input.Actor, "user:")) { + if input.Action == sourceActionSync && input.TriggerType == "" { + // Keep already queued Phase 2 payloads valid while making every new + // dispatch carry an explicit deployment trigger. + input.TriggerType = pagesSourceTriggerManualSync + } + if !validSourceActionPayload(input) { return nil, errors.New(errPagesSourceActionInvalid) } return json.Marshal(input) } +func validSourceActionPayload(input SourceActionPayload) bool { + if input.SourceID == 0 || input.ConfigVersion <= 0 { + return false + } + if input.Action != sourceActionCheck && input.Action != sourceActionSync { + return false + } + if !validPagesSourceActor(input.Actor) { + return false + } + if !validOptionalSourceRevision(input.TargetRevision) || + !validOptionalSourceRevision(input.ConfirmedRevision) { + return false + } + if input.Action == sourceActionCheck { + return input.TriggerType == "" && input.TargetRevision == "" && input.ConfirmedRevision == "" + } + return validSourceSyncPayload(input) +} + +func validSourceSyncPayload(input SourceActionPayload) bool { + if !validSourceDeploymentTrigger(input.TriggerType) || + (input.TargetRevision != "" && input.ConfirmedRevision != "") { + return false + } + switch input.TriggerType { + case pagesSourceTriggerScheduledAutoUpdate: + return input.Actor == pagesSourceCreatedBySystem && + input.TargetRevision != "" && input.ConfirmedRevision == "" + case pagesSourceTriggerManualSync: + if input.TargetRevision != "" || !strings.HasPrefix(input.Actor, "user:") { + return false + } + return true + default: + return false + } +} + // Execute validates again inside the worker and performs the source action. func (h *SourceActionHandler) Execute(ctx context.Context, payload []byte) (*task.TaskResult, error) { normalized, err := h.ValidatePayload(payload) @@ -169,9 +207,11 @@ func executeSourceSyncAction( var result *sourceSyncOutcome var err error if source.SourceType == PagesSourceTypeGitHubRelease { - result, err = syncGitHubSource(ctx, snapshot, input.Actor, input.TargetRevision, input.ConfirmedRevision) + result, err = syncGitHubSourceWithTrigger( + ctx, snapshot, input.Actor, input.TargetRevision, input.ConfirmedRevision, input.TriggerType, + ) } else { - result, err = syncRemoteSource(ctx, snapshot, input.Actor) + result, err = syncRemoteSourceWithTrigger(ctx, snapshot, input.Actor, input.TriggerType) } if err != nil { logger.ErrorF(ctx, "[PagesSource] sync failed: project_id=%d source_id=%d error=%v", snapshot.ProjectID, snapshot.SourceID, err) @@ -333,6 +373,25 @@ func dispatchSourceActionSnapshot( targetRevision string, confirmedRevision string, triggeredBy string, +) (*SourceActionReceipt, error) { + triggerType := "" + if action == sourceActionSync { + triggerType = pagesSourceTriggerManualSync + } + return dispatchSourceActionSnapshotWithTrigger( + ctx, source, action, actor, triggerType, targetRevision, confirmedRevision, triggeredBy, + ) +} + +func dispatchSourceActionSnapshotWithTrigger( + ctx context.Context, + source model.PagesProjectSource, + action string, + actor string, + triggerType string, + targetRevision string, + confirmedRevision string, + triggeredBy string, ) (*SourceActionReceipt, error) { if task.AsynqClient == nil { return nil, errors.New(errPagesSourceTaskDispatchFailed) @@ -343,6 +402,7 @@ func dispatchSourceActionSnapshot( ConfigVersion: source.ConfigVersion, Action: action, Actor: actor, + TriggerType: triggerType, TargetRevision: targetRevision, ConfirmedRevision: confirmedRevision, }) diff --git a/internal/apps/openflare/pages/source_tasks_test.go b/internal/apps/openflare/pages/source_tasks_test.go index 92458dc6..d51b9af7 100644 --- a/internal/apps/openflare/pages/source_tasks_test.go +++ b/internal/apps/openflare/pages/source_tasks_test.go @@ -19,6 +19,7 @@ func TestSourceActionPayloadValidationIsStrictAndCredentialFree(t *testing.T) { ConfigVersion: 3, Action: sourceActionSync, Actor: "user:42", + TriggerType: pagesSourceTriggerManualSync, } raw, err := json.Marshal(valid) if err != nil { @@ -35,6 +36,23 @@ func TestSourceActionPayloadValidationIsStrictAndCredentialFree(t *testing.T) { if got != valid { t.Errorf("ValidatePayload(valid) = %+v, want %+v", got, valid) } + legacy := valid + legacy.TriggerType = "" + legacyRaw, err := json.Marshal(legacy) + if err != nil { + t.Fatalf("json.Marshal(legacy payload) error = %v, want nil", err) + } + legacyNormalized, err := handler.ValidatePayload(legacyRaw) + if err != nil { + t.Fatalf("ValidatePayload(legacy payload) error = %v, want nil", err) + } + var legacyGot SourceActionPayload + if err := json.Unmarshal(legacyNormalized, &legacyGot); err != nil { + t.Fatalf("json.Unmarshal(legacy normalized payload) error = %v, want nil", err) + } + if legacyGot.TriggerType != pagesSourceTriggerManualSync { + t.Errorf("legacy payload trigger_type = %q, want %q", legacyGot.TriggerType, pagesSourceTriggerManualSync) + } for _, forbidden := range []string{"remote_url", "content_config_version", "expected_revision", "lease_token", "etag"} { if strings.Contains(string(normalized), forbidden) { t.Errorf("normalized payload = %s, want no forbidden field %q", normalized, forbidden) diff --git a/internal/db/migrator/goose/postgres/202607190002_seed_pages_source_scan.sql b/internal/db/migrator/goose/postgres/202607190002_seed_pages_source_scan.sql new file mode 100644 index 00000000..d50e45b3 --- /dev/null +++ b/internal/db/migrator/goose/postgres/202607190002_seed_pages_source_scan.sql @@ -0,0 +1,38 @@ +-- +goose Up +-- Earlier built-in schedules used explicit IDs, so advance the identity only +-- when it trails either existing rows or an already-higher sequence value. +SELECT setval( + pg_get_serial_sequence('w_schedules', 'id'), + GREATEST( + 1, + COALESCE((SELECT MAX(id) FROM w_schedules), 0), + COALESCE(( + SELECT sequences.last_value + FROM pg_sequences AS sequences + WHERE format('%I.%I', sequences.schemaname, sequences.sequencename)::regclass = + pg_get_serial_sequence('w_schedules', 'id')::regclass + ), 0) + ), + TRUE +); + +INSERT INTO w_schedules (name, task_type, cron, payload, is_active, created_at, updated_at) +SELECT + 'OpenFlare Pages 部署源扫描', + 'of_pages_source_scan', + '*/5 * * * *', + '{}', + TRUE, + CURRENT_TIMESTAMP, + CURRENT_TIMESTAMP +WHERE NOT EXISTS ( + SELECT 1 FROM w_schedules WHERE task_type = 'of_pages_source_scan' +); + +-- +goose Down +DELETE FROM w_schedules +WHERE task_type = 'of_pages_source_scan' + AND name = 'OpenFlare Pages 部署源扫描' + AND cron = '*/5 * * * *' + AND payload = '{}' + AND is_active = TRUE; diff --git a/internal/db/migrator/goose/sqlite/202607190002_seed_pages_source_scan.sql b/internal/db/migrator/goose/sqlite/202607190002_seed_pages_source_scan.sql new file mode 100644 index 00000000..dd0235bf --- /dev/null +++ b/internal/db/migrator/goose/sqlite/202607190002_seed_pages_source_scan.sql @@ -0,0 +1,21 @@ +-- +goose Up +INSERT INTO w_schedules (name, task_type, cron, payload, is_active, created_at, updated_at) +SELECT + 'OpenFlare Pages 部署源扫描', + 'of_pages_source_scan', + '*/5 * * * *', + '{}', + 1, + CURRENT_TIMESTAMP, + CURRENT_TIMESTAMP +WHERE NOT EXISTS ( + SELECT 1 FROM w_schedules WHERE task_type = 'of_pages_source_scan' +); + +-- +goose Down +DELETE FROM w_schedules +WHERE task_type = 'of_pages_source_scan' + AND name = 'OpenFlare Pages 部署源扫描' + AND cron = '*/5 * * * *' + AND payload = '{}' + AND is_active = 1; diff --git a/internal/db/migrator/pages_source_scan_migration_test.go b/internal/db/migrator/pages_source_scan_migration_test.go new file mode 100644 index 00000000..3ee01b33 --- /dev/null +++ b/internal/db/migrator/pages_source_scan_migration_test.go @@ -0,0 +1,160 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package migrator + +import ( + "database/sql" + "fmt" + "os" + "strings" + "testing" + "time" + + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/glebarez/sqlite" + "github.com/pressly/goose/v3" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gorm.io/driver/postgres" + "gorm.io/gorm" +) + +const ( + pagesSourceScanPreviousMigration = int64(202607190001) + pagesSourceScanMigration = int64(202607190002) + pagesSourceScanTaskType = "of_pages_source_scan" +) + +func TestPagesSourceScanScheduleMigrationSQLite(t *testing.T) { + dbPath := t.TempDir() + "/pages-source-scan-migration.db" + gormDB, err := gorm.Open(sqlite.Open(dbPath), &gorm.Config{ + DisableForeignKeyConstraintWhenMigrating: true, + }) + require.NoError(t, err) + sqlDB, err := gormDB.DB() + require.NoError(t, err) + sqlDB.SetMaxOpenConns(1) + t.Cleanup(func() { require.NoError(t, sqlDB.Close()) }) + + runPagesSourceScanScheduleMigration(t, gormDB, sqlDB, dialectSqlite, "goose/sqlite") +} + +func TestPagesSourceScanScheduleMigrationPostgres(t *testing.T) { + dsn := strings.TrimSpace(os.Getenv("OPENFLARE_TEST_POSTGRES_DSN")) + if dsn == "" { + t.Skip("OPENFLARE_TEST_POSTGRES_DSN is not set") + } + gormDB, err := gorm.Open(postgres.Open(dsn), &gorm.Config{ + DisableForeignKeyConstraintWhenMigrating: true, + }) + require.NoError(t, err) + sqlDB, err := gormDB.DB() + require.NoError(t, err) + sqlDB.SetMaxOpenConns(1) + + schema := fmt.Sprintf("pages_source_scan_migration_%d", time.Now().UnixNano()) + require.Regexp(t, `^[a-z0-9_]+$`, schema) + require.NoError(t, gormDB.Exec(`CREATE SCHEMA "`+schema+`"`).Error) + require.NoError(t, gormDB.Exec(`SET search_path TO "`+schema+`"`).Error) + t.Cleanup(func() { + assert.NoError(t, gormDB.Exec("SET search_path TO public").Error) + assert.NoError(t, gormDB.Exec(`DROP SCHEMA IF EXISTS "`+schema+`" CASCADE`).Error) + assert.NoError(t, sqlDB.Close()) + }) + + runPagesSourceScanScheduleMigration(t, gormDB, sqlDB, dialectPostgres, "goose/postgres") +} + +func runPagesSourceScanScheduleMigration( + t *testing.T, + gormDB *gorm.DB, + sqlDB *sql.DB, + dialect string, + dir string, +) { + t.Helper() + goose.SetBaseFS(migrationFS) + require.NoError(t, goose.SetDialect(dialect)) + require.NoError(t, goose.UpTo(sqlDB, dir, pagesSourceScanPreviousMigration)) + + var previousMaxID uint64 + require.NoError(t, gormDB.Table("w_schedules").Select("COALESCE(MAX(id), 0)").Scan(&previousMaxID).Error) + require.NoError(t, goose.UpTo(sqlDB, dir, pagesSourceScanMigration)) + seeded := assertPagesSourceScanSchedule(t, gormDB) + assert.NotZero(t, seeded.ID) + if dialect == dialectPostgres { + assert.Greater(t, seeded.ID, previousMaxID) + } + + require.NoError(t, goose.DownTo(sqlDB, dir, pagesSourceScanPreviousMigration)) + assertPagesSourceScanScheduleMissing(t, gormDB) + + custom := model.Schedule{ + ID: 900001, + Name: "用户保留的 Pages 扫描任务", + TaskType: pagesSourceScanTaskType, + Cron: "0 * * * *", + Payload: `{"custom":true}`, + IsActive: false, + } + require.NoError(t, gormDB.Create(&custom).Error) + require.NoError(t, goose.UpTo(sqlDB, dir, pagesSourceScanMigration)) + var schedules []model.Schedule + require.NoError(t, gormDB.Where("task_type = ?", pagesSourceScanTaskType).Find(&schedules).Error) + require.Len(t, schedules, 1) + assert.Equal(t, custom.ID, schedules[0].ID) + assert.Equal(t, custom.Name, schedules[0].Name) + + require.NoError(t, goose.DownTo(sqlDB, dir, pagesSourceScanPreviousMigration)) + var retained model.Schedule + require.NoError(t, gormDB.First(&retained, custom.ID).Error) + assert.Equal(t, custom.TaskType, retained.TaskType) +} + +func TestPagesSourceScanScheduleMigrationsUseDatabaseGeneratedIDs(t *testing.T) { + for _, name := range []string{ + "goose/postgres/202607190002_seed_pages_source_scan.sql", + "goose/sqlite/202607190002_seed_pages_source_scan.sql", + } { + t.Run(name, func(t *testing.T) { + content, err := migrationFS.ReadFile(name) + require.NoError(t, err) + normalized := strings.ToLower(string(content)) + assert.NotContains(t, normalized, "insert into w_schedules (id,") + assert.NotContains(t, normalized, "coalesce(max(id)") + }) + } + + postgresContent, err := migrationFS.ReadFile("goose/postgres/202607190002_seed_pages_source_scan.sql") + require.NoError(t, err) + compactPostgres := strings.Join(strings.Fields(strings.ToLower(string(postgresContent))), " ") + assert.Contains( + t, + compactPostgres, + "select setval( pg_get_serial_sequence('w_schedules', 'id'), greatest( 1,", + "sequence synchronization must retain a valid lower bound for an empty table", + ) +} + +func assertPagesSourceScanSchedule(t *testing.T, gormDB *gorm.DB) model.Schedule { + t.Helper() + var schedules []model.Schedule + require.NoError(t, gormDB.Where("task_type = ?", pagesSourceScanTaskType).Find(&schedules).Error) + require.Len(t, schedules, 1) + schedule := schedules[0] + assert.Equal(t, "OpenFlare Pages 部署源扫描", schedule.Name) + assert.Equal(t, "*/5 * * * *", schedule.Cron) + assert.Equal(t, "{}", schedule.Payload) + assert.True(t, schedule.IsActive) + return schedule +} + +func assertPagesSourceScanScheduleMissing(t *testing.T, gormDB *gorm.DB) { + t.Helper() + var count int64 + require.NoError(t, gormDB.Model(&model.Schedule{}). + Where("task_type = ?", pagesSourceScanTaskType). + Count(&count).Error) + assert.Zero(t, count) +} diff --git a/internal/task/handlers/register.go b/internal/task/handlers/register.go index 641d9dca..005bfd9e 100644 --- a/internal/task/handlers/register.go +++ b/internal/task/handlers/register.go @@ -53,6 +53,9 @@ func Register() { task.RegisterTaskMeta(openflare.UptimeKumaSyncMeta) // pages source actions are only dispatched by the Pages domain API/scanner. + task.RegisterHandler(pages.PagesSourceScanTask, &pages.SourceScanHandler{}) + task.RegisterTaskMeta(pages.PagesSourceScanMeta) + task.RegisterHandler(pages.PagesSourceActionTask, &pages.SourceActionHandler{}) task.RegisterTaskMeta(pages.PagesSourceActionMeta) From 67b051c2bcd069dd35dfcdbb5e4bc852574118e6 Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 19:09:57 +0800 Subject: [PATCH 7/8] =?UTF-8?q?feat(frontend):=20=E6=94=AF=E6=8C=81=20Page?= =?UTF-8?q?s=20=E8=87=AA=E5=8A=A8=E6=9B=B4=E6=96=B0=E4=BA=A4=E4=BA=92?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在 GitHub latest 来源中提供自动更新开关、检查间隔和运行状态。\n页面按来源到期时间低频刷新，并在自动发布或人工回滚后同步项目与部署历史。 --- .../detail/components/deployment-history.tsx | 2 +- .../detail/components/pages-source-card.tsx | 116 +++++- .../components/pages-source-details.tsx | 26 ++ .../detail/components/pages-source-dialog.tsx | 48 ++- .../components/pages-source-github-fields.tsx | 99 ++++- frontend/app/(main)/pages/detail/page.tsx | 3 +- frontend/lib/services/openflare/types.ts | 26 +- .../pages-source-auto-update.test.tsx | 355 ++++++++++++++++++ .../tests/openflare/pages-source-ui.test.tsx | 33 +- 9 files changed, 658 insertions(+), 50 deletions(-) create mode 100644 frontend/tests/openflare/pages-source-auto-update.test.tsx diff --git a/frontend/app/(main)/pages/detail/components/deployment-history.tsx b/frontend/app/(main)/pages/detail/components/deployment-history.tsx index 7fe01105..76227e2d 100644 --- a/frontend/app/(main)/pages/detail/components/deployment-history.tsx +++ b/frontend/app/(main)/pages/detail/components/deployment-history.tsx @@ -119,7 +119,7 @@ export function DeploymentHistory({ mutationFn: (deploymentId: number) => PagesService.activateDeployment(projectId, deploymentId), onSuccess: async () => { - toast.success('部署已激活'); + toast.success('历史部署已激活；自动更新（如已开启）已关闭'); await invalidateDeploymentState(); setPendingAction(null); }, diff --git a/frontend/app/(main)/pages/detail/components/pages-source-card.tsx b/frontend/app/(main)/pages/detail/components/pages-source-card.tsx index 9e101436..c9a19872 100644 --- a/frontend/app/(main)/pages/detail/components/pages-source-card.tsx +++ b/frontend/app/(main)/pages/detail/components/pages-source-card.tsx @@ -38,6 +38,7 @@ import { Skeleton } from '@/components/ui/skeleton'; import { Spinner } from '@/components/ui/spinner'; import { AdminTaskService } from '@/lib/services/admin'; import { + type PagesSource, type PagesSourceActionPayload, type PagesSourceActionReceipt, type PagesSourceStatus, @@ -58,6 +59,9 @@ import { const ACTION_POLL_INTERVAL = 2_000; const ACTION_MAX_WAIT = 16 * 60 * 1_000; +const LATEST_POLL_INTERVAL = 5 * 60 * 1_000; +const LATEST_NEAR_DUE_POLL_INTERVAL = 30_000; +const LATEST_OVERDUE_MAX_WAIT = 10 * 60 * 1_000; const SOURCE_STATUS: Record< PagesSourceStatus, @@ -79,6 +83,60 @@ interface ActiveAction { startedAt: number; } +export interface LatestSourceOverdueWindow { + nextCheckAt: string; + startedAt: number; +} + +export interface LatestSourcePollingDecision { + interval: number | false; + overdueWindow: LatestSourceOverdueWindow | null; +} + +export function getLatestSourceIdlePollingDecision( + source: PagesSource | undefined, + now: number, + overdueWindow: LatestSourceOverdueWindow | null, +): LatestSourcePollingDecision { + if ( + source?.source_type !== 'github_release' || + source.release_selector !== 'latest' + ) { + return { interval: false, overdueWindow: null }; + } + + const nextCheckAt = source.next_check_at; + const nextCheckTime = nextCheckAt ? Date.parse(nextCheckAt) : Number.NaN; + if (!nextCheckAt || !Number.isFinite(nextCheckTime)) { + return { interval: LATEST_POLL_INTERVAL, overdueWindow: null }; + } + + const timeUntilCheck = nextCheckTime - now; + if (timeUntilCheck > LATEST_POLL_INTERVAL) { + return { interval: LATEST_POLL_INTERVAL, overdueWindow: null }; + } + if (timeUntilCheck > 0) { + return { interval: LATEST_NEAR_DUE_POLL_INTERVAL, overdueWindow: null }; + } + + const currentWindow = + overdueWindow?.nextCheckAt === nextCheckAt + ? overdueWindow + : { nextCheckAt, startedAt: now }; + if (now - currentWindow.startedAt >= LATEST_OVERDUE_MAX_WAIT) { + return { interval: false, overdueWindow: currentWindow }; + } + return { + interval: LATEST_NEAR_DUE_POLL_INTERVAL, + overdueWindow: currentWindow, + }; +} + +function sourceDeploymentFingerprint(source: PagesSource) { + if (source.source_type === 'manual') return '|'; + return `${source.last_synced_at ?? ''}|${source.last_applied?.revision ?? ''}`; +} + function sourceActionLabel(action: PagesSourceActionReceipt['action']) { return action === 'check' ? '检查' : '同步并发布'; } @@ -87,6 +145,8 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { const queryClient = useQueryClient(); const handledExecutionID = useRef(null); const sourcePollingStartedAt = useRef(null); + const latestOverdueWindow = useRef(null); + const sourceDeploymentState = useRef(undefined); const [dialogOpen, setDialogOpen] = useState(false); const [dialogMode, setDialogMode] = useState('manual'); const [activeAction, setActiveAction] = useState(null); @@ -104,15 +164,40 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { (source.sync_status === 'checking' || source.sync_status === 'syncing') ) { sourcePollingStartedAt.current ??= Date.now(); + latestOverdueWindow.current = null; return Date.now() - sourcePollingStartedAt.current < ACTION_MAX_WAIT ? ACTION_POLL_INTERVAL : false; } sourcePollingStartedAt.current = null; - return false; + const decision = getLatestSourceIdlePollingDecision( + source, + Date.now(), + latestOverdueWindow.current, + ); + latestOverdueWindow.current = decision.overdueWindow; + return decision.interval; }, }); + const invalidateSourceState = useCallback( + () => + Promise.all([ + queryClient.invalidateQueries({ queryKey: sourceQueryKey(projectId) }), + queryClient.invalidateQueries({ + queryKey: projectQueryKey(projectId), + }), + queryClient.invalidateQueries({ + queryKey: deploymentsQueryKey(projectId), + }), + queryClient.invalidateQueries({ + queryKey: ['openflare', 'pages', 'deployment-files', projectId], + }), + queryClient.invalidateQueries({ queryKey: projectsQueryKey }), + ]), + [projectId, queryClient], + ); + const executionQuery = useQuery({ queryKey: [ 'admin', @@ -151,6 +236,21 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { return () => window.clearTimeout(timeout); }, [actionTimedOut, activeAction]); + useEffect(() => { + const source = sourceQuery.data; + if (!source) return; + const fingerprint = sourceDeploymentFingerprint(source); + const previousFingerprint = sourceDeploymentState.current; + sourceDeploymentState.current = fingerprint; + if ( + previousFingerprint === undefined || + previousFingerprint === fingerprint + ) { + return; + } + void invalidateSourceState(); + }, [invalidateSourceState, sourceQuery.data]); + useEffect(() => { const execution = executionQuery.data; if ( @@ -163,17 +263,7 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { if (handledExecutionID.current === execution.id) return; handledExecutionID.current = execution.id; - void Promise.all([ - queryClient.invalidateQueries({ queryKey: sourceQueryKey(projectId) }), - queryClient.invalidateQueries({ queryKey: projectQueryKey(projectId) }), - queryClient.invalidateQueries({ - queryKey: deploymentsQueryKey(projectId), - }), - queryClient.invalidateQueries({ - queryKey: ['openflare', 'pages', 'deployment-files', projectId], - }), - queryClient.invalidateQueries({ queryKey: projectsQueryKey }), - ]); + void invalidateSourceState(); const actionLabel = sourceActionLabel(activeAction.receipt.action); if (execution.status === 'succeeded') { @@ -183,7 +273,7 @@ export function PagesSourceCard({ projectId }: { projectId: number }) { } setActiveAction(null); setActionTimedOut(false); - }, [activeAction, executionQuery.data, projectId, queryClient]); + }, [activeAction, executionQuery.data, invalidateSourceState]); const checkMutation = useMutation({ mutationFn: () => PagesService.checkSource(projectId), diff --git a/frontend/app/(main)/pages/detail/components/pages-source-details.tsx b/frontend/app/(main)/pages/detail/components/pages-source-details.tsx index 100db35b..f7812b18 100644 --- a/frontend/app/(main)/pages/detail/components/pages-source-details.tsx +++ b/frontend/app/(main)/pages/detail/components/pages-source-details.tsx @@ -98,6 +98,32 @@ export function GitHubSourceDetails({ Release Asset {source.asset_name}
+ {source.release_selector === 'latest' ? ( + <> +
+ 自动更新 + + {source.auto_update_enabled ? '已开启' : '已关闭'} + +
+
+ 检查间隔 + + {source.check_interval_minutes} 分钟 + +
+
+ + 下次检查时间 + + + {source.next_check_at + ? formatDateTime(source.next_check_at) + : '等待调度'} + +
+ + ) : null}
远端已发现 diff --git a/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx b/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx index 3915bcdf..73968f92 100644 --- a/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx +++ b/frontend/app/(main)/pages/detail/components/pages-source-dialog.tsx @@ -1,6 +1,6 @@ 'use client'; -import { useEffect, useState } from 'react'; +import { useEffect, useRef, useState } from 'react'; import { useMutation, useQueryClient } from '@tanstack/react-query'; import { Eye, EyeOff } from 'lucide-react'; import { toast } from 'sonner'; @@ -78,6 +78,7 @@ const EMPTY_GITHUB_ERRORS: PagesGitHubSourceFormErrors = { repository: '', releaseTag: '', assetName: '', + checkInterval: '', }; function githubRepositoryURL(repository: string) { @@ -106,17 +107,25 @@ export function PagesSourceDialog({ releaseSelector: 'latest', releaseTag: '', assetName: DEFAULT_GITHUB_ASSET, + autoUpdateEnabled: false, + checkIntervalMinutes: String(DEFAULT_GITHUB_CHECK_INTERVAL), }); const [githubErrors, setGitHubErrors] = useState(EMPTY_GITHUB_ERRORS); const [confirmation, setConfirmation] = useState(null); + const initializedForOpen = useRef(false); useEffect(() => { if (!open) { + initializedForOpen.current = false; setRemoteURL(''); setRevealRemoteURL(false); return; } + // Runtime polling may replace the source view while the dialog is open. + // Initialize only on the open edge so it cannot overwrite an unsaved draft. + if (initializedForOpen.current) return; + initializedForOpen.current = true; const nextMode = initialMode ?? source.source_type; setMode(nextMode); setNetworkPolicy( @@ -145,6 +154,18 @@ export function PagesSourceDialog({ source.source_type === 'github_release' ? source.asset_name : DEFAULT_GITHUB_ASSET, + autoUpdateEnabled: + source.source_type === 'github_release' && + source.release_selector === 'latest' + ? source.auto_update_enabled + : false, + checkIntervalMinutes: + source.source_type === 'github_release' && + source.release_selector === 'latest' + ? String( + source.check_interval_minutes || DEFAULT_GITHUB_CHECK_INTERVAL, + ) + : String(DEFAULT_GITHUB_CHECK_INTERVAL), }); setGitHubErrors(EMPTY_GITHUB_ERRORS); setConfirmation(null); @@ -243,13 +264,27 @@ export function PagesSourceDialog({ const nextAssetNameError = validGitHubAssetName(githubForm.assetName) ? '' : 'Asset 文件名须为 1–255 字节，且不能是路径或包含控制、换行、双向文本字符'; + const checkIntervalMinutes = Number(githubForm.checkIntervalMinutes); + const nextCheckIntervalError = + githubForm.releaseSelector === 'latest' && + (!Number.isInteger(checkIntervalMinutes) || + checkIntervalMinutes < 5 || + checkIntervalMinutes > 1440) + ? '检查间隔须为 5–1440 分钟的整数' + : ''; setGitHubErrors({ repository: nextRepositoryError, releaseTag: nextReleaseTagError, assetName: nextAssetNameError, + checkInterval: nextCheckIntervalError, }); - if (nextRepositoryError || nextReleaseTagError || nextAssetNameError) { + if ( + nextRepositoryError || + nextReleaseTagError || + nextAssetNameError || + nextCheckIntervalError + ) { return; } @@ -261,13 +296,8 @@ export function PagesSourceDialog({ release_selector: 'latest', release_tag: '', asset_name: githubForm.assetName, - auto_update_enabled: false, - check_interval_minutes: - source.source_type === 'github_release' && - source.release_selector === 'latest' && - source.check_interval_minutes - ? source.check_interval_minutes - : DEFAULT_GITHUB_CHECK_INTERVAL, + auto_update_enabled: githubForm.autoUpdateEnabled, + check_interval_minutes: checkIntervalMinutes, } : { source_type: 'github_release', diff --git a/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx b/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx index 7a8d3422..84ec99a4 100644 --- a/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx +++ b/frontend/app/(main)/pages/detail/components/pages-source-github-fields.tsx @@ -1,10 +1,13 @@ import { Field, + FieldContent, FieldDescription, + FieldError, FieldLabel, FieldTitle, } from '@/components/ui/field'; import { Input } from '@/components/ui/input'; +import { Switch } from '@/components/ui/switch'; import { ToggleGroup, ToggleGroupItem } from '@/components/ui/toggle-group'; import type { PagesGitHubReleaseSelector } from '@/lib/services/openflare'; @@ -13,12 +16,15 @@ export interface PagesGitHubSourceFormValue { releaseSelector: PagesGitHubReleaseSelector; releaseTag: string; assetName: string; + autoUpdateEnabled: boolean; + checkIntervalMinutes: string; } export interface PagesGitHubSourceFormErrors { repository: string; releaseTag: string; assetName: string; + checkInterval: string; } interface PagesSourceGitHubFieldsProps { @@ -48,15 +54,19 @@ export function PagesSourceGitHubFields({ placeholder='https://github.com/owner/repo' value={value.repositoryURL} aria-invalid={Boolean(errors.repository)} + aria-describedby='pages-github-repository-description pages-github-repository-error' autoComplete='off' onChange={(event) => { onChange({ ...value, repositoryURL: event.target.value }); onErrorsChange({ ...errors, repository: '' }); }} /> - - {errors.repository || '仅支持公开 github.com 仓库。'} + + 仅支持公开 github.com 仓库。 + + {errors.repository} + @@ -69,8 +79,17 @@ export function PagesSourceGitHubFields({ className='grid w-full grid-cols-2' onValueChange={(selector) => { if (selector === 'latest' || selector === 'tag') { - onChange({ ...value, releaseSelector: selector }); - onErrorsChange({ ...errors, releaseTag: '' }); + onChange({ + ...value, + releaseSelector: selector, + autoUpdateEnabled: + selector === 'latest' ? value.autoUpdateEnabled : false, + }); + onErrorsChange({ + ...errors, + releaseTag: '', + checkInterval: '', + }); } }} > @@ -82,7 +101,9 @@ export function PagesSourceGitHubFields({ - 当前阶段由管理员手动检查并决定是否发布。 + {value.releaseSelector === 'latest' + ? '定时检查最新 Release；是否自动发布由下方开关控制。' + : '固定 Tag 仅支持管理员手动检查与同步。'} @@ -94,18 +115,74 @@ export function PagesSourceGitHubFields({ placeholder='v1.2.3' value={value.releaseTag} aria-invalid={Boolean(errors.releaseTag)} + aria-describedby='pages-github-tag-description pages-github-tag-error' autoComplete='off' onChange={(event) => { onChange({ ...value, releaseTag: event.target.value }); onErrorsChange({ ...errors, releaseTag: '' }); }} /> - - {errors.releaseTag || '精确检查并同步指定 tag。'} + + 精确检查并同步指定 tag。 + + {errors.releaseTag} + ) : null} + {value.releaseSelector === 'latest' ? ( + <> + + + + 自动更新 + + + 检查到新的 Release 后自动同步并发布。 + + + + onChange({ ...value, autoUpdateEnabled: checked }) + } + /> + + + + + 检查间隔（分钟） + + { + onChange({ + ...value, + checkIntervalMinutes: event.target.value, + }); + onErrorsChange({ ...errors, checkInterval: '' }); + }} + /> + + 可设置为 5–1440 分钟。 + + + {errors.checkInterval} + + + + ) : null} + Release Asset 文件名 @@ -115,15 +192,19 @@ export function PagesSourceGitHubFields({ placeholder={defaultAssetName} value={value.assetName} aria-invalid={Boolean(errors.assetName)} + aria-describedby='pages-github-asset-description pages-github-asset-error' autoComplete='off' onChange={(event) => { onChange({ ...value, assetName: event.target.value }); onErrorsChange({ ...errors, assetName: '' }); }} /> - - {errors.assetName || '按文件名精确匹配已上传的 Release Asset。'} + + 按文件名精确匹配已上传的 Release Asset。 + + {errors.assetName} + ); diff --git a/frontend/app/(main)/pages/detail/page.tsx b/frontend/app/(main)/pages/detail/page.tsx index e653471b..bdeb0c6f 100644 --- a/frontend/app/(main)/pages/detail/page.tsx +++ b/frontend/app/(main)/pages/detail/page.tsx @@ -252,8 +252,9 @@ function PagesDetailRoute() {
- + diff --git a/frontend/lib/services/openflare/types.ts b/frontend/lib/services/openflare/types.ts index 66eace1c..237bc85e 100644 --- a/frontend/lib/services/openflare/types.ts +++ b/frontend/lib/services/openflare/types.ts @@ -492,16 +492,29 @@ export interface PagesRemoteURLSource extends PagesSourceRuntimeView { remote_network_policy: PagesRemoteNetworkPolicy; } -export interface PagesGitHubReleaseSource extends PagesSourceRuntimeView { +interface PagesGitHubReleaseSourceBase extends PagesSourceRuntimeView { source_type: 'github_release'; github_repository: string; - release_selector: PagesGitHubReleaseSelector; - release_tag?: string; asset_name: string; - auto_update_enabled?: false; - check_interval_minutes?: number; } +interface PagesGitHubLatestReleaseSource extends PagesGitHubReleaseSourceBase { + release_selector: 'latest'; + release_tag?: ''; + auto_update_enabled: boolean; + check_interval_minutes: number; +} + +interface PagesGitHubTagReleaseSource extends PagesGitHubReleaseSourceBase { + release_selector: 'tag'; + release_tag: string; + auto_update_enabled: false; + check_interval_minutes?: 0; +} + +export type PagesGitHubReleaseSource = + PagesGitHubLatestReleaseSource | PagesGitHubTagReleaseSource; + /** * 部署源使用判别联合，后续仓库构建来源只需增加独立 git_repository variant， * 不需要向 Remote 或 GitHub Release 填入构建字段。 @@ -520,18 +533,19 @@ interface PagesGitHubSourceUpdateBase { source_type: 'github_release'; repository_url: string; asset_name: string; - auto_update_enabled: false; } export interface PagesGitHubLatestSourceUpdatePayload extends PagesGitHubSourceUpdateBase { release_selector: 'latest'; release_tag: ''; + auto_update_enabled: boolean; check_interval_minutes: number; } export interface PagesGitHubTagSourceUpdatePayload extends PagesGitHubSourceUpdateBase { release_selector: 'tag'; release_tag: string; + auto_update_enabled: false; check_interval_minutes: 0; } diff --git a/frontend/tests/openflare/pages-source-auto-update.test.tsx b/frontend/tests/openflare/pages-source-auto-update.test.tsx new file mode 100644 index 00000000..6ae7db61 --- /dev/null +++ b/frontend/tests/openflare/pages-source-auto-update.test.tsx @@ -0,0 +1,355 @@ +import { act } from 'react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import { render, screen, waitFor, within } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { DeploymentHistory } from '@/app/(main)/pages/detail/components/deployment-history'; +import { + getLatestSourceIdlePollingDecision, + PagesSourceCard, +} from '@/app/(main)/pages/detail/components/pages-source-card'; +import { + deploymentsQueryKey, + projectQueryKey, + projectsQueryKey, + sourceQueryKey, +} from '@/app/(main)/pages/components/pages-utils'; +import { + type PagesDeployment, + type PagesGitHubReleaseSource, + PagesService, +} from '@/lib/services/openflare'; + +vi.mock('@/lib/services/openflare', async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + PagesService: { + getSource: vi.fn(), + updateSource: vi.fn(), + deleteSource: vi.fn(), + checkSource: vi.fn(), + syncSource: vi.fn(), + listDeployments: vi.fn(), + listDeploymentFiles: vi.fn(), + activateDeployment: vi.fn(), + deleteDeployment: vi.fn(), + }, + }; +}); + +function renderWithQuery(ui: React.ReactNode) { + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false, gcTime: 0 }, + mutations: { retry: false }, + }, + }); + const rendered = render( + {ui}, + ); + return { + ...rendered, + queryClient, + rerenderWithQuery: (nextUI: React.ReactNode) => + rendered.rerender( + + {nextUI} + , + ), + }; +} + +const latestSource: PagesGitHubReleaseSource = { + source_type: 'github_release', + github_repository: 'openflare/site', + release_selector: 'latest', + asset_name: 'dist.zip', + auto_update_enabled: true, + check_interval_minutes: 120, + sync_status: 'idle', + update_available: false, + last_seen: { + revision: 'b'.repeat(64), + label: 'v1.2.3', + asset_name: 'dist.zip', + }, + last_applied: { + revision: 'a'.repeat(64), + label: 'v1.2.2', + asset_name: 'dist.zip', + }, + last_checked_at: '2026-07-19T10:00:00Z', + last_synced_at: '2026-07-19T09:00:00Z', + next_check_at: '2026-07-19T12:00:00Z', + last_error: '', +}; + +describe('Pages latest source automatic updates', () => { + beforeEach(() => { + vi.mocked(PagesService.getSource).mockReset(); + vi.mocked(PagesService.updateSource).mockReset(); + vi.mocked(PagesService.deleteSource).mockReset(); + vi.mocked(PagesService.checkSource).mockReset(); + vi.mocked(PagesService.syncSource).mockReset(); + vi.mocked(PagesService.listDeployments).mockReset(); + vi.mocked(PagesService.listDeploymentFiles).mockReset(); + vi.mocked(PagesService.activateDeployment).mockReset(); + vi.mocked(PagesService.deleteDeployment).mockReset(); + }); + + it('backfills latest settings and hides them after selecting a fixed tag', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(latestSource); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '编辑来源' })); + expect(screen.getByRole('switch', { name: '自动更新' })).toBeChecked(); + expect(screen.getByLabelText('检查间隔（分钟）')).toHaveValue(120); + + await user.click(screen.getByRole('radio', { name: '固定 Tag' })); + expect( + screen.queryByRole('switch', { name: '自动更新' }), + ).not.toBeInTheDocument(); + expect(screen.queryByLabelText('检查间隔（分钟）')).not.toBeInTheDocument(); + }); + + it('does not show automatic schedule details for a fixed tag source', async () => { + vi.mocked(PagesService.getSource).mockResolvedValue({ + ...latestSource, + release_selector: 'tag', + release_tag: 'v1.2.3', + auto_update_enabled: false, + check_interval_minutes: 0, + next_check_at: null, + }); + + renderWithQuery(); + + expect(await screen.findByText('固定 Tag · v1.2.3')).toBeVisible(); + expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); + expect(screen.queryByText('检查间隔')).not.toBeInTheDocument(); + expect(screen.queryByText('下次检查时间')).not.toBeInTheDocument(); + }); + + it('rejects a latest interval outside 5–1440 minutes', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(latestSource); + + renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '编辑来源' })); + const intervalInput = screen.getByLabelText('检查间隔（分钟）'); + await user.clear(intervalInput); + await user.type(intervalInput, '4'); + await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); + + expect(screen.getByRole('alert')).toHaveTextContent( + '检查间隔须为 5–1440 分钟的整数', + ); + expect(intervalInput).toHaveAttribute( + 'aria-describedby', + 'pages-github-check-interval-description pages-github-check-interval-error', + ); + expect(PagesService.updateSource).not.toHaveBeenCalled(); + }); + + it('keeps an unsaved source draft when idle polling updates runtime fields', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockResolvedValue(latestSource); + const { queryClient } = renderWithQuery(); + + await user.click(await screen.findByRole('button', { name: '编辑来源' })); + const assetInput = screen.getByLabelText('Release Asset 文件名'); + const intervalInput = screen.getByLabelText('检查间隔（分钟）'); + await user.clear(assetInput); + await user.type(assetInput, 'draft.zip'); + await user.clear(intervalInput); + await user.type(intervalInput, '30'); + await user.click(screen.getByRole('switch', { name: '自动更新' })); + + act(() => { + queryClient.setQueryData(sourceQueryKey(9), { + ...latestSource, + last_checked_at: '2026-07-19T10:30:00Z', + next_check_at: '2026-07-19T12:30:00Z', + }); + }); + + await waitFor(() => { + expect(assetInput).toHaveValue('draft.zip'); + expect(intervalInput).toHaveValue(30); + expect( + screen.getByRole('switch', { name: '自动更新' }), + ).not.toBeChecked(); + }); + }); + + it('remounts project-scoped source state when the route project changes', async () => { + const user = userEvent.setup(); + vi.mocked(PagesService.getSource).mockImplementation(async (projectId) => ({ + ...latestSource, + github_repository: + projectId === 9 ? 'openflare/project-nine' : 'openflare/project-ten', + })); + const { rerenderWithQuery } = renderWithQuery( + , + ); + + await user.click(await screen.findByRole('button', { name: '编辑来源' })); + expect(screen.getByRole('dialog')).toBeVisible(); + + rerenderWithQuery(); + + expect(screen.queryByRole('dialog')).not.toBeInTheDocument(); + expect(await screen.findByText('openflare/project-ten')).toBeVisible(); + }); + + it('uses low-frequency, near-due and bounded overdue polling', () => { + const now = Date.parse('2026-07-19T10:00:00Z'); + const farSource = { + ...latestSource, + next_check_at: new Date(now + 60 * 60 * 1_000).toISOString(), + }; + expect(getLatestSourceIdlePollingDecision(farSource, now, null)).toEqual({ + interval: 5 * 60 * 1_000, + overdueWindow: null, + }); + + const nearSource = { + ...latestSource, + next_check_at: new Date(now + 60_000).toISOString(), + }; + expect(getLatestSourceIdlePollingDecision(nearSource, now, null)).toEqual({ + interval: 30_000, + overdueWindow: null, + }); + + const overdueSource = { + ...latestSource, + next_check_at: new Date(now - 1).toISOString(), + }; + const overdue = getLatestSourceIdlePollingDecision( + overdueSource, + now, + null, + ); + expect(overdue.interval).toBe(30_000); + expect( + getLatestSourceIdlePollingDecision( + overdueSource, + now + 10 * 60 * 1_000, + overdue.overdueWindow, + ).interval, + ).toBe(false); + + expect( + getLatestSourceIdlePollingDecision( + { + ...latestSource, + release_selector: 'tag', + release_tag: 'v1.2.3', + auto_update_enabled: false, + check_interval_minutes: 0, + }, + now, + null, + ).interval, + ).toBe(false); + }); + + it('refreshes source consumers after a background deployment is applied', async () => { + vi.mocked(PagesService.getSource).mockResolvedValue(latestSource); + const { queryClient } = renderWithQuery(); + expect(await screen.findByText('openflare/site')).toBeVisible(); + + const invalidateQueries = vi.spyOn(queryClient, 'invalidateQueries'); + const updatedSource: PagesGitHubReleaseSource = { + ...latestSource, + last_applied: { + revision: 'b'.repeat(64), + label: 'v1.2.3', + asset_name: 'dist.zip', + }, + last_synced_at: '2026-07-19T10:30:00Z', + }; + vi.mocked(PagesService.getSource).mockResolvedValue(updatedSource); + + act(() => { + queryClient.setQueryData(sourceQueryKey(9), updatedSource); + }); + + await waitFor(() => { + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: sourceQueryKey(9), + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: projectQueryKey(9), + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: deploymentsQueryKey(9), + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: ['openflare', 'pages', 'deployment-files', 9], + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: projectsQueryKey, + }); + }); + }); + + it('warns that rollback closes auto update and refreshes four query groups', async () => { + const user = userEvent.setup(); + const deployment: PagesDeployment = { + id: 51, + project_id: 9, + deployment_number: 7, + checksum: 'd'.repeat(64), + status: 'uploaded', + file_count: 12, + total_size: 4_096, + created_by: 'user:1', + source_type: 'github_release', + source_label: 'v1.2.2', + trigger_type: 'scheduled_auto_update', + created_at: '2026-07-19T10:00:00Z', + activated_at: null, + }; + vi.mocked(PagesService.listDeployments).mockResolvedValue([deployment]); + vi.mocked(PagesService.activateDeployment).mockResolvedValue({} as never); + + const { queryClient } = renderWithQuery( + , + ); + await screen.findByText('GitHub · v1.2.2 · 定时更新'); + const invalidateQueries = vi.spyOn(queryClient, 'invalidateQueries'); + + await user.click(screen.getByRole('button', { name: '激活' })); + const dialog = screen.getByRole('alertdialog'); + expect( + within(dialog).getByText( + '激活其它历史部署会终止当前来源任务；若已开启自动更新，将同时关闭自动更新。', + ), + ).toBeVisible(); + await user.click(within(dialog).getByRole('button', { name: '确认' })); + + await waitFor(() => { + expect(PagesService.activateDeployment).toHaveBeenCalledWith(9, 51); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: deploymentsQueryKey(9), + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: projectQueryKey(9), + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: sourceQueryKey(9), + }); + expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: projectsQueryKey, + }); + }); + }); +}); diff --git a/frontend/tests/openflare/pages-source-ui.test.tsx b/frontend/tests/openflare/pages-source-ui.test.tsx index bdf61cc8..cf4d1383 100644 --- a/frontend/tests/openflare/pages-source-ui.test.tsx +++ b/frontend/tests/openflare/pages-source-ui.test.tsx @@ -150,7 +150,7 @@ describe('Pages source UI', () => { vi.mocked(AdminTaskService.getTaskExecution).mockReset(); }); - it('offers the three Phase 2 source types without future repository build controls', async () => { + it('offers the three source types without future repository build controls', async () => { const user = userEvent.setup(); vi.mocked(PagesService.getSource).mockResolvedValue({ source_type: 'manual', @@ -177,11 +177,13 @@ describe('Pages source UI', () => { expect( screen.getByText('仓库源码构建将在后续作为独立来源类型提供。'), ).toBeVisible(); + expect(screen.getByRole('switch', { name: '自动更新' })).not.toBeChecked(); + expect(screen.getByLabelText('检查间隔（分钟）')).toHaveValue(60); expect(screen.queryByText('构建命令')).not.toBeInTheDocument(); expect(screen.queryByText('输出目录')).not.toBeInTheDocument(); }); - it('submits the GitHub latest payload while keeping automatic controls hidden', async () => { + it('submits the GitHub latest automatic update settings', async () => { const user = userEvent.setup(); vi.mocked(PagesService.getSource).mockResolvedValue({ source_type: 'manual', @@ -204,8 +206,10 @@ describe('Pages source UI', () => { expect(screen.getByLabelText('Release Asset 文件名')).toHaveValue( 'dist.zip', ); - expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); - expect(screen.queryByText('检查间隔')).not.toBeInTheDocument(); + await user.click(screen.getByRole('switch', { name: '自动更新' })); + const intervalInput = screen.getByLabelText('检查间隔（分钟）'); + await user.clear(intervalInput); + await user.type(intervalInput, '15'); await user.click(screen.getByRole('button', { name: '保存 GitHub 来源' })); await waitFor(() => { @@ -215,8 +219,8 @@ describe('Pages source UI', () => { release_selector: 'latest', release_tag: '', asset_name: 'dist.zip', - auto_update_enabled: false, - check_interval_minutes: 60, + auto_update_enabled: true, + check_interval_minutes: 15, }); }); }); @@ -257,7 +261,8 @@ describe('Pages source UI', () => { ...githubLatestSource, release_selector: 'tag', release_tag: 'v1.2.3', - check_interval_minutes: undefined, + auto_update_enabled: false, + check_interval_minutes: 0, next_check_at: null, }, check_task: null, @@ -274,6 +279,10 @@ describe('Pages source UI', () => { 'https://github.com/openflare/site', ); await user.click(screen.getByRole('radio', { name: '固定 Tag' })); + expect( + screen.queryByRole('switch', { name: '自动更新' }), + ).not.toBeInTheDocument(); + expect(screen.queryByLabelText('检查间隔（分钟）')).not.toBeInTheDocument(); await user.type( screen.getByLabelText('Release tag'), 'release/candidate#1&channel=stable', @@ -380,7 +389,7 @@ describe('Pages source UI', () => { expect(PagesService.syncSource).not.toHaveBeenCalled(); }); - it('shows GitHub release state and hides Phase 3 controls', async () => { + it('shows GitHub latest automatic update state and schedule', async () => { vi.mocked(PagesService.getSource).mockResolvedValue(githubLatestSource); renderWithQuery(); @@ -389,9 +398,11 @@ describe('Pages source UI', () => { expect(screen.getByText('v1.2.3 · bbbbbbbbbbbb')).toBeVisible(); expect(screen.getByText('v1.2.2 · aaaaaaaaaaaa')).toBeVisible(); expect(screen.getByText('有可用更新')).toBeVisible(); - expect(screen.queryByText('下次检查时间')).not.toBeInTheDocument(); - expect(screen.queryByText('自动更新')).not.toBeInTheDocument(); - expect(screen.queryByText('检查间隔')).not.toBeInTheDocument(); + expect(screen.getByText('下次检查时间')).toBeVisible(); + expect(screen.getByText('自动更新')).toBeVisible(); + expect(screen.getByText('已关闭')).toBeVisible(); + expect(screen.getByText('检查间隔')).toBeVisible(); + expect(screen.getByText('60 分钟')).toBeVisible(); }); it('dispatches a GitHub check and starts TaskExecution polling', async () => { From ccea08fe4735ac59c1c3857e7c5ce857a4750d64 Mon Sep 17 00:00:00 2001 From: deqiying Date: Sun, 19 Jul 2026 19:25:28 +0800 Subject: [PATCH 8/8] =?UTF-8?q?docs(pages):=20=E6=94=B6=E5=8F=A3=E9=83=A8?= =?UTF-8?q?=E7=BD=B2=E6=BA=90=20V2=20=E5=AE=9E=E7=8E=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 同步 Pages、总体架构、Agent 与使用指南，记录阶段提交、验证结果和生产验收边界。 --- README.md | 2 +- docs/design/agent-design.md | 3 +- docs/design/architecture.md | 6 +- docs/design/index.md | 7 +- docs/design/pages-design.md | 115 +++++++++------- docs/guide/pages-usage.md | 148 +++++++++++++-------- docs/plan/20260719-pages-source-sync-v2.md | 91 ++++++++++--- docs/plan/index.md | 5 +- 8 files changed, 244 insertions(+), 133 deletions(-) diff --git a/README.md b/README.md index f168e95e..c67f3c7f 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ OpenFlare 是开源 CDN 编排与边缘安全平台。它支持反向代理、 * **安全内网穿透（Tunnels）**：开源版的 Cloudflare Tunnels。无须公网 IP 或暴露入向端口，通过 Relay 中继节点与 OpenFlared 客户端安全反向穿透内网 Web 服务至公网。 * **边缘 WAF 安全防护**：提供全局与自定义规则组，支持手动/自动/订阅型 IP 组、MaxMind GeoIP 国家级地域准入、IP 组成员 Checksum 差分同步（无需 Nginx 重载）以及自定义拦截响应。 * **防 CC 与人机挑战（PoW）**：内置高性能客户端密码学 Proof of Work 挑战（类似 Turnstile），在网关边缘秒级拦截并阻断僵尸网络与爬虫。 -* **Pages 静态托管**：直接上传预构建 ZIP 包，由边缘 Agent 拉取并通过 OpenResty 本地提供服务，支持 SPA Fallback 与内置 API 反向代理配置。 +* **Pages 静态托管**：支持上传或从受限 Remote URL、公开 GitHub Release asset 同步预构建产物；GitHub latest 可定时检查并可选自动发布。所有来源统一生成不可变部署，由边缘 Agent 拉取并通过 OpenResty 本地提供服务，支持回滚、SPA Fallback 与 API 反向代理。 * **TLS 证书自动化**：支持证书动态上传、多域名证书自动匹配绑定，以及通过 ACME 协议向 Let's Encrypt 自动申请与续期证书。 * **Uptime Kuma 监控同步**：与 Uptime Kuma 集成，自动差分同步监控站点列表，实时感知节点存活与服务可用状态。 * **SSO 单点登录**：支持 GitHub OAuth 与标准 OIDC 协议，无缝接入企业身份提供商实现统一登录。 diff --git a/docs/design/agent-design.md b/docs/design/agent-design.md index 4bb2f665..d42bdd4d 100644 --- a/docs/design/agent-design.md +++ b/docs/design/agent-design.md @@ -98,7 +98,7 @@ Agent 对数据面 OpenResty 的管控实现了端到端的闭环，包含配置 * `certs/`：证书存放目录（文件命名为 `{cert_id}.crt` 和 `{cert_id}.key`）。 * `waf/` 与 `pow/`：WAF 及防 CC 挑战所需的专用 Lua 运行时脚本。 * `waf_config.json` 与 `waf_ip_groups.json`：WAF 过滤引擎所需的结构化规则配置文件。 -* `pages_dir`：Pages 静态站点部署目录，默认位于 `data_dir/var/lib/openflare/pages`。当激活配置引用 Pages **项目**时，Agent 按 `project_id` 请求控制面「最新激活包」（hash + package，下载后再校验 hash 防竞态），解压到 `projects/{project_id}/releases/{hash}`，切换 `current` 后**立即删除同项目其它历史 release**（仅保留最新）。项目内切换激活无需重发主配置；多项目对账时单项目失败不阻塞其它项目。 +* `pages_dir`：Pages 静态站点部署目录，默认位于 `data_dir/var/lib/openflare/pages`。当激活配置引用 Pages **项目**时，Agent 按 `project_id` 请求控制面「最新激活包」（hash + package），以流式方式写入临时文件并执行实际响应上限与 SHA-256 校验，再安全解压到 `projects/{project_id}/releases/{hash}`。解压后会复核文件数与总字节，绝对防御上限为 2 GiB 包、1,000 个文件、单文件及总量 8 GiB；随后原子切换 `current` 并**立即删除同项目其它历史 release**（仅保留最新）。项目内切换激活无需重发主配置；多项目对账时单项目失败不阻塞其它项目。 ### 2. 精细化的重载动作 1. **备份当前配置**：在写入新文件之前，Agent 会将现有的配置文件复制到 `.backup` 临时目录下，保留完整的现场快照。 @@ -174,3 +174,4 @@ graph TD 2. **严格的 Token 过滤与前缀验证**：Agent 侧向 Server 请求资源时，接口端点固定以 `/api/v1/agent/` 为前缀，并强制携带 `X-Agent-Token` 进行签名或令牌核验。 3. **节点自治原则**：Agent 须具备完备的离线工作能力。在与 Server 失去连接期间，本地 OpenResty 必须依靠本地已落地的配置保持反向代理服务的绝对正常运行。 4. **观测只上报事实**：访问日志以明细形式上送；主机指标上报计数器/瞬时读数。禁止在 Agent 内计算业务 UV、Top 域名、24h 已提供数据等结论性指标（由 Server 聚合）。详见 [边缘可观测与业务流量统计](./observability-design.md)。 +5. **Pages 只消费控制面产物**：Remote URL、GitHub Release、自动 scanner，以及未来仓库 checkout/build executor 均属于 Server 职责。Agent 不接收外部 URL、访问令牌、仓库凭据或任意 clone/install/build 命令，只拉取已经激活且带完整性元数据的部署包。 diff --git a/docs/design/architecture.md b/docs/design/architecture.md index f58a915c..8cea6e51 100644 --- a/docs/design/architecture.md +++ b/docs/design/architecture.md @@ -80,7 +80,7 @@ OpenResty (Agent, TLS/WAF) * 提供管理端 REST API（`/api/v1/d/*`），通过 **Session Cookie** 鉴权，可选 `X-Access-Token` 访问令牌。 * 边缘节点协议走 `/api/v1/agent|relay|tunnel/*`，分别使用 `X-Agent-Token` / `X-Tunnel-Token` 鉴权。 * 包含配置编译器（Compiler），将数据库中的规则、证书与全局参数统一编译为不可变的配置快照及 OpenResty 物理配置文件文本。 -* 统一接收 Pages 本地上传、Remote URL 与公开 GitHub Release 预构建产物，完成来源检查、受限下载、归档校验、不可变 deployment 与原子激活，并向 Agent 提供受控的 latest 下载接口。未来仓库源码构建由独立 Server build executor 扩展，Agent 不执行第三方拉取或构建命令。 +* 统一接收 Pages 本地上传、Remote URL 与公开 GitHub Release 预构建产物，完成来源检查、受限下载、归档校验和不可变 deployment；manual 上传生成待显式激活的 candidate，持久来源 sync 才 create-or-load 并原子激活。Server 向 Agent 提供受控的 latest 下载接口；内部 scanner 负责 GitHub latest 的限量检查、租约恢复、可选自动发布与孤儿上传记录补偿，通用任务管理入口不能修改该排程。未来仓库源码构建由独立 Server build executor 扩展，Agent 不执行第三方拉取或构建命令。 * 后台集成 Uptime Kuma 监控同步服务，自动为可用站点维护 HTTP 探测任务。 * 启动入口为根目录 `main.go` + `internal/cmd/`（`api` / `worker` / `scheduler` / `all`）；OpenFlare 业务在 `internal/apps/openflare/`，边缘协议处理在 `internal/apps/openflare/{agent,relay,flared}/`。 * *详细设计请参阅：[Agent 与发布模型设计](./agent-design.md) 以及 [Uptime Kuma 监控同步设计](./kuma-design.md)* @@ -126,6 +126,7 @@ OpenResty (Agent, TLS/WAF) ### 2. 静态托管与 API 代理流 * 静态资源解压落地于 Agent 节点的 `projects/{project_id}/current` 下（按项目 latest 拉取，仅保留最新包），OpenResty 通过 `root`/`index`/`try_files` 在边缘直接提供静态资源服务。 * 当启用 API 代理时，OpenResty 自动根据站点配置的 `api_proxy_path`（如 `/api`）将 API 请求重写并转发（`proxy_pass`）给后端动态接口。 +* 管理员操作和内部 scanner 都只生成受约束的 artifact candidate，并复用统一 inspect、`upload.Ingest` 与 deployment pipeline。manual 上传创建新的未激活 candidate；持久来源 sync/scanner 才 create-or-load 并原子激活。未来 repository build executor 也只能向同一 artifact pipeline 输出产物；Agent 始终只是 active deployment 消费者。 * *部署包校验、解压逃逸防御及 Nginx 规则渲染详见：[Pages 静态托管设计文档](./pages-design.md)* ### 3. WAF 安全过滤流 @@ -160,7 +161,7 @@ OpenResty 健康与连接数 --> 边缘健康（瞬时，不作 24h 业务总量 当前系统核心实体包括： * **反代与配置**：`zones` (根域管理边界), `zone_domains` (明确域名与证书/路由关联), `proxy_routes` (路由策略), `origins` (源站), `config_versions` (配置版本), `tls_certificates` (证书). 详见 [Zone 与域名资源设计](./zone-design.md)。 -* **Pages 静态托管**：`pages_projects` (Pages项目), `pages_project_sources` / `pages_project_source_runtime` (可变来源配置与运行态), `pages_deployments` (不可变部署), `pages_deployment_files` (部署文件清单). +* **Pages 静态托管**：`of_pages_projects` (Pages项目), `of_pages_project_sources` / `of_pages_project_source_runtime` (可变来源配置与运行态), `of_pages_deployments` (不可变部署), `of_pages_deployment_files` (部署文件清单). * **节点与穿透**：`nodes` (节点), `tunnels` (隧道客户端), `node_system_profiles` (系统概况), `apply_logs` (应用日志). * **WAF 与安全**：`waf_rule_groups` (WAF规则组), `waf_ip_groups` (WAF IP组), `waf_rule_group_bindings` (网站WAF绑定). * **系统与账号**：`acme_accounts` (ACME账户), `dns_accounts` (DNS账户), `geoip_update_configs` (GeoIP更新配置). @@ -179,6 +180,7 @@ OpenResty 健康与连接数 --> 边缘健康（瞬时，不作 24h 业务总量 | 运行时配置与控制库解耦 | WAF 规则发布时编译并随 OpenResty reload 加载；动态 IP 组通过 checksum 驱动的内存快照独立刷新 | | 业务流量以访问日志为唯一真相 | Agent 禁止业务预聚合；看板与 Zone 共用 Server 侧聚合，避免 openresty_tx 与 bytes_sent 双轨 | | 业务交付 / 边缘健康 / 主机资源分层 | 已提供数据≠宿主机网卡出站≠OpenResty 连接数，UI 与 API 分名分区 | +| Pages artifact 与仓库构建分离 | 现有来源只导入预构建产物；未来 checkout/build 由 Server 隔离 executor 完成并复用 artifact pipeline，Agent 不执行第三方构建 | --- diff --git a/docs/design/index.md b/docs/design/index.md index b2fb386d..821a053a 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -27,7 +27,7 @@ OpenFlare 适合需要统一管理多台 OpenResty 代理节点的团队，具 | **配置版本控制** | 支持全局单一激活版本的预览、发布、不可变快照历史与秒级一键回滚 | [Agent 与发布模型](./agent-design.md) | | **WAF 安全防护** | 支持可视化 DAG 编排规则、手动/自动/订阅型 IP 组、GeoIP 匹配与 PoW CC 防护 | [WAF 设计](./waf-design.md) / [WAF 可编排规则设计](./waf-orchestration-design.md) / [WAF 使用指南](../guide/waf-usage.md) | | **内网穿透** | 通过中继节点（Relay）与内网客户端（OpenFlared），反向穿透暴露内网 Web 服务 | [内网穿透设计](./tunnel-design.md) / [穿透使用指南](../guide/tunnel-usage.md) | -| **Pages 静态托管** | 直接上传前端压缩包（zip / tar.gz / tar.xz / 7z 等），由边缘节点拉取并由 OpenResty 本地服务，支持 API 反代与 SPA Fallback | [Pages 静态托管设计](./pages-design.md) | +| **Pages 静态托管** | 支持上传或从 Remote URL、公开 GitHub Release 同步预构建产物；GitHub latest 可定时检查并可选自动发布。不可变部署由边缘节点拉取并由 OpenResty 本地服务，支持回滚、API 反代与 SPA Fallback | [Pages 静态托管设计](./pages-design.md) / [Pages 使用指南](../guide/pages-usage.md) | | **TLS 证书自动续期** | 将证书显式绑定到 Zone 域名，并通过 ACME 协议向 Let's Encrypt 申请/续期证书 | [Zone 与域名资源设计](./zone-design.md) | | **多节点监控与观测** | 访问日志为业务流量唯一真相；Agent 只上报明细与主机读数，Server 统一聚合；与 Zone/看板对账 | [观测数据传输模型](./observability-transport-model.md) / [边缘可观测与业务流量统计](./observability-design.md) / [上报协议与表结构](./observability-data-model.md) / [系统架构](./architecture.md) | @@ -53,8 +53,9 @@ OpenFlare 适合需要统一管理多台 OpenResty 代理节点的团队，具 * **Tunnel 与 Node 体系隔离**：Tunnel 客户端在内网发起出向建连，与控制面托管的边缘 Node（公网节点）是独立的实体，使用专属的 `tunnel_token` 进行鉴权。 ### 4. Pages 静态托管边界 -* **Direct Upload 托管模式**：仅支持直接上传预构建的 ZIP 静态资源包。不支持外部 Git 仓库自动构建、边缘 Serverless 函数、动态 SSR 服务或生成的二级预览域名。 -* **包体硬上限限制**：为了保障边缘节点安全，ZIP 压缩包体最大 25 MiB，解压文件树不超过 1,000 个且总体积不超过 100 MiB。禁止上传含有任何软链接或目录跨越（Zip-Slip）的安全高危压缩包。 +* **预构建产物来源**：项目可保持手动上传，或配置一个 Remote URL / 公开 GitHub Release asset 来源。Remote 与固定 tag 只支持手动操作；只有 GitHub latest 进入定时检查并可选择自动更新。来源可切换，但不可变 deployment 与当前生产版本不会随 source 编辑或删除而丢失。 +* **归档与资源上限**：支持 `zip`、`tar.gz` / `tgz`、`tar.xz` / `txz`、`tar.bz2` / `tbz2`、`tar`、`7z`。压缩包上限由 `pages_max_package_size_mb` 控制（默认 100 MiB，范围 1～2048）；展开后的单文件和总量上限为包上限的 4 倍且最低 100 MiB，最多 1,000 个常规文件。Server 与 Agent 都校验实际字节，并拒绝路径逃逸、软/硬链接与特殊文件。 +* **构建与运行时边界**：当前不从外部 Git 仓库拉取源码或执行构建，也不提供边缘 Serverless、动态 SSR 或二级预览域名。未来仓库集成必须使用独立 `git_repository` Provider 与 Server 侧隔离 build executor，只向统一 artifact 管线输出受限产物；Agent 不接收仓库凭据、外部 URL 或 clone/install/build 命令。 ### 5. 系统与版本边界 * **全局单一激活版本**：所有节点拉取并消费同一份全局激活配置。不进行按节点分组的差异化配置发布。 diff --git a/docs/design/pages-design.md b/docs/design/pages-design.md index 07343276..6adf2a64 100644 --- a/docs/design/pages-design.md +++ b/docs/design/pages-design.md @@ -12,7 +12,7 @@ 2. **多节点分发困难**：当控制面管理多台边缘节点时，将静态文件同步分发到所有节点，并确保文件一致性，需要维护复杂的同步脚本（如 rsync 等）。 3. **回滚缺乏一致性**：一旦新前端包发布失败或存在严重缺陷，不仅要恢复静态文件，还要恢复对应的反代规则，很难做到原子回滚。 -为了解决这些问题，OpenFlare 引入了受 Cloudflare Pages 启发的 **Pages 静态托管** 功能。该功能将“前端部署包上传”与“网站代理规则配置”合二为一，依托 OpenFlare 的 pull-based（拉取式）协同架构，实现静态文件分发与反代配置发布的强一致性、不可变性与一键秒级回滚。 +为了解决这些问题，OpenFlare 引入了受 Cloudflare Pages 启发的 **Pages 静态托管** 功能。该功能将“预构建产物导入”与“网站代理规则配置”纳入同一控制面，依托 OpenFlare 的 pull-based（拉取式）协同架构，以不可变 deployment、单节点原子切换和周期对账实现多 Agent 最终收敛，并支持快速回滚。 --- @@ -20,7 +20,8 @@ Pages 静态托管子系统包含以下核心能力： * **预构建产物部署**：支持直接上传静态资源压缩包，也可为项目保存一个 Remote URL 或公开 GitHub Release asset 来源。外部来源只由 Server 访问，成功同步后统一创建或复用不可变 deployment 并原子激活。 -* **不可变部署快照**：每次上传产生一个带唯一 ID 和 SHA-256 Checksum 的不可变部署记录。支持按系统配置保留最近 N 个历史部署，并可随时激活和回滚。 +* **不可变部署快照**：本地上传每次创建新的候选 deployment；持久来源同步按 source identity/revision 创建或复用 deployment 并激活。所有部署都有唯一 ID 和整包 SHA-256，支持按系统配置保留最近 N 个历史版本并随时回滚。 +* **检查与自动更新**：GitHub latest 可按项目间隔定时检查；默认只提示可用更新，管理员显式开启后才按检查到的精确 revision 自动同步并发布。 * **SPA Fallback 支持**：支持对单页应用（SPA）进行 Fallback 路由配置，请求找不到静态文件时自动重定向到入口文件。 * **内置 API 反代服务**：支持在 Pages 规则内一键启用 API 代理，消除跨域问题，将请求转发给指定的后端服务。 * **安全包校验与解压缩**：内置路径逃逸防御、防软链接劫持、文件大小/数量上限与可配置上传包体积控制，保障节点物理安全。 @@ -30,7 +31,7 @@ Pages 静态托管子系统包含以下核心能力： 项目当前支持 manual、Remote URL、GitHub Release 三种来源视图。无 source 记录即 manual；切换或删除 source 不删除历史 deployment，也不改变当前 active deployment。Remote URL 只允许手动“同步并发布”；GitHub Release 支持 latest/tag 手动检查与同步，只有 latest 可选择定时检查和自动更新。 -source 是可变配置，deployment 是不可变事实。source 配置与运行态游标、状态、租约分别存储；deployment 只保存创建时的安全 provenance 快照。所有来源最终都进入同一条“下载或接收产物 → 真实字节与入口校验 → `upload.Ingest` → deployment → 原子激活”管线，Agent 不感知来源类型。 +source 是可变配置，deployment 是不可变事实。source 配置与运行态游标、状态、租约分别存储；deployment 只保存创建时的安全 provenance 快照。所有产物都复用“下载或接收产物 → 真实字节与入口校验 → `upload.Ingest` → deployment”的 artifact pipeline：manual 上传停在 candidate，等待管理员显式激活；持久来源 sync 才在同一业务事务中 create-or-load 并原子激活。Agent 只消费 active deployment，不感知来源类型。 后续从 Git 仓库拉取源码并自动构建时，将新增独立 `git_repository` provider 与隔离的 build executor。它输出受限的预构建产物后继续复用上述导入管线；不得把 clone、依赖安装或任意构建命令下发给 Agent，也不得把 branch/build/env 字段塞入现有 `github_release` source。当前 V2 不增加这些未来字段或空任务，只稳定 provider 输出、source discriminated view 与 deployment provenance 三个扩展边界。 @@ -50,35 +51,41 @@ graph TD OpenResty -->|2. 转发 API 代理| BackEnd[4. 后端 API 服务] %% 控制流与心跳 - Server[OpenFlare Server 控制面] <-->|Agent API / Heartbeat| Agent[openflare-agent 进程] - Server -.->|5. 存储部署包| LocalStore[(Server 本地存储)] + Admin[管理员 / CI] -->|上传或配置来源| Server[OpenFlare Server 控制面] + Providers[Remote / GitHub Provider] -->|受限 artifact candidate| Server + Scanner[内部 scanner / action task] -->|检查与自动同步| Server + Server <-->|Agent API / Heartbeat| Agent[openflare-agent 进程] + Server -.->|统一 upload.Ingest| UploadStore[(平台 upload backend)] Agent -->|1. 发现新版本| Server Agent -->|2. 下载部署包| Server - Agent -->|3. 校验并解压缩| StaticFiles - Agent -->|4. 应用并 Reload| OpenResty + Agent -->|3. 校验、解压并原子切换| StaticFiles style Browser fill:#f9f,stroke:#333,stroke-width:2px style StaticFiles fill:#9f9,stroke:#333,stroke-width:2px style Server fill:#f96,stroke:#333,stroke-width:2px ``` -* **控制面（Control Plane）**：Server 接收前端上传的部署包，并将包存储于本地磁盘，元数据写入数据库。配置发布时，编译出带有 `pages_deployment` 详情的不可变全局版本快照。 -* **数据面（Data Plane）**：Agent 在心跳同步中发现版本更新并引用了 Pages 部署，通过专属 API 下载对应的部署包并执行校验解压缩。OpenResty 拦截域名请求，在本地提供静态文件服务。 +* **控制面（Control Plane）**：Server 接收本地上传，或通过受限 Provider 获取 Remote/GitHub 预构建产物；action task 与内部 scanner 负责检查、同步和自动更新。所有产物经统一 inspect 与 `upload.Ingest` 写入平台存储后端；manual 上传创建新的 candidate，持久来源 sync 则 create-or-load deployment 并原子激活。配置发布时只编译稳定的项目锚点与静态服务元数据。 +* **数据面（Data Plane）**：Agent 在心跳/WS 对账中发现配置引用的 Pages 项目，通过专属 API 拉取该项目当前激活包并执行校验解压缩。OpenResty 在本地提供静态文件服务；Agent 不感知产物来自上传、Remote、GitHub 或未来 build executor。 --- ## 数据模型与元数据设计 ### 1. 核心数据库实体 -* **Pages 项目 (`pages_projects`)**： +* **Pages 项目 (`of_pages_projects`)**： * 记录项目的业务名称、Slug 标识（URL 友好型）、启用状态、静态服务根目录（RootDir，可为空）、入口文件名（EntryFile，默认 `index.html`）、SPA Fallback 设置，以及 API 反向代理配置（APIProxyPath, APIProxyPass, APIProxyRewrite）。 -* **Pages 部署 (`pages_deployments`)**： - * 记录单次上传生成的不可变快照。包含：部署号 (DeploymentNumber, 递增序列)、SHA-256 Checksum 校验和、部署状态 (uploaded/active)、部署包的本地存储路径、解压后的文件数与总字节数。 -* **部署文件清单 (`pages_deployment_files`)**： - * 存储每次部署的完整静态文件树路径与文件大小（来自压缩包声明的未压缩大小），供控制台展示与统计。 - * 不再为包内每个文件计算内容哈希；完整性由**整包** SHA-256（`pages_deployments.checksum`）保证，Agent 拉取时只校验整包 hash。 - * 控制面 inspect 通过文件句柄 / 随机访问读取归档索引（zip/7z 中央目录；tar 流式读 header 并丢弃 body），避免将整包 `ReadFile` 进内存，也避免逐文件解压算 hash。 +* **部署源配置 (`of_pages_project_sources`)**： + * 每个项目最多一条可变来源配置，使用 `source_type` 区分 Remote URL 与 GitHub Release。`config_version` 用于 fence 旧任务；Remote 完整 URL 只保存在配置表中，不会进入响应、日志、任务 payload 或 deployment provenance。V2 不承诺数据库列加密。 +* **部署源运行态 (`of_pages_project_source_runtime`)**： + * 与 source 1:1 保存 ETag、seen/applied revision、最近检查/同步、下次检查、错误和 lease。状态固定为 `idle | checking | update_available | syncing | failed | attention`，排队/完成状态由 `TaskExecution` 承担。 +* **Pages 部署 (`of_pages_deployments`)**： + * 记录不可变部署事实：项目内递增部署号、整包 SHA-256、`upload_id`、文件数/总字节、创建者，以及可空的 source identity/revision、来源安全快照与 trigger。`artifact_path` 仅为旧数据兼容字段，不再是新部署的存储真相。 +* **部署文件清单 (`of_pages_deployment_files`)**： + * 存储每次部署的完整常规文件路径与实际字节数，供控制台展示与统计。 + * 不再为包内每个文件计算内容哈希；完整性由**整包** SHA-256（`of_pages_deployments.checksum`）保证，Agent 拉取时校验整包 hash。 + * 控制面 inspect 通过文件句柄读取归档，流式消费每个常规文件体并核对声明大小与实际字节，避免将整包 `ReadFile` 进内存，也避免逐文件落盘计算 hash。 ### 2. 路由关联与快照 `proxy_routes` 路由规则通过 `upstream_type = "pages"` 及 `pages_project_id` 关联 Pages 项目。当路由类型为 `pages` 且该项目存在已激活的部署时，才允许将该路由加入发布流程。 @@ -97,7 +104,7 @@ graph TD "api_proxy_path": "/api", "api_proxy_pass": "http://api.internal:8000", "api_proxy_rewrite": "/api/(.*) /$1", - "local_root": "__OPENFLARE_PAGES_DIR__/deployments/12/current" + "local_root": "__OPENFLARE_PAGES_DIR__/projects/1/current" } ``` @@ -108,7 +115,7 @@ graph TD * Agent 按项目请求「最新激活包」（类似 `github/release/latest`）： * `GET /api/v1/agent/pages/projects/:project_id/latest/hash` * `GET /api/v1/agent/pages/projects/:project_id/latest/package` - * 控制面根据该项目**当前激活部署**返回哈希与压缩包；Agent 不关心具体 deployment_id。 + * 控制面根据该项目**当前激活部署**返回 deployment ID、哈希、包大小与展开清单元数据。Agent 用 deployment ID 与其它 latest 元数据识别下载期间的指针竞态，但主配置和本地目录的稳定锚点仍是 project ID。 * 因此：在项目内切换激活部署后，**不必发布主配置**；Agent 在周期性对账时轮询 latest hash，发现变化即下载并切换 `current`。 * 快照中的 `pages_deployment` 字段仍可记录发布时元数据（入口文件、SPA/API 代理等），但不作为 Agent 拉包的版本锁定。 @@ -117,40 +124,49 @@ graph TD ## Server 端 (控制面) 职责与生命周期 ### 1. 部署包安全校验与分析 -为了避免不可信的用户上传恶意压缩包攻击服务器，控制面在 `UploadDeployment` 时执行严格校验： +为了避免不可信产物攻击服务器，控制面对本地上传和所有外部来源执行同一套严格校验： * **格式支持**：`zip`、`tar.gz` / `tgz`、`tar.xz` / `txz`、`tar.bz2` / `tbz2`、`tar`、`7z`。 -* **大小限制**：压缩包体积由系统配置 `pages_max_package_size_mb` 控制（默认 100 MiB，范围 1～2048）；展开后总体积上限为「包大小 × 4」且不低于 100 MiB（按归档**声明**的未压缩大小累计，默认不流式重读每个文件内容）。 +* **大小限制**：压缩包体积由系统配置 `pages_max_package_size_mb` 控制（默认 100 MiB，范围 1～2048）；展开后的单文件与总体积上限为「包大小 × 4」且不低于 100 MiB。inspect 始终流式读取常规文件体，核对声明大小与实际字节并按实际值执行上限。 * **数量限制**：压缩包中包含的静态文件总数不得超过 1,000 个。 * **软链接阻断**：遍历归档文件，一旦检测到任何软链接，立即抛出错误并拒绝上传，防御软链接劫持攻击。 * **路径逃逸防御**：对每个压缩文件路径进行 `Clean` 并检查是否包含 `..` 或以 `/` 开头，防御目录跨越漏洞，防止写入系统敏感路径。 * **入口文件校验**：项目指定的入口文件（例如 `index.html`，可在 `project.RootDir` 下）必须在部署包中存在，否则拒绝上传。 * **公共根目录去噪**：许多打包工具会包含一个多余的主文件夹作为公共根前缀。控制面自动探测公共根前缀并将其安全剥离。 * **整包完整性**：上传/导入时对压缩包字节计算一次 SHA-256，写入部署记录；Agent 拉包后按整包 hash 对账。包内单文件不做内容哈希。 -* **可选体积实测**：`InspectOptions.VerifySizes` 可对流式统计实际字节并与声明大小比对（仍不算 hash）；默认关闭以降低上传 CPU/IO。 -* **历史保留**：系统配置 `pages_max_history_count`（默认 20，0 表示不限制）在每次上传成功后执行裁剪。语义为：**每个项目最多保留 N 条部署**；当前激活部署始终保留；其余名额按部署 ID 从新到旧填充；超出的非激活部署连同文件清单与存储对象一并删除。上传已成功时裁剪失败只记日志、不回滚上传；并发上传下可能短暂超过 N，后续上传的裁剪会收敛回 N。主配置版本回滚不依赖旧 Pages 包（见上节双轨关系）。 +* **实际体积复核**：`InspectOptions.VerifySizes` 只保留兼容意义；当前 inspect 无论该值为何都会读取常规文件体、核对声明值并累计实际大小，但仍不为单文件计算内容 hash。 +* **历史保留**：系统配置 `pages_max_history_count`（默认 20，0 表示不限制）在部署成功后执行裁剪。通常语义为：**每个项目最多保留 N 条部署**；当前激活部署始终保留，其余名额按部署 ID 从新到旧填充。`history_count=1` 时，manual 上传会临时保留 active 与最新 candidate 两条，下一次上传替换旧 candidate；candidate 激活后恢复严格上限。超出的非激活 deployment 与文件清单会删除，对应 upload record 通过平台原语幂等软删除；Pages 不直接物理删除可能被 dedup 共享的 blob。部署已成功时裁剪失败只记日志、不回滚激活；并发操作下可能短暂超过 N，后续裁剪会收敛回 N。主配置版本回滚不依赖旧 Pages 包（见上节双轨关系）。 ### 2. 部署包存储规划 -控制面通过统一上传框架（`upload.Ingest`）存储原始部署包，并在数据库中记录 `upload_id` 与文件清单。**大体积静态包不写入 config_versions 记录和任何配置推送通道**，以保障控制面数据同步的轻量与高效。 +控制面通过统一上传框架（`upload.Ingest`）把本地、Remote 和 GitHub 产物存入配置的本地/S3 后端，并在数据库中记录 `upload_id` 与文件清单。**大体积静态包不写入 config_versions 记录和任何配置推送通道**，以保障控制面数据同步的轻量与高效。 + +### 3. 来源检查、自动更新与上传补偿 + +* `openflare:pages_source_action` 执行管理员 check/sync 或 scanner 派发的精确 revision sync；payload 不携带 URL、Token、ETag 或 lease token。手动 sync 只接受真实用户 actor，自动 sync 只接受系统 actor 与 `scheduled_auto_update` trigger。 +* `openflare:pages_source_scan` 是固定 `*/5 * * * *` 的 internal-only TaskHandler，只接受 `{}`，不会出现在通用任务类型与排程管理界面。每轮按“恢复过期 lease → 补偿 orphan upload → 扫描到期来源”执行。 +* scanner 按 `next_check_at, source_id` 稳定排序，每批最多串行检查 20 个 GitHub latest source；ETag/304 仍推进检查时间，403/429 记录状态码和实际退避截止时间，单来源失败不阻塞后续来源。 +* 发现更新总会先保存 seen cursor。只有 `auto_update_enabled=true` 且状态为普通 `update_available` 时，才携带本次检查得到的精确 revision 派发同步；`attention`、Remote 和固定 tag 不会自动发布。人工激活其它 deployment 会 fence 在途任务并关闭 auto。 +* orphan 补偿每轮最多检查 100 条至少隔离 2 小时的 upload record，并要求 system owner、Pages 保留 type、V2 marker、无 deployment 引用。候选在 `project → source → runtime → upload` 锁序内复查，只通过上传框架软删除 record 和更新统计，不直接物理删除可能被 dedup 共享的 blob。 --- ## Agent 端 (数据落地) 职责与自愈 -Agent 运行在各边缘代理节点上，在应用配置版本前，必须先将 Pages 静态资源“原子”地拉取到节点本地。 +Agent 运行在各边缘代理节点上：首次应用引用 Pages 项目的配置时，以及后续周期性 latest 对账时，都会把当前激活的静态资源“原子”地拉取到节点本地。 ### 1. 按项目拉取 latest 1. Agent 从激活主配置中解析 `UpstreamType == "pages"` 的路由，收集稳定锚点 **`pages_project_id`**。 2. 对每个项目调用 `GET /api/v1/agent/pages/projects/:project_id/latest/hash` 获取控制面当前激活包哈希（类似 latest 指针）。 -3. 若本地 `projects/{project_id}/releases/{hash}` 尚未就绪，再下载 `.../latest/package`。下载后 **再次请求 hash** 与包内容 SHA-256 对齐，避免激活切换造成的竞态；不一致则有限次重试。 +3. 若本地 `projects/{project_id}/releases/{hash}` 尚未就绪，再把 `.../latest/package` 流式下载到临时文件，执行真实响应上限与 SHA-256；下载后 **再次请求 hash**，避免激活切换造成的竞态，不一致则有限次重试。 4. 请求头携带节点 `X-Agent-Token`。 ### 2. 安全解压缩、原子切换与只保留最新 -1. 下载字节计算 SHA-256，须与「下载后再次查询」的 latest hash 一致。 -2. 解压至 `projects/{project_id}/releases/{hash}.tmp`（支持 zip / tar.* / 7z）。Agent 信任控制面业务校验，仅做路径逃逸/软链防护。 -3. 写入 `.openflare-pages.json` 后 rename 为 `releases/{hash}`。 -4. **原子切换** `projects/{project_id}/current` 指向新 release（优先 symlink，失败则拷贝）。 -5. **仅当新包已就绪且 current 切换成功后**，删除该项目下其它 `releases/*`（含 `.tmp`），**不保留历史部署包**。边缘节点每个项目永远只保留一份最新内容。 -6. 多项目对账时 **隔离失败**：单个项目失败记日志并继续其它项目，最后汇总返回错误。 +1. 包体绝对上限为 2 GiB；下载内容的 SHA-256 须与「下载后再次查询」的 latest hash 一致，整个包不会进入 `[]byte`。 +2. 解压至 `projects/{project_id}/releases/.{hash}-.tmp` 随机 staging 目录（支持 zip / tar.* / 7z），拒绝路径逃逸、链接和特殊文件。Agent 同时服从 Server metadata 上限与本地绝对上限：最多 1,000 个文件，单文件及总量最多 8 GiB。 +3. 解压完成后遍历实际文件树，精确复核文件数与总字节是否等于 Server metadata；不一致时拒绝切换。 +4. 写入 `.openflare-pages.json` 后 rename 为 `releases/{hash}`。 +5. **原子切换** `projects/{project_id}/current` 指向新 release（优先 symlink，失败则拷贝）。 +6. **仅当新包已就绪且 current 切换成功后**，删除该项目下其它 `releases/*`（含 `.tmp`），**不保留历史部署包**。边缘节点每个项目永远只保留一份最新内容。 +7. 多项目对账时 **隔离失败**：单个项目失败记日志并继续其它项目，最后汇总返回错误。 --- @@ -221,26 +237,23 @@ server { ## 交互逻辑与同步流程 -一次完整的 Pages 上传与全局生效的生命周期如下： +一次完整的预构建产物导入与生效生命周期如下。首次绑定项目需要发布主配置；后续 active deployment 变化通过项目 latest 独立收敛： ```text - [ 前端管理员 ] [ Server (控制面) ] [ Agent (数据落地) ] [ OpenResty ] - | | | | - |--- 1. 上传 ZIP 包 ----->| | | - | |--- 2. 安全校验与解压分析 ----| | - | |--- 3. 归档包与持久化清单 ---| | - | | | | - |--- 4. 绑定路由并发布 -->| | | - | |--- 5. 生成新配置版本并广播 ->| | - | | | | - | | |--- 6. 下载 ZIP 部署包 -->| - | | |<-- 7. 返回文件数据 -------| - | | | | - | | |--- 8. 强一致性 Checksum -| - | | |--- 9. 安全解压缩 -------| - | | |--- 10. 原子切换 current -| - | | |--- 11. 测试与重载配置 ---->| - | | |<-- 12. 重载成功 ---------| - | |<-- 13. 上报 Apply Success | | - | | | | + [管理员 / scanner] [Server 控制面] [Agent] [OpenResty] + | | | | + |-- manual 上传 ------>|-- inspect / Ingest ---->| | + | |-- 创建 candidate | | + |-- 显式激活 candidate ->|-- 切换 active | | + | | | | + |-- source sync ------>|-- inspect / Ingest | | + | |-- create/load + 原子激活 | | + | | | | + |-- 首次绑定项目并发布 ->|-- 广播项目锚点 -------->|-- 写入/重载路由 ---------->| + | | | | + |-- 后续激活/同步/回滚 ->|-- active latest 改变 ---| | + | |<-- latest 元数据对账 ----| | + | |--- 流式返回 package ---->| | + | | |-- 校验、解压、复核 --------| + | | |-- 原子切换 current -------->| ``` diff --git a/docs/guide/pages-usage.md b/docs/guide/pages-usage.md index 1a8936f7..752593ec 100644 --- a/docs/guide/pages-usage.md +++ b/docs/guide/pages-usage.md @@ -1,86 +1,116 @@ # Pages 静态托管使用 -你会学到：如何在 OpenFlare 中使用 Pages 静态托管功能部署前端项目（如 React、Vue 等 SPA 或 VitePress、Hugo 等静态站点），配置单页应用 (SPA) Fallback 路由以及接口反向代理 (API Proxy)，并理解不可变部署与 Agent 侧原子切换的底层逻辑。 +你会学到：如何通过本地上传、Remote URL 或公开 GitHub Release asset 部署预构建静态站点，配置 SPA Fallback 与 API 反向代理，并安全地检查更新、自动发布和回滚。 --- -## 核心机制与工作流 +## 核心机制与页面结构 -OpenFlare Pages 提供受 Cloudflare Pages 启发的 **Direct Upload (直接上传)** 静态网站托管服务。它与常规代理站点的不同之处在于，数据面的边缘节点 (Agent) 会将静态文件拉取并解压到节点本地，直接通过本地的 OpenResty 提供高性能的静态文件服务，无需维护额外的 Nginx 宿主机静态目录同步。 +OpenFlare Pages 受 Cloudflare Pages 的 Direct Upload 与部署历史交互启发，但当前处理的是**预构建产物**，不是仓库源码构建。项目详情按“当前生产部署 → 部署源 → 部署历史”组织：来源配置可以变化，已经创建的 deployment 保持不可变。 ```text - [ 管理员 / CI ] ────── 1. 上传 ZIP 压缩包 ──────► [ OpenFlare Server ] - │ - [ 访客浏览器 ] ◄────── 4. 访问页面 / 静态资源 ────────── [ Agent 节点 / OpenResty ] - ▲ - │ - 2. 检查 Checksum 并拉取 ZIP - 3. 解压并原子切换 current 链接 +本地上传 ─> 统一校验 / upload.Ingest ─> 新 candidate ─> 管理员显式激活 ─┐ +Remote URL ── Server 受限下载 ────────┐ │ +GitHub Release asset ─ Server 解析 ───┴─> create/load deployment ─────┤ + └─> source sync 原子激活 ────────┘ + | + v + Agent 按项目 latest 拉取 + | + v + OpenResty 本地静态服务 ``` -1. **直接上传部署包**：在控制面上传预构建好的网站 `.zip` 压缩包，Server 会生成一条带有唯一 SHA-256 校验和 (Checksum) 的不可变部署记录。 -2. **发布与推送**：在路由配置中将源站类型 (Upstream Type) 设为 `Pages 静态托管` 并绑定项目。发布配置版本后，Server 会广播给所有 Agent 节点。 -3. **安全拉取与部署**：Agent 节点识别到新配置引用了新的 Pages 部署，增量下载 ZIP 包，校验 Checksum 保证一致性，并在本地解压、完成原子目录切换，重载 OpenResty 使服务生效。 +外部 URL、GitHub 元数据和自动检查都只由 Server 处理。Agent 只从控制面拉取当前激活的部署包，不接收外部来源凭据，也不执行 `git clone`、依赖安装或构建命令。 ---- +## 第一步：创建项目 -## 第一步：上传部署包与创建 Pages 项目 +1. 登录管理端，进入 **「Pages」**，点击 **「创建项目」**。 +2. 填写项目名称与唯一 Slug。 +3. 配置内容入口： + * **入口文件名**：默认 `index.html`。 + * **静态资源根路径（RootDir）**：产物位于 `dist/` 等子目录时填写该相对路径；产物就在归档根目录时留空。 +4. 按需设置 SPA Fallback 与 API 代理。RootDir 和入口文件是项目级配置，会统一应用于所有来源。 -1. 登录管理端控制面板，进入左侧导航 **「Pages」** 菜单，点击 **「创建项目」**。 -2. 填写项目基本信息： - * **项目名称**：业务名称（如 `我的前端应用`）。 - * **项目标识 (Slug)**：URL 友好的唯一英文标识（如 `my-react-app`），将作为存储目录的文件夹名。 -3. 设定站点目录结构与入口： - * **入口文件名**：默认为 `index.html`。 - * **静态资源根路径 (RootDir)**：如果你的打包产物在压缩包的子目录下（例如打包出来的 zip 里包含一个 `dist/` 目录），则需要在这里填入子路径（如 `dist`）。若打包产物直接在 zip 根目录，留空即可。 -4. **上传 ZIP 压缩包**： - * 上传你的项目静态资源打包生成的 `.zip` 文件。 +## 第二步：选择部署源 -> [!IMPORTANT] -> **部署包安全限制规范** -> 为了保障控制面和边缘节点的系统安全与性能，上传的部署包必须满足以下硬性指标，否则会被系统拒绝： -> * **大小限制**：ZIP 压缩包体积不得超过 **25 MiB**，解压后的总文件大小不得超过 **100 MiB**。 -> * **数量限制**：解压后的文件总数不得超过 **1,000 个**。 -> * **软链接拦截**：ZIP 包内禁止包含任何软链接 (Symbolic Link)，防御软链接劫持攻击。 -> * **Zip-Slip 防御**：压缩包中所有文件路径会被强制规范化，禁止使用 `..` 或以 `/` 开头，防止解压路径穿越攻击。 -> * **入口文件检查**：你指定的入口文件（在静态资源根路径下，如 `dist/index.html`）**必须在压缩包中存在**。 +### 1. 手动上传 ---- +不配置持久来源时，项目保持手动模式。点击 **「上传部署包」** 选择预构建归档；上传成功会创建一条候选 deployment，再从部署历史中显式激活。重复上传不会修改已有 deployment。 -## 第二步：配置高级路由规则 +支持 `zip`、`tar.gz` / `tgz`、`tar.xz` / `txz`、`tar.bz2` / `tbz2`、`tar` 与 `7z`。 -在项目详情的配置页面中，你可以根据前端项目类型开启以下高级特性： +### 2. Remote URL -### 1. 单页应用 (SPA) Fallback 路由 -对于使用 React Router、Vue Router 等进行前端路由的单页应用 (SPA)，当用户直接刷新类似 `/profile/settings` 的子路径时，边缘节点本地并不存在该物理文件，会导致 404 错误。 -* **配置方式**：在项目设置中开启 **「SPA Fallback」**，并将路径设为入口文件（如 `/index.html`）。 -* **生效逻辑**：开启后，如果访客请求的静态资源在物理上不存在，OpenResty 会自动降级重定向渲染入口文件，将路由交由前端 JavaScript 接管，避免 404 报错。 +在部署源卡片中选择 **Remote URL**，填写 HTTP(S) 地址并选择网络策略： -### 2. 内置 API 反向代理 -为了避免前端请求后端 API 时遭遇跨域 (CORS) 限制，Pages 托管支持在同一个域名下直通后端 API。 -* **配置方式**： - * **API 代理路径 (APIProxyPath)**：匹配的 URL 前缀（如 `/api`）。 - * **后端服务地址 (APIProxyPass)**：后端 API 的源站地址（如 `http://10.0.0.5:8080`）。 - * **重写规则 (APIProxyRewrite)**：可选。如果需要剥离前缀或重写路径，可使用正则匹配。例如： - * 剥离前缀：将请求 `/api/users` 重写为 `/users` 发送给后端，配置为 `^/api/(.*)$ /$1`。 -* **生效逻辑**：所有以 `/api` 开头的请求会被直接转发至后端服务，而其他请求则继续由静态托管服务处理。 +* **public**：默认策略，拒绝 loopback、私网、链路本地地址、DNS rebinding、自签 TLS，以及重定向到非公网目标。 +* **trusted_internal**：仅用于明确受信的内网或自签服务；保存前需要再次确认风险。 ---- +保存后地址只以脱敏形式展示。编辑其它配置时无需重新填写；只有选择更换地址时才提交新 URL。Remote 来源只提供 **「同步并发布」**：每次由 Server 下载、校验并原子激活，不支持“检查更新”、定时检查或自动更新。 -## 第三步：绑定代理路由并发布 +### 3. GitHub Release -Pages 项目配置并上传好部署包后，需要绑定到对外公开的域名上才能被访客访问。 +GitHub 来源仅支持公开 `github.com` 仓库。填写： -1. 导航至左侧菜单 **「规则管理」**，创建或编辑一条代理规则。 -2. 切换到 **「反向代理」** 选项卡： - * **源站类型**：选择 **「Pages」**。 - * **选择 Pages 项目**：选择你刚才创建的项目，并关联要激活的部署版本（默认会自动关联最新上传成功的部署）。 -3. 点击右上角 **「配置预览」** -> 确认无误后点击 **「发布并激活」**。 +* `https://github.com/{owner}/{repo}` 格式的仓库地址； +* **最新 Release** 或 **固定 Tag**； +* 精确、区分大小写的 Release Asset 文件名，默认 `dist.zip`。 -## 运维与回滚 +两种选择都可手动 **「检查更新」** 和 **「同步并发布」**。区别如下： -* **不可变部署与回滚**：每次在 Pages 项目下上传 `.zip` 文件，系统都会产生一个全新且唯一的部署版本。如果在历史部署列表中将上一版本设为激活并重新发布，可实现边缘节点的秒级回滚。 -* **原子切换与自愈**：边缘节点（Agent）在拉取静态资源包时，会执行校验与流式解压，并通过原子切换物理目录来保障服务的无缝过渡。同时，Agent 会定时清理不再引用的历史部署包。 +* **latest**：可设置 5～1440 分钟检查间隔，默认 60 分钟；自动更新默认关闭。开启后，scanner 发现新 revision 才会异步同步并发布。 +* **tag**：只支持管理员手动检查和同步，不参与定时 scanner。 + +“检查更新”只解析 Release/asset 并更新版本游标，不下载部署包；“同步并发布”才会下载、校验、创建或复用 deployment 并激活。如果同一个 Release 下的 asset 被替换，来源会进入 **「需要确认」**，必须确认页面显示的精确 revision 后才能发布，避免静默覆盖。 + +GitHub Release 在这里是预构建产物源，不等同于连接代码仓库自动构建。未来仓库集成会使用独立的 `git_repository` 来源和 Server build executor，再把构建产物送入同一部署管线。 + +### 4. 切换或删除来源 + +可以在手动、Remote 和 GitHub Release 之间切换。修改或删除来源不会删除当前生产部署和历史 deployment；切回手动模式后可继续上传并显式激活。 + +## 部署包安全限制 + +部署包必须满足以下约束： + +* 压缩包大小由系统配置 `pages_max_package_size_mb` 控制，默认 100 MiB，可配置 1～2048 MiB。 +* 展开后的单文件和总量上限为“包大小上限 × 4”，且最低为 100 MiB；最多 1,000 个常规文件。 +* 控制面会流式读取常规文件体，核对声明大小与实际字节，并校验项目入口文件。 +* 归档中的绝对路径、`..` 路径逃逸、软链接、硬链接和特殊文件都会被拒绝。 + +Agent 下载时还会执行 SHA-256、真实响应字节上限、解压后文件数与总大小复核；失败不会切换现有 `current`。 + +## 第三步：配置高级路由规则 + +### 1. SPA Fallback + +使用 React Router、Vue Router 等前端路由时，开启 **「SPA Fallback」** 并设置入口路径（通常为 `/index.html`）。访客直接访问不存在的物理路径时，OpenResty 会回退到入口文件交由前端路由处理。 + +### 2. API 反向代理 + +Pages 可在同一域名下把指定前缀转发到后端 API： + +* **APIProxyPath**：匹配前缀，例如 `/api`。 +* **APIProxyPass**：后端地址，例如 `http://10.0.0.5:8080`。 +* **APIProxyRewrite**：可选的路径重写规则。 + +匹配 API 前缀的请求走反向代理，其余请求继续由静态站点处理。 + +## 第四步：绑定路由并首次发布 + +1. 创建或编辑一条代理规则。 +2. 将源站类型设为 **Pages**，并选择 Pages **项目**。 +3. 预览配置后发布并激活。 + +路由绑定的是稳定的项目 ID，不是某个 deployment。首次发布让 Agent 获得项目锚点；此后本地上传、来源同步、自动更新或人工回滚只会改变项目的 active deployment，Agent 会通过 latest hash 对账收敛，无需重新发布主配置。 + +## 运维、状态与回滚 + +* 来源卡片展示最近检查/同步、已发现与已应用 revision、下次检查和安全错误。检查或同步任务运行时，页面会轮询任务状态；latest 空闲时只在接近检查时间时低频刷新。 +* 自动更新失败不会替换旧 active deployment；单个来源失败也不会阻塞 scanner 处理其它项目。 +* 在部署历史中激活其它 deployment 即完成人工回滚。系统会 fence 在途来源任务，并关闭该来源的自动更新，避免下一轮 latest 又覆盖人工选择；重复激活当前版本是 no-op。 +* Agent 下载到临时文件并校验 SHA-256，安全解压后原子切换 `current`。任一步失败都保留旧内容，多项目对账时单项目失败不影响其它项目。 > [!TIP] -> 关于不可变部署、目录结构设计、增量拉取和安全防逃逸校验等底层架构与自愈细节，请参阅 [Pages 静态托管设计](../design/pages-design.md)。 +> 关于来源状态机、自动 scanner、上传补偿、不可变部署和 Agent 原子切换，请参阅 [Pages 静态托管设计](../design/pages-design.md)。 diff --git a/docs/plan/20260719-pages-source-sync-v2.md b/docs/plan/20260719-pages-source-sync-v2.md index 03978901..e37fe907 100644 --- a/docs/plan/20260719-pages-source-sync-v2.md +++ b/docs/plan/20260719-pages-source-sync-v2.md @@ -1,7 +1,7 @@ # Pages 项目部署源与 GitHub Releases 自动更新 V2 实现方案 日期：2026-07-19 -状态：实施中 +状态：代码实施完成（范围内自动化验证完成；生产环境验收见 §7） 方案版本：V2（设计修订版，不代表新增 `/api/v2`） 关联材料： @@ -32,9 +32,9 @@ V2 保留原方案正确的主链路：外部来源只由 Server 控制面访问 ## 1. 目标与背景 (Goal & Context) -### 1.1 当前实现与问题 +### 1.1 方案制定时的实现与问题 -当前 Pages 已支持： +方案制定时，Pages 已支持： * 管理员本地上传压缩包； * 同步调用 `POST /api/v1/d/pages/:id/deployments/upload-from-url` 完成一次性 URL 导入； @@ -149,7 +149,7 @@ flowchart LR Current --> OpenResty["OpenResty 静态服务"] ``` -scanner 本身是一个正式 TaskHandler，并非绕过任务框架。它在单次执行中先限量恢复 lease/orphan record，再串行检查到期的 GitHub latest source，避免一次 cron 批量投递 20 个并行 GitHub 请求。手动操作和自动下载使用统一 action task；scanner 不执行长时间 package 下载。 +scanner 本身是一个正式 TaskHandler，并非绕过任务框架。它在单次执行中先扫描并精确 CAS 恢复全部过期 lease，再限量补偿最多 100 条 orphan record，最后串行检查最多 20 个到期的 GitHub latest source，避免一次 cron 批量投递并行 GitHub 请求。手动操作和自动下载使用统一 action task；scanner 不执行长时间 package 下载。 ### 2.3 领域对象与不变量 @@ -1057,11 +1057,27 @@ scanner TaskResult 和结构化日志至少记录： #### [NEW] `internal/apps/openflare/pages/source_sync.go` -* runtime 状态、lease、check/sync service、原子 create-or-load/activate、补偿。 +* Remote/GitHub 统一 ingest、deployment create-or-load、原子激活与失败补偿。 + +#### [NEW] `internal/apps/openflare/pages/source_runtime.go` + +* source execution snapshot、短/长 lease、heartbeat、失败终态、过期 lease 精确 CAS 恢复。 + +#### [NEW] `internal/apps/openflare/pages/github_source.go`、`github_source_action.go` + +* GitHub 配置归一化，以及 latest/tag check、ETag/304、精确 revision sync、attention 与 provider 退避。 #### [NEW] `internal/apps/openflare/pages/source_tasks.go` -* scanner/action TaskMeta、payload validation、Handler 与限量 orphan record reconciliation。 +* action TaskMeta、旧/新 payload normalization、actor/trigger 边界与 Handler。 + +#### [NEW] `internal/apps/openflare/pages/source_scanner.go` + +* internal-only scanner、过期 lease 恢复、20 条稳定批次、403/429 退避、backlog 与精确 revision 自动派发。 + +#### [NEW] `internal/apps/openflare/pages/source_orphan_cleanup.go`、`internal/model/openflare_pages_cleanup.go` + +* 100 条/2 小时隔离的 orphan upload 候选查询、统一锁序复检、幂等软删除与缓存修复。 #### [MODIFY] `internal/apps/openflare/pages/logics.go` @@ -1207,6 +1223,7 @@ SQLite `0001` 的 Down 必须通过重建受影响表完整移除新增列、约 * `frontend/tests/openflare/pages-service.test.ts` * `frontend/tests/openflare/pages-source-ui.test.tsx` +* `frontend/tests/openflare/pages-source-auto-update.test.tsx` ### 3.5 文档与生成物（代码实施时） @@ -1370,7 +1387,7 @@ make code-check ## 5. 分阶段实施 -### 阶段 0：安全与一致性前置（独立合并/发布） +### 阶段 0：安全与一致性前置（已完成：`4e8ec232`） * RootDir/EntryFile 严格路径与 LocalRoot 端到端一致； * Server 真实归档限制和 tar 流式实现； @@ -1381,7 +1398,7 @@ make code-check 验收：不引入 source 表/API 的情况下，现有本地上传与旧 URL 路径全部回归；大包内存与路径安全测试通过。 -### 阶段 1：数据模型与 Remote 手动同步 +### 阶段 1：数据模型与 Remote 手动同步（已完成：`38b05169`） * `0001` DDL migration、model、source CRUD/view； * runtime 六态、lease、config/content fence； @@ -1393,7 +1410,7 @@ make code-check 本阶段 API/DTO 变化完成后立即运行 `make swagger` 并将生成物纳入阶段验证，不把 Swagger 漂移累积到阶段 4。 -### 阶段 2：GitHub 手动检查与同步 +### 阶段 2：GitHub 手动检查与同步（已完成：`c39a3edc`） * GitHub client、latest/tag、ETag、asset/digest、rate limit； * action check、首次异步 check、update_available； @@ -1405,7 +1422,7 @@ make code-check 本阶段 API/DTO 变化后再次运行 `make swagger`，保证阶段 2 可独立合并发布。 -### 阶段 3：latest scanner 与自动更新 +### 阶段 3：latest scanner 与自动更新（已完成：`848884d8`、`999428cf`、`67b051c2`） * scanner task、`0002` schedule seed、过期 lease 恢复； * serial batch、jitter、退避、自动 sync dispatch； @@ -1418,20 +1435,20 @@ make code-check 本阶段 API/DTO 变化后再次运行 `make swagger`，阶段 4 只做最终一致性复检。 -### 阶段 4：文档、生成物与全门禁 +### 阶段 4：文档、生成物与验证记录（代码收口已完成） * 同步 Pages design/architecture/guide/README 与中文 changelog； * 生成 Swagger； -* 运行前后端测试、`make prettier`、`make code-check`； -* 按 §4.7 完成手工矩阵并记录未覆盖的真实外部场景。 +* 运行前后端测试、`make prettier`、`make code-check`，并如实记录全仓失败与未执行边界； +* 整理 §4.7 手工矩阵，并记录本地环境未覆盖的真实外部场景。 每个阶段只提交本阶段明确路径并独立验证；阶段 0 不与后续 source 功能捆绑成一个大提交。 --- -## 6. 完成定义 +## 6. 生产验收完成定义 -只有同时满足以下条件，V2 才视为实现完成： +只有同时满足以下条件，V2 才视为生产验收完成： * 三类来源能力边界与 UI/API 完全一致； * 数据模型为 config/runtime 分离的瘦表，状态不超过六态； @@ -1442,3 +1459,47 @@ make code-check * history=1、本地 candidate、source sync 和清理补偿均有自动化覆盖； * PostgreSQL、SQLite、后端、Agent、前端及项目门禁全部通过； * 实际代码、Swagger、中文设计/使用文档和 changelog 同步。 + +本轮状态中的“代码实施完成”表示功能、迁移、前后端交互、生成物和文档均已落地，范围内自动化门禁已经通过。真实 PostgreSQL、真实外部来源和多 Agent 故障矩阵仍是生产验收条件；未执行项及全仓存量失败不会在本文中伪报为通过，统一记录如下。 + +--- + +## 7. 实施结果与验证记录 + +### 7.1 已交付 + +* 完成部署包安全与一致性前置：项目 RootDir 生效、归档实际字节限制、Agent 流式下载与复核、历史裁剪、上传记录补偿及共享对象安全边界。 +* 完成 Remote URL 与公开 GitHub Release source/runtime 模型、CRUD、手动 check/sync、revision 幂等、不可变 deployment 与原子激活。 +* 完成 GitHub latest scanner、稳定批次、lease 恢复、ETag/304、403/429 退避、精确 revision 自动派发和 orphan upload 延迟补偿。 +* 完成 Cloudflare Pages 风格的“当前生产部署 → 部署源 → 部署历史”前端交互，并保留独立 `git_repository` Provider、Server build executor 和统一 artifact pipeline 的后续边界。 +* 完成 Swagger、中文 changelog、Pages 设计、总体架构、Agent 设计、使用指南与 README 同步。 + +### 7.2 已通过的自动化验证 + +* `make swagger`：通过，生成物已随实现提交。 +* `make prettier`：通过；格式化产生的三个无关存量文件变化已恢复，未混入提交。 +* `make code-check`：通过；`golangci-lint`、前端 TypeScript 与全量 ESLint 均无错误。 +* Pages、Agent、GitHub Release integration、归档、上传和迁移相关 Go 定向测试通过；关键并发路径的 race 测试通过。 +* SQLite Pages source 与 scanner schedule migration 的 Up/Down/Up 通过。 +* 前端 Pages 四个测试文件共 34 条用例全部通过，相关 TypeScript 与 ESLint 检查通过。 +* 全仓 Go 测试中 `internal/apps/openflare/pages`、`internal/apps/openflare/agent`、`internal/integration/githubrelease`、`internal/db/migrator`、`internal/apps/upload/*` 与 `pkg/pagesarchive` 均通过。 + +### 7.3 全仓测试中仍存在的范围外失败 + +* `go test ./... -count=1` 未全绿：`internal/apps/admin/system_config` 仍按旧快照断言 32 条默认配置和 3 条 business 配置，实际为 34 和 5；本分支未修改该模块。 +* `internal/apps/flared/frpc` 的 `TestUnexpectedExit0CPUProtection`、`TestBackoffReset` 及 `internal/apps/relay/frps` 的 `TestUnexpectedExitAndAutorestart` 存在进程状态时序失败。 +* `internal/apps/openflare/tasks` 的 `TestRunSSLRenewJobTriggersDueCertificates` 连接本机 Redis 时收到 `NOAUTH Authentication required`，证书状态因此未进入 `applying`。 +* 前端全仓 Vitest 共 101 条通过、1 条失败：`frontend/tests/zone/zone-page.test.tsx:116` 仍查找旧文案“唯一访问者”；Pages 的 34 条测试不受影响。 + +这些失败点不属于本次 Pages V2 功能路径，因此未通过扩大范围修改存量模块来掩盖；它们仍应在各自模块后续收敛。 + +### 7.4 本地环境未执行的生产验收 + +* `OPENFLARE_TEST_POSTGRES_DSN` 未设置，因此 PostgreSQL migration Up/Down/Up、JSONB orphan 候选矩阵和真实行锁竞争未执行；SQLite 对应路径已通过。 +* 未访问真实公开 GitHub Release，也未以真实服务验证 GitHub rate limit、asset redirect、Remote public DNS rebinding 或 `trusted_internal` 自签 TLS；普通测试均使用可注入 client/`httptest.Server` 覆盖协议分支。 +* 未执行多 Agent 实机收敛，以及 Server/Worker 在下载、Ingest、最终提交阶段的进程级中断矩阵。 +* 前端交互由 React Testing Library 覆盖，未执行真实浏览器 E2E。 + +### 7.5 剩余验证债务 + +当前 orphan/deployment 竞态测试能验证锁序与事务结果，但 SQLite 串行执行不能替代 PostgreSQL 下两个独立事务的真实行锁竞争。生产发布前应在可用 PostgreSQL 测试实例上补跑 migration、JSONB 候选和双事务交错测试，并按 §4.7 完成真实网络与多 Agent 最小矩阵。 diff --git a/docs/plan/index.md b/docs/plan/index.md index 07ee80d4..0ef1905f 100644 --- a/docs/plan/index.md +++ b/docs/plan/index.md @@ -19,7 +19,10 @@ * [访问日志 cache_status 明细可见](./20260718-access-log-cache-status.md)：上报 `$upstream_cache_status`，明细展示命中/回源/未缓存三态。 * [边缘缓存默认 static 策略](./20260718-edge-cache-static-default.md)：开启缓存默认仅静态扩展名；存量 url→all。 * [访问日志 IP 明细 Tab](./20260719-access-log-ip-tab.md)：第三 Tab 按 IP 聚合列表（时间窗/流量/2xx 比例）；IP 情报迁入独立详情；日志详情仅请求字段。 -* [Pages 项目部署源与 GitHub Releases 自动更新 V2](./20260719-pages-source-sync-v2.md)：统一 Remote URL / GitHub Release 来源、不可变部署、自动检查更新与安全回滚，并预留独立仓库构建 Provider 边界。 + +## 已完成的计划 + +* [Pages 项目部署源与 GitHub Releases 自动更新 V2](./20260719-pages-source-sync-v2.md)：已完成 Remote URL / GitHub Release 来源、不可变部署、自动检查更新与安全回滚，并预留独立仓库构建 Provider 边界；生产环境验收边界见计划内验证记录。 ## 使用建议