diff --git a/docs/changelog/index.md b/docs/changelog/index.md index def5e967..ecd99ece 100644 --- a/docs/changelog/index.md +++ b/docs/changelog/index.md @@ -21,6 +21,19 @@ sidebar: false ## [unreleased] +### 新增 + +- Pages 现支持上传 zip、tar.gz、tar.xz、tar.bz2、tar、7z 等常用压缩格式的部署包。 +- 管理员可在运维设置中配置 Pages 部署包大小上限与每个项目的历史部署保留数量。 + +### 变更 + +- Pages 部署包上传体积限制改为系统动态配置,默认仍为 100 MiB,可按环境调整。 +- 上传新部署后会按保留策略自动清理超出数量的历史部署包,减少磁盘占用。 +- Agent 在同步 Pages 部署时信任控制面已完成的包校验,不再重复限制文件数与展开体积,仅校验下载完整性并安全解压到本地。 +- 明确 Pages 历史部署保留语义为每个项目最多 N 条(激活必留、其余按从新到旧填充),裁剪失败会记录日志且不回滚已成功上传。 +- 主配置版本与 Pages 部署双轨管理:下发给 Agent 的配置会将 Pages 路由重绑定到项目当前激活部署,主配置回滚不再依赖已裁剪的旧部署包。 + ## [v3.3.0] - 2026-07-14 ### 新增 diff --git a/docs/design/agent-design.md b/docs/design/agent-design.md index de1130dd..3957d831 100644 --- a/docs/design/agent-design.md +++ b/docs/design/agent-design.md @@ -98,7 +98,7 @@ Agent 对数据面 OpenResty 的管控实现了端到端的闭环,包含配置 * `certs/`:证书存放目录(文件命名为 `{cert_id}.crt` 和 `{cert_id}.key`)。 * `waf/` 与 `pow/`:WAF 及防 CC 挑战所需的专用 Lua 运行时脚本。 * `waf_config.json` 与 `waf_ip_groups.json`:WAF 过滤引擎所需的结构化规则配置文件。 -* `pages_dir`:Pages 静态站点部署目录,默认位于 `data_dir/var/lib/openflare/pages`。当激活配置引用 Pages 部署时,Agent 会下载部署 zip、校验 checksum、解压到部署 release 目录,并切换 `deployments/{deployment_id}/current` 供 OpenResty `root`/`try_files` 读取。 +* `pages_dir`:Pages 静态站点部署目录,默认位于 `data_dir/var/lib/openflare/pages`。当激活配置引用 Pages 部署时,Agent 会下载部署包(zip / tar.gz / tar.xz / tar.bz2 / tar / 7z)、校验下载 checksum、解压到部署 release 目录,并切换 `deployments/{deployment_id}/current` 供 OpenResty `root`/`try_files` 读取。业务侧体积/文件数校验由控制面完成,Agent 信任控制面结果,解压时仅保留本机路径安全防护。 ### 2. 精细化的重载动作 1. **备份当前配置**:在写入新文件之前,Agent 会将现有的配置文件复制到 `.backup` 临时目录下,保留完整的现场快照。 diff --git a/docs/design/index.md b/docs/design/index.md index 4bdbaf6e..9b429870 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -26,7 +26,7 @@ OpenFlare 适合需要统一管理多台 OpenResty 代理节点的团队,具 | **配置版本控制** | 支持全局单一激活版本的预览、发布、不可变快照历史与秒级一键回滚 | [Agent 与发布模型](./agent-design.md) | | **WAF 安全防护** | 支持可视化 DAG 编排规则、手动/自动/订阅型 IP 组、GeoIP 匹配与 PoW CC 防护 | [WAF 设计](./waf-design.md) / [WAF 可编排规则设计](./waf-orchestration-design.md) / [WAF 使用指南](../guide/waf-usage.md) | | **内网穿透** | 通过中继节点(Relay)与内网客户端(OpenFlared),反向穿透暴露内网 Web 服务 | [内网穿透设计](./tunnel-design.md) / [穿透使用指南](../guide/tunnel-usage.md) | -| **Pages 静态托管** | 直接上传前端 zip 包,由边缘节点拉取并由 OpenResty 本地服务,支持 API 反代与 SPA Fallback | [Pages 静态托管设计](./pages-design.md) | +| **Pages 静态托管** | 直接上传前端压缩包(zip / tar.gz / tar.xz / 7z 等),由边缘节点拉取并由 OpenResty 本地服务,支持 API 反代与 SPA Fallback | [Pages 静态托管设计](./pages-design.md) | | **TLS 证书自动续期** | 将证书显式绑定到 Zone 域名,并通过 ACME 协议向 Let's Encrypt 申请/续期证书 | [Zone 与域名资源设计](./zone-design.md) | | **多节点监控与观测** | 收集节点资源快照、健康事件,聚合请求指标与访问日志明细 | [系统架构](./architecture.md) | diff --git a/docs/design/pages-design.md b/docs/design/pages-design.md index 410edb95..781c2bad 100644 --- a/docs/design/pages-design.md +++ b/docs/design/pages-design.md @@ -19,11 +19,12 @@ ## 核心功能 Pages 静态托管子系统包含以下核心能力: -* **Direct Upload 部署模式**:支持直接上传预构建的 `.zip` 静态资源包,省去复杂的 Git 集成和构建环境依赖。 -* **不可变部署快照**:每次上传产生一个带唯一 ID 和 SHA-256 Checksum 的不可变部署记录。历史包永久保留,支持随时激活和回滚。 +* **Direct Upload 部署模式**:支持直接上传预构建的静态资源压缩包(`zip`、`tar.gz`、`tar.xz`、`tar.bz2`、`tar`、`7z`),省去复杂的 Git 集成和构建环境依赖。 +* **不可变部署快照**:每次上传产生一个带唯一 ID 和 SHA-256 Checksum 的不可变部署记录。支持按系统配置保留最近 N 个历史部署,并可随时激活和回滚。 * **SPA Fallback 支持**:支持对单页应用(SPA)进行 Fallback 路由配置,请求找不到静态文件时自动重定向到入口文件。 * **内置 API 反代服务**:支持在 Pages 规则内一键启用 API 代理,消除跨域问题,将请求转发给指定的后端服务。 -* **安全包校验与解压缩**:内置 Zip-Slip 路径逃逸防御、防软链接劫持、文件大小/数量硬上限控制,保障节点物理安全。 +* **安全包校验与解压缩**:内置路径逃逸防御、防软链接劫持、文件大小/数量上限与可配置上传包体积控制,保障节点物理安全。 +* **可配置限额**:管理员可在运维设置中调整「部署包大小上限」与「历史部署保留数」。 --- @@ -40,7 +41,7 @@ graph TD %% 控制流与心跳 Server[OpenFlare Server 控制面] <-->|Agent API / Heartbeat| Agent[openflare-agent 进程] - Server -.->|5. 存储 ZIP 部署包| LocalStore[(Server 本地存储)] + Server -.->|5. 存储部署包| LocalStore[(Server 本地存储)] Agent -->|1. 发现新版本| Server Agent -->|2. 下载部署包| Server @@ -88,21 +89,29 @@ graph TD } ``` +### 3. 与主配置版本的双轨关系 +* **主配置版本**(`config_versions`)与 **Pages 部署** 是两套独立的版本体系。 +* 快照里记录的 `pages_deployment` 仅反映**发布当时**的激活部署,用于审计与当时渲染结果存档。 +* **运行时**:下发给 Agent 的配置会把各 Pages 路由**重绑定到该项目当前激活部署**(最新包)。主配置回滚/重新激活旧版本时,**不要求**仍能拉到旧 Pages 包,只保证指向当前最新激活部署。 +* 因此 Pages 历史裁剪可以安全删除非激活部署,无需为「主配置回滚到旧 Pages 包」预留存储。 + --- ## Server 端 (控制面) 职责与生命周期 -### 1. ZIP 包安全校验与分析 -为了避免不可信的用户上传恶意压缩包攻击服务器,控制面在 `UploadPagesDeployment` 时执行严格的流式校验: -* **大小限制**:ZIP 压缩包不得超过 25 MiB(保守的 V1 默认值),且展开后的解压总体积不得超过 100 MiB。 +### 1. 部署包安全校验与分析 +为了避免不可信的用户上传恶意压缩包攻击服务器,控制面在 `UploadDeployment` 时执行严格校验: +* **格式支持**:`zip`、`tar.gz` / `tgz`、`tar.xz` / `txz`、`tar.bz2` / `tbz2`、`tar`、`7z`。 +* **大小限制**:压缩包体积由系统配置 `pages_max_package_size_mb` 控制(默认 100 MiB,范围 1~2048);展开后总体积上限为「包大小 × 4」且不低于 100 MiB。 * **数量限制**:压缩包中包含的静态文件总数不得超过 1,000 个。 -* **软链接阻断**:遍历 ZIP 文件,一旦检测到任何软链接 (`os.ModeSymlink`),立即抛出错误并拒绝上传,防御软链接劫持攻击。 -* **Zip-Slip 防御**:对每个压缩文件路径进行 `Clean` 并检查是否包含 `..` 或以 `/` 开头,防御目录跨越漏洞,防止写入系统敏感路径。 -* **入口文件校验**:项目指定的入口文件(例如 `index.html`,可在 `project.RootDir` 下)必须在 ZIP 压缩包中存在,否则拒绝上传。 -* **公共根目录去噪**:许多打包工具(如 GitHub 导出的 zip)会包含一个多余的主文件夹作为公共根前缀。控制面自动探测公共根前缀并将其安全剥离。 +* **软链接阻断**:遍历归档文件,一旦检测到任何软链接,立即抛出错误并拒绝上传,防御软链接劫持攻击。 +* **路径逃逸防御**:对每个压缩文件路径进行 `Clean` 并检查是否包含 `..` 或以 `/` 开头,防御目录跨越漏洞,防止写入系统敏感路径。 +* **入口文件校验**:项目指定的入口文件(例如 `index.html`,可在 `project.RootDir` 下)必须在部署包中存在,否则拒绝上传。 +* **公共根目录去噪**:许多打包工具会包含一个多余的主文件夹作为公共根前缀。控制面自动探测公共根前缀并将其安全剥离。 +* **历史保留**:系统配置 `pages_max_history_count`(默认 20,0 表示不限制)在每次上传成功后执行裁剪。语义为:**每个项目最多保留 N 条部署**;当前激活部署始终保留;其余名额按部署 ID 从新到旧填充;超出的非激活部署连同文件清单与存储对象一并删除。上传已成功时裁剪失败只记日志、不回滚上传;并发上传下可能短暂超过 N,后续上传的裁剪会收敛回 N。主配置版本回滚不依赖旧 Pages 包(见上节双轨关系)。 ### 2. 部署包存储规划 -控制面仅将 zip 文件存储在本地存储目录 `artifacts/{project_slug}/{checksum}.zip`,并在数据库中记录路径和清单。**大体积静态包不写入 config_versions 记录和任何配置推送通道**,以保障控制面数据同步的轻量与高效。 +控制面通过统一上传框架(`upload.Ingest`)存储原始部署包,并在数据库中记录 `upload_id` 与文件清单。**大体积静态包不写入 config_versions 记录和任何配置推送通道**,以保障控制面数据同步的轻量与高效。 --- @@ -118,7 +127,7 @@ Agent 运行在各边缘代理节点上,在应用配置版本前,必须先 ### 2. 安全解压缩与原子切换 为了保证配置应用过程的“无缝”且能在出错时立即回滚: 1. Agent 将下载的部署包数据写入临时目录,并重新计算 SHA-256 Checksum。如果与配置指明的 checksum 不符,立即报错并阻断发布流程。 -2. 解压部署包至临时目录 `releases/{checksum}.tmp`。解压时同样执行 Zip-Slip 目录跨越和软链接校验防御。 +2. 解压部署包至临时目录 `releases/{checksum}.tmp`。解压支持 zip / tar.* / 7z。**Agent 默认信任控制面**:不再重复校验文件数/体积等业务限额(控制面上传时已完成);仅做本机落盘安全处理(路径逃逸、软链接拒绝)与下载 checksum 完整性校验。 3. 解压成功后,写入标记文件 `.openflare-pages.json`。 4. 清理 `releases/{checksum}` 目录,将整个临时目录重命名为 `releases/{checksum}`。 5. **原子切换**:建立拷贝当前部署的物理副本到目标位置 `deployments/{deployment_id}/current`。切换前先备份上一版本的 `current`,一旦重载配置失败,Agent 能够快速恢复 `current` 目录并回滚 OpenResty。 diff --git a/frontend/app/(main)/admin/settings/components/openflare-ops-utils.ts b/frontend/app/(main)/admin/settings/components/openflare-ops-utils.ts index 8b7f7ed4..eecb073f 100644 --- a/frontend/app/(main)/admin/settings/components/openflare-ops-utils.ts +++ b/frontend/app/(main)/admin/settings/components/openflare-ops-utils.ts @@ -20,6 +20,8 @@ export type OpenFlareOpsFields = { uptime_kuma_timeout: string; database_auto_cleanup_enabled: boolean; database_auto_cleanup_retention_days: string; + pages_max_package_size_mb: string; + pages_max_history_count: string; }; export const defaultOpenFlareOpsFields: OpenFlareOpsFields = { @@ -42,6 +44,8 @@ export const defaultOpenFlareOpsFields: OpenFlareOpsFields = { uptime_kuma_timeout: '48', database_auto_cleanup_enabled: false, database_auto_cleanup_retention_days: '30', + pages_max_package_size_mb: '100', + pages_max_history_count: '20', }; export const INSTALLER_SCRIPT_URL = @@ -90,6 +94,8 @@ export function mapOptionsToOpsFields( ), database_auto_cleanup_retention_days: optionMap.database_auto_cleanup_retention_days ?? '30', + pages_max_package_size_mb: optionMap.pages_max_package_size_mb ?? '100', + pages_max_history_count: optionMap.pages_max_history_count ?? '20', }; } @@ -226,3 +232,30 @@ export function databaseAutoCleanupEntries( }, ]; } + +export function validatePagesFields(fields: OpenFlareOpsFields) { + const packageSize = Number.parseInt(fields.pages_max_package_size_mb, 10); + const historyCount = Number.parseInt(fields.pages_max_history_count, 10); + if (Number.isNaN(packageSize) || packageSize < 1 || packageSize > 2048) { + throw new Error('Pages 部署包大小上限必须为 1~2048 MiB。'); + } + if (Number.isNaN(historyCount) || historyCount < 0) { + throw new Error( + 'Pages 历史保留数必须为大于等于 0 的整数(0 表示不限制)。', + ); + } +} + +export function pagesOptionEntries(fields: OpenFlareOpsFields): OptionItem[] { + validatePagesFields(fields); + return [ + { + key: 'pages_max_package_size_mb', + value: String(Number.parseInt(fields.pages_max_package_size_mb, 10)), + }, + { + key: 'pages_max_history_count', + value: String(Number.parseInt(fields.pages_max_history_count, 10)), + }, + ]; +} diff --git a/frontend/app/(main)/admin/settings/components/openflare-ops.tsx b/frontend/app/(main)/admin/settings/components/openflare-ops.tsx index 3dd4b329..3794fa9c 100644 --- a/frontend/app/(main)/admin/settings/components/openflare-ops.tsx +++ b/frontend/app/(main)/admin/settings/components/openflare-ops.tsx @@ -64,6 +64,7 @@ import { mapOptionsToOpsFields, type OpenFlareOpsFields, optionsToMap, + pagesOptionEntries, uptimeKumaOptionEntries, } from './openflare-ops-utils'; import { UptimeKumaSiteSelectModal } from './uptimekuma-site-modal'; @@ -258,6 +259,17 @@ export function OpenFlareOpsSettings() { } }; + const savePagesSettings = () => { + try { + saveMutation.mutate({ + section: 'pages', + entries: pagesOptionEntries(fields), + }); + } catch (error) { + toast.error(error instanceof Error ? error.message : '参数校验失败'); + } + }; + if (optionsQuery.isLoading) { return ( + + +
+ Pages 静态托管 + + 配置部署包上传体积上限与每个项目的历史部署保留数量。 + +
+ +
+ +
+ + updateField('pages_max_package_size_mb', value) + } + placeholder='100' + /> + + updateField('pages_max_history_count', value) + } + placeholder='20' + /> +
+

+ 每个项目最多保留 N 条部署:激活部署始终保留,其余按从新到旧填充;超出的非激活部署会在上传成功后自动清理。支持 + zip、tar.gz、tar.xz、tar.bz2、tar、7z 格式。 +

+
+
+
diff --git a/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx b/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx index c0bc8bda..a4573a4b 100644 --- a/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx +++ b/frontend/app/(main)/pages/components/deployment-upload-dialog.tsx @@ -27,6 +27,27 @@ import { projectsQueryKey, } from './pages-utils'; +const PAGES_PACKAGE_ACCEPT = + '.zip,.tar.gz,.tgz,.tar.xz,.txz,.tar.bz2,.tbz2,.tbz,.tar,.7z'; + +const PAGES_PACKAGE_EXTENSIONS = [ + '.zip', + '.tar.gz', + '.tgz', + '.tar.xz', + '.txz', + '.tar.bz2', + '.tbz2', + '.tbz', + '.tar', + '.7z', +] as const; + +function isSupportedPagesPackage(fileName: string) { + const lower = fileName.toLowerCase(); + return PAGES_PACKAGE_EXTENSIONS.some((ext) => lower.endsWith(ext)); +} + interface DeploymentUploadDialogProps { open: boolean; onOpenChange: (open: boolean) => void; @@ -58,7 +79,7 @@ export function DeploymentUploadDialog({ const uploadMutation = useMutation({ mutationFn: () => { - if (!file) throw new Error('请选择 zip 部署包'); + if (!file) throw new Error('请选择部署包'); return PagesService.uploadDeployment(projectId, { file, onProgress: setUploadProgress, @@ -83,8 +104,8 @@ export function DeploymentUploadDialog({ const handleFileSelect = (selected: File | null) => { if (!selected) return; - if (!selected.name.toLowerCase().endsWith('.zip')) { - toast.error('仅支持 zip 格式的文件'); + if (!isSupportedPagesPackage(selected.name)) { + toast.error('仅支持 zip、tar.gz、tar.xz、tar.bz2、tar、7z 格式的部署包'); return; } setFile(selected); @@ -96,7 +117,8 @@ export function DeploymentUploadDialog({ 上传部署包 - 上传已构建的 zip 静态资源包,部署后可在列表中激活。 + 上传已构建的静态资源压缩包(zip / tar.gz / tar.xz / tar.bz2 / tar / + 7z),部署后可在列表中激活。 @@ -121,7 +143,10 @@ export function DeploymentUploadDialog({ }} > -

拖拽 zip 文件到此处,或点击选择文件

+

拖拽部署包到此处,或点击选择文件

+

+ 支持 zip、tar.gz、tar.xz、tar.bz2、tar、7z +

diff --git a/go.mod b/go.mod index b910302f..d0992695 100644 --- a/go.mod +++ b/go.mod @@ -10,6 +10,7 @@ require ( github.com/aws/aws-sdk-go-v2/config v1.32.16 github.com/aws/aws-sdk-go-v2/credentials v1.19.15 github.com/aws/aws-sdk-go-v2/service/s3 v1.99.1 + github.com/bodgit/sevenzip v1.6.5 github.com/bwmarrin/snowflake v0.3.0 github.com/coreos/go-oidc/v3 v3.17.0 github.com/deepteams/webp v1.2.3 @@ -39,7 +40,9 @@ require ( github.com/swaggo/files v1.0.1 github.com/swaggo/gin-swagger v1.6.1 github.com/swaggo/swag v1.16.6 + github.com/ulikunitz/xz v0.5.15 github.com/uptrace/opentelemetry-go-extra/otelzap v0.3.2 + github.com/yuin/gopher-lua v1.1.1 go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin v0.61.0 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 go.opentelemetry.io/otel v1.43.0 @@ -47,14 +50,13 @@ require ( go.opentelemetry.io/otel/sdk v1.43.0 go.opentelemetry.io/otel/trace v1.43.0 go.uber.org/zap v1.27.1 - golang.org/x/crypto v0.51.0 + golang.org/x/crypto v0.53.0 golang.org/x/image v0.42.0 - golang.org/x/mod v0.36.0 - golang.org/x/net v0.54.0 + golang.org/x/mod v0.37.0 + golang.org/x/net v0.56.0 golang.org/x/oauth2 v0.36.0 - golang.org/x/sync v0.21.0 + golang.org/x/sync v0.22.0 gopkg.in/natefinch/lumberjack.v2 v2.2.1 - gorm.io/driver/clickhouse v0.7.0 gorm.io/driver/postgres v1.6.0 gorm.io/driver/sqlite v1.6.0 gorm.io/gorm v1.31.1 @@ -66,7 +68,7 @@ require ( filippo.io/edwards25519 v1.2.0 // indirect github.com/ClickHouse/ch-go v0.71.0 // indirect github.com/KyleBanks/depth v1.2.1 // indirect - github.com/andybalholm/brotli v1.2.1 // indirect + github.com/andybalholm/brotli v1.2.2 // indirect github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.9 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.22 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.22 // indirect @@ -81,6 +83,8 @@ require ( github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.20 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.42.0 // indirect github.com/aws/smithy-go v1.25.0 // indirect + github.com/bodgit/plumbing v1.3.0 // indirect + github.com/bodgit/windows v1.0.1 // indirect github.com/boj/redistore v1.4.1 // indirect github.com/bytedance/gopkg v0.1.3 // indirect github.com/bytedance/sonic v1.14.2 // indirect @@ -124,6 +128,7 @@ require ( github.com/gorilla/sessions v1.4.0 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect github.com/hashicorp/go-version v1.9.0 // indirect + github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect @@ -132,7 +137,7 @@ require ( github.com/jinzhu/inflection v1.0.0 // indirect github.com/jinzhu/now v1.1.5 // indirect github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12 // indirect - github.com/klauspost/compress v1.18.5 // indirect + github.com/klauspost/compress v1.19.0 // indirect github.com/klauspost/cpuid/v2 v2.3.0 // indirect github.com/leodido/go-urn v1.4.0 // indirect github.com/mattn/go-isatty v0.0.21 // indirect @@ -144,7 +149,7 @@ require ( github.com/ncruces/go-strftime v1.0.0 // indirect github.com/paulmach/orb v0.13.0 // indirect github.com/pelletier/go-toml/v2 v2.2.4 // indirect - github.com/pierrec/lz4/v4 v4.1.26 // indirect + github.com/pierrec/lz4/v4 v4.1.27 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/quic-go/qpack v0.5.1 // indirect github.com/quic-go/quic-go v0.55.0 // indirect @@ -156,11 +161,11 @@ require ( github.com/spf13/afero v1.15.0 // indirect github.com/spf13/cast v1.10.0 // indirect github.com/spf13/pflag v1.0.10 // indirect + github.com/stangelandcl/ppmd v0.1.1 // indirect github.com/subosito/gotenv v1.6.0 // indirect github.com/twitchyliquid64/golang-asm v0.15.1 // indirect github.com/ugorji/go/codec v1.3.1 // indirect github.com/uptrace/opentelemetry-go-extra/otelutil v0.3.2 // indirect - github.com/yuin/gopher-lua v1.1.1 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect go.opentelemetry.io/otel/log v0.12.2 // indirect @@ -169,16 +174,18 @@ require ( go.uber.org/mock v0.6.0 // indirect go.uber.org/multierr v1.11.0 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect + go4.org v0.0.0-20260112195520-a5071408f32f // indirect golang.org/x/arch v0.22.0 // indirect - golang.org/x/sys v0.44.0 // indirect - golang.org/x/text v0.38.0 // indirect + golang.org/x/sys v0.46.0 // indirect + golang.org/x/text v0.40.0 // indirect golang.org/x/time v0.15.0 // indirect - golang.org/x/tools v0.45.0 // indirect + golang.org/x/tools v0.47.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260420184626-e10c466a9529 // indirect google.golang.org/grpc v1.80.0 // indirect google.golang.org/protobuf v1.36.11 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect + gorm.io/driver/clickhouse v0.7.0 // indirect gorm.io/driver/mysql v1.6.0 // indirect modernc.org/libc v1.72.1 // indirect modernc.org/mathutil v1.7.1 // indirect diff --git a/go.sum b/go.sum index 51dc78a3..ace3b8a9 100644 --- a/go.sum +++ b/go.sum @@ -10,8 +10,8 @@ github.com/alicebob/miniredis/v2 v2.38.0 h1:nZAzCR+Lj+Vxk4ZXzm2NuKq2O33RXj1XxJ2e github.com/alicebob/miniredis/v2 v2.38.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.5.1 h1:vtiFd0hhPAbyYJjztl0wYUq/PqEGkIlDmVuTIy6zw8Y= github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.5.1/go.mod h1:FTzydeQVmR24FI0D6XWUOMKckjXehM/jgMn1xC+DA9M= -github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro= -github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY= +github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM= +github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY= github.com/aws/aws-sdk-go-v2 v1.41.6 h1:1AX0AthnBQzMx1vbmir3Y4WsnJgiydmnJjiLu+LvXOg= github.com/aws/aws-sdk-go-v2 v1.41.6/go.mod h1:dy0UzBIfwSeot4grGvY1AqFWN5zgziMmWGzysDnHFcQ= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.9 h1:adBsCIIpLbLmYnkQU+nAChU5yhVTvu5PerROm+/Kq2A= @@ -48,6 +48,12 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.42.0 h1:ks8KBcZPh3PYISr5dAiXCM5/Thcu github.com/aws/aws-sdk-go-v2/service/sts v1.42.0/go.mod h1:pFw33T0WLvXU3rw1WBkpMlkgIn54eCB5FYLhjDc9Foo= github.com/aws/smithy-go v1.25.0 h1:Sz/XJ64rwuiKtB6j98nDIPyYrV1nVNJ4YU74gttcl5U= github.com/aws/smithy-go v1.25.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= +github.com/bodgit/plumbing v1.3.0 h1:pf9Itz1JOQgn7vEOE7v7nlEfBykYqvUYioC61TwWCFU= +github.com/bodgit/plumbing v1.3.0/go.mod h1:JOTb4XiRu5xfnmdnDJo6GmSbSbtSyufrsyZFByMtKEs= +github.com/bodgit/sevenzip v1.6.5 h1:7H7BxgmeX0j6UX42lH+KXQ92WgMQJ49DoocFdfHbCng= +github.com/bodgit/sevenzip v1.6.5/go.mod h1:GhuB6Lq1xCpP1sps+horjZ8lgiKPJcy2zUX3prla9wc= +github.com/bodgit/windows v1.0.1 h1:tF7K6KOluPYygXa3Z2594zxlkbKPAOvqr97etrGNIz4= +github.com/bodgit/windows v1.0.1/go.mod h1:a6JLwrB4KrTR5hBpp8FI9/9W9jJfeQ2h4XDXU74ZCdM= github.com/boj/redistore v1.4.1 h1:lP9ZZWqKMq2RIqexlZX1w1ODSnegL+puxGIujkU5tIw= github.com/boj/redistore v1.4.1/go.mod h1:c0Tvw6aMjslog4jHIAcNv6EtJM849YoOAhMY7JBbWpI= github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs= @@ -206,8 +212,8 @@ github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ= github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8= github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12 h1:9Nu54bhS/H/Kgo2/7xNSUuC5G28VR8ljfrLKU2G4IjU= github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12/go.mod h1:TBzl5BIHNXfS9+C35ZyJaklL7mLDbgUkcgXzSLa8Tk0= -github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE= -github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/klauspost/compress v1.19.0 h1:sXLILfc9jV2QYWkzFOPWStmcUVH2RHEB1JCdY2oVvCQ= +github.com/klauspost/compress v1.19.0/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= @@ -242,8 +248,8 @@ github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0 github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY= github.com/peterbourgon/diskv/v3 v3.0.1 h1:x06SQA46+PKIUftmEujdwSEpIx8kR+M9eLYsUxeYveU= github.com/peterbourgon/diskv/v3 v3.0.1/go.mod h1:kJ5Ny7vLdARGU3WUuy6uzO6T0nb/2gWcT1JiBvRmb5o= -github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY= -github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= +github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk= +github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= @@ -285,14 +291,19 @@ github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU= github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY= +github.com/stangelandcl/ppmd v0.1.1 h1:c25QazhlWUn5nmR1QOzafKhQxBicAr7GGCKER2aJ8H8= +github.com/stangelandcl/ppmd v0.1.1/go.mod h1:Rrv7M+/2P5jYr/GMLhBl7Ug3uJ1bUiVzr5LbbaV6xgY= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= +github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4= +github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= @@ -311,6 +322,8 @@ github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08= github.com/ugorji/go/codec v1.3.1 h1:waO7eEiFDwidsBN6agj1vJQ4AG7lh2yqXyOXqhgQuyY= github.com/ugorji/go/codec v1.3.1/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4= +github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY= +github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= github.com/uptrace/opentelemetry-go-extra/otelutil v0.3.2 h1:3/aHKUq7qaFMWxyQV0W2ryNgg8x8rVeKVA20KJUkfS0= github.com/uptrace/opentelemetry-go-extra/otelutil v0.3.2/go.mod h1:Zit4b8AQXaXvA68+nzmbyDzqiyFRISyw1JiD5JqUBjw= github.com/uptrace/opentelemetry-go-extra/otelzap v0.3.2 h1:cj/Z6FKTTYBnstI0Lni9PA+k2foounKIPUmj1LBwNiQ= @@ -356,37 +369,39 @@ go.uber.org/zap v1.27.1 h1:08RqriUEv8+ArZRYSTXy1LeBScaMpVSTBhCeaZYfMYc= go.uber.org/zap v1.27.1/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= +go4.org v0.0.0-20260112195520-a5071408f32f h1:ziUVAjmTPwQMBmYR1tbdRFJPtTcQUI12fH9QQjfb0Sw= +go4.org v0.0.0-20260112195520-a5071408f32f/go.mod h1:ZRJnO5ZI4zAwMFp+dS1+V6J6MSyAowhRqAE+DPa1Xp0= golang.org/x/arch v0.22.0 h1:c/Zle32i5ttqRXjdLyyHZESLD/bB90DCU1g9l/0YBDI= golang.org/x/arch v0.22.0/go.mod h1:dNHoOeKiyja7GTvF9NJS1l3Z2yntpQNzgrjh1cU103A= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI= -golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8= +golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto= +golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio= golang.org/x/image v0.42.0 h1:1gSs6ehNWXLbkHBIPcWztk3D/6aIA/8hauiAYtlodVY= golang.org/x/image v0.42.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4= -golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ= +golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ= +golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w= -golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ= +golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o= +golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM= -golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= +golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ= -golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= +golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= @@ -394,15 +409,15 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE= -golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4= +golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= +golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8= -golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0= +golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q= +golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= diff --git a/internal/apps/agent/sync/pages.go b/internal/apps/agent/sync/pages.go index e4867df0..07adb259 100644 --- a/internal/apps/agent/sync/pages.go +++ b/internal/apps/agent/sync/pages.go @@ -2,8 +2,6 @@ package sync import ( - "archive/zip" - "bytes" "context" "crypto/sha256" "encoding/hex" @@ -11,21 +9,19 @@ import ( "errors" "fmt" "io" - "math" "os" - "path" "path/filepath" "strings" "github.com/Rain-kl/Wavelet/internal/apps/agent/protocol" "github.com/Rain-kl/Wavelet/internal/apps/agent/state" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" ) const ( - pagesMaxExtractedFileBytes = 100 * 1024 * 1024 - pagesDirPerm = 0o755 - pagesFilePerm = 0o644 - pagesManifestFilePerm = 0o644 + pagesDirPerm = 0o755 + pagesFilePerm = 0o644 + pagesManifestFilePerm = 0o644 ) type pagesSourceDocument struct { @@ -223,89 +219,26 @@ func referencedPagesDeployments(config *protocol.ActiveConfigResponse) ([]pagesD return result, nil } -func findCommonRootPrefix(files []*zip.File) (string, error) { - var firstFilePath string - hasMultipleFiles := false - for _, item := range files { - relativePath, skip, err := normalizePagesArchivePath(item.Name) - if err != nil { - return "", err - } - if skip { - continue - } - normalizedPath := filepath.ToSlash(relativePath) - if firstFilePath == "" { - firstFilePath = normalizedPath - } else { - hasMultipleFiles = true - } - } - if firstFilePath == "" { - return "", nil - } - parts := strings.Split(firstFilePath, "/") - if len(parts) <= 1 { - return "", nil - } - commonPrefix := parts[0] + "/" - if hasMultipleFiles { - for _, item := range files { - relativePath, skip, err := normalizePagesArchivePath(item.Name) - if err != nil { - return "", err - } - if skip { - continue - } - normalizedPath := filepath.ToSlash(relativePath) - if !strings.HasPrefix(normalizedPath, commonPrefix) { - return "", nil - } - } - } - return commonPrefix, nil -} - func extractPagesPackage(packageBytes []byte, releaseDir string, deployment pagesDeploymentSource) error { tmpDir := releaseDir + ".tmp" _ = os.RemoveAll(tmpDir) if err := os.MkdirAll(tmpDir, pagesDirPerm); err != nil { return err } - reader, err := zip.NewReader(bytes.NewReader(packageBytes), int64(len(packageBytes))) + format, err := pagesarchive.DetectFormat("", packageBytes) if err != nil { _ = os.RemoveAll(tmpDir) - return fmt.Errorf("open Pages zip: %w", err) + return fmt.Errorf("detect Pages package format: %w", err) } - commonPrefix, err := findCommonRootPrefix(reader.File) - if err != nil { + // Control plane already inspected and accepted this package. Agent only + // verifies download integrity (checksum) and performs local-safe extract + // (path escape / symlink guards). Size and file-count limits are not re-applied. + if err := pagesarchive.ExtractBytes(packageBytes, format, tmpDir, pagesarchive.ExtractOptions{ + StripCommonRoot: true, + EnforceLimits: false, + }); err != nil { _ = os.RemoveAll(tmpDir) - return err - } - for _, item := range reader.File { - relativePath, skip, err := normalizePagesArchivePath(item.Name) - if err != nil { - _ = os.RemoveAll(tmpDir) - return err - } - if skip { - continue - } - if commonPrefix != "" { - slashPath := filepath.ToSlash(relativePath) - if strings.HasPrefix(slashPath, commonPrefix) { - relativePath = filepath.FromSlash(strings.TrimPrefix(slashPath, commonPrefix)) - } - } - if item.FileInfo().Mode()&os.ModeSymlink != 0 { - _ = os.RemoveAll(tmpDir) - return fmt.Errorf("pages package contains unsupported symlink: %s", relativePath) - } - if err := extractPagesFile(item, filepath.Join(tmpDir, relativePath)); err != nil { - _ = os.RemoveAll(tmpDir) - return err - } + return fmt.Errorf("extract Pages package: %w", err) } if err := writePagesMarker(tmpDir, deployment); err != nil { _ = os.RemoveAll(tmpDir) @@ -315,42 +248,6 @@ func extractPagesPackage(packageBytes []byte, releaseDir string, deployment page return os.Rename(tmpDir, releaseDir) } -func copyPagesZipEntryContent(dst io.Writer, src io.Reader, declaredSize uint64) (int64, error) { - if declaredSize > pagesMaxExtractedFileBytes || declaredSize > uint64(math.MaxInt64) { - return 0, errors.New("pages file size out of bounds") - } - if declaredSize > 0 { - return io.CopyN(dst, src, int64(declaredSize)) //nolint:gosec // declaredSize is bounded to math.MaxInt64 above - } - limited := io.LimitReader(src, pagesMaxExtractedFileBytes+1) - written, err := io.Copy(dst, limited) - if written > pagesMaxExtractedFileBytes { - return written, errors.New("pages file size out of bounds") - } - return written, err -} - -func extractPagesFile(item *zip.File, targetPath string) error { - if err := os.MkdirAll(filepath.Dir(targetPath), pagesDirPerm); err != nil { - return err - } - source, err := item.Open() - if err != nil { - return err - } - defer func() { _ = source.Close() }() - target, err := os.OpenFile(targetPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, pagesFilePerm) //nolint:gosec // targetPath is under managed PagesDir from validated zip entry - if err != nil { - return err - } - defer func() { _ = target.Close() }() - _, err = copyPagesZipEntryContent(target, source, item.UncompressedSize64) - if err != nil { - return fmt.Errorf("%s: %w", item.Name, err) - } - return nil -} - func switchPagesCurrentDir(baseDir string, deploymentID uint, releaseDir string) error { currentDir := pagesCurrentDir(baseDir, deploymentID) previousDir := currentDir + ".previous" @@ -440,24 +337,6 @@ func copyPagesDir(sourceDir string, targetDir string) error { }) } -func normalizePagesArchivePath(raw string) (string, bool, error) { - name := strings.TrimSpace(filepath.ToSlash(raw)) - if name == "" || strings.HasSuffix(name, "/") { - return "", true, nil - } - if strings.HasPrefix(name, "/") { - return "", false, fmt.Errorf("pages package contains absolute path: %s", raw) - } - cleaned := path.Clean(name) - if cleaned == "." { - return "", true, nil - } - if cleaned == ".." || strings.HasPrefix(cleaned, "../") || strings.Contains(cleaned, "/../") { - return "", false, fmt.Errorf("pages package path escapes deployment root: %s", raw) - } - return filepath.FromSlash(cleaned), false, nil -} - func markerMatches(dir string, deployment pagesDeploymentSource) bool { data, err := os.ReadFile(filepath.Join(dir, ".openflare-pages.json")) //nolint:gosec // dir is managed PagesDir if err != nil { diff --git a/internal/apps/agent/sync/service_test.go b/internal/apps/agent/sync/service_test.go index 2797a990..fdae7b7b 100644 --- a/internal/apps/agent/sync/service_test.go +++ b/internal/apps/agent/sync/service_test.go @@ -525,7 +525,7 @@ func TestSyncOnceRejectsPagesZipSlipBeforeApply(t *testing.T) { service.SetPagesDir(t.TempDir()) err := service.SyncOnce(context.Background(), &protocol.ActiveConfigMeta{Version: "20260309-102", Checksum: "pages-config-checksum"}) - if err == nil || !strings.Contains(err.Error(), "escapes deployment root") { + if err == nil || (!strings.Contains(err.Error(), "escapes deployment root") && !strings.Contains(err.Error(), "escapes directory")) { t.Fatalf("expected zip-slip rejection, got %v", err) } if len(manager.applyRouteContents) != 0 { diff --git a/internal/apps/openflare/agent/config.go b/internal/apps/openflare/agent/config.go index b16c20d0..a9656ffd 100644 --- a/internal/apps/openflare/agent/config.go +++ b/internal/apps/openflare/agent/config.go @@ -9,6 +9,7 @@ import ( "errors" "strings" + "github.com/Rain-kl/Wavelet/internal/apps/openflare/pages" "github.com/Rain-kl/Wavelet/internal/model" openrestyrender "github.com/Rain-kl/Wavelet/pkg/render/openresty" "gorm.io/gorm" @@ -38,10 +39,20 @@ func getActiveConfigForAgent(ctx context.Context) (*ConfigResponse, error) { } } + // Main config version history is independent of Pages deployment history. + // Agents always receive pages routes bound to each project's current active + // deployment so config rollback never depends on pruned packages. + sourceJSON := version.SnapshotJSON + if rebound, rebindErr := pages.RebindSnapshotPagesToCurrentActive(ctx, version.SnapshotJSON); rebindErr != nil { + return nil, rebindErr + } else if strings.TrimSpace(rebound) != "" { + sourceJSON = rebound + } + return &ConfigResponse{ Version: version.Version, Checksum: version.Checksum, - SourceConfigJSON: version.SnapshotJSON, + SourceConfigJSON: sourceJSON, SupportFiles: sourceSupportFiles(supportFiles), CreatedAt: version.CreatedAt, }, nil diff --git a/internal/apps/openflare/option/validate.go b/internal/apps/openflare/option/validate.go index dee250d4..4de58baa 100644 --- a/internal/apps/openflare/option/validate.go +++ b/internal/apps/openflare/option/validate.go @@ -59,6 +59,9 @@ func validateOptionWithState(ctx context.Context, option model.OpenFlareOption, if err := validateAgentOption(option.Key, option.Value); err != nil { return err } + if err := validatePagesOption(option.Key, option.Value); err != nil { + return err + } return validateUptimeKumaOption(ctx, option.Key, option.Value, state) } @@ -109,6 +112,23 @@ func validateAgentOption(key, value string) error { return nil } +func validatePagesOption(key, value string) error { + trimmed := strings.TrimSpace(value) + switch key { + case model.ConfigKeyPagesMaxPackageSizeMB: + intValue, err := strconv.Atoi(trimmed) + if err != nil || intValue < 1 || intValue > 2048 { + return fmt.Errorf("%s 必须为 1~2048 的整数(MiB)", key) + } + case model.ConfigKeyPagesMaxHistoryCount: + intValue, err := strconv.Atoi(trimmed) + if err != nil || intValue < 0 { + return fmt.Errorf("%s 必须为大于等于 0 的整数(0 表示不限制)", key) + } + } + return nil +} + func validateUptimeKumaOption(ctx context.Context, key, value string, state map[string]string) error { trimmed := strings.TrimSpace(value) switch key { diff --git a/internal/apps/openflare/pages/errs.go b/internal/apps/openflare/pages/errs.go index 4de9edf9..e5eaf1d4 100644 --- a/internal/apps/openflare/pages/errs.go +++ b/internal/apps/openflare/pages/errs.go @@ -14,9 +14,13 @@ const ( errPagesDeploymentMismatch = "pages 部署不属于该项目" errPagesDeleteActiveDeploy = "不能删除当前激活的 Pages 部署" errPagesPackageMissing = "缺少 Pages 部署包" - errPagesPackageNotZip = "pages 部署包必须是 .zip 文件" - errPagesPackageInvalidZip = "pages 部署包不是有效 zip 文件" + errPagesPackageNotZip = "pages 部署包必须是 .zip 文件" // legacy alias kept for tests + errPagesPackageUnsupported = "pages 部署包仅支持 zip、tar.gz、tar.xz、tar.bz2、tar、7z 格式" + errPagesPackageInvalidZip = "pages 部署包不是有效 zip 文件" // legacy alias + errPagesPackageInvalid = "pages 部署包不是有效的压缩文件" errPagesPackageEmpty = "pages 部署包不能为空" + errPagesPackageExtractedTooLarge = "pages 部署包展开后体积超过限制" + errPagesPackageFileTooLarge = "pages 部署包内文件过大" errPagesAPIProxyPathRequired = "启用 API 反代时,匹配路径不能为空" errPagesAPIProxyPathPrefix = "API 反代匹配路径必须以 '/' 开头" errPagesAPIProxyPassRequired = "启用 API 反代时,后端服务地址不能为空" //nolint:gosec // false positive: this is an error message, not hardcoded credentials diff --git a/internal/apps/openflare/pages/helpers.go b/internal/apps/openflare/pages/helpers.go index 086178bd..b1f2c959 100644 --- a/internal/apps/openflare/pages/helpers.go +++ b/internal/apps/openflare/pages/helpers.go @@ -4,14 +4,12 @@ package pages import ( - "archive/zip" "context" "crypto/sha256" "encoding/hex" "errors" "fmt" "io" - "math" "mime/multipart" "os" "path" @@ -22,17 +20,22 @@ import ( "github.com/Rain-kl/Wavelet/internal/apps/upload" "github.com/Rain-kl/Wavelet/internal/model" "github.com/Rain-kl/Wavelet/internal/repository" + "github.com/Rain-kl/Wavelet/pkg/logger" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" ) const ( - pagesMaxDeploymentFiles = 1000 - pagesMaxDeploymentBytes = 100 * 1024 * 1024 - defaultPagesEntryFile = "index.html" - defaultPagesFallbackPath = "/index.html" - pagesDeploymentUploadType = "openflare_pages_deployment" - mimeTypeApplicationZip = "application/zip" - pagesMaxPathLength = 512 - bytesPerKiB = 1024 + pagesMaxDeploymentFiles = 1000 + defaultPagesMaxPackageSizeMB = 100 + maxPagesMaxPackageSizeMB = 2048 + defaultPagesMaxHistoryCount = 20 + defaultPagesEntryFile = "index.html" + defaultPagesFallbackPath = "/index.html" + pagesDeploymentUploadType = "openflare_pages_deployment" + pagesMaxPathLength = 512 + bytesPerMiB = 1024 * 1024 + pagesExtractedSizeMultiplier = 4 + pagesMinExtractedSizeBytes = 100 * bytesPerMiB ) var pagesSlugPattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,126}[a-z0-9]$|^[a-z0-9]$`) @@ -42,6 +45,14 @@ type deploymentManifest struct { FileCount int TotalSize int64 EntryFile string + Format pagesarchive.Format +} + +type pagesLimits struct { + PackageBytes int64 + ExtractedBytes int64 + MaxFiles int + HistoryCount int } func isUniqueConstraintError(err error) bool { @@ -51,6 +62,38 @@ func isUniqueConstraintError(err error) bool { return strings.Contains(strings.ToLower(err.Error()), "unique") } +func resolvePagesLimits(ctx context.Context) pagesLimits { + packageMB := defaultPagesMaxPackageSizeMB + if value, err := repository.GetIntByKey(ctx, model.ConfigKeyPagesMaxPackageSizeMB); err == nil && value > 0 { + packageMB = value + } + if packageMB > maxPagesMaxPackageSizeMB { + packageMB = maxPagesMaxPackageSizeMB + } + + historyCount := defaultPagesMaxHistoryCount + if value, err := repository.GetIntByKey(ctx, model.ConfigKeyPagesMaxHistoryCount); err == nil { + if value < 0 { + historyCount = 0 + } else { + historyCount = value + } + } + + packageBytes := int64(packageMB) * bytesPerMiB + extractedBytes := packageBytes * pagesExtractedSizeMultiplier + if extractedBytes < pagesMinExtractedSizeBytes { + extractedBytes = pagesMinExtractedSizeBytes + } + + return pagesLimits{ + PackageBytes: packageBytes, + ExtractedBytes: extractedBytes, + MaxFiles: pagesMaxDeploymentFiles, + HistoryCount: historyCount, + } +} + func normalizePagesSlug(raw string) string { value := strings.ToLower(strings.TrimSpace(raw)) var builder strings.Builder @@ -76,7 +119,7 @@ func validateAndNormalizePagesRootDir(raw string) (string, error) { return "", nil } if len(value) > pagesMaxPathLength { - return "", errors.New("pages 根目录长度不能超过 512") // error 消息首字母小写 + return "", errors.New("pages 根目录长度不能超过 512") } if strings.Contains(value, "\\") || strings.ContainsAny(value, "\"';") { return "", errors.New("pages 根目录包含不支持的字符") @@ -151,29 +194,50 @@ func normalizePagesEntryFile(raw string) string { return strings.TrimPrefix(value, "/") } -func persistPagesUploadTemp(fileHeader *multipart.FileHeader) (string, string, int64, error) { +func persistPagesUploadTemp(fileHeader *multipart.FileHeader, maxPackageBytes int64) (string, string, int64, pagesarchive.Format, error) { + format, ok := pagesarchive.DetectFormatFromName(fileHeader.Filename) + if !ok { + return "", "", 0, "", errors.New(errPagesPackageUnsupported) + } file, err := fileHeader.Open() if err != nil { - return "", "", 0, err + return "", "", 0, "", err } defer func() { _ = file.Close() }() - temp, err := os.CreateTemp("", "openflare-pages-*.zip") + temp, err := os.CreateTemp("", "openflare-pages-*."+safeTempSuffix(format)) if err != nil { - return "", "", 0, err + return "", "", 0, "", err } defer func() { _ = temp.Close() }() hash := sha256.New() - limited := io.LimitReader(file, pagesMaxDeploymentBytes+1) + limited := io.LimitReader(file, maxPackageBytes+1) written, err := io.Copy(io.MultiWriter(temp, hash), limited) if err != nil { _ = os.Remove(temp.Name()) - return "", "", 0, err + return "", "", 0, "", err } - if written > pagesMaxDeploymentBytes { + if written > maxPackageBytes { _ = os.Remove(temp.Name()) - return "", "", 0, fmt.Errorf("pages 部署包不能超过 %d MiB", pagesMaxDeploymentBytes/bytesPerKiB/bytesPerKiB) + return "", "", 0, "", fmt.Errorf("pages 部署包不能超过 %d MiB", maxPackageBytes/bytesPerMiB) + } + return temp.Name(), hex.EncodeToString(hash.Sum(nil)), written, format, nil +} + +func safeTempSuffix(format pagesarchive.Format) string { + switch format { + case pagesarchive.FormatTarGz: + return "tar.gz" + case pagesarchive.FormatTarXz: + return "tar.xz" + case pagesarchive.FormatTarBz2: + return "tar.bz2" + case pagesarchive.FormatSevenZip: + return "7z" + case pagesarchive.FormatTar: + return "tar" + default: + return "zip" } - return temp.Name(), hex.EncodeToString(hash.Sum(nil)), written, nil } func pagesLegacyRelativeCandidates(project *model.PagesProject, deployment *model.PagesDeployment) []string { @@ -185,6 +249,7 @@ func pagesLegacyRelativeCandidates(project *model.PagesProject, deployment *mode if slug == "" || checksum == "" { return nil } + // Legacy artifacts were always stored as .zip. fileName := checksum + ".zip" return []string{ filepath.Join("artifacts", slug, fileName), @@ -199,14 +264,16 @@ func ingestPagesDeploymentPackage( checksum string, projectSlug string, fileName string, + format pagesarchive.Format, ) (upload.IngestResult, error) { systemUser := repository.GetSystemUser(ctx) accessMode := 0 + extension := pagesarchive.NormalizeNameExtension(fileName, format) return upload.IngestFromLocalPath(ctx, localPath, upload.IngestRequest{ UserID: systemUser.ID, FileName: fileName, - MimeType: mimeTypeApplicationZip, - Extension: "zip", + MimeType: pagesarchive.MIMEType(format), + Extension: extension, Hash: checksum, Type: pagesDeploymentUploadType, AccessMode: &accessMode, @@ -215,6 +282,7 @@ func ingestPagesDeploymentPackage( Metadata: model.UploadMetadata{ Extra: map[string]any{ "project_slug": projectSlug, + "format": string(format), }, }, }) @@ -224,167 +292,96 @@ func removeDeploymentArtifact(ctx context.Context, deployment *model.PagesDeploy if deployment == nil { return } - if deployment.UploadID > 0 { - _, _ = upload.Remove(ctx, deployment.UploadID) + if deployment.UploadID == 0 { + return + } + if _, err := upload.Remove(ctx, deployment.UploadID); err != nil { + // Soft-delete / storage cleanup failure must not undo DB prune; log for ops. + logger.WarnF(ctx, + "[Pages] remove deployment artifact failed: deployment_id=%d upload_id=%d error=%v", + deployment.ID, deployment.UploadID, err, + ) } } -func findCommonRootPrefix(files []*zip.File) (string, error) { - var firstFilePath string - hasMultipleFiles := false - for _, item := range files { - normalizedPath, skip, err := normalizePagesZipPath(item.Name) - if err != nil { - return "", err - } - if skip { - continue - } - if firstFilePath == "" { - firstFilePath = normalizedPath - } else { - hasMultipleFiles = true - } - } - if firstFilePath == "" { - return "", nil - } - parts := strings.Split(firstFilePath, "/") - if len(parts) <= 1 { - return "", nil - } - commonPrefix := parts[0] + "/" - if hasMultipleFiles { - for _, item := range files { - normalizedPath, skip, err := normalizePagesZipPath(item.Name) - if err != nil { - return "", err - } - if skip { - continue - } - if !strings.HasPrefix(normalizedPath, commonPrefix) { - return "", nil - } - } - } - return commonPrefix, nil -} - -func inspectPagesZip(zipPath string, rootDir string, entryFile string) (*deploymentManifest, error) { - reader, err := zip.OpenReader(zipPath) - if err != nil { - return nil, errors.New(errPagesPackageInvalidZip) - } - defer func() { _ = reader.Close() }() - - commonPrefix, err := findCommonRootPrefix(reader.File) - if err != nil { - return nil, err - } - - manifest := &deploymentManifest{ - Files: []model.PagesDeploymentFile{}, +func inspectPagesPackage(packagePath string, format pagesarchive.Format, rootDir string, entryFile string, limits pagesLimits) (*deploymentManifest, error) { + archiveManifest, err := pagesarchive.InspectFile(packagePath, format, pagesarchive.InspectOptions{ + RootDir: rootDir, EntryFile: entryFile, + Limits: pagesarchive.Limits{ + MaxFiles: limits.MaxFiles, + MaxFileBytes: limits.ExtractedBytes, + MaxTotalBytes: limits.ExtractedBytes, + }, + }) + if err != nil { + return nil, mapPagesArchiveError(err) } - targetEntryPath := entryFile - if rootDir != "" { - targetEntryPath = path.Join(rootDir, entryFile) + manifest := &deploymentManifest{ + Files: make([]model.PagesDeploymentFile, 0, len(archiveManifest.Files)), + FileCount: archiveManifest.FileCount, + TotalSize: archiveManifest.TotalSize, + EntryFile: entryFile, + Format: format, } - entrySeen := false - for _, item := range reader.File { - normalizedPath, skip, err := normalizePagesZipPath(item.Name) - if err != nil { - return nil, err - } - if skip { - continue - } - if commonPrefix != "" { - normalizedPath = strings.TrimPrefix(normalizedPath, commonPrefix) - } - if item.FileInfo().Mode()&os.ModeSymlink != 0 { - return nil, fmt.Errorf("pages 部署包不支持符号链接: %s", normalizedPath) - } - if item.UncompressedSize64 > pagesMaxDeploymentBytes { - return nil, fmt.Errorf("pages 文件过大: %s", normalizedPath) - } - manifest.FileCount++ - if manifest.FileCount > pagesMaxDeploymentFiles { - return nil, fmt.Errorf("pages 部署文件数不能超过 %d", pagesMaxDeploymentFiles) - } - checksum, fileSize, err := checksumZipFile(item) - if err != nil { - return nil, fmt.Errorf("%s: %w", normalizedPath, err) - } - manifest.TotalSize += fileSize - if manifest.TotalSize > pagesMaxDeploymentBytes { - return nil, fmt.Errorf("pages 部署展开后不能超过 %d MiB", pagesMaxDeploymentBytes/bytesPerKiB/bytesPerKiB) - } - if normalizedPath == targetEntryPath { - entrySeen = true - } + for _, file := range archiveManifest.Files { manifest.Files = append(manifest.Files, model.PagesDeploymentFile{ - Path: normalizedPath, - Size: fileSize, - Checksum: checksum, + Path: file.Path, + Size: file.Size, + Checksum: file.Checksum, }) } - if manifest.FileCount == 0 { - return nil, errors.New(errPagesPackageEmpty) - } - if !entrySeen { - return nil, fmt.Errorf("pages 部署包缺少入口文件 %s", targetEntryPath) - } return manifest, nil } -func normalizePagesZipPath(raw string) (string, bool, error) { - name := strings.TrimSpace(filepath.ToSlash(raw)) - if name == "" { - return "", true, nil +func mapPagesArchiveError(err error) error { + if err == nil { + return nil } - if strings.HasSuffix(name, "/") { - return "", true, nil + message := err.Error() + switch { + case strings.Contains(message, "unsupported pages package format"): + return errors.New(errPagesPackageUnsupported) + case strings.Contains(message, "open zip"), strings.Contains(message, "open gzip"), + strings.Contains(message, "open xz"), strings.Contains(message, "open 7z"), + strings.Contains(message, "read tar"): + return errors.New(errPagesPackageInvalid) + case strings.Contains(message, "empty"): + return errors.New(errPagesPackageEmpty) + case strings.Contains(message, "missing entry file"): + return err + case strings.Contains(message, "file count exceeds"): + return fmt.Errorf("pages 部署文件数不能超过 %d", pagesMaxDeploymentFiles) + case strings.Contains(message, "extracted size exceeds"): + return errors.New(errPagesPackageExtractedTooLarge) + case strings.Contains(message, "file too large"), strings.Contains(message, "size out of bounds"): + return errors.New(errPagesPackageFileTooLarge) + case strings.Contains(message, "symlink"): + return err + case strings.Contains(message, "absolute path"), strings.Contains(message, "escapes directory"): + return err + default: + return err } - if strings.HasPrefix(name, "/") || path.IsAbs(name) { - return "", false, fmt.Errorf("pages 部署包不能包含绝对路径: %s", raw) - } - cleaned := path.Clean(name) - if cleaned == "." { - return "", true, nil - } - if cleaned == ".." || strings.HasPrefix(cleaned, "../") || strings.Contains(cleaned, "/../") { - return "", false, fmt.Errorf("pages 部署包路径不能逃逸目录: %s", raw) - } - return cleaned, false, nil } -func copyPagesZipEntryContent(dst io.Writer, src io.Reader, declaredSize uint64) (int64, error) { - if declaredSize > pagesMaxDeploymentBytes || declaredSize > uint64(math.MaxInt64) { - return 0, errors.New("pages file size out of bounds") +func packageDownloadName(deploymentID uint, fileName string, contentType string) string { + if format, ok := pagesarchive.DetectFormatFromName(fileName); ok { + return fmt.Sprintf("pages-deployment-%d.%s", deploymentID, pagesarchive.Extension(format)) } - if declaredSize > 0 { - return io.CopyN(dst, src, int64(declaredSize)) //nolint:gosec // declaredSize is bounded to math.MaxInt64 above + // Fall back by content type. + switch strings.ToLower(strings.TrimSpace(contentType)) { + case "application/gzip", "application/x-gzip": + return fmt.Sprintf("pages-deployment-%d.tar.gz", deploymentID) + case "application/x-xz": + return fmt.Sprintf("pages-deployment-%d.tar.xz", deploymentID) + case "application/x-bzip2": + return fmt.Sprintf("pages-deployment-%d.tar.bz2", deploymentID) + case "application/x-7z-compressed": + return fmt.Sprintf("pages-deployment-%d.7z", deploymentID) + case "application/x-tar": + return fmt.Sprintf("pages-deployment-%d.tar", deploymentID) + default: + return fmt.Sprintf("pages-deployment-%d.zip", deploymentID) } - limited := io.LimitReader(src, pagesMaxDeploymentBytes+1) - written, err := io.Copy(dst, limited) - if written > pagesMaxDeploymentBytes { - return written, errors.New("pages file size out of bounds") - } - return written, err -} - -func checksumZipFile(item *zip.File) (string, int64, error) { - file, err := item.Open() - if err != nil { - return "", 0, err - } - defer func() { _ = file.Close() }() - hash := sha256.New() - written, err := copyPagesZipEntryContent(hash, file, item.UncompressedSize64) - if err != nil { - return "", written, err - } - return hex.EncodeToString(hash.Sum(nil)), written, nil } diff --git a/internal/apps/openflare/pages/logics.go b/internal/apps/openflare/pages/logics.go index 6011241b..883fc2f7 100644 --- a/internal/apps/openflare/pages/logics.go +++ b/internal/apps/openflare/pages/logics.go @@ -12,13 +12,14 @@ import ( "mime/multipart" "net/url" "os" - "path/filepath" "strings" "time" "github.com/Rain-kl/Wavelet/internal/apps/upload" "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/pkg/logger" + "github.com/Rain-kl/Wavelet/pkg/pagesarchive" "gorm.io/gorm" ) @@ -251,20 +252,18 @@ func UploadDeployment(ctx context.Context, projectID uint, fileHeader *multipart if fileHeader == nil { return nil, errors.New(errPagesPackageMissing) } - if !strings.EqualFold(filepath.Ext(fileHeader.Filename), ".zip") { - return nil, errors.New(errPagesPackageNotZip) - } + limits := resolvePagesLimits(ctx) rootDir, err := validateAndNormalizePagesRootDir(project.RootDir) if err != nil { return nil, err } entryFile := normalizePagesEntryFile(project.EntryFile) - tempPath, checksum, _, err := persistPagesUploadTemp(fileHeader) + tempPath, checksum, _, format, err := persistPagesUploadTemp(fileHeader, limits.PackageBytes) if err != nil { return nil, err } defer func() { _ = os.Remove(tempPath) }() - manifest, err := inspectPagesZip(tempPath, rootDir, entryFile) + manifest, err := inspectPagesPackage(tempPath, format, rootDir, entryFile, limits) if err != nil { return nil, err } @@ -274,6 +273,7 @@ func UploadDeployment(ctx context.Context, projectID uint, fileHeader *multipart checksum, project.Slug, fileHeader.Filename, + format, ) if err != nil { return nil, err @@ -320,10 +320,160 @@ func UploadDeployment(ctx context.Context, projectID uint, fileHeader *multipart return nil, err } ingestCommitted = true + + // History prune must not fail the already-committed upload. Log and continue; + // a later upload (or a retry pass inside prune) will re-attempt cleanup. + if pruneErr := pruneProjectDeploymentHistory(ctx, project.ID, limits.HistoryCount); pruneErr != nil { + logger.ErrorF(ctx, + "[Pages] prune deployment history failed: project_id=%d keep=%d error=%v", + project.ID, limits.HistoryCount, pruneErr, + ) + } + view := buildDeploymentView(deployment) return &view, nil } +// pruneProjectDeploymentHistory enforces the retention policy for one project. +// +// Policy (keepCount > 0): +// - At most keepCount deployment rows remain for the project. +// - The current active deployment is always retained (if any). +// - Remaining slots are filled by newest deployments first (id desc). +// - All other non-kept deployments are deleted with their file lists and artifacts. +// +// keepCount <= 0 means unlimited history. +// +// Concurrency: DB row deletes run in a single transaction after a consistent read +// of project + deployments. Concurrent uploads may briefly exceed keepCount; the +// next successful prune brings the project back within the limit (eventual). +func pruneProjectDeploymentHistory(ctx context.Context, projectID uint, keepCount int) error { + if keepCount <= 0 { + return nil + } + + // Two passes: first pass after upload, second pass heals a concurrent race + // that inserted another deployment between our list and delete. + var lastErr error + for pass := 0; pass < 2; pass++ { + deleted, err := pruneProjectDeploymentHistoryOnce(ctx, projectID, keepCount) + if err != nil { + lastErr = err + break + } + if deleted == 0 { + break + } + } + return lastErr +} + +// pruneProjectDeploymentHistoryOnce performs one list → select → delete cycle. +// Returns the number of deployments deleted from the database. +func pruneProjectDeploymentHistoryOnce(ctx context.Context, projectID uint, keepCount int) (int, error) { + project, err := model.GetPagesProjectByID(ctx, projectID) + if err != nil { + return 0, fmt.Errorf("load pages project: %w", err) + } + deployments, err := model.ListPagesDeployments(ctx, projectID) + if err != nil { + return 0, fmt.Errorf("list pages deployments: %w", err) + } + if len(deployments) <= keepCount { + return 0, nil + } + + var activeID uint + if project.ActiveDeploymentID != nil { + activeID = *project.ActiveDeploymentID + } + toDelete := selectDeploymentsToPrune(deployments, activeID, keepCount) + if len(toDelete) == 0 { + return 0, nil + } + + // Delete metadata in one transaction so partial prune does not leave + // orphan file-list rows without a parent deployment. + if err := db.DB(ctx).Transaction(func(tx *gorm.DB) error { + for index := range toDelete { + deployment := toDelete[index] + // Never delete the active deployment even if project pointer raced. + if activeID != 0 && deployment.ID == activeID { + continue + } + if project.ActiveDeploymentID != nil && deployment.ID == *project.ActiveDeploymentID { + continue + } + if err := tx.Where("deployment_id = ?", deployment.ID).Delete(&model.PagesDeploymentFile{}).Error; err != nil { + return fmt.Errorf("delete deployment files id=%d: %w", deployment.ID, err) + } + if err := tx.Where("id = ? AND project_id = ?", deployment.ID, projectID). + Delete(&model.PagesDeployment{}).Error; err != nil { + return fmt.Errorf("delete deployment id=%d: %w", deployment.ID, err) + } + } + return nil + }); err != nil { + return 0, err + } + + // Artifacts are best-effort outside the transaction (object storage I/O). + for index := range toDelete { + deployment := toDelete[index] + if activeID != 0 && deployment.ID == activeID { + continue + } + removeDeploymentArtifact(ctx, &deployment) + } + + logger.InfoF(ctx, + "[Pages] pruned deployment history: project_id=%d keep=%d deleted=%d", + projectID, keepCount, len(toDelete), + ) + return len(toDelete), nil +} + +// selectDeploymentsToPrune returns deployments that should be removed under the +// "at most keepCount, always keep active, fill with newest" policy. +// deployments must be ordered newest-first (id desc). +func selectDeploymentsToPrune(deployments []model.PagesDeployment, activeID uint, keepCount int) []model.PagesDeployment { + if keepCount <= 0 || len(deployments) <= keepCount { + return nil + } + + keepIDs := make(map[uint]struct{}, keepCount) + // 1) Active is always retained and occupies one slot when present. + if activeID != 0 { + // Only count active if it still exists in the list. + for _, deployment := range deployments { + if deployment.ID == activeID { + keepIDs[activeID] = struct{}{} + break + } + } + } + // 2) Fill remaining slots from newest to oldest. + for _, deployment := range deployments { + if len(keepIDs) >= keepCount { + break + } + keepIDs[deployment.ID] = struct{}{} + } + + toDelete := make([]model.PagesDeployment, 0, len(deployments)-len(keepIDs)) + for _, deployment := range deployments { + if _, keep := keepIDs[deployment.ID]; keep { + continue + } + // Safety: never mark active for deletion. + if activeID != 0 && deployment.ID == activeID { + continue + } + toDelete = append(toDelete, deployment) + } + return toDelete +} + // ActivateDeployment 激活 Pages 部署。 func ActivateDeployment(ctx context.Context, projectID uint, deploymentID uint) (*View, error) { project, err := model.GetPagesProjectByID(ctx, projectID) @@ -422,10 +572,14 @@ func openDeploymentPackageFromUpload(ctx context.Context, uploadID uint64, deplo } contentType := opened.ContentType if contentType == "" { - contentType = mimeTypeApplicationZip + contentType = opened.Upload.MimeType } + if contentType == "" { + contentType = "application/octet-stream" + } + fileName := packageDownloadName(deploymentID, opened.Upload.FileName, contentType) return DeploymentPackage{ - FileName: fmt.Sprintf("pages-deployment-%d.zip", deploymentID), + FileName: fileName, ContentType: contentType, ContentLength: opened.ContentLength, Body: opened.Body, @@ -478,6 +632,7 @@ func hydrateLegacyDeploymentUpload( deployment.Checksum, project.Slug, fmt.Sprintf("pages-deployment-%d.zip", deployment.ID), + pagesarchive.FormatZip, ) if err != nil { return nil, err @@ -493,7 +648,26 @@ func hydrateLegacyDeploymentUpload( return &ingestResult.Upload, nil } +// ensureDeploymentInActiveSnapshot allows Agent package download when the +// deployment is the project's current active deployment and that project is +// used by at least one pages route in the active main config. +// +// Main config versions pin historical pages_deployment ids for audit only. +// Runtime download always follows the live active Pages deployment (dual +// version control); rolling back main config must not require old packages. func ensureDeploymentInActiveSnapshot(ctx context.Context, deploymentID uint) error { + deployment, err := model.GetPagesDeploymentByID(ctx, deploymentID) + if err != nil { + return err + } + project, err := model.GetPagesProjectByID(ctx, deployment.ProjectID) + if err != nil { + return err + } + if project.ActiveDeploymentID == nil || *project.ActiveDeploymentID != deployment.ID { + return errors.New(errPagesPackageNotInActiveConfig) + } + version, err := model.GetActiveConfigVersion(ctx) if err != nil { if errors.Is(err, gorm.ErrRecordNotFound) { @@ -506,10 +680,30 @@ func ensureDeploymentInActiveSnapshot(ctx context.Context, deploymentID uint) er return err } for _, route := range routes { - if route.UpstreamType != "pages" || route.PagesDeployment == nil { + if !strings.EqualFold(strings.TrimSpace(route.UpstreamType), "pages") { continue } - if route.PagesDeployment.DeploymentID == deploymentID { + if route.PagesProjectID != nil && *route.PagesProjectID == project.ID { + return nil + } + if route.PagesDeployment == nil { + continue + } + if route.PagesDeployment.ProjectID == project.ID { + return nil + } + // Frozen snapshot may only carry deployment_id; resolve project via that row. + if route.PagesDeployment.DeploymentID == 0 { + continue + } + if route.PagesDeployment.DeploymentID == deployment.ID { + return nil + } + snapDeployment, snapErr := model.GetPagesDeploymentByID(ctx, route.PagesDeployment.DeploymentID) + if snapErr != nil { + continue + } + if snapDeployment.ProjectID == project.ID { return nil } } @@ -518,10 +712,12 @@ func ensureDeploymentInActiveSnapshot(ctx context.Context, deploymentID uint) er type snapshotPagesDeployment struct { DeploymentID uint `json:"deployment_id"` + ProjectID uint `json:"project_id"` } type snapshotRouteRef struct { UpstreamType string `json:"upstream_type"` + PagesProjectID *uint `json:"pages_project_id"` PagesDeployment *snapshotPagesDeployment `json:"pages_deployment"` } diff --git a/internal/apps/openflare/pages/logics_test.go b/internal/apps/openflare/pages/logics_test.go index ccef6a54..efed9f18 100644 --- a/internal/apps/openflare/pages/logics_test.go +++ b/internal/apps/openflare/pages/logics_test.go @@ -4,8 +4,10 @@ package pages import ( + "archive/tar" "archive/zip" "bytes" + "compress/gzip" "context" "fmt" "io" @@ -17,6 +19,7 @@ import ( "github.com/Rain-kl/Wavelet/internal/db" "github.com/Rain-kl/Wavelet/internal/model" + "github.com/Rain-kl/Wavelet/internal/repository" "github.com/Rain-kl/Wavelet/internal/storage" "github.com/glebarez/sqlite" "github.com/stretchr/testify/assert" @@ -40,6 +43,7 @@ func setupPagesTestDB(t *testing.T) func() { &model.PagesDeployment{}, &model.PagesDeploymentFile{}, &model.ConfigVersion{}, + &model.SystemConfig{}, )) require.NoError(t, sqliteDB.Create(&model.User{ ID: 999, @@ -48,8 +52,25 @@ func setupPagesTestDB(t *testing.T) func() { Nickname: "系统", IsActive: true, }).Error) + require.NoError(t, sqliteDB.Create([]model.SystemConfig{ + { + Key: model.ConfigKeyPagesMaxPackageSizeMB, + Value: "100", + Type: "business", + Description: "Pages 部署包上传大小上限(MiB)", + }, + { + Key: model.ConfigKeyPagesMaxHistoryCount, + Value: "0", // unlimited for existing tests + Type: "business", + Description: "Pages 每个项目最大历史部署保留数(0 表示不限制)", + }, + }).Error) db.SetDB(sqliteDB) + // Clear process-global system config RAM cache so tests do not see stale values. + _ = repository.InvalidateSystemConfigCache(context.Background(), model.ConfigKeyPagesMaxPackageSizeMB) + _ = repository.InvalidateSystemConfigCache(context.Background(), model.ConfigKeyPagesMaxHistoryCount) return func() { db.SetDB(nil) } @@ -339,6 +360,149 @@ func testPagesZip(t *testing.T, files map[string]string) []byte { return buffer.Bytes() } +func TestUploadDeploymentAcceptsTarGz(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + ctx := context.Background() + + project, err := CreateProject(ctx, Input{ + Name: "TarGz Site", + Slug: "tar-gz-site", + Enabled: true, + }) + require.NoError(t, err) + + deployment, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "site.tar.gz", testPagesTarGz(t, map[string]string{ + "index.html": "tar-ok", + "app.js": "1", + })), "root") + require.NoError(t, err) + assert.Equal(t, 2, deployment.FileCount) + assert.NotZero(t, deployment.UploadID) +} + +func TestSelectDeploymentsToPruneKeepsActiveAndNewest(t *testing.T) { + // ids 4(newest) ... 1(oldest); active is oldest id=1; keep=2 → keep {1,4}, prune {3,2} + deployments := []model.PagesDeployment{ + {ID: 4, ProjectID: 1}, + {ID: 3, ProjectID: 1}, + {ID: 2, ProjectID: 1}, + {ID: 1, ProjectID: 1}, + } + toDelete := selectDeploymentsToPrune(deployments, 1, 2) + require.Len(t, toDelete, 2) + assert.Equal(t, uint(3), toDelete[0].ID) + assert.Equal(t, uint(2), toDelete[1].ID) + + // active is newest; keep=2 → keep {4,3}, prune {2,1} + toDelete = selectDeploymentsToPrune(deployments, 4, 2) + require.Len(t, toDelete, 2) + assert.Equal(t, uint(2), toDelete[0].ID) + assert.Equal(t, uint(1), toDelete[1].ID) + + // no active; keep=2 → keep {4,3} + toDelete = selectDeploymentsToPrune(deployments, 0, 2) + require.Len(t, toDelete, 2) + assert.Equal(t, uint(2), toDelete[0].ID) + assert.Equal(t, uint(1), toDelete[1].ID) + + // keep=1 with active → only active, prune the rest + toDelete = selectDeploymentsToPrune(deployments, 2, 1) + require.Len(t, toDelete, 3) + for _, item := range toDelete { + assert.NotEqual(t, uint(2), item.ID) + } + + // already within limit + assert.Nil(t, selectDeploymentsToPrune(deployments[:2], 4, 2)) + // unlimited + assert.Nil(t, selectDeploymentsToPrune(deployments, 1, 0)) +} + +func TestPruneProjectDeploymentHistory(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + _, disableStorage := setupPagesStorageMock(t) + defer disableStorage() + ctx := context.Background() + + require.NoError(t, db.DB(ctx).Model(&model.SystemConfig{}). + Where("key = ?", model.ConfigKeyPagesMaxHistoryCount). + Update("value", "2").Error) + require.NoError(t, repository.InvalidateSystemConfigCache(ctx, model.ConfigKeyPagesMaxHistoryCount)) + + project, err := CreateProject(ctx, Input{ + Name: "History Site", + Slug: "history-site", + Enabled: true, + }) + require.NoError(t, err) + + var ids []uint + for i := 0; i < 3; i++ { + deployment, uploadErr := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "site.zip", testPagesZip(t, map[string]string{ + "index.html": fmt.Sprintf("v%d", i), + })), "root") + require.NoError(t, uploadErr) + ids = append(ids, deployment.ID) + } + // After 3 uploads with keep=2 and no active: only 2 newest remain. + deployments, err := model.ListPagesDeployments(ctx, project.ID) + require.NoError(t, err) + require.Len(t, deployments, 2) + assert.Equal(t, ids[2], deployments[0].ID) + assert.Equal(t, ids[1], deployments[1].ID) + + // Activate the older of the remaining two, then upload again. + _, err = ActivateDeployment(ctx, project.ID, ids[1]) + require.NoError(t, err) + + latest, err := UploadDeployment(ctx, project.ID, testPagesMultipartFile(t, "site.zip", testPagesZip(t, map[string]string{ + "index.html": "v-latest", + })), "root") + require.NoError(t, err) + + deployments, err = model.ListPagesDeployments(ctx, project.ID) + require.NoError(t, err) + require.Len(t, deployments, 2, "must be at most N=2, not active+N newest") + + storedProject, err := model.GetPagesProjectByID(ctx, project.ID) + require.NoError(t, err) + require.NotNil(t, storedProject.ActiveDeploymentID) + assert.Equal(t, ids[1], *storedProject.ActiveDeploymentID) + + kept := map[uint]struct{}{} + for _, item := range deployments { + kept[item.ID] = struct{}{} + } + _, hasActive := kept[ids[1]] + _, hasLatest := kept[latest.ID] + assert.True(t, hasActive, "active deployment must be retained") + assert.True(t, hasLatest, "newest deployment must fill remaining slot") +} + +func testPagesTarGz(t *testing.T, files map[string]string) []byte { + t.Helper() + + var buffer bytes.Buffer + gzWriter := gzip.NewWriter(&buffer) + tarWriter := tar.NewWriter(gzWriter) + for name, content := range files { + require.NoError(t, tarWriter.WriteHeader(&tar.Header{ + Name: name, + Mode: 0o644, + Size: int64(len(content)), + })) + _, err := tarWriter.Write([]byte(content)) + require.NoError(t, err) + } + require.NoError(t, tarWriter.Close()) + require.NoError(t, gzWriter.Close()) + return buffer.Bytes() +} + func testPagesMultipartFile(t *testing.T, fileName string, content []byte) *multipart.FileHeader { t.Helper() diff --git a/internal/apps/openflare/pages/rebind.go b/internal/apps/openflare/pages/rebind.go new file mode 100644 index 00000000..1e177c98 --- /dev/null +++ b/internal/apps/openflare/pages/rebind.go @@ -0,0 +1,236 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "encoding/json" + "fmt" + "strings" + + "github.com/Rain-kl/Wavelet/internal/model" + openrestyrender "github.com/Rain-kl/Wavelet/pkg/render/openresty" + "gorm.io/gorm" +) + +// RebindSnapshotPagesToCurrentActive rewrites pages_deployment fields so every +// pages route points at the project's current active deployment. +// +// Main config versions and Pages deployments are independent. Rolling back a +// main config version must not require old Pages packages; Agents always follow +// the live active deployment for each referenced project. +// +// Returns the original JSON unchanged when there are no pages routes. Does not +// mutate stored config_versions rows. Non-pages route fields are preserved. +func RebindSnapshotPagesToCurrentActive(ctx context.Context, snapshotJSON string) (string, error) { + text := strings.TrimSpace(snapshotJSON) + if text == "" { + return snapshotJSON, nil + } + + if strings.HasPrefix(text, "[") { + var routes []map[string]json.RawMessage + if err := json.Unmarshal([]byte(text), &routes); err != nil { + return "", fmt.Errorf("parse pages snapshot routes: %w", err) + } + changed, err := rebindPagesRouteMaps(ctx, routes) + if err != nil { + return "", err + } + if !changed { + return snapshotJSON, nil + } + encoded, err := json.Marshal(routes) + if err != nil { + return "", err + } + return string(encoded), nil + } + + var raw map[string]json.RawMessage + if err := json.Unmarshal([]byte(text), &raw); err != nil { + return "", fmt.Errorf("parse pages snapshot document: %w", err) + } + routesRaw, ok := raw["routes"] + if !ok || len(routesRaw) == 0 { + return snapshotJSON, nil + } + var routes []map[string]json.RawMessage + if err := json.Unmarshal(routesRaw, &routes); err != nil { + return "", fmt.Errorf("parse pages snapshot routes: %w", err) + } + changed, err := rebindPagesRouteMaps(ctx, routes) + if err != nil { + return "", err + } + if !changed { + return snapshotJSON, nil + } + encodedRoutes, err := json.Marshal(routes) + if err != nil { + return "", err + } + raw["routes"] = encodedRoutes + encoded, err := json.Marshal(raw) + if err != nil { + return "", err + } + return string(encoded), nil +} + +func rebindPagesRouteMaps(ctx context.Context, routes []map[string]json.RawMessage) (bool, error) { + changed := false + for index := range routes { + route := routes[index] + if route == nil { + continue + } + upstreamType, _ := rawJSONString(route["upstream_type"]) + if !strings.EqualFold(strings.TrimSpace(upstreamType), "pages") { + continue + } + siteName, _ := rawJSONString(route["site_name"]) + projectID, err := resolveProjectIDFromRouteMap(route) + if err != nil { + if siteName == "" { + siteName = "pages" + } + return false, fmt.Errorf("路由 %s %w", siteName, err) + } + project, activeDeployment, err := loadActivePagesProject(ctx, projectID, siteName) + if err != nil { + return false, err + } + deployment := buildLivePagesDeployment(project, activeDeployment) + projectIDCopy := project.ID + originURL := fmt.Sprintf("openflare-pages://project/%d", project.ID) + + if err := putJSON(route, "pages_project_id", projectIDCopy); err != nil { + return false, err + } + if err := putJSON(route, "pages_deployment", deployment); err != nil { + return false, err + } + if err := putJSON(route, "origin_url", originURL); err != nil { + return false, err + } + if err := putJSON(route, "upstreams", []string{originURL}); err != nil { + return false, err + } + routes[index] = route + changed = true + } + return changed, nil +} + +const jsonNullLiteral = "null" + +func isPresentJSON(raw json.RawMessage) bool { + return len(raw) > 0 && string(raw) != jsonNullLiteral +} + +func resolveProjectIDFromRouteMap(route map[string]json.RawMessage) (uint, error) { + if raw, ok := route["pages_project_id"]; ok && isPresentJSON(raw) { + var projectID uint + if err := json.Unmarshal(raw, &projectID); err == nil && projectID != 0 { + return projectID, nil + } + } + if raw, ok := route["pages_deployment"]; ok && isPresentJSON(raw) { + var deployment struct { + ProjectID uint `json:"project_id"` + } + if err := json.Unmarshal(raw, &deployment); err == nil && deployment.ProjectID != 0 { + return deployment.ProjectID, nil + } + } + return 0, fmt.Errorf("pages 配置无效: 缺少 pages_project_id") +} + +func loadActivePagesProject(ctx context.Context, projectID uint, siteName string) (*model.PagesProject, *model.PagesDeployment, error) { + if siteName == "" { + siteName = "pages" + } + project, err := model.GetPagesProjectByID(ctx, projectID) + if err != nil { + if errorsIsNotFound(err) { + return nil, nil, fmt.Errorf("路由 %s Pages 配置无效: pages 项目不存在", siteName) + } + return nil, nil, err + } + if !project.Enabled { + return nil, nil, fmt.Errorf("路由 %s Pages 配置无效: pages 项目未启用", siteName) + } + if project.ActiveDeploymentID == nil || *project.ActiveDeploymentID == 0 { + return nil, nil, fmt.Errorf("路由 %s Pages 配置无效: pages 项目没有激活部署", siteName) + } + activeDeployment, err := model.GetPagesDeploymentByID(ctx, *project.ActiveDeploymentID) + if err != nil { + if errorsIsNotFound(err) { + return nil, nil, fmt.Errorf("路由 %s Pages 配置无效: pages 激活部署不存在", siteName) + } + return nil, nil, err + } + if activeDeployment.ProjectID != project.ID { + return nil, nil, fmt.Errorf("路由 %s Pages 配置无效: pages 激活部署不匹配", siteName) + } + if strings.TrimSpace(activeDeployment.Checksum) == "" { + return nil, nil, fmt.Errorf("路由 %s Pages 配置无效: pages 部署校验和缺失", siteName) + } + return project, activeDeployment, nil +} + +func buildLivePagesDeployment(project *model.PagesProject, active *model.PagesDeployment) *openrestyrender.PagesDeployment { + entryFile := strings.TrimSpace(project.EntryFile) + if entryFile == "" { + entryFile = defaultPagesEntryFile + } + fallbackPath := strings.TrimSpace(project.SPAFallbackPath) + if fallbackPath == "" { + fallbackPath = defaultPagesFallbackPath + } + return &openrestyrender.PagesDeployment{ + ProjectID: project.ID, + ProjectSlug: strings.TrimSpace(project.Slug), + DeploymentID: active.ID, + DeploymentNumber: active.DeploymentNumber, + Checksum: strings.TrimSpace(active.Checksum), + EntryFile: entryFile, + SPAFallbackEnabled: project.SPAFallbackEnabled, + SPAFallbackPath: fallbackPath, + APIProxyEnabled: project.APIProxyEnabled, + APIProxyPath: strings.TrimSpace(project.APIProxyPath), + APIProxyPass: strings.TrimSpace(project.APIProxyPass), + APIProxyRewrite: strings.TrimSpace(project.APIProxyRewrite), + LocalRoot: fmt.Sprintf( + "%s/deployments/%d/current", + openrestyrender.PagesDirPlaceholder, + active.ID, + ), + } +} + +func rawJSONString(raw json.RawMessage) (string, bool) { + if !isPresentJSON(raw) { + return "", false + } + var value string + if err := json.Unmarshal(raw, &value); err != nil { + return "", false + } + return value, true +} + +func putJSON(route map[string]json.RawMessage, key string, value any) error { + encoded, err := json.Marshal(value) + if err != nil { + return err + } + route[key] = encoded + return nil +} + +func errorsIsNotFound(err error) bool { + return err != nil && (err == gorm.ErrRecordNotFound || strings.Contains(strings.ToLower(err.Error()), "record not found")) +} diff --git a/internal/apps/openflare/pages/rebind_test.go b/internal/apps/openflare/pages/rebind_test.go new file mode 100644 index 00000000..30b3f090 --- /dev/null +++ b/internal/apps/openflare/pages/rebind_test.go @@ -0,0 +1,87 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pages + +import ( + "context" + "encoding/json" + "testing" + + "github.com/Rain-kl/Wavelet/internal/db" + "github.com/Rain-kl/Wavelet/internal/model" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestRebindSnapshotPagesToCurrentActive(t *testing.T) { + cleanup := setupPagesTestDB(t) + defer cleanup() + ctx := context.Background() + + project, err := CreateProject(ctx, Input{ + Name: "Rebind Site", + Slug: "rebind-site", + Enabled: true, + }) + require.NoError(t, err) + + old := &model.PagesDeployment{ + ProjectID: project.ID, + DeploymentNumber: 1, + Checksum: "old-checksum", + Status: model.PagesDeploymentStatusUploaded, + FileCount: 1, + } + require.NoError(t, db.DB(ctx).Create(old).Error) + active := &model.PagesDeployment{ + ProjectID: project.ID, + DeploymentNumber: 2, + Checksum: "new-checksum", + Status: model.PagesDeploymentStatusActive, + FileCount: 1, + } + require.NoError(t, db.DB(ctx).Create(active).Error) + require.NoError(t, db.DB(ctx).Model(&model.PagesProject{}). + Where("id = ?", project.ID). + Update("active_deployment_id", active.ID).Error) + + // Frozen snapshot still points at the old deployment (simulates old main config). + frozen := map[string]any{ + "routes": []map[string]any{ + { + "site_name": "rebind", + "origin_url": "openflare-pages://project/1", + "enabled": true, + "upstream_type": "pages", + "pages_project_id": project.ID, + "pages_deployment": map[string]any{ + "project_id": project.ID, + "deployment_id": old.ID, + "checksum": "old-checksum", + "local_root": "__OPENFLARE_PAGES_DIR__/deployments/1/current", + }, + "extra_keep_me": "yes", + }, + }, + "waf": map[string]any{"rule_groups": []any{}}, + } + frozenJSON, err := json.Marshal(frozen) + require.NoError(t, err) + + reboundJSON, err := RebindSnapshotPagesToCurrentActive(ctx, string(frozenJSON)) + require.NoError(t, err) + + var rebound map[string]any + require.NoError(t, json.Unmarshal([]byte(reboundJSON), &rebound)) + _, hasWAF := rebound["waf"] + assert.True(t, hasWAF) + + routes := rebound["routes"].([]any) + require.Len(t, routes, 1) + route := routes[0].(map[string]any) + assert.Equal(t, "yes", route["extra_keep_me"]) + deployment := route["pages_deployment"].(map[string]any) + assert.EqualValues(t, active.ID, deployment["deployment_id"]) + assert.Equal(t, "new-checksum", deployment["checksum"]) +} diff --git a/internal/db/migrator/goose/postgres/202607170001_add_pages_system_configs.sql b/internal/db/migrator/goose/postgres/202607170001_add_pages_system_configs.sql new file mode 100644 index 00000000..fea0b27d --- /dev/null +++ b/internal/db/migrator/goose/postgres/202607170001_add_pages_system_configs.sql @@ -0,0 +1,9 @@ +-- +goose Up +INSERT INTO w_system_configs (key, value, type, visibility, description, created_at, updated_at) +VALUES + ('pages_max_package_size_mb', '100', 'business', 0, 'Pages 部署包上传大小上限(MiB)', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP), + ('pages_max_history_count', '20', 'business', 0, 'Pages 每个项目最大历史部署保留数(0 表示不限制)', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) +ON CONFLICT (key) DO NOTHING; + +-- +goose Down +DELETE FROM w_system_configs WHERE key IN ('pages_max_package_size_mb', 'pages_max_history_count'); diff --git a/internal/db/migrator/goose/sqlite/202607170001_add_pages_system_configs.sql b/internal/db/migrator/goose/sqlite/202607170001_add_pages_system_configs.sql new file mode 100644 index 00000000..fea0b27d --- /dev/null +++ b/internal/db/migrator/goose/sqlite/202607170001_add_pages_system_configs.sql @@ -0,0 +1,9 @@ +-- +goose Up +INSERT INTO w_system_configs (key, value, type, visibility, description, created_at, updated_at) +VALUES + ('pages_max_package_size_mb', '100', 'business', 0, 'Pages 部署包上传大小上限(MiB)', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP), + ('pages_max_history_count', '20', 'business', 0, 'Pages 每个项目最大历史部署保留数(0 表示不限制)', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) +ON CONFLICT (key) DO NOTHING; + +-- +goose Down +DELETE FROM w_system_configs WHERE key IN ('pages_max_package_size_mb', 'pages_max_history_count'); diff --git a/internal/model/system_configs.go b/internal/model/system_configs.go index 6b42add0..9b7f1063 100644 --- a/internal/model/system_configs.go +++ b/internal/model/system_configs.go @@ -50,6 +50,10 @@ const ( ConfigKeyDatabaseAutoCleanupEnabled = "database_auto_cleanup_enabled" // 数据库自动清理开关 ConfigKeyDatabaseAutoCleanupRetentionDays = "database_auto_cleanup_retention_days" // 数据库保留天数 + // Pages 静态托管配置 + ConfigKeyPagesMaxPackageSizeMB = "pages_max_package_size_mb" // Pages 部署包上传大小上限(MiB) + ConfigKeyPagesMaxHistoryCount = "pages_max_history_count" // Pages 每个项目最大历史部署保留数(0 表示不限制) + // UptimeKuma 集成配置 ConfigKeyUptimeKumaEnabled = "uptime_kuma_enabled" // UptimeKuma 集成开关 ConfigKeyUptimeKumaURL = "uptime_kuma_url" // UptimeKuma URL diff --git a/internal/testhelper/test_helper.go b/internal/testhelper/test_helper.go index 6c0d4327..391234a4 100644 --- a/internal/testhelper/test_helper.go +++ b/internal/testhelper/test_helper.go @@ -295,6 +295,18 @@ func getSeedConfigsPart2() []model.SystemConfig { Type: configTypeBusiness, Description: "FRPS 内置 Web 界面端口", }, + { + Key: model.ConfigKeyPagesMaxPackageSizeMB, + Value: "100", + Type: configTypeBusiness, + Description: "Pages 部署包上传大小上限(MiB)", + }, + { + Key: model.ConfigKeyPagesMaxHistoryCount, + Value: "20", + Type: configTypeBusiness, + Description: "Pages 每个项目最大历史部署保留数(0 表示不限制)", + }, } } diff --git a/pkg/pagesarchive/entry.go b/pkg/pagesarchive/entry.go new file mode 100644 index 00000000..51e31a58 --- /dev/null +++ b/pkg/pagesarchive/entry.go @@ -0,0 +1,101 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "math" + "os" + "path/filepath" +) + +// Limits bounds archive inspection / extraction work. +type Limits struct { + // MaxFiles is the maximum number of regular files allowed. + MaxFiles int + // MaxFileBytes is the maximum size of a single extracted file. + MaxFileBytes int64 + // MaxTotalBytes is the maximum sum of all extracted file sizes. + MaxTotalBytes int64 +} + +// FileEntry is a regular file discovered inside a deployment package. +type FileEntry struct { + Path string + Size int64 + Checksum string +} + +// Manifest is the inspected content of a Pages deployment package. +type Manifest struct { + Files []FileEntry + FileCount int + TotalSize int64 +} + +// Entry describes one archive member for extraction. +type Entry struct { + // Name is the original path inside the archive. + Name string + // IsDir marks directory entries. + IsDir bool + // IsSymlink marks symbolic links (unsupported for Pages). + IsSymlink bool + // Size is the declared uncompressed size when known; 0 means unknown. + Size uint64 + // Open returns a reader for the entry body. Caller must Close it. + Open func() (io.ReadCloser, error) +} + +// copyLimited copies src to dst. +// When maxBytes <= 0, size limits are not enforced (trusted extract path). +func copyLimited(dst io.Writer, src io.Reader, declaredSize uint64, maxBytes int64) (int64, error) { + if maxBytes <= 0 { + if declaredSize > 0 { + if declaredSize > uint64(math.MaxInt64) { + return 0, fmt.Errorf("pages file size out of bounds") + } + //nolint:gosec // declaredSize is bounded to MaxInt64 above + return io.CopyN(dst, src, int64(declaredSize)) + } + return io.Copy(dst, src) + } + if declaredSize > uint64(maxBytes) || declaredSize > uint64(math.MaxInt64) { //nolint:gosec // maxBytes positive + return 0, fmt.Errorf("pages file size out of bounds") + } + if declaredSize > 0 { + //nolint:gosec // declaredSize is bounded to MaxInt64 above + return io.CopyN(dst, src, int64(declaredSize)) + } + limited := io.LimitReader(src, maxBytes+1) + written, err := io.Copy(dst, limited) + if written > maxBytes { + return written, fmt.Errorf("pages file size out of bounds") + } + return written, err +} + +func checksumReader(src io.Reader, declaredSize uint64, maxBytes int64) (string, int64, error) { + hash := sha256.New() + written, err := copyLimited(hash, src, declaredSize, maxBytes) + if err != nil { + return "", written, err + } + return hex.EncodeToString(hash.Sum(nil)), written, nil +} + +func writeEntryFile(targetPath string, src io.Reader, declaredSize uint64, maxBytes int64, perm os.FileMode) (int64, error) { + if err := os.MkdirAll(filepath.Dir(targetPath), dirPerm); err != nil { + return 0, err + } + target, err := os.OpenFile(targetPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, perm) //nolint:gosec // caller validates path under release dir + if err != nil { + return 0, err + } + defer func() { _ = target.Close() }() + return copyLimited(target, src, declaredSize, maxBytes) +} diff --git a/pkg/pagesarchive/extract.go b/pkg/pagesarchive/extract.go new file mode 100644 index 00000000..c5caddf4 --- /dev/null +++ b/pkg/pagesarchive/extract.go @@ -0,0 +1,154 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "fmt" + "os" + "path/filepath" +) + +// ExtractOptions controls package extraction. +type ExtractOptions struct { + // Limits bounds files and sizes during extraction when EnforceLimits is true. + Limits Limits + // StripCommonRoot strips a single shared top-level directory when present. + StripCommonRoot bool + // EnforceLimits enables MaxFiles / MaxFileBytes / MaxTotalBytes checks. + // When false, the caller is assumed to have already validated the package + // (e.g. Agent trusts control-plane inspection). Path-escape and symlink + // guards still apply so local extraction cannot leave destDir. + EnforceLimits bool +} + +// ExtractBytes extracts a deployment package into destDir. +func ExtractBytes(data []byte, format Format, destDir string, opts ExtractOptions) error { + if format == "" { + var err error + format, err = DetectFormat("", data) + if err != nil { + return err + } + } + entries, err := listEntries(data, format) + if err != nil { + return err + } + return extractEntries(entries, destDir, opts) +} + +// ExtractFile reads path and extracts it into destDir. +func ExtractFile(filePath string, format Format, destDir string, opts ExtractOptions) error { + data, err := os.ReadFile(filePath) //nolint:gosec // controlled path + if err != nil { + return err + } + return ExtractBytes(data, format, destDir, opts) +} + +func extractEntries(entries []Entry, destDir string, opts ExtractOptions) error { + limits := Limits{} + if opts.EnforceLimits { + limits = normalizeLimits(opts.Limits) + } + commonPrefix := "" + if opts.StripCommonRoot { + commonPrefix = FindCommonRootPrefix(collectFileNames(entries)) + } + + var totalSize int64 + var fileCount int + for _, entry := range entries { + written, counted, err := extractSingleEntry(entry, destDir, commonPrefix, limits, opts.EnforceLimits) + if err != nil { + return err + } + if !counted { + continue + } + fileCount++ + if opts.EnforceLimits && fileCount > limits.MaxFiles { + return fmt.Errorf("pages deployment file count exceeds %d", limits.MaxFiles) + } + totalSize += written + if opts.EnforceLimits && totalSize > limits.MaxTotalBytes { + return fmt.Errorf("pages extracted size exceeds limit") + } + } + if fileCount == 0 { + return fmt.Errorf("pages package is empty") + } + return nil +} + +func extractSingleEntry( + entry Entry, + destDir, commonPrefix string, + limits Limits, + enforceLimits bool, +) (written int64, counted bool, err error) { + relativePath, skip, err := NormalizeEntryPath(entry.Name) + if err != nil { + return 0, false, err + } + if skip { + return 0, false, nil + } + if commonPrefix != "" { + relativePath = StripPrefix(relativePath, commonPrefix) + if relativePath == "" { + return 0, false, nil + } + } + if entry.IsSymlink { + return 0, false, fmt.Errorf("pages package contains unsupported symlink: %s", relativePath) + } + + targetPath := filepath.Join(destDir, filepath.FromSlash(relativePath)) + if !isWithinDir(destDir, targetPath) { + return 0, false, fmt.Errorf("pages package path escapes directory: %s", entry.Name) + } + + if entry.IsDir { + if err := os.MkdirAll(targetPath, dirPerm); err != nil { + return 0, false, err + } + return 0, false, nil + } + + maxFileBytes := int64(0) // unlimited when not enforcing + if enforceLimits { + if exceedsFileByteLimit(entry.Size, limits.MaxFileBytes) { + return 0, false, fmt.Errorf("pages file too large: %s", relativePath) + } + maxFileBytes = limits.MaxFileBytes + } + src, err := entry.Open() + if err != nil { + return 0, false, fmt.Errorf("%s: %w", relativePath, err) + } + written, writeErr := writeEntryFile(targetPath, src, entry.Size, maxFileBytes, filePerm) + _ = src.Close() + if writeErr != nil { + return 0, false, fmt.Errorf("%s: %w", relativePath, writeErr) + } + return written, true, nil +} + +func isWithinDir(baseDir, targetPath string) bool { + cleanBase := filepath.Clean(baseDir) + cleanTarget := filepath.Clean(targetPath) + rel, err := filepath.Rel(cleanBase, cleanTarget) + if err != nil { + return false + } + return rel != ".." && !hasParentRel(rel) +} + +func hasParentRel(rel string) bool { + if rel == ".." { + return true + } + return len(rel) >= 3 && (rel[:3] == "../" || rel[:3] == "..\\") +} diff --git a/pkg/pagesarchive/format.go b/pkg/pagesarchive/format.go new file mode 100644 index 00000000..99fde833 --- /dev/null +++ b/pkg/pagesarchive/format.go @@ -0,0 +1,191 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +// Package pagesarchive provides multi-format archive detection, inspection and +// extraction helpers for OpenFlare Pages deployment packages. +package pagesarchive + +import ( + "bytes" + "fmt" + "path/filepath" + "strings" +) + +// Format identifies a supported Pages deployment package archive format. +type Format string + +const ( + // FormatZip is a ZIP archive. + FormatZip Format = "zip" + // FormatTar is an uncompressed tar archive. + FormatTar Format = "tar" + // FormatTarGz is a gzip-compressed tar archive. + FormatTarGz Format = "tar.gz" + // FormatTarXz is an xz-compressed tar archive. + FormatTarXz Format = "tar.xz" + // FormatTarBz2 is a bzip2-compressed tar archive. + FormatTarBz2 Format = "tar.bz2" + // FormatSevenZip is a 7z archive. + FormatSevenZip Format = "7z" + + ustarMagicOffset = 257 + ustarMagicMinLen = 262 +) + +// DetectFormatFromName returns the archive format inferred from a file name. +// Returns an empty Format and false when the extension is unsupported. +func DetectFormatFromName(fileName string) (Format, bool) { + name := strings.ToLower(strings.TrimSpace(fileName)) + switch { + case strings.HasSuffix(name, ".tar.gz"), strings.HasSuffix(name, ".tgz"): + return FormatTarGz, true + case strings.HasSuffix(name, ".tar.xz"), strings.HasSuffix(name, ".txz"): + return FormatTarXz, true + case strings.HasSuffix(name, ".tar.bz2"), strings.HasSuffix(name, ".tbz2"), strings.HasSuffix(name, ".tbz"): + return FormatTarBz2, true + case strings.HasSuffix(name, ".tar"): + return FormatTar, true + case strings.HasSuffix(name, ".7z"): + return FormatSevenZip, true + case strings.HasSuffix(name, ".zip"): + return FormatZip, true + default: + return "", false + } +} + +// DetectFormatFromBytes returns the archive format inferred from magic bytes. +// Prefer DetectFormatFromName when a reliable file name is available. +func DetectFormatFromBytes(data []byte) (Format, bool) { + if isZipMagic(data) { + return FormatZip, true + } + if isSevenZipMagic(data) { + return FormatSevenZip, true + } + if isXZMagic(data) { + return FormatTarXz, true + } + if isGzipMagic(data) { + return FormatTarGz, true + } + if isBzip2Magic(data) { + return FormatTarBz2, true + } + if looksLikeTar(data) { + return FormatTar, true + } + return "", false +} + +// DetectFormat prefers the file name when present, otherwise magic bytes. +func DetectFormat(fileName string, data []byte) (Format, error) { + if format, ok := DetectFormatFromName(fileName); ok { + return format, nil + } + if format, ok := DetectFormatFromBytes(data); ok { + return format, nil + } + return "", fmt.Errorf("unsupported pages package format") +} + +// Extension returns the canonical file extension for a format (without leading dot). +func Extension(format Format) string { + switch format { + case FormatZip: + return "zip" + case FormatTar: + return "tar" + case FormatTarGz: + return "tar.gz" + case FormatTarXz: + return "tar.xz" + case FormatTarBz2: + return "tar.bz2" + case FormatSevenZip: + return "7z" + default: + return "bin" + } +} + +// MIMEType returns a reasonable content type for the archive format. +func MIMEType(format Format) string { + switch format { + case FormatZip: + return "application/zip" + case FormatTar: + return "application/x-tar" + case FormatTarGz: + return "application/gzip" + case FormatTarXz: + return "application/x-xz" + case FormatTarBz2: + return "application/x-bzip2" + case FormatSevenZip: + return "application/x-7z-compressed" + default: + return "application/octet-stream" + } +} + +// SupportedExtensions lists human-readable extensions for UI copy and accept attributes. +func SupportedExtensions() []string { + return []string{".zip", ".tar.gz", ".tgz", ".tar.xz", ".txz", ".tar.bz2", ".tbz2", ".tar", ".7z"} +} + +// AcceptAttribute returns a comma-separated accept list for file inputs. +func AcceptAttribute() string { + return strings.Join(SupportedExtensions(), ",") +} + +// NormalizeNameExtension returns a storage-safe extension for the given format/name. +func NormalizeNameExtension(fileName string, format Format) string { + if format != "" { + return Extension(format) + } + if formatFromName, ok := DetectFormatFromName(fileName); ok { + return Extension(formatFromName) + } + ext := strings.TrimPrefix(filepath.Ext(fileName), ".") + if ext == "" { + return "bin" + } + return strings.ToLower(ext) +} + +func isZipMagic(data []byte) bool { + return len(data) >= 4 && + data[0] == 0x50 && data[1] == 0x4b && + (data[2] == 0x03 || data[2] == 0x05 || data[2] == 0x07) && + (data[3] == 0x04 || data[3] == 0x06 || data[3] == 0x08) +} + +func isSevenZipMagic(data []byte) bool { + return len(data) >= 6 && + data[0] == 0x37 && data[1] == 0x7a && data[2] == 0xbc && + data[3] == 0xaf && data[4] == 0x27 && data[5] == 0x1c +} + +func isXZMagic(data []byte) bool { + return len(data) >= 6 && + data[0] == 0xfd && data[1] == 0x37 && data[2] == 0x7a && + data[3] == 0x58 && data[4] == 0x5a && data[5] == 0x00 +} + +func isGzipMagic(data []byte) bool { + return len(data) >= 2 && data[0] == 0x1f && data[1] == 0x8b +} + +func isBzip2Magic(data []byte) bool { + return len(data) >= 3 && data[0] == 0x42 && data[1] == 0x5a && data[2] == 0x68 +} + +func looksLikeTar(data []byte) bool { + // POSIX ustar magic at offset 257 ("ustar\0" or "ustar "). + if len(data) < ustarMagicMinLen { + return false + } + return bytes.Equal(data[ustarMagicOffset:ustarMagicOffset+5], []byte("ustar")) +} diff --git a/pkg/pagesarchive/inspect.go b/pkg/pagesarchive/inspect.go new file mode 100644 index 00000000..c91d6370 --- /dev/null +++ b/pkg/pagesarchive/inspect.go @@ -0,0 +1,148 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "fmt" + "os" + "path" + "strings" +) + +// InspectOptions controls package inspection. +type InspectOptions struct { + // RootDir is an optional project root subdirectory that must contain EntryFile. + RootDir string + // EntryFile is the required entry file name (e.g. index.html). + EntryFile string + // Limits bounds files and sizes. + Limits Limits +} + +// InspectFile opens path and inspects it as a Pages deployment package. +func InspectFile(filePath string, format Format, opts InspectOptions) (*Manifest, error) { + data, err := os.ReadFile(filePath) //nolint:gosec // filePath is a controlled temp upload path + if err != nil { + return nil, err + } + return InspectBytes(data, format, opts) +} + +// InspectBytes inspects an in-memory deployment package. +func InspectBytes(data []byte, format Format, opts InspectOptions) (*Manifest, error) { + if format == "" { + var err error + format, err = DetectFormat("", data) + if err != nil { + return nil, err + } + } + entries, err := listEntries(data, format) + if err != nil { + return nil, err + } + return buildManifest(entries, opts) +} + +func buildManifest(entries []Entry, opts InspectOptions) (*Manifest, error) { + limits := normalizeLimits(opts.Limits) + commonPrefix := FindCommonRootPrefix(collectFileNames(entries)) + targetEntryPath := resolveTargetEntryPath(opts.RootDir, opts.EntryFile) + + manifest := &Manifest{Files: make([]FileEntry, 0)} + entrySeen := false + + for _, entry := range entries { + normalizedPath, skip, err := prepareEntryPath(entry, commonPrefix) + if err != nil { + return nil, err + } + if skip { + continue + } + if exceedsFileByteLimit(entry.Size, limits.MaxFileBytes) { + return nil, fmt.Errorf("pages file too large: %s", normalizedPath) + } + + fileEntry, err := inspectRegularFile(entry, normalizedPath, limits) + if err != nil { + return nil, err + } + manifest.FileCount++ + if manifest.FileCount > limits.MaxFiles { + return nil, fmt.Errorf("pages deployment file count exceeds %d", limits.MaxFiles) + } + manifest.TotalSize += fileEntry.Size + if manifest.TotalSize > limits.MaxTotalBytes { + return nil, fmt.Errorf("pages extracted size exceeds limit") + } + if normalizedPath == targetEntryPath { + entrySeen = true + } + manifest.Files = append(manifest.Files, fileEntry) + } + + if manifest.FileCount == 0 { + return nil, fmt.Errorf("pages package is empty") + } + if !entrySeen { + return nil, fmt.Errorf("pages package is missing entry file %s", targetEntryPath) + } + return manifest, nil +} + +func collectFileNames(entries []Entry) []string { + names := make([]string, 0, len(entries)) + for _, entry := range entries { + if entry.IsDir || entry.IsSymlink { + continue + } + names = append(names, entry.Name) + } + return names +} + +func resolveTargetEntryPath(rootDir, entryFile string) string { + normalizedEntry := strings.TrimSpace(entryFile) + if normalizedEntry == "" { + normalizedEntry = "index.html" + } + normalizedRoot := strings.Trim(strings.TrimSpace(rootDir), "/") + if normalizedRoot == "" { + return normalizedEntry + } + return path.Join(normalizedRoot, normalizedEntry) +} + +func prepareEntryPath(entry Entry, commonPrefix string) (string, bool, error) { + normalizedPath, skip, err := NormalizeEntryPath(entry.Name) + if err != nil { + return "", false, err + } + if skip || entry.IsDir { + return "", true, nil + } + normalizedPath = StripPrefix(normalizedPath, commonPrefix) + if entry.IsSymlink { + return "", false, fmt.Errorf("pages package contains unsupported symlink: %s", normalizedPath) + } + return normalizedPath, false, nil +} + +func inspectRegularFile(entry Entry, normalizedPath string, limits Limits) (FileEntry, error) { + src, err := entry.Open() + if err != nil { + return FileEntry{}, fmt.Errorf("%s: %w", normalizedPath, err) + } + checksum, fileSize, checksumErr := checksumReader(src, entry.Size, limits.MaxFileBytes) + _ = src.Close() + if checksumErr != nil { + return FileEntry{}, fmt.Errorf("%s: %w", normalizedPath, checksumErr) + } + return FileEntry{ + Path: normalizedPath, + Size: fileSize, + Checksum: checksum, + }, nil +} diff --git a/pkg/pagesarchive/limits.go b/pkg/pagesarchive/limits.go new file mode 100644 index 00000000..a96dcaf4 --- /dev/null +++ b/pkg/pagesarchive/limits.go @@ -0,0 +1,38 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +const ( + defaultMaxFiles = 1000 + defaultMaxFileBytes = 100 * 1024 * 1024 + defaultMaxTotalBytes = 100 * 1024 * 1024 + dirPerm = 0o750 + filePerm = 0o644 +) + +// normalizeLimits applies defaults for control-plane inspection. +// Callers that already validated the package should use EnforceLimits=false instead. +func normalizeLimits(limits Limits) Limits { + if limits.MaxFiles <= 0 { + limits.MaxFiles = defaultMaxFiles + } + if limits.MaxFileBytes <= 0 { + limits.MaxFileBytes = defaultMaxFileBytes + } + if limits.MaxTotalBytes <= 0 { + limits.MaxTotalBytes = defaultMaxTotalBytes + } + return limits +} + +func exceedsFileByteLimit(size uint64, maxBytes int64) bool { + // maxBytes <= 0 means unlimited (trusted extract path). + if maxBytes <= 0 { + return false + } + if size == 0 { + return false + } + return size > uint64(maxBytes) //nolint:gosec // maxBytes is positive +} diff --git a/pkg/pagesarchive/list.go b/pkg/pagesarchive/list.go new file mode 100644 index 00000000..da6c2982 --- /dev/null +++ b/pkg/pagesarchive/list.go @@ -0,0 +1,220 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "archive/tar" + "archive/zip" + "bytes" + "compress/bzip2" + "compress/gzip" + "fmt" + "io" + "os" + + "github.com/bodgit/sevenzip" + "github.com/ulikunitz/xz" +) + +type archiveFile interface { + Name() string + IsDir() bool + IsSymlink() bool + Size() uint64 + Open() (io.ReadCloser, error) +} + +type zipArchiveFile struct { + file *zip.File +} + +func (z zipArchiveFile) Name() string { return z.file.Name } +func (z zipArchiveFile) IsDir() bool { return z.file.FileInfo().IsDir() } +func (z zipArchiveFile) IsSymlink() bool { + return z.file.Mode()&os.ModeSymlink != 0 +} +func (z zipArchiveFile) Size() uint64 { return z.file.UncompressedSize64 } +func (z zipArchiveFile) Open() (io.ReadCloser, error) { + return z.file.Open() +} + +type sevenZipArchiveFile struct { + file *sevenzip.File +} + +func (z sevenZipArchiveFile) Name() string { return z.file.Name } +func (z sevenZipArchiveFile) IsDir() bool { return z.file.FileInfo().IsDir() } +func (z sevenZipArchiveFile) IsSymlink() bool { + return z.file.Mode()&os.ModeSymlink != 0 +} +func (z sevenZipArchiveFile) Size() uint64 { return z.file.UncompressedSize } +func (z sevenZipArchiveFile) Open() (io.ReadCloser, error) { + return z.file.Open() +} + +func listEntries(data []byte, format Format) ([]Entry, error) { + switch format { + case FormatZip: + return listZipEntries(data) + case FormatTar: + return listTarEntries(bytes.NewReader(data)) + case FormatTarGz: + gzReader, err := gzip.NewReader(bytes.NewReader(data)) + if err != nil { + return nil, fmt.Errorf("open gzip pages package: %w", err) + } + defer func() { _ = gzReader.Close() }() + return listTarEntries(gzReader) + case FormatTarXz: + xzReader, err := xz.NewReader(bytes.NewReader(data)) + if err != nil { + return nil, fmt.Errorf("open xz pages package: %w", err) + } + return listTarEntries(xzReader) + case FormatTarBz2: + return listTarEntries(bzip2.NewReader(bytes.NewReader(data))) + case FormatSevenZip: + return listSevenZipEntries(data) + default: + return nil, fmt.Errorf("unsupported pages package format: %s", format) + } +} + +func entriesFromArchiveFiles(files []archiveFile) []Entry { + entries := make([]Entry, 0, len(files)) + for _, item := range files { + file := item + entries = append(entries, Entry{ + Name: file.Name(), + IsDir: file.IsDir(), + IsSymlink: file.IsSymlink(), + Size: file.Size(), + Open: file.Open, + }) + } + return entries +} + +func listZipEntries(data []byte) ([]Entry, error) { + reader, err := zip.NewReader(bytes.NewReader(data), int64(len(data))) + if err != nil { + return nil, fmt.Errorf("open zip pages package: %w", err) + } + files := make([]archiveFile, 0, len(reader.File)) + for _, item := range reader.File { + files = append(files, zipArchiveFile{file: item}) + } + return entriesFromArchiveFiles(files), nil +} + +func listSevenZipEntries(data []byte) ([]Entry, error) { + reader, err := sevenzip.NewReader(bytes.NewReader(data), int64(len(data))) + if err != nil { + return nil, fmt.Errorf("open 7z pages package: %w", err) + } + files := make([]archiveFile, 0, len(reader.File)) + for _, item := range reader.File { + files = append(files, sevenZipArchiveFile{file: item}) + } + return entriesFromArchiveFiles(files), nil +} + +func listTarEntries(r io.Reader) ([]Entry, error) { + tarReader := tar.NewReader(r) + // Tar is sequential: materialize regular file bodies so entries can be opened later. + type materialised struct { + header *tar.Header + body []byte + } + items := make([]materialised, 0) + for { + header, err := tarReader.Next() + if err == io.EOF { + break + } + if err != nil { + return nil, fmt.Errorf("read tar pages package: %w", err) + } + item, skip, err := materialiseTarHeader(tarReader, header) + if err != nil { + return nil, err + } + if skip { + continue + } + items = append(items, item) + } + + entries := make([]Entry, 0, len(items)) + for _, item := range items { + entries = append(entries, tarEntryFromMaterialised(item.header, item.body)) + } + return entries, nil +} + +func materialiseTarHeader(tarReader *tar.Reader, header *tar.Header) (item struct { + header *tar.Header + body []byte +}, skip bool, err error) { + switch header.Typeflag { + case tar.TypeDir, tar.TypeSymlink, tar.TypeLink: + return struct { + header *tar.Header + body []byte + }{header: header}, false, nil + case tar.TypeReg, tar.TypeRegA: //nolint:staticcheck // TypeRegA still appears in older archives + body, readErr := readTarBody(tarReader, header) + if readErr != nil { + return item, false, readErr + } + return struct { + header *tar.Header + body []byte + }{header: header, body: body}, false, nil + default: + if header.Size > 0 { + if _, copyErr := io.CopyN(io.Discard, tarReader, header.Size); copyErr != nil { + return item, false, fmt.Errorf("skip tar entry %s: %w", header.Name, copyErr) + } + } + return item, true, nil + } +} + +func readTarBody(tarReader *tar.Reader, header *tar.Header) ([]byte, error) { + if header.Size > 0 { + body := make([]byte, header.Size) + if _, err := io.ReadFull(tarReader, body); err != nil { + return nil, fmt.Errorf("read tar entry %s: %w", header.Name, err) + } + return body, nil + } + body, err := io.ReadAll(tarReader) + if err != nil { + return nil, fmt.Errorf("read tar entry %s: %w", header.Name, err) + } + return body, nil +} + +func tarEntryFromMaterialised(header *tar.Header, body []byte) Entry { + size := header.Size + if int64(len(body)) > size { + size = int64(len(body)) + } + entry := Entry{ + Name: header.Name, + IsDir: header.Typeflag == tar.TypeDir, + IsSymlink: header.Typeflag == tar.TypeSymlink || header.Typeflag == tar.TypeLink, + Size: uint64(size), //nolint:gosec // non-negative sizes + Open: func() (io.ReadCloser, error) { + return io.NopCloser(bytes.NewReader(body)), nil + }, + } + if entry.IsDir || entry.IsSymlink { + entry.Open = func() (io.ReadCloser, error) { + return io.NopCloser(bytes.NewReader(nil)), nil + } + } + return entry +} diff --git a/pkg/pagesarchive/pagesarchive_test.go b/pkg/pagesarchive/pagesarchive_test.go new file mode 100644 index 00000000..8e3c654e --- /dev/null +++ b/pkg/pagesarchive/pagesarchive_test.go @@ -0,0 +1,187 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "archive/tar" + "archive/zip" + "bytes" + "compress/gzip" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/ulikunitz/xz" +) + +func TestDetectFormatFromName(t *testing.T) { + cases := map[string]Format{ + "site.zip": FormatZip, + "site.TAR.GZ": FormatTarGz, + "site.tgz": FormatTarGz, + "site.tar.xz": FormatTarXz, + "site.txz": FormatTarXz, + "site.tar.bz2": FormatTarBz2, + "site.tar": FormatTar, + "site.7z": FormatSevenZip, + } + for name, want := range cases { + got, ok := DetectFormatFromName(name) + assert.True(t, ok, name) + assert.Equal(t, want, got, name) + } + _, ok := DetectFormatFromName("site.rar") + assert.False(t, ok) +} + +func TestInspectAndExtractZip(t *testing.T) { + data := testZip(t, map[string]string{ + "dist/index.html": "ok", + "dist/app.js": "console.log(1)", + }) + manifest, err := InspectBytes(data, FormatZip, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{MaxFiles: 100, MaxFileBytes: 1 << 20, MaxTotalBytes: 1 << 20}, + }) + require.NoError(t, err) + assert.Equal(t, 2, manifest.FileCount) + paths := make(map[string]struct{}, len(manifest.Files)) + for _, file := range manifest.Files { + paths[file.Path] = struct{}{} + } + assert.Contains(t, paths, "index.html") + assert.Contains(t, paths, "app.js") + + dest := t.TempDir() + require.NoError(t, ExtractBytes(data, FormatZip, dest, ExtractOptions{ + StripCommonRoot: true, + EnforceLimits: true, + Limits: Limits{MaxFiles: 100, MaxFileBytes: 1 << 20, MaxTotalBytes: 1 << 20}, + })) + body, err := os.ReadFile(filepath.Join(dest, "index.html")) //nolint:gosec + require.NoError(t, err) + assert.Equal(t, "ok", string(body)) +} + +func TestExtractTrustedSkipsSizeLimits(t *testing.T) { + // Content larger than a tiny limit would fail if limits were enforced. + large := strings.Repeat("x", 64) + data := testZip(t, map[string]string{ + "index.html": large, + }) + dest := t.TempDir() + require.NoError(t, ExtractBytes(data, FormatZip, dest, ExtractOptions{ + // Agent trusts control-plane validation: no size/count re-check. + EnforceLimits: false, + })) + body, err := os.ReadFile(filepath.Join(dest, "index.html")) //nolint:gosec + require.NoError(t, err) + assert.Equal(t, large, string(body)) +} + +func TestInspectAndExtractTarGz(t *testing.T) { + data := testTarGz(t, map[string]string{ + "index.html": "tar", + "style.css": "body{}", + }) + format, err := DetectFormat("site.tar.gz", data) + require.NoError(t, err) + assert.Equal(t, FormatTarGz, format) + + manifest, err := InspectBytes(data, format, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{MaxFiles: 100, MaxFileBytes: 1 << 20, MaxTotalBytes: 1 << 20}, + }) + require.NoError(t, err) + assert.Equal(t, 2, manifest.FileCount) + + dest := t.TempDir() + require.NoError(t, ExtractBytes(data, format, dest, ExtractOptions{ + EnforceLimits: true, + Limits: Limits{MaxFiles: 100, MaxFileBytes: 1 << 20, MaxTotalBytes: 1 << 20}, + })) + body, err := os.ReadFile(filepath.Join(dest, "index.html")) //nolint:gosec + require.NoError(t, err) + assert.Equal(t, "tar", string(body)) +} + +func TestInspectTarXz(t *testing.T) { + data := testTarXz(t, map[string]string{ + "index.html": "xz", + }) + manifest, err := InspectBytes(data, FormatTarXz, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{MaxFiles: 10, MaxFileBytes: 1 << 20, MaxTotalBytes: 1 << 20}, + }) + require.NoError(t, err) + assert.Equal(t, 1, manifest.FileCount) +} + +func TestRejectZipSlip(t *testing.T) { + data := testZip(t, map[string]string{ + "../evil.txt": "x", + "index.html": "ok", + }) + _, err := InspectBytes(data, FormatZip, InspectOptions{ + EntryFile: "index.html", + Limits: Limits{MaxFiles: 10, MaxFileBytes: 1 << 20, MaxTotalBytes: 1 << 20}, + }) + require.Error(t, err) +} + +func testZip(t *testing.T, files map[string]string) []byte { + t.Helper() + var buffer bytes.Buffer + writer := zip.NewWriter(&buffer) + for name, content := range files { + file, err := writer.Create(name) + require.NoError(t, err) + _, err = file.Write([]byte(content)) + require.NoError(t, err) + } + require.NoError(t, writer.Close()) + return buffer.Bytes() +} + +func testTarGz(t *testing.T, files map[string]string) []byte { + t.Helper() + var buffer bytes.Buffer + gzWriter := gzip.NewWriter(&buffer) + tarWriter := tar.NewWriter(gzWriter) + for name, content := range files { + require.NoError(t, tarWriter.WriteHeader(&tar.Header{ + Name: name, + Mode: 0o644, + Size: int64(len(content)), + })) + _, err := tarWriter.Write([]byte(content)) + require.NoError(t, err) + } + require.NoError(t, tarWriter.Close()) + require.NoError(t, gzWriter.Close()) + return buffer.Bytes() +} + +func testTarXz(t *testing.T, files map[string]string) []byte { + t.Helper() + var buffer bytes.Buffer + xzWriter, err := xz.NewWriter(&buffer) + require.NoError(t, err) + tarWriter := tar.NewWriter(xzWriter) + for name, content := range files { + require.NoError(t, tarWriter.WriteHeader(&tar.Header{ + Name: name, + Mode: 0o644, + Size: int64(len(content)), + })) + _, err := tarWriter.Write([]byte(content)) + require.NoError(t, err) + } + require.NoError(t, tarWriter.Close()) + require.NoError(t, xzWriter.Close()) + return buffer.Bytes() +} diff --git a/pkg/pagesarchive/path.go b/pkg/pagesarchive/path.go new file mode 100644 index 00000000..4735ff86 --- /dev/null +++ b/pkg/pagesarchive/path.go @@ -0,0 +1,85 @@ +// Copyright 2026 Arctel.net +// SPDX-License-Identifier: Apache-2.0 + +package pagesarchive + +import ( + "fmt" + "path" + "path/filepath" + "strings" +) + +// NormalizeEntryPath cleans an archive entry path and rejects zip-slip / absolute paths. +// skip=true means the entry should be ignored (empty path or directory marker). +func NormalizeEntryPath(raw string) (cleaned string, skip bool, err error) { + name := strings.TrimSpace(filepath.ToSlash(raw)) + if name == "" { + return "", true, nil + } + if strings.HasSuffix(name, "/") { + return "", true, nil + } + if strings.HasPrefix(name, "/") || path.IsAbs(name) { + return "", false, fmt.Errorf("pages package contains absolute path: %s", raw) + } + // Reject Windows drive / UNC-style paths that may appear after ToSlash. + if len(name) >= 2 && name[1] == ':' { + return "", false, fmt.Errorf("pages package contains absolute path: %s", raw) + } + cleanedPath := path.Clean(name) + if cleanedPath == "." { + return "", true, nil + } + if cleanedPath == ".." || strings.HasPrefix(cleanedPath, "../") || strings.Contains(cleanedPath, "/../") { + return "", false, fmt.Errorf("pages package path escapes directory: %s", raw) + } + return cleanedPath, false, nil +} + +// FindCommonRootPrefix returns a trailing-slash directory prefix shared by all file paths. +// When files do not share a single root folder the result is empty. +func FindCommonRootPrefix(paths []string) string { + var firstFilePath string + hasMultipleFiles := false + for _, item := range paths { + normalizedPath, skip, err := NormalizeEntryPath(item) + if err != nil || skip { + continue + } + if firstFilePath == "" { + firstFilePath = normalizedPath + } else { + hasMultipleFiles = true + } + } + if firstFilePath == "" { + return "" + } + parts := strings.Split(firstFilePath, "/") + if len(parts) <= 1 { + return "" + } + commonPrefix := parts[0] + "/" + if !hasMultipleFiles { + return commonPrefix + } + for _, item := range paths { + normalizedPath, skip, err := NormalizeEntryPath(item) + if err != nil || skip { + continue + } + if !strings.HasPrefix(normalizedPath, commonPrefix) { + return "" + } + } + return commonPrefix +} + +// StripPrefix removes a directory prefix from a cleaned path when present. +func StripPrefix(normalizedPath, prefix string) string { + if prefix == "" { + return normalizedPath + } + return strings.TrimPrefix(normalizedPath, prefix) +}