refactor(migrator): use goose SQL only and auto-import zones on upgrade

移除 Go goose 迁移(bridge/legacy data/zone import),改为 goose SQL 占位
与结构迁移脚本;启动时 Migrate 在删旧列前自动导入历史域名,去掉
migrate-zones 手动命令及文档中的人工导入步骤。
This commit is contained in:
ryan
2026-07-12 15:36:00 +08:00
parent 5b52acdd6c
commit 03b81e5f74
19 changed files with 571 additions and 1252 deletions
+1 -1
View File
@@ -26,7 +26,7 @@ sidebar: false
- 新增第一阶段 Zone 与正规化 Zone 域名数据库表及路由绑定模型,为后续以稳定 ID 管理网站与域名关联提供基础。
- 新增 Zone 管理 API 与显式历史域名导入命令,使用公共后缀列表验证注册根域和域名归属。
- 管理端网站入口改为 Zone 列表与 `/websites/:zoneId` 详情(概览 / 域名 / 路由 / 证书 / 设置),反代路由通过 Zone 域名选择器绑定。
- 新增 Zone 域名迁移指南(`docs/guide/zone-domain-migration.md`),覆盖备份、`wavelet migrate-zones`、预览等价性与回滚步骤。
- 新增 Zone 域名迁移指南(`docs/guide/zone-domain-migration.md`);历史域名在 Server 启动时由 goose 自动导入,无需单独命令。
### 修改
+3 -3
View File
@@ -84,15 +84,15 @@ WAF、Pages、上游与发布版本仍属于 `proxy_routes`。Zone 概览只聚
本次改造分两个发布阶段,以免 SQL 用错误的“末两段域名”规则处理多级公共后缀。操作细则见 [Zone 域名迁移与发布验收](../guide/zone-domain-migration.md)。
1. **第一阶段 DDL**:PostgreSQL 与 SQLite 同版本 Goose 创建 `of_zones` / `of_zone_domains`;暂时保留 `of_managed_domains` 与路由冗余列。
2. **数据导入**:`wavelet migrate-zones` 使用 `publicsuffix.EffectiveTLDPlusOne`,从既有路由域名及(无路由域名时的)`managed_domains` 创建 Zone / Zone 域名,按旧 `domain_cert_ids` 位置写入 `zone_domains.cert_id`。冲突时整单回滚并输出报告,禁止在有冲突时发布。
2. **数据导入(自动)**:Server 启动时 `migrator.Migrate()` 先应用 goose SQL 至 `202607120002`,再自动导入旧路由域名 / `managed_domains`(`publicsuffix` 解析注册根域,写入 `cert_id` 与 `proxy_route_id`),最后继续后续 SQL。冲突时启动失败;修复后重启可幂等重试。无需手动命令。
3. **代码切换**:控制面 API、配置快照、渲染、前端均以 Zone 域名为唯一来源;路由写入仅使用 `zone_domain_ids`。
4. **第二阶段清理**:独立 Goose 迁移 `202607130001_drop_legacy_route_domain_columns` 删除 `of_managed_domains` 与 `of_proxy_routes` 的 `domain` / `domains` / `cert_id` / `cert_ids` / `domain_cert_ids`。Down 仅恢复开发库空结构,不回填历史数据。
4. **第二阶段清理**:Goose SQL `202607130001_drop_legacy_route_domain_columns` 删除 `of_managed_domains` 与 `of_proxy_routes` 冗余列。Down 仅恢复开发库空结构,不回填历史数据。
### 运行时模型边界
* 持久化:域名与证书只存在于 `of_zone_domains`;`of_proxy_routes` 仅保存路由策略(上游、缓存、限流、WAF 绑定键等)。
* 渲染:配置快照在内存中组装临时 `Domains` / `DomainCertIDs` 供 OpenResty 渲染,不写回数据库。
* 导入命令:第二阶段后若旧列/旧表已不存在则跳过对应源,对已导入 Zone 域名保持幂等。
* 结构迁移仅使用 `internal/db/migrator/goose/{postgres,sqlite}/*.sql`;启动时自动导入历史域名,第二阶段后旧列不存在则为空操作。
## 验证
+1 -1
View File
@@ -11,7 +11,7 @@ OpenFlare 是一套自托管的 OpenResty 控制面。它把反向代理网站
1. [快速开始](./quick-start.md):用 Docker Compose 启动 Server,登录管理端,并接入第一个 Agent。
2. [发布第一份配置](./first-site.md):快速新建一条最基础的 HTTP 反代站点规则,并验证节点生效状态。
3. [新建反代配置](./proxy-config.md):一步一步了解如何从证书导入与申请开始,配置 HTTPS 加密与上游源站管理。
4. [Zone 域名迁移](./zone-domain-migration.md):从旧托管域名/路由内嵌域名升级到 Zone 模型,含备份、导入、预览与回滚。
4. [Zone 域名迁移](./zone-domain-migration.md):从旧托管域名/路由内嵌域名升级到 Zone 模型(goose 自动导入),含备份、验收与回滚说明。
5. [Pages 静态托管使用](./pages-usage.md):了解静态项目 ZIP 上传限制、SPA Fallback、以及内置 API 反向代理配置。
6. [内网穿透与隧道使用](./tunnel-usage.md):部署 Relay 与 Client,实现安全、无公网 IP 反向穿透。
7. [WAF 安全防护使用](./waf-usage.md):配置 WAF 规则组,掌握 IP 黑白名单、自动/订阅 IP 组、地域限制与 PoW CC 防护。
+28 -74
View File
@@ -1,95 +1,49 @@
# Zone 域名迁移与发布验收
从旧版 `managed_domains` / 反代路由内嵌域名列迁移到 Zone + Zone 域名模型时,按本指南操作。**导入报告存在冲突时禁止继续发布。**
从旧版 `managed_domains` / 反代路由内嵌域名列迁移到 Zone + Zone 域名模型时,数据导入与表结构升级均由 **Server 启动时的 goose 自动迁移**完成,无需单独执行导入命令。
## 前置条件
## 升级时发生了什么
* 已备份 PostgreSQL / SQLite 数据库与当前激活配置版本(可导出管理端「配置版本」中的激活快照)。
* Server 二进制已升级到包含 `of_zones` / `of_zone_domains` 表迁移的版本。
* 维护窗口内可暂停非必要配置发布。
启动(或滚动升级)包含 Zone 改造的 Server 版本时,**无需手动命令**,`migrator.Migrate()` 自动:
## 1. 备份
1. 应用 goose SQL:创建 `of_zones` / `of_zone_domains`(若尚未存在)。
2. **自动导入**旧路由域名列(及无路由域名时的 `of_managed_domains`)为 Zone / Zone 域名,并绑定 `proxy_route_id` / `cert_id`(公共后缀列表解析注册根域)。
3. 继续 goose SQL:删除 `of_managed_domains` 与 `of_proxy_routes` 冗余域名/证书列。
导入幂等:已存在的域名会跳过或补绑路由。
**若历史数据无法解析(冲突域名、无效根域、证书不存在等),启动失败。** 修复数据或恢复备份后再次启动即可重试。
## 建议操作
### 1. 升级前备份
```bash
# PostgreSQL 示例
pg_dump "$DATABASE_URL" > openflare-pre-zone-$(date +%Y%m%d).sql
# 或复制备份卷 / 快照;SQLite 则直接复制 data 目录中的库文件
# 或复制备份卷 / 快照;SQLite 则复制 data 目录中的库文件
```
在管理端确认当前**激活版本号**并记下 checksum,便于回滚对比。
可选:在管理端记下当前**激活配置版本号**与 checksum,便于配置回滚对比。
## 2. 执行历史导入
### 2. 升级并启动 Server
```bash
# 二进制名称为 wavelet(或你的部署包中的同名入口)
wavelet migrate-zones
```
部署新版本并启动即可。观察启动日志中的 goose 成功信息;若出现「迁移 Zone 失败(N 个冲突)」则按日志中的冲突项修复源数据后重启。
命令会:
### 3. 升级后检查
1. 先跑 goose 迁移(确保 Zone 表存在)。
2. 以事务从旧路由域名(及无路由域名时的 `of_managed_domains`)导入 Zone / Zone 域名。
3. 使用公共后缀列表解析注册根域;冲突时整单回滚并输出报告。
1. 管理端 **网站** `/websites`:Zone 根域与域名计数是否合理。
2. Zone 详情:域名、证书、关联路由 ID。
3. **反代路由**:域名绑定来自 Zone 域名,而非旧手写字段。
**成功标志:** 进程退出码 0,且日志/标准输出无「冲突」列表。
### 4. 配置预览与发布
**失败时:** 阅读冲突项(无法解析的根域、通配符 FQDN、全局域名冲突、证书不存在等),修复源数据后重新执行。`migrate-zones` 幂等:已导入的域名会跳过,不会重复创建。
**有冲突时不要发布配置、不要执行第二阶段删列迁移。**
## 3. 导入后检查
1. 打开管理端 **网站** `/websites`:确认 Zone 根域与域名计数合理。
2. 进入各 Zone 详情:域名、证书绑定、关联路由 ID 是否正确。
3. 打开 **反代路由**:域名区应展示 Zone 域名绑定,而不是手写域名。
## 4. 配置预览与快照等价性
在升级前若已导出激活快照,导入后:
1. 在管理端打开配置差异 / 预览(或调用配置 diff / preview API)。
2. **逐路由**核对:
* 明确 `server_name` 集合(全部 FQDN)
* 证书支持文件路径 / 证书 ID 与域名对应关系
* WAF 绑定的 Route ID(`site_name` 与路由 ID 不变)
* Pages 项目引用
3. **允许**旧快照 JSON 中路由上的冗余 `domain` / `domains` / `cert_ids` 字段消失。
4. **不允许**数据面语义变化(域名集合、证书覆盖、上游、WAF、Pages 绑定)。
不一致时:停止发布,修正 Zone 域名/证书绑定后重新预览。
## 5. 发布与回滚
1. 预览通过后,在管理端执行**配置发布**,记录新版本号。
2. 用根域与各子域发起 HTTP(S) 请求,确认节点应用成功。
3. **回滚:** 在配置版本中重新激活导入前的版本;节点会拉取旧快照。数据库侧若需回退,使用升级前备份恢复(第二阶段删列后 Down 迁移不回填业务数据)。
## 6. 第二阶段:删除旧表与冗余列
仅在以下条件全部满足后执行:
* `migrate-zones` 无冲突
* 至少完成一次预览对比与发布(及必要时的回滚演练)
* 运维确认不再依赖 `of_managed_domains` 与 `of_proxy_routes` 上的 `domain` / `domains` / `cert_*` 列
然后升级到包含 `202607130001_drop_legacy_route_domain_columns` 的版本并启动 Server(自动 goose)。该迁移将:
* 删除 `of_proxy_routes` 的 `domain`、`domains`、`cert_id`、`cert_ids`、`domain_cert_ids`
* 删除表 `of_managed_domains`
**不可逆业务数据:** Down 仅在开发库重建空结构,不恢复历史域名行。
## 7. 命令速查
| 步骤 | 命令 / 操作 |
| --- | --- |
| 备份 | `pg_dump` / 复制 SQLite 文件 |
| 导入 | `wavelet migrate-zones` |
| 预览 | 管理端配置差异 / Preview API |
| 发布 | 管理端发布激活版本 |
| 回滚配置 | 管理端激活旧版本 |
| 回滚库 | 恢复备份(勿依赖 Down 填数) |
1. 在管理端查看配置差异 / 预览。
2. **逐路由**核对:`server_name` 集合、证书路径、WAF Route ID、Pages 引用。
3. **允许**旧快照 JSON 中路由上的冗余 `domain` / `domains` / `cert_ids` 消失。
4. **不允许**数据面语义变化。
5. 预览通过后发布;需要时在配置版本中激活升级前版本做配置回滚。数据库回退请使用升级前备份(Down 迁移不回填业务域名数据)。
## 相关文档
@@ -36,8 +36,8 @@ func NormalizeDomains(rawDomains []string) ([]string, error) {
return normalized, nil
}
// DecodeDomains parses legacy route domain fields for the explicit migration
// command. Runtime consumers must read ZoneDomain bindings instead.
// DecodeDomains parses legacy route domain fields for the goose upgrade importer.
// Runtime consumers must read ZoneDomain bindings instead.
func DecodeDomains(raw string, fallbackDomain string) ([]string, error) {
text := strings.TrimSpace(raw)
if text == "" {
+298 -118
View File
@@ -5,14 +5,14 @@ package zone
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"github.com/Rain-kl/Wavelet/internal/apps/openflare/routeidentity"
"github.com/Rain-kl/Wavelet/internal/db"
"github.com/Rain-kl/Wavelet/internal/model"
"gorm.io/gorm"
)
// ImportReport describes the idempotent legacy migration result.
@@ -31,138 +31,244 @@ func (r ImportReport) LogAndReturn(err error) error {
}
type legacyDomain struct {
Domain string
CertID *uint
Remark string
Domain string
CertID *uint
Remark string
ProxyRouteID *uint
}
// legacyRouteRow reads pre-cleanup of_proxy_routes columns via raw scan.
type legacyRouteRow struct {
ID uint `gorm:"column:id"`
Domain string `gorm:"column:domain"`
Domains string `gorm:"column:domains"`
DomainCertIDs string `gorm:"column:domain_cert_ids"`
Remark string `gorm:"column:remark"`
}
// legacyManagedRow reads of_managed_domains while the table still exists.
type legacyManagedRow struct {
Domain string `gorm:"column:domain"`
CertID *uint `gorm:"column:cert_id"`
Remark string `gorm:"column:remark"`
}
// ImportLegacy imports legacy proxy-route / managed-domain rows into Zone tables.
// After the phase-2 schema cleanup, missing legacy columns or tables are skipped.
// ImportLegacyTx imports legacy proxy-route / managed-domain rows into Zone tables
// within an existing SQL transaction (goose runs this on Server upgrade).
// postgres selects $n placeholders; otherwise SQLite-style ? is used.
// Missing legacy columns or tables are skipped so re-runs after phase-2 cleanup are no-ops.
//
//nolint:cyclop // the transactional importer intentionally validates every legacy source in one pass.
func ImportLegacy(ctx context.Context) (report ImportReport, resultErr error) {
conn := db.DB(ctx)
if conn == nil {
return report, fmt.Errorf("database is not initialized")
//nolint:cyclop,gocyclo // single-pass legacy importer validates every source before write.
func ImportLegacyTx(ctx context.Context, tx *sql.Tx, postgres bool) (report ImportReport, err error) {
if tx == nil {
return report, errors.New("transaction is required")
}
resultErr = conn.Transaction(func(tx *gorm.DB) error {
items := make([]legacyDomain, 0)
hasRouteDomains := false
q := func(sqlText string) string { return rebindSQL(sqlText, postgres) }
if tx.Migrator().HasColumn("of_proxy_routes", "domain") &&
tx.Migrator().HasColumn("of_proxy_routes", "domains") {
var routes []legacyRouteRow
if err := tx.Table("of_proxy_routes").
Select("id, domain, domains, domain_cert_ids, remark").
Find(&routes).Error; err != nil {
return err
}
for _, route := range routes {
domains, err := routeidentity.DecodeDomains(route.Domains, route.Domain)
if err != nil {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("route %d: %v", route.ID, err))
continue
}
if len(domains) > 0 {
hasRouteDomains = true
}
certIDs := decodeLegacyCertIDs(route.DomainCertIDs, len(domains))
for i, domain := range domains {
var certID *uint
if i < len(certIDs) && certIDs[i] > 0 {
v := certIDs[i]
certID = &v
}
items = append(items, legacyDomain{Domain: domain, CertID: certID, Remark: route.Remark})
}
items := make([]legacyDomain, 0)
hasRouteDomains := false
hasDomainCol, err := hasTableColumn(ctx, tx, q, postgres, "of_proxy_routes", "domain")
if err != nil {
return report, err
}
hasDomainsCol, err := hasTableColumn(ctx, tx, q, postgres, "of_proxy_routes", "domains")
if err != nil {
return report, err
}
if hasDomainCol && hasDomainsCol {
var collectErr error
items, hasRouteDomains, report.Conflicts, collectErr = collectLegacyRouteDomainsImpl(ctx, tx, q)
if collectErr != nil {
return report, collectErr
}
}
if !hasRouteDomains {
exists, tableErr := hasTable(ctx, tx, q, postgres, "of_managed_domains")
if tableErr != nil {
return report, tableErr
}
if exists {
managed, managedErr := collectLegacyManagedDomains(ctx, tx, q)
if managedErr != nil {
return report, managedErr
}
items = append(items, managed...)
}
}
for _, item := range items {
domain, normErr := normalizeDomain(item.Domain)
if normErr != nil {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: %v", item.Domain, normErr))
continue
}
root, rootErr := zoneRoot(domain)
if rootErr != nil {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: %v", domain, rootErr))
continue
}
if !hasRouteDomains && tx.Migrator().HasTable("of_managed_domains") {
var legacy []legacyManagedRow
if err := tx.Table("of_managed_domains").
Select("domain, cert_id, remark").
Find(&legacy).Error; err != nil {
return err
}
for _, item := range legacy {
items = append(items, legacyDomain(item))
var existingID uint
var existingZoneDomain string
scanErr := tx.QueryRowContext(ctx, q(`
SELECT zd.id, z.domain
FROM of_zone_domains zd
JOIN of_zones z ON z.id = zd.zone_id
WHERE zd.domain = ?
`), domain).Scan(&existingID, &existingZoneDomain)
if scanErr == nil {
if existingZoneDomain != root {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: global domain conflict", domain))
} else if item.ProxyRouteID != nil {
if _, bindErr := tx.ExecContext(ctx, q(`
UPDATE of_zone_domains
SET proxy_route_id = COALESCE(proxy_route_id, ?),
cert_id = COALESCE(cert_id, ?)
WHERE id = ?
`), *item.ProxyRouteID, nullableUint(item.CertID), existingID); bindErr != nil {
return report, bindErr
}
}
continue
}
if !errors.Is(scanErr, sql.ErrNoRows) {
return report, scanErr
}
for _, item := range items {
domain, err := normalizeDomain(item.Domain)
if err != nil {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: %v", item.Domain, err))
continue
}
root, err := zoneRoot(domain)
if err != nil {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: %v", domain, err))
continue
}
var existing model.ZoneDomain
err = tx.Where("domain = ?", domain).First(&existing).Error
if err == nil {
var z model.Zone
if tx.First(&z, existing.ZoneID).Error != nil || z.Domain != root {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: global domain conflict", domain))
}
continue
}
if err != nil && !isNotFound(err) {
return err
}
var zone model.Zone
err = tx.Where("domain = ?", root).First(&zone).Error
if isNotFound(err) {
zone = model.Zone{Domain: root}
if err = tx.Create(&zone).Error; err != nil {
return err
}
report.Zones++
} else if err != nil {
return err
}
if item.CertID != nil {
var cert model.TLSCertificate
if err = tx.First(&cert, *item.CertID).Error; err != nil {
zoneID, zoneErr := ensureZone(ctx, tx, q, root, &report)
if zoneErr != nil {
return report, zoneErr
}
if item.CertID != nil {
var certID uint
if certErr := tx.QueryRowContext(ctx, q(`SELECT id FROM of_tls_certificates WHERE id = ?`), *item.CertID).
Scan(&certID); certErr != nil {
if errors.Is(certErr, sql.ErrNoRows) {
report.Conflicts = append(report.Conflicts, fmt.Sprintf("%s: %s", domain, errCertificateNotFound))
continue
}
return report, certErr
}
if err = tx.Create(&model.ZoneDomain{
ZoneID: zone.ID,
Domain: domain,
CertID: item.CertID,
Remark: item.Remark,
}).Error; err != nil {
return err
}
if _, insErr := tx.ExecContext(ctx, q(`
INSERT INTO of_zone_domains (zone_id, proxy_route_id, domain, cert_id, remark, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
`), zoneID, nullableUint(item.ProxyRouteID), domain, nullableUint(item.CertID), item.Remark); insErr != nil {
return report, insErr
}
report.Domains++
}
if len(report.Conflicts) > 0 {
return report, fmt.Errorf("legacy data has conflicts")
}
return report, nil
}
func ensureZone(
ctx context.Context,
tx *sql.Tx,
q func(string) string,
root string,
report *ImportReport,
) (uint, error) {
var zoneID uint
err := tx.QueryRowContext(ctx, q(`SELECT id FROM of_zones WHERE domain = ?`), root).Scan(&zoneID)
if err == nil {
return zoneID, nil
}
if !errors.Is(err, sql.ErrNoRows) {
return 0, err
}
if _, execErr := tx.ExecContext(ctx, q(`
INSERT INTO of_zones (domain, remark, created_at, updated_at)
VALUES (?, '', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
`), root); execErr != nil {
return 0, execErr
}
if err := tx.QueryRowContext(ctx, q(`SELECT id FROM of_zones WHERE domain = ?`), root).Scan(&zoneID); err != nil {
return 0, err
}
report.Zones++
return zoneID, nil
}
func collectLegacyRouteDomainsImpl(
ctx context.Context,
tx *sql.Tx,
q func(string) string,
) (items []legacyDomain, hasRouteDomains bool, conflicts []string, err error) {
// Probe domain_cert_ids: if SELECT fails, fall back without it.
queryWithCert := q(`SELECT id, domain, domains, COALESCE(domain_cert_ids, '[]'), remark FROM of_proxy_routes`)
rows, err := tx.QueryContext(ctx, queryWithCert)
useCert := true
if err != nil {
useCert = false
rows, err = tx.QueryContext(ctx, q(`SELECT id, domain, domains, remark FROM of_proxy_routes`))
if err != nil {
return nil, false, nil, err
}
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var (
id uint
domain string
domains string
certIDs string
remark string
)
if useCert {
if err := rows.Scan(&id, &domain, &domains, &certIDs, &remark); err != nil {
return nil, false, nil, err
}
report.Domains++
} else {
if err := rows.Scan(&id, &domain, &domains, &remark); err != nil {
return nil, false, nil, err
}
certIDs = "[]"
}
if len(report.Conflicts) > 0 {
return fmt.Errorf("legacy data has conflicts")
decoded, decodeErr := routeidentity.DecodeDomains(domains, domain)
if decodeErr != nil {
conflicts = append(conflicts, fmt.Sprintf("route %d: %v", id, decodeErr))
continue
}
return nil
})
return report, resultErr
if len(decoded) > 0 {
hasRouteDomains = true
}
ids := decodeLegacyCertIDs(certIDs, len(decoded))
routeID := id
for i, d := range decoded {
var certID *uint
if i < len(ids) && ids[i] > 0 {
v := ids[i]
certID = &v
}
items = append(items, legacyDomain{
Domain: d,
CertID: certID,
Remark: remark,
ProxyRouteID: &routeID,
})
}
}
return items, hasRouteDomains, conflicts, rows.Err()
}
func collectLegacyManagedDomains(ctx context.Context, tx *sql.Tx, q func(string) string) ([]legacyDomain, error) {
rows, err := tx.QueryContext(ctx, q(`SELECT domain, cert_id, remark FROM of_managed_domains`))
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
items := make([]legacyDomain, 0)
for rows.Next() {
var (
domain string
certID sql.NullInt64
remark string
)
if err := rows.Scan(&domain, &certID, &remark); err != nil {
return nil, err
}
item := legacyDomain{Domain: domain, Remark: remark}
if certID.Valid && certID.Int64 > 0 {
v := uint(certID.Int64)
item.CertID = &v
}
items = append(items, item)
}
return items, rows.Err()
}
func decodeLegacyCertIDs(raw string, count int) []uint {
@@ -176,4 +282,78 @@ func decodeLegacyCertIDs(raw string, count int) []uint {
return values
}
func isNotFound(err error) bool { return err == gorm.ErrRecordNotFound }
func nullableUint(v *uint) any {
if v == nil {
return nil
}
return *v
}
func rebindSQL(query string, postgres bool) string {
if !postgres {
return query
}
var b strings.Builder
b.Grow(len(query) + len(query)/4)
n := 0
for i := 0; i < len(query); i++ {
if query[i] == '?' {
n++
b.WriteByte('$')
b.WriteString(strconv.Itoa(n))
continue
}
b.WriteByte(query[i])
}
return b.String()
}
func hasTable(
ctx context.Context,
tx *sql.Tx,
_ func(string) string,
postgres bool,
table string,
) (bool, error) {
var count int
var err error
if postgres {
err = tx.QueryRowContext(ctx, `
SELECT COUNT(*) FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = $1
`, table).Scan(&count)
} else {
err = tx.QueryRowContext(ctx,
`SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?`, table,
).Scan(&count)
}
if err != nil {
return false, err
}
return count > 0, nil
}
func hasTableColumn(
ctx context.Context,
tx *sql.Tx,
_ func(string) string,
postgres bool,
table, column string,
) (bool, error) {
var count int
var err error
if postgres {
err = tx.QueryRowContext(ctx, `
SELECT COUNT(*) FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = $1 AND column_name = $2
`, table, column).Scan(&count)
} else {
err = tx.QueryRowContext(ctx,
`SELECT COUNT(*) FROM pragma_table_info(?) WHERE name = ?`, table, column,
).Scan(&count)
}
if err != nil {
return false, err
}
return count > 0, nil
}
@@ -0,0 +1,154 @@
// Copyright 2026 Arctel.net
// SPDX-License-Identifier: Apache-2.0
package zone
import (
"context"
"database/sql"
"testing"
"github.com/Rain-kl/Wavelet/internal/db"
"github.com/glebarez/sqlite"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
)
func setupLegacyImportDB(t *testing.T) (*sql.DB, func()) {
t.Helper()
gormDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
require.NoError(t, err)
sqlDB, err := gormDB.DB()
require.NoError(t, err)
// Pre-phase-2 schema: legacy route columns + managed domains + zone tables.
stmts := []string{
`CREATE TABLE of_zones (
id INTEGER PRIMARY KEY AUTOINCREMENT,
domain TEXT NOT NULL UNIQUE,
remark TEXT NOT NULL DEFAULT '',
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
)`,
`CREATE TABLE of_zone_domains (
id INTEGER PRIMARY KEY AUTOINCREMENT,
zone_id INTEGER NOT NULL,
proxy_route_id INTEGER,
domain TEXT NOT NULL UNIQUE,
cert_id INTEGER,
remark TEXT NOT NULL DEFAULT '',
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
)`,
`CREATE TABLE of_proxy_routes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site_name TEXT NOT NULL DEFAULT '',
domain TEXT NOT NULL DEFAULT '',
domains TEXT NOT NULL DEFAULT '[]',
domain_cert_ids TEXT NOT NULL DEFAULT '[]',
origin_url TEXT NOT NULL DEFAULT '',
remark TEXT NOT NULL DEFAULT ''
)`,
`CREATE TABLE of_tls_certificates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL DEFAULT ''
)`,
`CREATE TABLE of_managed_domains (
id INTEGER PRIMARY KEY AUTOINCREMENT,
domain TEXT NOT NULL,
cert_id INTEGER,
remark TEXT NOT NULL DEFAULT ''
)`,
}
for _, stmt := range stmts {
_, err := sqlDB.Exec(stmt)
require.NoError(t, err)
}
previous := db.DB(context.Background())
db.SetDB(gormDB)
return sqlDB, func() {
db.SetDB(previous)
_ = sqlDB.Close()
}
}
func TestImportLegacyTxBindsRouteDomains(t *testing.T) {
sqlDB, cleanup := setupLegacyImportDB(t)
defer cleanup()
ctx := context.Background()
_, err := sqlDB.Exec(`INSERT INTO of_tls_certificates (id, name) VALUES (7, 'cert')`)
require.NoError(t, err)
_, err = sqlDB.Exec(`
INSERT INTO of_proxy_routes (id, site_name, domain, domains, domain_cert_ids, origin_url, remark)
VALUES (3, 'api', 'api.example.com', '["api.example.com","www.example.com"]', '[7,7]', 'http://origin', 'r')
`)
require.NoError(t, err)
tx, err := sqlDB.Begin()
require.NoError(t, err)
report, err := ImportLegacyTx(ctx, tx, false)
require.NoError(t, err)
require.NoError(t, tx.Commit())
assert.Equal(t, 1, report.Zones)
assert.Equal(t, 2, report.Domains)
var zoneDomain string
require.NoError(t, sqlDB.QueryRow(`SELECT domain FROM of_zones`).Scan(&zoneDomain))
assert.Equal(t, "example.com", zoneDomain)
var count int
require.NoError(t, sqlDB.QueryRow(`SELECT COUNT(*) FROM of_zone_domains WHERE proxy_route_id = 3`).Scan(&count))
assert.Equal(t, 2, count)
// Idempotent re-run
tx, err = sqlDB.Begin()
require.NoError(t, err)
report2, err := ImportLegacyTx(ctx, tx, false)
require.NoError(t, err)
require.NoError(t, tx.Commit())
assert.Equal(t, 0, report2.Domains)
}
func TestImportLegacyTxNoOpWithoutLegacyColumns(t *testing.T) {
gormDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
require.NoError(t, err)
sqlDB, err := gormDB.DB()
require.NoError(t, err)
defer sqlDB.Close()
_, err = sqlDB.Exec(`
CREATE TABLE of_zones (
id INTEGER PRIMARY KEY AUTOINCREMENT,
domain TEXT NOT NULL UNIQUE,
remark TEXT NOT NULL DEFAULT '',
created_at DATETIME, updated_at DATETIME
);
CREATE TABLE of_zone_domains (
id INTEGER PRIMARY KEY AUTOINCREMENT,
zone_id INTEGER NOT NULL,
proxy_route_id INTEGER,
domain TEXT NOT NULL UNIQUE,
cert_id INTEGER,
remark TEXT NOT NULL DEFAULT '',
created_at DATETIME, updated_at DATETIME
);
CREATE TABLE of_proxy_routes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site_name TEXT NOT NULL DEFAULT '',
origin_url TEXT NOT NULL DEFAULT ''
);
`)
require.NoError(t, err)
tx, err := sqlDB.Begin()
require.NoError(t, err)
report, err := ImportLegacyTx(context.Background(), tx, false)
require.NoError(t, err)
require.NoError(t, tx.Commit())
assert.Equal(t, 0, report.Zones)
assert.Equal(t, 0, report.Domains)
}
-21
View File
@@ -1,21 +0,0 @@
// Copyright 2026 Arctel.net
// SPDX-License-Identifier: Apache-2.0
package cmd
import (
"context"
"github.com/Rain-kl/Wavelet/internal/apps/openflare/zone"
"github.com/Rain-kl/Wavelet/internal/db/migrator"
"github.com/spf13/cobra"
)
var migrateZonesCmd = &cobra.Command{
Use: "migrate-zones", Short: "导入旧域名数据到 Zone",
PreRun: func(_ *cobra.Command, _ []string) { migrator.Migrate() },
RunE: func(_ *cobra.Command, _ []string) error {
report, err := zone.ImportLegacy(context.Background())
return report.LogAndReturn(err)
},
}
+1 -1
View File
@@ -72,7 +72,7 @@ func init() {
schedulerCmd.PreRun = migratePreRun
// 2. 集中将这些命令注册为真正的子命令,以解决 Cobra 的 unknown command 校验限制
rootCmd.AddCommand(allCmd, apiCmd, workerCmd, schedulerCmd, migrateZonesCmd)
rootCmd.AddCommand(allCmd, apiCmd, workerCmd, schedulerCmd)
}
// Execute 执行根命令
@@ -1,204 +0,0 @@
// Copyright 2026 Arctel.net
// SPDX-License-Identifier: Apache-2.0
package migrator
import (
"context"
"database/sql"
"fmt"
"strings"
"github.com/pressly/goose/v3"
)
func init() {
goose.AddMigrationContext(up202606050001, down202606050001)
}
func up202606050001(ctx context.Context, tx *sql.Tx) error {
dialect := gooseDialect()
if dialect == dialectSqlite {
if err := fixSqliteGooseDbVersion(ctx, tx); err != nil {
return err
}
}
tableExistsQuery := tableExistsSQL(dialect)
checkTable := func(name string) (bool, error) {
var count int
err := tx.QueryRowContext(ctx, tableExistsQuery, name).Scan(&count)
return count > 0, err
}
// Check if this is a legacy database (by checking if the "options" table exists).
// On a clean install, "options" won't exist, so this will be a quick no-op.
hasOptions, err := checkTable("options")
if err != nil {
return fmt.Errorf("check table options failed: %w", err)
}
if !hasOptions {
// Clean install or already migrated, no action needed.
return nil
}
// Helper to drop table if it exists
dropTable := func(tableName string) error {
dropSQL := fmt.Sprintf("DROP TABLE IF EXISTS %s", tableName)
if dialect == dialectPostgres {
dropSQL += cascadeSuffix
}
if _, err := tx.ExecContext(ctx, dropSQL); err != nil {
return fmt.Errorf("drop table %s failed: %w", tableName, err)
}
return nil
}
// 1. Drop legacy observability and log tables to free space and avoid conflict
obsPrefixes := []string{
"node_system_profiles",
"node_health_events",
"node_access_logs_",
"node_observation_openresties_",
"node_observation_frpcs_",
"node_observation_frps_",
"node_metric_snapshots_",
"node_request_reports_",
}
for _, prefix := range obsPrefixes {
if err := dropTablesWithPrefix(ctx, tx, tablesWithPrefixSQL(dialect), prefix, dropTable); err != nil {
return err
}
}
// 2. Rename custom and platform legacy tables to legacy_ prefix
tablesToRename := []string{
"users",
"auth_sources",
"external_accounts",
"options",
"origins",
"apply_logs",
"proxy_routes",
"nodes",
"waf_rule_groups",
"waf_rule_group_bindings",
"waf_ip_groups",
"tls_certificates",
"managed_domains",
"dns_accounts",
"acme_accounts",
"config_versions",
"pages_projects",
"pages_deployments",
"pages_deployment_files",
}
for _, oldName := range tablesToRename {
if err := renameLegacyTable(ctx, tx, oldName, checkTable, dropTable); err != nil {
return err
}
}
return nil
}
func dropTablesWithPrefix(ctx context.Context, tx *sql.Tx, query, prefix string, dropTable func(string) error) error {
rows, err := tx.QueryContext(ctx, query, prefix+"%")
if err != nil {
return fmt.Errorf("query legacy tables for prefix %s failed: %w", prefix, err)
}
defer func() {
_ = rows.Close()
}()
var tables []string
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
return err
}
tables = append(tables, name)
}
for _, table := range tables {
if err := dropTable(table); err != nil {
return err
}
}
return nil
}
func renameLegacyTable(ctx context.Context, tx *sql.Tx, oldName string, checkTable func(string) (bool, error), dropTable func(string) error) error {
exists, err := checkTable(oldName)
if err != nil {
return err
}
if !exists {
return nil
}
newName := "legacy_" + oldName
newExists, err := checkTable(newName)
if err != nil {
return err
}
if newExists {
if err := dropTable(newName); err != nil {
return err
}
}
renameSQL := fmt.Sprintf("ALTER TABLE %s RENAME TO %s", oldName, newName)
if _, err := tx.ExecContext(ctx, renameSQL); err != nil {
return fmt.Errorf("rename table %s to %s failed: %w", oldName, newName, err)
}
return nil
}
func down202606050001(_ context.Context, _ *sql.Tx) error {
// Down migration is a no-op as the legacy DB is rolled forward during migration.
return nil
}
func fixSqliteGooseDbVersion(ctx context.Context, tx *sql.Tx) error {
var createSQL string
err := tx.QueryRowContext(ctx, "SELECT sql FROM sqlite_master WHERE type='table' AND name='goose_db_version'").Scan(&createSQL)
if err != nil {
if err == sql.ErrNoRows {
return nil
}
return err
}
if strings.Contains(strings.ToUpper(createSQL), "AUTOINCREMENT") {
return nil // Already upgraded
}
if _, err := tx.ExecContext(ctx, "ALTER TABLE goose_db_version RENAME TO old_goose_db_version"); err != nil {
return fmt.Errorf("rename goose_db_version failed: %w", err)
}
newTableSQL := `CREATE TABLE goose_db_version (
id INTEGER PRIMARY KEY AUTOINCREMENT,
version_id INTEGER NOT NULL,
is_applied INTEGER NOT NULL,
tstamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);`
if _, err := tx.ExecContext(ctx, newTableSQL); err != nil {
return fmt.Errorf("create new goose_db_version failed: %w", err)
}
copySQL := `INSERT INTO goose_db_version (id, version_id, is_applied, tstamp)
SELECT id, version_id, is_applied, tstamp FROM old_goose_db_version;`
if _, err := tx.ExecContext(ctx, copySQL); err != nil {
return fmt.Errorf("copy goose_db_version data failed: %w", err)
}
if _, err := tx.ExecContext(ctx, "DROP TABLE old_goose_db_version"); err != nil {
return fmt.Errorf("drop old_goose_db_version failed: %w", err)
}
return nil
}
@@ -1,550 +0,0 @@
// Copyright 2026 Arctel.net
// SPDX-License-Identifier: Apache-2.0
package migrator
import (
"context"
"database/sql"
"fmt"
"github.com/pressly/goose/v3"
)
func init() {
goose.AddMigrationContext(up202606200006, down202606200006)
}
type migrationTask struct {
legacyName string
sqliteSQL string
pgSQL string
}
func up202606200006(ctx context.Context, tx *sql.Tx) error {
dialect := gooseDialect()
tableExistsQuery := tableExistsSQL(dialect)
checkTable := func(name string) (bool, error) {
var count int
err := tx.QueryRowContext(ctx, tableExistsQuery, name).Scan(&count)
return count > 0, err
}
// Helper to migrate table if it exists
migrateTable := func(task migrationTask) error {
exists, err := checkTable(task.legacyName)
if err != nil {
return err
}
if !exists {
return nil
}
var sqlToRun string
if dialect == dialectSqlite {
sqlToRun = task.sqliteSQL
} else {
sqlToRun = task.pgSQL
}
if _, err := tx.ExecContext(ctx, sqlToRun); err != nil {
return fmt.Errorf("execute migration query for %s failed: %w", task.legacyName, err)
}
return nil
}
tasks := getMigrationTasks()
for _, task := range tasks {
if err := migrateTable(task); err != nil {
return err
}
}
// Drop legacy tables that exist
legacyTables := []string{
"legacy_users",
"legacy_auth_sources",
"legacy_external_accounts",
"legacy_options",
"legacy_origins",
"legacy_apply_logs",
"legacy_proxy_routes",
"legacy_nodes",
"legacy_waf_rule_groups",
"legacy_waf_rule_group_bindings",
"legacy_waf_ip_groups",
"legacy_tls_certificates",
"legacy_managed_domains",
"legacy_dns_accounts",
"legacy_acme_accounts",
"legacy_config_versions",
"legacy_pages_projects",
"legacy_pages_deployments",
"legacy_pages_deployment_files",
}
for _, table := range legacyTables {
exists, err := checkTable(table)
if err != nil {
return err
}
if exists {
dropSQL := fmt.Sprintf("DROP TABLE IF EXISTS %s", table)
if dialect == dialectPostgres {
dropSQL += cascadeSuffix
}
if _, err := tx.ExecContext(ctx, dropSQL); err != nil {
return fmt.Errorf("drop legacy table %s failed: %w", table, err)
}
}
}
if dialect == dialectPostgres {
if _, err := tx.ExecContext(ctx, `
SELECT setval(
pg_get_serial_sequence('of_waf_rule_group_bindings', 'id'),
GREATEST(COALESCE((SELECT MAX(id) FROM of_waf_rule_group_bindings), 0), 1),
COALESCE((SELECT MAX(id) FROM of_waf_rule_group_bindings), 0) > 0
)
`); err != nil {
return fmt.Errorf("sync of_waf_rule_group_bindings sequence failed: %w", err)
}
}
return nil
}
func getMigrationTasks() []migrationTask {
var tasks []migrationTask
tasks = append(tasks, getUserAndOptionTasks()...)
tasks = append(tasks, getOriginAndRouteTasks()...)
tasks = append(tasks, getWafTasks()...)
tasks = append(tasks, getTLSTasks()...)
tasks = append(tasks, getPagesAndConfigTasks()...)
return tasks
}
func getUserAndOptionTasks() []migrationTask {
return []migrationTask{
{
legacyName: "legacy_users",
sqliteSQL: `INSERT OR REPLACE INTO w_users (id, username, password, nickname, email, is_active, is_admin, created_at, updated_at)
SELECT id, username, password, display_name, email,
CASE WHEN status = 1 THEN 1 ELSE 0 END,
CASE WHEN role = 100 THEN 1 ELSE 0 END,
CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
FROM legacy_users;`,
pgSQL: `INSERT INTO w_users (id, username, password, nickname, email, is_active, is_admin, created_at, updated_at)
SELECT id, username, password, display_name, email,
CASE WHEN status = 1 THEN TRUE ELSE FALSE END,
CASE WHEN role = 100 THEN TRUE ELSE FALSE END,
CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
FROM legacy_users
ON CONFLICT (id) DO UPDATE SET
username = EXCLUDED.username,
password = EXCLUDED.password,
nickname = EXCLUDED.nickname,
email = EXCLUDED.email,
is_active = EXCLUDED.is_active,
is_admin = EXCLUDED.is_admin,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_auth_sources",
sqliteSQL: `INSERT OR REPLACE INTO w_auth_sources (id, name, type, display_name, is_active, client_id, client_secret, openid_discovery_url, scopes, icon_url, created_at, updated_at)
SELECT id, name, type, display_name, is_active, client_id, client_secret, openid_discovery_url, scopes, icon_url, created_at, updated_at
FROM legacy_auth_sources;`,
pgSQL: `INSERT INTO w_auth_sources (id, name, type, display_name, is_active, client_id, client_secret, openid_discovery_url, scopes, icon_url, created_at, updated_at)
SELECT id, name, type, display_name, is_active::boolean, client_id, client_secret, openid_discovery_url, scopes, icon_url, created_at, updated_at
FROM legacy_auth_sources
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
type = EXCLUDED.type,
display_name = EXCLUDED.display_name,
is_active = EXCLUDED.is_active,
client_id = EXCLUDED.client_id,
client_secret = EXCLUDED.client_secret,
openid_discovery_url = EXCLUDED.openid_discovery_url,
scopes = EXCLUDED.scopes,
icon_url = EXCLUDED.icon_url,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_external_accounts",
sqliteSQL: `INSERT OR REPLACE INTO w_external_accounts (id, auth_source_id, user_id, external_id, external_username, email, created_at, updated_at)
SELECT id, auth_source_id, user_id, external_id, external_username, email, created_at, updated_at
FROM legacy_external_accounts;`,
pgSQL: `INSERT INTO w_external_accounts (id, auth_source_id, user_id, external_id, external_username, email, created_at, updated_at)
SELECT id, auth_source_id, user_id, external_id, external_username, email, created_at, updated_at
FROM legacy_external_accounts
ON CONFLICT (id) DO UPDATE SET
auth_source_id = EXCLUDED.auth_source_id,
user_id = EXCLUDED.user_id,
external_id = EXCLUDED.external_id,
external_username = EXCLUDED.external_username,
email = EXCLUDED.email,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_options",
sqliteSQL: `INSERT OR REPLACE INTO of_options (key, value)
SELECT key, COALESCE(value, '') FROM legacy_options;`,
pgSQL: `INSERT INTO of_options (key, value)
SELECT key, COALESCE(value, '') FROM legacy_options
ON CONFLICT (key) DO UPDATE SET
value = EXCLUDED.value;`,
},
}
}
func getOriginAndRouteTasks() []migrationTask {
return []migrationTask{
{
legacyName: "legacy_origins",
sqliteSQL: `INSERT OR REPLACE INTO of_origins (id, name, address, remark, created_at, updated_at)
SELECT id, name, address, COALESCE(remark, ''), created_at, updated_at
FROM legacy_origins;`,
pgSQL: `INSERT INTO of_origins (id, name, address, remark, created_at, updated_at)
SELECT id, name, address, COALESCE(remark, ''), created_at, updated_at
FROM legacy_origins
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
address = EXCLUDED.address,
remark = EXCLUDED.remark,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_apply_logs",
sqliteSQL: `INSERT OR REPLACE INTO of_apply_logs (id, node_id, version, result, message, checksum, main_config_checksum, route_config_checksum, support_file_count, created_at)
SELECT id, node_id, version, result, message, COALESCE(checksum, ''), COALESCE(main_config_checksum, ''), COALESCE(route_config_checksum, ''), COALESCE(support_file_count, 0), created_at
FROM legacy_apply_logs;`,
pgSQL: `INSERT INTO of_apply_logs (id, node_id, version, result, message, checksum, main_config_checksum, route_config_checksum, support_file_count, created_at)
SELECT id, node_id, version, result, message, COALESCE(checksum, ''), COALESCE(main_config_checksum, ''), COALESCE(route_config_checksum, ''), COALESCE(support_file_count, 0), created_at
FROM legacy_apply_logs
ON CONFLICT (id) DO UPDATE SET
node_id = EXCLUDED.node_id,
version = EXCLUDED.version,
result = EXCLUDED.result,
message = EXCLUDED.message,
checksum = EXCLUDED.checksum,
main_config_checksum = EXCLUDED.main_config_checksum,
route_config_checksum = EXCLUDED.route_config_checksum,
support_file_count = EXCLUDED.support_file_count;`,
},
{
legacyName: "legacy_proxy_routes",
sqliteSQL: `INSERT OR REPLACE INTO of_proxy_routes (id, site_name, domain, domains, origin_id, origin_url, origin_host, upstreams, enabled, enable_https, cert_id, cert_ids, domain_cert_ids, redirect_http, limit_conn_per_server, limit_conn_per_ip, limit_rate, cache_enabled, cache_policy, cache_rules, custom_headers, basic_auth_enabled, basic_auth_username, basic_auth_password, remark, upstream_type, tunnel_node_id, tunnel_target_addr, tunnel_target_protocol, pages_project_id, created_at, updated_at)
SELECT id, COALESCE(site_name, ''), domain, COALESCE(domains, '[]'), origin_id, origin_url, COALESCE(origin_host, ''), COALESCE(upstreams, '[]'), enabled, enable_https, cert_id, COALESCE(cert_ids, '[]'), COALESCE(domain_cert_ids, '[]'), redirect_http, limit_conn_per_server, limit_conn_per_ip, COALESCE(limit_rate, ''), cache_enabled, COALESCE(cache_policy, ''), COALESCE(cache_rules, '[]'), COALESCE(custom_headers, '[]'), basic_auth_enabled, COALESCE(basic_auth_username, ''), COALESCE(basic_auth_password, ''), COALESCE(remark, ''), COALESCE(upstream_type, 'direct'), tunnel_node_id, COALESCE(tunnel_target_addr, ''), COALESCE(tunnel_target_protocol, ''), pages_project_id, created_at, updated_at
FROM legacy_proxy_routes;`,
pgSQL: `INSERT INTO of_proxy_routes (id, site_name, domain, domains, origin_id, origin_url, origin_host, upstreams, enabled, enable_https, cert_id, cert_ids, domain_cert_ids, redirect_http, limit_conn_per_server, limit_conn_per_ip, limit_rate, cache_enabled, cache_policy, cache_rules, custom_headers, basic_auth_enabled, basic_auth_username, basic_auth_password, remark, upstream_type, tunnel_node_id, tunnel_target_addr, tunnel_target_protocol, pages_project_id, created_at, updated_at)
SELECT id, COALESCE(site_name, ''), domain, COALESCE(domains, '[]'), origin_id, origin_url, COALESCE(origin_host, ''), COALESCE(upstreams, '[]'), enabled, enable_https, cert_id, COALESCE(cert_ids, '[]'), COALESCE(domain_cert_ids, '[]'), redirect_http, limit_conn_per_server, limit_conn_per_ip, COALESCE(limit_rate, ''), cache_enabled, COALESCE(cache_policy, ''), COALESCE(cache_rules, '[]'), COALESCE(custom_headers, '[]'), basic_auth_enabled, COALESCE(basic_auth_username, ''), COALESCE(basic_auth_password, ''), COALESCE(remark, ''), COALESCE(upstream_type, 'direct'), tunnel_node_id, COALESCE(tunnel_target_addr, ''), COALESCE(tunnel_target_protocol, ''), pages_project_id, created_at, updated_at
FROM legacy_proxy_routes
ON CONFLICT (id) DO UPDATE SET
site_name = EXCLUDED.site_name,
domain = EXCLUDED.domain,
domains = EXCLUDED.domains,
origin_id = EXCLUDED.origin_id,
origin_url = EXCLUDED.origin_url,
origin_host = EXCLUDED.origin_host,
upstreams = EXCLUDED.upstreams,
enabled = EXCLUDED.enabled,
enable_https = EXCLUDED.enable_https,
cert_id = EXCLUDED.cert_id,
cert_ids = EXCLUDED.cert_ids,
domain_cert_ids = EXCLUDED.domain_cert_ids,
redirect_http = EXCLUDED.redirect_http,
limit_conn_per_server = EXCLUDED.limit_conn_per_server,
limit_conn_per_ip = EXCLUDED.limit_conn_per_ip,
limit_rate = EXCLUDED.limit_rate,
cache_enabled = EXCLUDED.cache_enabled,
cache_policy = EXCLUDED.cache_policy,
cache_rules = EXCLUDED.cache_rules,
custom_headers = EXCLUDED.custom_headers,
basic_auth_enabled = EXCLUDED.basic_auth_enabled,
basic_auth_username = EXCLUDED.basic_auth_username,
basic_auth_password = EXCLUDED.basic_auth_password,
remark = EXCLUDED.remark,
upstream_type = EXCLUDED.upstream_type,
tunnel_node_id = EXCLUDED.tunnel_node_id,
tunnel_target_addr = EXCLUDED.tunnel_target_addr,
tunnel_target_protocol = EXCLUDED.tunnel_target_protocol,
pages_project_id = EXCLUDED.pages_project_id,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_nodes",
sqliteSQL: `INSERT OR REPLACE INTO of_nodes (id, node_id, name, ip, ip_manual_override, geo_name, geo_latitude, geo_longitude, geo_manual_override, access_token, auto_update_enabled, update_requested, update_channel, update_tag, restart_openresty_requested, version, ext_version, openresty_status, openresty_message, status, current_version, last_seen_at, last_error, created_at, updated_at, node_type, relay_bind_port, relay_vhost_http_port, relay_auth_token, relay_agent_access_addr, relay_client_access_addr, relay_client_proxy_url, capabilities_json, relay_status, relay_web_server_enabled)
SELECT id, node_id, name, COALESCE(ip, ''), COALESCE(ip_manual_override, 0), COALESCE(geo_name, ''), geo_latitude, geo_longitude, COALESCE(geo_manual_override, 0), COALESCE(access_token, ''), COALESCE(auto_update_enabled, 0), COALESCE(update_requested, 0), COALESCE(update_channel, 'stable'), COALESCE(update_tag, ''), COALESCE(restart_openresty_requested, 0), COALESCE(version, ''), COALESCE(ext_version, ''), COALESCE(openresty_status, 'unknown'), openresty_message, COALESCE(status, 'offline'), COALESCE(current_version, ''), last_seen_at, last_error, created_at, updated_at, COALESCE(node_type, 'edge_node'), COALESCE(relay_bind_port, 0), COALESCE(relay_vhost_http_port, 0), COALESCE(relay_auth_token, ''), COALESCE(relay_agent_access_addr, ''), COALESCE(relay_client_access_addr, ''), COALESCE(relay_client_proxy_url, ''), COALESCE(capabilities_json, '[]'), COALESCE(relay_status, 'unknown'), COALESCE(relay_web_server_enabled, 0)
FROM legacy_nodes;`,
pgSQL: `INSERT INTO of_nodes (id, node_id, name, ip, ip_manual_override, geo_name, geo_latitude, geo_longitude, geo_manual_override, access_token, auto_update_enabled, update_requested, update_channel, update_tag, restart_openresty_requested, version, ext_version, openresty_status, openresty_message, status, current_version, last_seen_at, last_error, created_at, updated_at, node_type, relay_bind_port, relay_vhost_http_port, relay_auth_token, relay_agent_access_addr, relay_client_access_addr, relay_client_proxy_url, capabilities_json, relay_status, relay_web_server_enabled)
SELECT id, node_id, name, COALESCE(ip, ''), COALESCE(ip_manual_override, FALSE), COALESCE(geo_name, ''), geo_latitude, geo_longitude, COALESCE(geo_manual_override, FALSE), COALESCE(access_token, ''), COALESCE(auto_update_enabled, FALSE), COALESCE(update_requested, FALSE), COALESCE(update_channel, 'stable'), COALESCE(update_tag, ''), COALESCE(restart_openresty_requested, FALSE), COALESCE(version, ''), COALESCE(ext_version, ''), COALESCE(openresty_status, 'unknown'), openresty_message, COALESCE(status, 'offline'), COALESCE(current_version, ''), last_seen_at, last_error, created_at, updated_at, COALESCE(node_type, 'edge_node'), COALESCE(relay_bind_port, 0), COALESCE(relay_vhost_http_port, 0), COALESCE(relay_auth_token, ''), COALESCE(relay_agent_access_addr, ''), COALESCE(relay_client_access_addr, ''), COALESCE(relay_client_proxy_url, ''), COALESCE(capabilities_json, '[]'), COALESCE(relay_status, 'unknown'), COALESCE(relay_web_server_enabled, FALSE)
FROM legacy_nodes
ON CONFLICT (id) DO UPDATE SET
node_id = EXCLUDED.node_id,
name = EXCLUDED.name,
ip = EXCLUDED.ip,
ip_manual_override = EXCLUDED.ip_manual_override,
geo_name = EXCLUDED.geo_name,
geo_latitude = EXCLUDED.geo_latitude,
geo_longitude = EXCLUDED.geo_longitude,
geo_manual_override = EXCLUDED.geo_manual_override,
access_token = EXCLUDED.access_token,
auto_update_enabled = EXCLUDED.auto_update_enabled,
update_requested = EXCLUDED.update_requested,
update_channel = EXCLUDED.update_channel,
update_tag = EXCLUDED.update_tag,
restart_openresty_requested = EXCLUDED.restart_openresty_requested,
version = EXCLUDED.version,
ext_version = EXCLUDED.ext_version,
openresty_status = EXCLUDED.openresty_status,
openresty_message = EXCLUDED.openresty_message,
status = EXCLUDED.status,
current_version = EXCLUDED.current_version,
last_seen_at = EXCLUDED.last_seen_at,
last_error = EXCLUDED.last_error,
updated_at = EXCLUDED.updated_at,
node_type = EXCLUDED.node_type,
relay_bind_port = EXCLUDED.relay_bind_port,
relay_vhost_http_port = EXCLUDED.relay_vhost_http_port,
relay_auth_token = EXCLUDED.relay_auth_token,
relay_agent_access_addr = EXCLUDED.relay_agent_access_addr,
relay_client_access_addr = EXCLUDED.relay_client_access_addr,
relay_client_proxy_url = EXCLUDED.relay_client_proxy_url,
capabilities_json = EXCLUDED.capabilities_json,
relay_status = EXCLUDED.relay_status,
relay_web_server_enabled = EXCLUDED.relay_web_server_enabled;`,
},
}
}
func getWafTasks() []migrationTask {
return []migrationTask{
{
legacyName: "legacy_waf_rule_groups",
sqliteSQL: `INSERT OR REPLACE INTO of_waf_rule_groups (id, name, enabled, is_global, block_status_code, block_response_body, ip_whitelist, ip_blacklist, ip_whitelist_groups, ip_blacklist_groups, country_whitelist, country_blacklist, region_whitelist, region_blacklist, pow_enabled, pow_config, remark, created_at, updated_at)
SELECT id, name, COALESCE(enabled, 1), COALESCE(is_global, 0), COALESCE(block_status_code, 418), COALESCE(block_response_body, ''), COALESCE(ip_whitelist, '[]'), COALESCE(ip_blacklist, '[]'), COALESCE(ip_whitelist_groups, '[]'), COALESCE(ip_blacklist_groups, '[]'), COALESCE(country_whitelist, '[]'), COALESCE(country_blacklist, '[]'), COALESCE(region_whitelist, '[]'), COALESCE(region_blacklist, '[]'), COALESCE(pow_enabled, 0), COALESCE(pow_config, '{}'), COALESCE(remark, ''), created_at, updated_at
FROM legacy_waf_rule_groups;`,
pgSQL: `INSERT INTO of_waf_rule_groups (id, name, enabled, is_global, block_status_code, block_response_body, ip_whitelist, ip_blacklist, ip_whitelist_groups, ip_blacklist_groups, country_whitelist, country_blacklist, region_whitelist, region_blacklist, pow_enabled, pow_config, remark, created_at, updated_at)
SELECT id, name, COALESCE(enabled::boolean, TRUE), COALESCE(is_global::boolean, FALSE), COALESCE(block_status_code, 418), COALESCE(block_response_body, ''), COALESCE(ip_whitelist, '[]'), COALESCE(ip_blacklist, '[]'), COALESCE(ip_whitelist_groups, '[]'), COALESCE(ip_blacklist_groups, '[]'), COALESCE(country_whitelist, '[]'), COALESCE(country_blacklist, '[]'), COALESCE(region_whitelist, '[]'), COALESCE(region_blacklist, '[]'), COALESCE(pow_enabled::boolean, FALSE), COALESCE(pow_config, '{}'), COALESCE(remark, ''), created_at, updated_at
FROM legacy_waf_rule_groups
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
enabled = EXCLUDED.enabled,
is_global = EXCLUDED.is_global,
block_status_code = EXCLUDED.block_status_code,
block_response_body = EXCLUDED.block_response_body,
ip_whitelist = EXCLUDED.ip_whitelist,
ip_blacklist = EXCLUDED.ip_blacklist,
ip_whitelist_groups = EXCLUDED.ip_whitelist_groups,
ip_blacklist_groups = EXCLUDED.ip_blacklist_groups,
country_whitelist = EXCLUDED.country_whitelist,
country_blacklist = EXCLUDED.country_blacklist,
region_whitelist = EXCLUDED.region_whitelist,
region_blacklist = EXCLUDED.region_blacklist,
pow_enabled = EXCLUDED.pow_enabled,
pow_config = EXCLUDED.pow_config,
remark = EXCLUDED.remark,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_waf_rule_group_bindings",
sqliteSQL: `INSERT OR REPLACE INTO of_waf_rule_group_bindings (id, rule_group_id, proxy_route_id, created_at)
SELECT id, rule_group_id, proxy_route_id, created_at
FROM legacy_waf_rule_group_bindings;`,
pgSQL: `INSERT INTO of_waf_rule_group_bindings (id, rule_group_id, proxy_route_id, created_at)
SELECT id, rule_group_id, proxy_route_id, created_at
FROM legacy_waf_rule_group_bindings
ON CONFLICT (id) DO UPDATE SET
rule_group_id = EXCLUDED.rule_group_id,
proxy_route_id = EXCLUDED.proxy_route_id;`,
},
{
legacyName: "legacy_waf_ip_groups",
sqliteSQL: `INSERT OR REPLACE INTO of_waf_ip_groups (id, name, type, enabled, ip_list, auto_config, ext_ips, subscription_url, subscription_format, subscription_mapping_rule, sync_interval_minutes, last_synced_at, next_sync_at, last_sync_status, last_sync_message, remark, created_at, updated_at)
SELECT id, name, type, COALESCE(enabled, 1), COALESCE(ip_list, '[]'), COALESCE(auto_config, '{}'), COALESCE(ext_ips, '[]'), COALESCE(subscription_url, ''), COALESCE(subscription_format, 'text'), COALESCE(subscription_mapping_rule, ''), COALESCE(sync_interval_minutes, 1440), last_synced_at, next_sync_at, COALESCE(last_sync_status, ''), COALESCE(last_sync_message, ''), COALESCE(remark, ''), created_at, updated_at
FROM legacy_waf_ip_groups;`,
pgSQL: `INSERT INTO of_waf_ip_groups (id, name, type, enabled, ip_list, auto_config, ext_ips, subscription_url, subscription_format, subscription_mapping_rule, sync_interval_minutes, last_synced_at, next_sync_at, last_sync_status, last_sync_message, remark, created_at, updated_at)
SELECT id, name, type, COALESCE(enabled::boolean, TRUE), COALESCE(ip_list, '[]'), COALESCE(auto_config, '{}'), COALESCE(ext_ips, '[]'), COALESCE(subscription_url, ''), COALESCE(subscription_format, 'text'), COALESCE(subscription_mapping_rule, ''), COALESCE(sync_interval_minutes, 1440), last_synced_at, next_sync_at, COALESCE(last_sync_status, ''), COALESCE(last_sync_message, ''), COALESCE(remark, ''), created_at, updated_at
FROM legacy_waf_ip_groups
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
type = EXCLUDED.type,
enabled = EXCLUDED.enabled,
ip_list = EXCLUDED.ip_list,
auto_config = EXCLUDED.auto_config,
ext_ips = EXCLUDED.ext_ips,
subscription_url = EXCLUDED.subscription_url,
subscription_format = EXCLUDED.subscription_format,
subscription_mapping_rule = EXCLUDED.subscription_mapping_rule,
sync_interval_minutes = EXCLUDED.sync_interval_minutes,
last_synced_at = EXCLUDED.last_synced_at,
next_sync_at = EXCLUDED.next_sync_at,
last_sync_status = EXCLUDED.last_sync_status,
last_sync_message = EXCLUDED.last_sync_message,
remark = EXCLUDED.remark,
updated_at = EXCLUDED.updated_at;`,
},
}
}
func getTLSTasks() []migrationTask {
return []migrationTask{
{
legacyName: "legacy_tls_certificates",
sqliteSQL: `INSERT OR REPLACE INTO of_tls_certificates (id, name, cert_pem, key_pem, not_before, not_after, remark, provider, acme_account_id, dns_account_id, key_algorithm, auto_renew, primary_domain, other_domains, disable_cname, skip_dns, dns1, dns2, apply_status, apply_message, created_at, updated_at)
SELECT id, name, cert_pem, key_pem, not_before, not_after, COALESCE(remark, ''), COALESCE(provider, 'upload'), COALESCE(acme_account_id, 0), COALESCE(dns_account_id, 0), COALESCE(key_algorithm, ''), COALESCE(auto_renew, 0), COALESCE(primary_domain, ''), COALESCE(other_domains, ''), COALESCE(disable_cname, 0), COALESCE(skip_dns, 0), COALESCE(dns1, ''), COALESCE(dns2, ''), COALESCE(apply_status, 'ready'), COALESCE(apply_message, ''), created_at, updated_at
FROM legacy_tls_certificates;`,
pgSQL: `INSERT INTO of_tls_certificates (id, name, cert_pem, key_pem, not_before, not_after, remark, provider, acme_account_id, dns_account_id, key_algorithm, auto_renew, primary_domain, other_domains, disable_cname, skip_dns, dns1, dns2, apply_status, apply_message, created_at, updated_at)
SELECT id, name, cert_pem, key_pem, not_before, not_after, COALESCE(remark, ''), COALESCE(provider, 'upload'), COALESCE(acme_account_id, 0), COALESCE(dns_account_id, 0), COALESCE(key_algorithm, ''), COALESCE(auto_renew, FALSE), COALESCE(primary_domain, ''), COALESCE(other_domains, ''), COALESCE(disable_cname, FALSE), COALESCE(skip_dns, FALSE), COALESCE(dns1, ''), COALESCE(dns2, ''), COALESCE(apply_status, 'ready'), COALESCE(apply_message, ''), created_at, updated_at
FROM legacy_tls_certificates
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
cert_pem = EXCLUDED.cert_pem,
key_pem = EXCLUDED.key_pem,
not_before = EXCLUDED.not_before,
not_after = EXCLUDED.not_after,
remark = EXCLUDED.remark,
provider = EXCLUDED.provider,
acme_account_id = EXCLUDED.acme_account_id,
dns_account_id = EXCLUDED.dns_account_id,
key_algorithm = EXCLUDED.key_algorithm,
auto_renew = EXCLUDED.auto_renew,
primary_domain = EXCLUDED.primary_domain,
other_domains = EXCLUDED.other_domains,
disable_cname = EXCLUDED.disable_cname,
skip_dns = EXCLUDED.skip_dns,
dns1 = EXCLUDED.dns1,
dns2 = EXCLUDED.dns2,
apply_status = EXCLUDED.apply_status,
apply_message = EXCLUDED.apply_message,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_managed_domains",
sqliteSQL: `INSERT OR REPLACE INTO of_managed_domains (id, domain, cert_id, enabled, remark, created_at, updated_at)
SELECT id, domain, cert_id, COALESCE(enabled, 1), COALESCE(remark, ''), created_at, updated_at
FROM legacy_managed_domains;`,
pgSQL: `INSERT INTO of_managed_domains (id, domain, cert_id, enabled, remark, created_at, updated_at)
SELECT id, domain, cert_id, COALESCE(enabled, TRUE), COALESCE(remark, ''), created_at, updated_at
FROM legacy_managed_domains
ON CONFLICT (id) DO UPDATE SET
domain = EXCLUDED.domain,
cert_id = EXCLUDED.cert_id,
enabled = EXCLUDED.enabled,
remark = EXCLUDED.remark,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_dns_accounts",
sqliteSQL: `INSERT OR REPLACE INTO of_dns_accounts (id, name, type, authorization, created_at, updated_at)
SELECT id, name, type, authorization, created_at, updated_at
FROM legacy_dns_accounts;`,
pgSQL: `INSERT INTO of_dns_accounts (id, name, type, "authorization", created_at, updated_at)
SELECT id, name, type, "authorization", created_at, updated_at
FROM legacy_dns_accounts
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
type = EXCLUDED.type,
"authorization" = EXCLUDED."authorization",
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_acme_accounts",
sqliteSQL: `INSERT OR REPLACE INTO of_acme_accounts (id, email, url, private_key, created_at, updated_at)
SELECT id, COALESCE(email, ''), COALESCE(url, ''), COALESCE(private_key, ''), created_at, updated_at
FROM legacy_acme_accounts;`,
pgSQL: `INSERT INTO of_acme_accounts (id, email, url, private_key, created_at, updated_at)
SELECT id, COALESCE(email, ''), COALESCE(url, ''), COALESCE(private_key, ''), created_at, updated_at
FROM legacy_acme_accounts
ON CONFLICT (id) DO UPDATE SET
email = EXCLUDED.email,
url = EXCLUDED.url,
private_key = EXCLUDED.private_key,
updated_at = EXCLUDED.updated_at;`,
},
}
}
func getPagesAndConfigTasks() []migrationTask {
return []migrationTask{
{
legacyName: "legacy_config_versions",
sqliteSQL: `INSERT OR REPLACE INTO of_config_versions (id, version, snapshot_json, main_config, rendered_config, support_files_json, checksum, is_active, created_by, created_at)
SELECT id, version, snapshot_json, COALESCE(main_config, ''), rendered_config, COALESCE(support_files_json, '[]'), checksum, COALESCE(is_active, 0), created_by, created_at
FROM legacy_config_versions;`,
pgSQL: `INSERT INTO of_config_versions (id, version, snapshot_json, main_config, rendered_config, support_files_json, checksum, is_active, created_by, created_at)
SELECT id, version, snapshot_json, COALESCE(main_config, ''), rendered_config, COALESCE(support_files_json, '[]'), checksum, COALESCE(is_active, FALSE), created_by, created_at
FROM legacy_config_versions
ON CONFLICT (id) DO UPDATE SET
version = EXCLUDED.version,
snapshot_json = EXCLUDED.snapshot_json,
main_config = EXCLUDED.main_config,
rendered_config = EXCLUDED.rendered_config,
support_files_json = EXCLUDED.support_files_json,
checksum = EXCLUDED.checksum,
is_active = EXCLUDED.is_active,
created_by = EXCLUDED.created_by;`,
},
{
legacyName: "legacy_pages_projects",
sqliteSQL: `INSERT OR REPLACE INTO of_pages_projects (id, name, slug, description, enabled, spa_fallback_enabled, spa_fallback_path, api_proxy_enabled, api_proxy_path, api_proxy_pass, api_proxy_rewrite, active_deployment_id, root_dir, entry_file, created_at, updated_at)
SELECT id, name, slug, COALESCE(description, ''), COALESCE(enabled, 1), COALESCE(spa_fallback_enabled, 0), COALESCE(spa_fallback_path, '/index.html'), COALESCE(api_proxy_enabled, 0), COALESCE(api_proxy_path, ''), COALESCE(api_proxy_pass, ''), COALESCE(api_proxy_rewrite, ''), active_deployment_id, COALESCE(root_dir, ''), COALESCE(entry_file, 'index.html'), created_at, updated_at
FROM legacy_pages_projects;`,
pgSQL: `INSERT INTO of_pages_projects (id, name, slug, description, enabled, spa_fallback_enabled, spa_fallback_path, api_proxy_enabled, api_proxy_path, api_proxy_pass, api_proxy_rewrite, active_deployment_id, root_dir, entry_file, created_at, updated_at)
SELECT id, name, slug, COALESCE(description, ''), COALESCE(enabled, TRUE), COALESCE(spa_fallback_enabled, FALSE), COALESCE(spa_fallback_path, '/index.html'), COALESCE(api_proxy_enabled, FALSE), COALESCE(api_proxy_path, ''), COALESCE(api_proxy_pass, ''), COALESCE(api_proxy_rewrite, ''), active_deployment_id, COALESCE(root_dir, ''), COALESCE(entry_file, 'index.html'), created_at, updated_at
FROM legacy_pages_projects
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
slug = EXCLUDED.slug,
description = EXCLUDED.description,
enabled = EXCLUDED.enabled,
spa_fallback_enabled = EXCLUDED.spa_fallback_enabled,
spa_fallback_path = EXCLUDED.spa_fallback_path,
api_proxy_enabled = EXCLUDED.api_proxy_enabled,
api_proxy_path = EXCLUDED.api_proxy_path,
api_proxy_pass = EXCLUDED.api_proxy_pass,
api_proxy_rewrite = EXCLUDED.api_proxy_rewrite,
active_deployment_id = EXCLUDED.active_deployment_id,
root_dir = EXCLUDED.root_dir,
entry_file = EXCLUDED.entry_file,
updated_at = EXCLUDED.updated_at;`,
},
{
legacyName: "legacy_pages_deployments",
sqliteSQL: `INSERT OR REPLACE INTO of_pages_deployments (id, project_id, deployment_number, checksum, status, artifact_path, file_count, total_size, created_by, created_at, activated_at)
SELECT id, project_id, deployment_number, checksum, COALESCE(status, 'uploaded'), artifact_path, COALESCE(file_count, 0), COALESCE(total_size, 0), COALESCE(created_by, ''), created_at, activated_at
FROM legacy_pages_deployments;`,
pgSQL: `INSERT INTO of_pages_deployments (id, project_id, deployment_number, checksum, status, artifact_path, file_count, total_size, created_by, created_at, activated_at)
SELECT id, project_id, deployment_number, checksum, COALESCE(status, 'uploaded'), artifact_path, COALESCE(file_count, 0), COALESCE(total_size, 0), COALESCE(created_by, ''), created_at, activated_at
FROM legacy_pages_deployments
ON CONFLICT (id) DO UPDATE SET
project_id = EXCLUDED.project_id,
deployment_number = EXCLUDED.deployment_number,
checksum = EXCLUDED.checksum,
status = EXCLUDED.status,
artifact_path = EXCLUDED.artifact_path,
file_count = EXCLUDED.file_count,
total_size = EXCLUDED.total_size,
created_by = EXCLUDED.created_by,
activated_at = EXCLUDED.activated_at;`,
},
{
legacyName: "legacy_pages_deployment_files",
sqliteSQL: `INSERT OR REPLACE INTO of_pages_deployment_files (id, deployment_id, path, size, checksum, created_at)
SELECT id, deployment_id, path, COALESCE(size, 0), checksum, created_at
FROM legacy_pages_deployment_files;`,
pgSQL: `INSERT INTO of_pages_deployment_files (id, deployment_id, path, size, checksum, created_at)
SELECT id, deployment_id, path, COALESCE(size, 0), checksum, created_at
FROM legacy_pages_deployment_files
ON CONFLICT (id) DO UPDATE SET
deployment_id = EXCLUDED.deployment_id,
path = EXCLUDED.path,
size = EXCLUDED.size,
checksum = EXCLUDED.checksum;`,
},
}
}
func down202606200006(_ context.Context, _ *sql.Tx) error {
// Down migration is a no-op
return nil
}
-261
View File
@@ -1,261 +0,0 @@
// Copyright 2026 Arctel.net
// SPDX-License-Identifier: Apache-2.0
package migrator
import (
"testing"
"time"
"github.com/Rain-kl/Wavelet/internal/config"
"github.com/Rain-kl/Wavelet/internal/db"
"github.com/alicebob/miniredis/v2"
"github.com/glebarez/sqlite"
"github.com/redis/go-redis/v9"
"github.com/redis/go-redis/v9/maintnotifications"
"gorm.io/gorm"
)
func TestMigrateUpgradesLegacyDatabaseAndPreservesData(t *testing.T) {
// 1. Initialize an empty in-memory SQLite DB
sqliteDB, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{
DisableForeignKeyConstraintWhenMigrating: true,
})
if err != nil {
t.Fatalf("gorm.Open(sqlite) error = %v", err)
}
rawDB, err := sqliteDB.DB()
if err != nil {
t.Fatalf("sqliteDB.DB() error = %v", err)
}
// 2. Set up the legacy schema manually
setupLegacySchema := []string{
`CREATE TABLE users (
id integer(64) NOT NULL,
username text,
password text NOT NULL,
display_name text,
role integer(64),
status integer(64),
token text,
email text,
github_id text,
wechat_id text,
CONSTRAINT users_pkey PRIMARY KEY (id)
);`,
`CREATE TABLE options (
key text NOT NULL,
value text,
CONSTRAINT options_pkey PRIMARY KEY (key)
);`,
`CREATE TABLE origins (
id integer(64) NOT NULL,
name text(255) NOT NULL,
address text(255) NOT NULL,
remark text(255),
created_at text(6),
updated_at text(6),
CONSTRAINT origins_pkey PRIMARY KEY (id)
);`,
`CREATE TABLE apply_logs (
id integer(64) NOT NULL,
node_id text(64) NOT NULL,
version text(32) NOT NULL,
result text(32) NOT NULL,
message text,
checksum text(64) NOT NULL,
main_config_checksum text(64) NOT NULL,
route_config_checksum text(64) NOT NULL,
support_file_count integer(64) NOT NULL,
created_at text(6),
CONSTRAINT apply_logs_pkey PRIMARY KEY (id)
);`,
// Legacy log tables (which should be dropped)
`CREATE TABLE node_access_logs_00 (
id integer(64) NOT NULL,
node_id text(64) NOT NULL,
logged_at text(6) NOT NULL
);`,
`CREATE TABLE node_system_profiles (
id integer(64) NOT NULL,
node_id text(64) NOT NULL,
hostname text(255)
);`,
}
for _, query := range setupLegacySchema {
if _, err := rawDB.Exec(query); err != nil {
t.Fatalf("Exec setup query failed: %v", err)
}
}
// 3. Populate with legacy mock data
insertMockData := []string{
// Legacy admin user
`INSERT INTO users (id, username, password, display_name, role, status, email)
VALUES (1, 'ryan', 'hashed_pass_123', 'Root User ryan', 100, 1, 'ryan@example.com');`,
// Legacy normal user
`INSERT INTO users (id, username, password, display_name, role, status, email)
VALUES (2, 'jack', 'hashed_pass_456', 'Normal User jack', 10, 1, 'jack@example.com');`,
// Legacy config options(GeoIPProvider 会被迁移到 w_system_configs.geoip_provider)
`INSERT INTO options (key, value) VALUES ('GeoIPProvider', 'ip-api');`,
// Legacy origin
`INSERT INTO origins (id, name, address, remark, created_at, updated_at)
VALUES (10, 'my_origin', '127.0.0.1:8080', 'original origin', '2026-06-01 12:00:00', '2026-06-01 12:00:00');`,
// Legacy apply log
`INSERT INTO apply_logs (id, node_id, version, result, message, checksum, main_config_checksum, route_config_checksum, support_file_count, created_at)
VALUES (100, 'node_1', 'v1.0.0', 'success', 'applied successfully', 'hash1', 'hash2', 'hash3', 2, '2026-06-01 12:00:00');`,
}
for _, query := range insertMockData {
if _, err := rawDB.Exec(query); err != nil {
t.Fatalf("Exec mock data query failed: %v", err)
}
}
// 4. Set up mock services (Redis & Config)
mr, err := miniredis.Run()
if err != nil {
t.Fatalf("miniredis.Run() error = %v", err)
}
redisClient := redis.NewClient(&redis.Options{
Addr: mr.Addr(),
MaintNotificationsConfig: &maintnotifications.Config{
Mode: maintnotifications.ModeDisabled,
},
})
previousDBEnabled := config.Config.Database.Enabled
previousRedis := db.Redis
config.Config.Database.Enabled = false
db.SetDB(sqliteDB)
db.Redis = redisClient
t.Cleanup(func() {
config.Config.Database.Enabled = previousDBEnabled
db.SetDB(nil)
db.Redis = previousRedis
_ = redisClient.Close()
mr.Close()
})
// 5. Run the Goose migrations
Migrate()
// 6. Assertions on migrated data schema & records
// Check that the w_users table exists and contains correct columns/records
var usersCount int64
if err := sqliteDB.Table("w_users").Count(&usersCount).Error; err != nil {
t.Fatalf("count w_users error: %v", err)
}
if usersCount < 2 {
t.Errorf("expected at least 2 users, got %d", usersCount)
}
// Verify user 1 (ryan) details
type userStruct struct {
ID int64
Username string
Nickname string
Email string
IsActive bool
IsAdmin bool
}
var ryanUser userStruct
if err := sqliteDB.Table("w_users").Where("id = ?", 1).First(&ryanUser).Error; err != nil {
t.Fatalf("query ryan user failed: %v", err)
}
if ryanUser.Username != "ryan" || ryanUser.Nickname != "Root User ryan" || !ryanUser.IsActive || !ryanUser.IsAdmin {
t.Errorf("ryan user migrated incorrectly: %+v", ryanUser)
}
// Verify user 2 (jack) details
var jackUser userStruct
if err := sqliteDB.Table("w_users").Where("id = ?", 2).First(&jackUser).Error; err != nil {
t.Fatalf("query jack user failed: %v", err)
}
if jackUser.Username != "jack" || jackUser.Nickname != "Normal User jack" || !jackUser.IsActive || jackUser.IsAdmin {
t.Errorf("jack user migrated incorrectly: %+v", jackUser)
}
// Verify legacy option migrated all the way to w_system_configs(of_options 已被删除)
var geoIPProviderVal string
if err := sqliteDB.Table("w_system_configs").Where("key = ?", "geoip_provider").Select("value").Scan(&geoIPProviderVal).Error; err != nil {
t.Fatalf("query geoip_provider failed: %v", err)
}
if geoIPProviderVal != "ip-api" {
t.Errorf("geoip_provider value incorrect: got %s, want ip-api", geoIPProviderVal)
}
// Verify origin migrated to of_origins
type originStruct struct {
ID int64
Name string
Address string
Remark string
}
var testOrigin originStruct
if err := sqliteDB.Table("of_origins").Where("id = ?", 10).First(&testOrigin).Error; err != nil {
t.Fatalf("query my_origin failed: %v", err)
}
if testOrigin.Name != "my_origin" || testOrigin.Address != "127.0.0.1:8080" || testOrigin.Remark != "original origin" {
t.Errorf("origin migrated incorrectly: %+v", testOrigin)
}
// Verify apply log migrated to of_apply_logs
type applyLogStruct struct {
ID int64
NodeID string
Version string
Result string
Message string
CreatedAt time.Time
}
var testApplyLog applyLogStruct
if err := sqliteDB.Table("of_apply_logs").Where("id = ?", 100).First(&testApplyLog).Error; err != nil {
t.Fatalf("query apply_logs failed: %v", err)
}
if testApplyLog.NodeID != "node_1" || testApplyLog.Version != "v1.0.0" || testApplyLog.Result != "success" || testApplyLog.Message != "applied successfully" {
t.Errorf("apply log migrated incorrectly: %+v", testApplyLog)
}
// 7. Verify that all legacy_ prefix tables have been dropped
legacyTablesToCheck := []string{
"legacy_users",
"legacy_options",
"legacy_origins",
"legacy_apply_logs",
"legacy_proxy_routes",
"legacy_nodes",
"legacy_waf_rule_groups",
"legacy_waf_rule_group_bindings",
"legacy_waf_ip_groups",
"legacy_tls_certificates",
"legacy_managed_domains",
"legacy_dns_accounts",
"legacy_acme_accounts",
"legacy_config_versions",
"legacy_pages_projects",
"legacy_pages_deployments",
"legacy_pages_deployment_files",
"users",
"options",
"origins",
"apply_logs",
"node_access_logs_00",
"node_system_profiles",
}
for _, table := range legacyTablesToCheck {
var count int
if err := sqliteDB.Raw("SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&count).Error; err != nil {
t.Fatalf("check table %s existence failed: %v", table, err)
}
if count > 0 {
t.Errorf("expected table %s to be dropped, but it still exists", table)
}
}
}
@@ -0,0 +1,7 @@
-- +goose Up
-- Historical Wavelet→OpenFlare table rename bridge (formerly Go migration).
-- Fresh installs and current of_* schemas need no action.
SELECT 1;
-- +goose Down
SELECT 1;
@@ -0,0 +1,7 @@
-- +goose Up
-- Historical copy from legacy_* tables into of_* / w_* (formerly Go migration).
-- Already-applied environments keep their data; new installs have no legacy_* sources.
SELECT 1;
-- +goose Down
SELECT 1;
@@ -0,0 +1,8 @@
-- +goose Up
-- Marker: Zone 域名从旧路由列 / of_managed_domains 的导入由 migrator.Migrate()
-- 在全部 goose SQL 应用后自动执行(publicsuffix 根域解析无法用纯 SQL 正确完成)。
-- 本文件仅占位版本号,保证升级链路有序:建表 → 导入 → 删旧列。
SELECT 1;
-- +goose Down
SELECT 1;
@@ -0,0 +1,7 @@
-- +goose Up
-- Historical Wavelet→OpenFlare table rename bridge (formerly Go migration).
-- Fresh installs and current of_* schemas need no action.
SELECT 1;
-- +goose Down
SELECT 1;
@@ -0,0 +1,7 @@
-- +goose Up
-- Historical copy from legacy_* tables into of_* / w_* (formerly Go migration).
-- Already-applied environments keep their data; new installs have no legacy_* sources.
SELECT 1;
-- +goose Down
SELECT 1;
@@ -0,0 +1,8 @@
-- +goose Up
-- Marker: Zone 域名从旧路由列 / of_managed_domains 的导入由 migrator.Migrate()
-- 在全部 goose SQL 应用后自动执行(publicsuffix 根域解析无法用纯 SQL 正确完成)。
-- 本文件仅占位版本号,保证升级链路有序:建表 → 导入 → 删旧列。
SELECT 1;
-- +goose Down
SELECT 1;
+39 -16
View File
@@ -12,6 +12,7 @@ import (
"fmt"
"log"
"github.com/Rain-kl/Wavelet/internal/apps/openflare/zone"
"github.com/Rain-kl/Wavelet/internal/config"
"github.com/Rain-kl/Wavelet/internal/db"
"github.com/Rain-kl/Wavelet/internal/repository"
@@ -34,7 +35,9 @@ func dbType() string {
const (
dialectSqlite = "sqlite3"
dialectPostgres = "postgres"
cascadeSuffix = " CASCADE"
// zoneImportSQLVersion is the goose SQL marker after of_zones creation and
// before drop of legacy route domain columns. Zone data import runs here.
zoneImportSQLVersion int64 = 202607120002
)
func gooseDialect() string {
@@ -51,7 +54,7 @@ func migrationDir() string {
return "goose/postgres"
}
// Migrate 执行数据库迁移
// Migrate 执行数据库迁移:全部结构变更走 goose SQL;Zone 历史域名导入在 SQL 之后自动执行。
func Migrate() {
gormDB := db.DB(context.Background())
if gormDB == nil {
@@ -70,6 +73,15 @@ func Migrate() {
if err := resyncGooseVersionSequence(sqlDB); err != nil {
log.Fatalf("[%s] resync goose_db_version sequence failed: %v\n", dbType(), err)
}
// 1) SQL up to zone-import marker (includes of_zones DDL; still has legacy columns).
if err := goose.UpTo(sqlDB, migrationDir(), zoneImportSQLVersion); err != nil {
log.Fatalf("[%s] goose migrate (up to zone import) failed: %v\n", dbType(), err)
}
// 2) Auto-import legacy domains (publicsuffix; idempotent; no-op after phase-2 drop).
if err := importZoneDomainsAfterGoose(sqlDB); err != nil {
log.Fatalf("[%s] zone domain import failed: %v\n", dbType(), err)
}
// 3) Remaining SQL (drop legacy columns / managed_domains, later migrations).
if err := goose.Up(sqlDB, migrationDir()); err != nil {
log.Fatalf("[%s] goose migrate failed: %v\n", dbType(), err)
}
@@ -79,6 +91,31 @@ func Migrate() {
log.Printf("[%s] goose migrate success\n", dbType())
}
// importZoneDomainsAfterGoose 在 SQL 迁移完成后自动导入旧路由/托管域名。
// 必须使用 publicsuffix 解析注册根域,故不能放在纯 SQL 中;幂等且在旧列删除后为空操作。
func importZoneDomainsAfterGoose(sqlDB *sql.DB) error {
ctx := context.Background()
tx, err := sqlDB.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("begin zone import transaction: %w", err)
}
report, err := zone.ImportLegacyTx(ctx, tx, gooseDialect() == dialectPostgres)
if err != nil {
_ = tx.Rollback()
return report.LogAndReturn(err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit zone import: %w", err)
}
if report.Zones > 0 || report.Domains > 0 {
log.Printf(
"[%s] imported zone domains automatically: zones=%d domains=%d\n",
dbType(), report.Zones, report.Domains,
)
}
return nil
}
// resyncGooseVersionSequence 修复 PostgreSQL 下 goose_db_version.id 自增序列落后于
// MAX(id) 的问题(常见于从 dump 恢复或历史迁移以显式 id 复制数据后)。序列落后会
// 导致 goose 记录新版本号时 INSERT 命中 goose_db_version_pkey 唯一约束冲突。
@@ -115,17 +152,3 @@ func clearSystemConfigCache() {
log.Printf("[%s] clear system config cache failed: %v\n", dbType(), err)
}
}
func tableExistsSQL(dialect string) string {
if dialect == dialectPostgres {
return "SELECT count(*) FROM information_schema.tables WHERE table_schema='public' AND table_name=$1"
}
return "SELECT count(*) FROM sqlite_master WHERE type='table' AND name=?"
}
func tablesWithPrefixSQL(dialect string) string {
if dialect == dialectPostgres {
return "SELECT table_name FROM information_schema.tables WHERE table_schema='public' AND table_name LIKE $1"
}
return "SELECT name FROM sqlite_master WHERE type='table' AND name LIKE ?"
}