mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-10-09 00:56:37 +08:00
[优化] 重构数据库迁移逻辑,整合遗留和Goose迁移处理
This commit is contained in:
@@ -179,4 +179,4 @@ go build -o openflare-agent ./cmd/agent
|
|||||||
4. 涉及配置、部署、API 或产品边界时同步更新文档。
|
4. 涉及配置、部署、API 或产品边界时同步更新文档。
|
||||||
5. 风险较高的修改补充测试或等效联调验证。
|
5. 风险较高的修改补充测试或等效联调验证。
|
||||||
|
|
||||||
数据库结构变更必须提升数据库版本号,并补充从上一版本到新版本的显式迁移方法和校验逻辑。
|
数据库结构变更必须提升数据库版本号,并补充显式迁移方法和校验逻辑。v8-v17 保留为旧升级框架兼容链;v17 之后统一使用 goose,新的 goose 框架代码必须集中在 `openflare_server/model/goose` 包下;每次数据库升级都要在该包下新增独立的 `goose_<timestamp>_<description>.go` 文件,不得把具体迁移逻辑集中堆在 goose 注册入口中,也不得把新 goose 框架代码放回 `openflare_server/model` 根包。
|
||||||
|
|||||||
@@ -112,17 +112,19 @@ Frontend:
|
|||||||
|
|
||||||
每次提升数据库版本号时,必须补充从上一版本升级到新版本的显式迁移方法。迁移方法必须包含升级后的校验逻辑;只有校验通过,才能写入新的数据库版本记录。
|
每次提升数据库版本号时,必须补充从上一版本升级到新版本的显式迁移方法。迁移方法必须包含升级后的校验逻辑;只有校验通过,才能写入新的数据库版本记录。
|
||||||
|
|
||||||
v1-v7 视为历史初始基线,不再维护逐版本升级文件。从 v8 起,数据库迁移必须放在 `openflare_server/model/migrate` 目录中,并以目标版本命名文件,例如 `v16.go`。每个版本文件通过 `init()` 注册自己的迁移,当前数据库版本取已注册迁移的最大目标版本。不得为了整理文件而改变已发布 v8+ 迁移的语义。
|
v1-v7 视为历史初始基线,不再维护逐版本升级文件。v8-v17 是旧升级框架的兼容迁移链,只保留在 `openflare_server/model/migrate` 目录中用于老库升级。旧库启动时必须先按旧框架升级到 `legacyMigrationTerminalVersion`,再桥接到 goose;不得为了整理文件而改变已发布 v8-v17 迁移的语义。
|
||||||
|
|
||||||
|
从 v17 之后,数据库升级统一使用 goose。新的 goose provider、桥接逻辑、注册入口和具体迁移文件必须全部放在 `openflare_server/model/goose` 包下,`openflare_server/model` 根包只保留纯净实体类、旧框架兼容适配和必要的上下文注入。每次新增数据库升级都必须新建一个单独的 Go 文件,文件名使用 `openflare_server/model/goose/goose_<timestamp>_<description>.go`,例如 `openflare_server/model/goose/goose_202606020001_add_node_capabilities_json.go`。迁移文件必须同时包含该版本的 goose migration 构造函数、升级逻辑和校验逻辑;`model/goose/migrations.go` 只能作为注册入口和公共构造工具,禁止把具体迁移逻辑集中堆放在该文件中。
|
||||||
|
|
||||||
执行数据库升级时必须按以下步骤完成:
|
执行数据库升级时必须按以下步骤完成:
|
||||||
|
|
||||||
1. 判断是否需要升级数据库版本:凡是新增/删除/重命名表、字段、索引、约束、列类型、分表规则,或改变持久化数据语义,都必须升级。
|
1. 判断是否需要升级数据库版本:凡是新增/删除/重命名表、字段、索引、约束、列类型、分表规则,或改变持久化数据语义,都必须升级。
|
||||||
2. 新增 `openflare_server/model/migrate/vN.go`,其中 `N` 为目标版本号。文件头部必须包含注释,说明本次升级了什么内容,以及为什么需要升级。
|
2. 新增 `openflare_server/model/goose/goose_<timestamp>_<description>.go`,其中 `<timestamp>` 为 goose 版本号。文件头部或迁移构造函数附近必须包含注释,说明本次升级了什么内容,以及为什么需要升级。
|
||||||
3. 在 `vN.go` 中实现 `VN()`,并在 `init()` 中调用 `Register(VN())`。`FromVersion` 必须等于 `N-1`,`ToVersion` 必须等于 `N`。
|
3. 在该文件中实现独立迁移构造函数,并返回通过 `newGORMMigration(...)` 创建的 migration;随后只在 `openflare_server/model/goose/migrations.go` 的 `registeredMigrations(...)` 中新增一条注册项。
|
||||||
4. 在 `migrateVN` 中写入升级逻辑。可通过 `Context` 调用 `ApplyCurrentSchema`、历史 backfill、默认数据初始化等公共能力;复杂数据修复必须显式处理,不得只依赖 `AutoMigrate`。
|
4. 在同一个单独迁移文件中写入升级逻辑。可通过 goose `Context` 调用 `ApplyCurrentSchema`、历史 backfill、默认数据初始化等公共能力;复杂数据修复必须显式处理,不得只依赖 `AutoMigrate`。
|
||||||
5. 在 `validateVN` 中写入升级后的校验逻辑。校验至少要覆盖新增表/字段/索引是否存在、关键默认数据是否存在、必要的数据回填是否成功。
|
5. 在同一个单独迁移文件中写入升级后的校验逻辑。校验至少要覆盖新增表/字段/索引是否存在、关键默认数据是否存在、必要的数据回填是否成功。
|
||||||
6. 如果新迁移需要新的公共 backfill 或校验辅助函数,将其放在 `openflare_server/model/migrations.go` 或更合适的 model 文件中,并通过 `Context` 暴露给 `model/migrate`,避免子包反向 import `model` 造成循环依赖。
|
6. 如果新迁移需要新的公共 backfill 或校验辅助函数,优先放在该迁移文件中;只有多个迁移共同复用时,才放到 `openflare_server/model/goose` 包内的公共文件中。不要把新 goose 框架代码放回 `openflare_server/model` 根包。
|
||||||
7. 补充迁移测试:至少覆盖从 `N-1` 老库升级到 `N` 后 schema version、字段/表结构、关键数据回填和校验结果。注册表连续性由 `model/migrate` 测试兜底,但具体业务迁移仍必须有测试。
|
7. 补充迁移测试:至少覆盖从旧框架终点或上一 goose 版本升级后 schema version、字段/表结构、关键数据回填和校验结果。还应保留旧库从 v15/v17 桥接到 goose 的回归覆盖。
|
||||||
8. 同步更新设计/开发文档;如果管理端 API、配置项或用户可见行为变化,还要同步更新对应指南、配置参考和 Swagger 文档。
|
8. 同步更新设计/开发文档;如果管理端 API、配置项或用户可见行为变化,还要同步更新对应指南、配置参考和 Swagger 文档。
|
||||||
|
|
||||||
新包启动后必须先检查数据库当前版本,再按顺序逐步升级到目标版本;禁止跳过中间升级步骤直接写目标版本。
|
新包启动后必须先检查数据库当前版本,再按顺序逐步升级到目标版本;禁止跳过中间升级步骤直接写目标版本。
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
package model
|
package goose
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"database/sql"
|
"database/sql"
|
||||||
@@ -9,6 +9,17 @@ import (
|
|||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
type BridgeContext interface {
|
||||||
|
Context
|
||||||
|
AutoMigrateLegacySchemaMetadata(db *gorm.DB) error
|
||||||
|
InitializeFreshDatabaseSchema(db *gorm.DB, backend string) error
|
||||||
|
IsDatabaseEmpty(db *gorm.DB) (bool, error)
|
||||||
|
RepairCurrentSchemaState(db *gorm.DB, backend string) error
|
||||||
|
SaveLegacyDatabaseSchemaVersion(db *gorm.DB, version int) error
|
||||||
|
UpgradeLegacyDatabaseSchema(db *gorm.DB, backend string, version int) error
|
||||||
|
ValidateCurrentDatabaseSchema(db *gorm.DB, backend string) error
|
||||||
|
}
|
||||||
|
|
||||||
type schemaMigrationState int
|
type schemaMigrationState int
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -19,8 +30,8 @@ const (
|
|||||||
schemaMigrationStateMixed
|
schemaMigrationStateMixed
|
||||||
)
|
)
|
||||||
|
|
||||||
func detectLegacySchemaState(db *gorm.DB) (schemaMigrationState, error) {
|
func detectSchemaState(db *gorm.DB, ctx BridgeContext) (schemaMigrationState, error) {
|
||||||
hasLegacyTable := db.Migrator().HasTable(&DatabaseSchemaVersion{})
|
hasLegacyTable := db.Migrator().HasTable("database_schema_versions")
|
||||||
hasGooseTable := db.Migrator().HasTable("goose_db_version")
|
hasGooseTable := db.Migrator().HasTable("goose_db_version")
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
@@ -32,7 +43,7 @@ func detectLegacySchemaState(db *gorm.DB) (schemaMigrationState, error) {
|
|||||||
return schemaMigrationStateGooseOnly, nil
|
return schemaMigrationStateGooseOnly, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
empty, err := isDatabaseEmpty(db)
|
empty, err := ctx.IsDatabaseEmpty(db)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
@@ -42,7 +53,7 @@ func detectLegacySchemaState(db *gorm.DB) (schemaMigrationState, error) {
|
|||||||
return schemaMigrationStateLegacyBootstrap, nil
|
return schemaMigrationStateLegacyBootstrap, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadGooseDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
func LoadDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
||||||
if db == nil || !db.Migrator().HasTable("goose_db_version") {
|
if db == nil || !db.Migrator().HasTable("goose_db_version") {
|
||||||
return 0, false, nil
|
return 0, false, nil
|
||||||
}
|
}
|
||||||
@@ -64,8 +75,29 @@ func loadGooseDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
|||||||
return int(version), true, nil
|
return int(version), true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func bootstrapLegacySchemaVersion(db *gorm.DB) error {
|
func loadLegacyDatabaseSchemaVersion(db *gorm.DB) (int, bool, error) {
|
||||||
if err := autoMigrateLegacySchemaMetadata(db); err != nil {
|
if db == nil || !db.Migrator().HasTable("database_schema_versions") {
|
||||||
|
return 0, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var version int
|
||||||
|
err := db.Table("database_schema_versions").
|
||||||
|
Where("id = ?", 1).
|
||||||
|
Select("version").
|
||||||
|
Limit(1).
|
||||||
|
Row().
|
||||||
|
Scan(&version)
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return 0, false, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return 0, false, err
|
||||||
|
}
|
||||||
|
return version, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func bootstrapLegacySchemaVersion(db *gorm.DB, ctx BridgeContext) error {
|
||||||
|
if err := ctx.AutoMigrateLegacySchemaMetadata(db); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
version, exists, err := loadLegacyDatabaseSchemaVersion(db)
|
version, exists, err := loadLegacyDatabaseSchemaVersion(db)
|
||||||
@@ -73,16 +105,16 @@ func bootstrapLegacySchemaVersion(db *gorm.DB) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if exists {
|
if exists {
|
||||||
if version > legacyMigrationTerminalVersion {
|
if int64(version) > LegacyBridgeVersion {
|
||||||
return fmt.Errorf("legacy schema version %d is newer than supported terminal version %d", version, legacyMigrationTerminalVersion)
|
return fmt.Errorf("legacy schema version %d is newer than supported terminal version %d", version, LegacyBridgeVersion)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return saveLegacyDatabaseSchemaVersion(db, legacyDatabaseSchemaVersion)
|
return ctx.SaveLegacyDatabaseSchemaVersion(db, 7)
|
||||||
}
|
}
|
||||||
|
|
||||||
func upgradeLegacyToLegacyTerminal(db *gorm.DB, backend string) error {
|
func upgradeLegacyToTerminal(db *gorm.DB, backend string, ctx BridgeContext) error {
|
||||||
if err := bootstrapLegacySchemaVersion(db); err != nil {
|
if err := bootstrapLegacySchemaVersion(db, ctx); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
version, exists, err := loadLegacyDatabaseSchemaVersion(db)
|
version, exists, err := loadLegacyDatabaseSchemaVersion(db)
|
||||||
@@ -92,70 +124,64 @@ func upgradeLegacyToLegacyTerminal(db *gorm.DB, backend string) error {
|
|||||||
if !exists {
|
if !exists {
|
||||||
return fmt.Errorf("legacy schema version record is missing after bootstrap")
|
return fmt.Errorf("legacy schema version record is missing after bootstrap")
|
||||||
}
|
}
|
||||||
return upgradeLegacyDatabaseSchema(db, backend, version)
|
return ctx.UpgradeLegacyDatabaseSchema(db, backend, version)
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateGooseBridgeState(db *gorm.DB) error {
|
func validateGooseBridgeState(db *gorm.DB) error {
|
||||||
version, exists, err := loadGooseDatabaseVersion(db)
|
version, exists, err := LoadDatabaseVersion(db)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if !exists {
|
if !exists {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if version < legacyMigrationTerminalVersion {
|
if int64(version) < LegacyBridgeVersion {
|
||||||
return fmt.Errorf("goose schema version %d is below legacy bridge baseline %d", version, legacyMigrationTerminalVersion)
|
return fmt.Errorf("goose schema version %d is below legacy bridge baseline %d", version, LegacyBridgeVersion)
|
||||||
}
|
}
|
||||||
if int64(version) > currentGooseTargetVersion() {
|
if int64(version) > CurrentTargetVersion() {
|
||||||
return fmt.Errorf("goose schema version %d is newer than application target version %d", version, currentGooseTargetVersion())
|
return fmt.Errorf("goose schema version %d is newer than application target version %d", version, CurrentTargetVersion())
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func finalizeLegacyToGooseBridge(db *gorm.DB) error {
|
func finalizeLegacyToGooseBridge(db *gorm.DB) error {
|
||||||
gooseVersion, exists, err := loadGooseDatabaseVersion(db)
|
gooseVersion, exists, err := LoadDatabaseVersion(db)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if !exists || gooseVersion < legacyMigrationTerminalVersion {
|
if !exists || int64(gooseVersion) < LegacyBridgeVersion {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if !db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
if !db.Migrator().HasTable("database_schema_versions") {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if err := db.Migrator().DropTable(&DatabaseSchemaVersion{}); err != nil {
|
if err := db.Exec("DROP TABLE IF EXISTS database_schema_versions").Error; err != nil {
|
||||||
return fmt.Errorf("drop legacy schema versions table failed: %w", err)
|
return fmt.Errorf("drop legacy schema versions table failed: %w", err)
|
||||||
}
|
}
|
||||||
slog.Info("completed legacy-to-goose migration bridge", "goose_version", gooseVersion)
|
slog.Info("completed legacy-to-goose migration bridge", "goose_version", gooseVersion)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func repairCurrentSchemaState(db *gorm.DB, backend string) error {
|
func ValidateRegisteredSchema(db *gorm.DB) error {
|
||||||
if err := dropLegacyNodeColumns(db, backend); err != nil {
|
if err := validateNodeCapabilitiesJSON(db); err != nil {
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := ensureDefaultGitHubAuthSource(db); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := ensureDefaultWAFRuleGroup(db); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
func EnsureDatabaseSchemaUpToDate(db *gorm.DB, backend string, ctx BridgeContext) error {
|
||||||
state, err := detectLegacySchemaState(db)
|
state, err := detectSchemaState(db, ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
switch state {
|
switch state {
|
||||||
case schemaMigrationStateFresh:
|
case schemaMigrationStateFresh:
|
||||||
if err := initializeFreshDatabaseSchema(db, backend); err != nil {
|
if err := ctx.InitializeFreshDatabaseSchema(db, backend); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
case schemaMigrationStateLegacyOnly:
|
case schemaMigrationStateLegacyOnly:
|
||||||
if err := upgradeLegacyToLegacyTerminal(db, backend); err != nil {
|
if err := upgradeLegacyToTerminal(db, backend, ctx); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
case schemaMigrationStateGooseOnly:
|
case schemaMigrationStateGooseOnly:
|
||||||
@@ -163,7 +189,7 @@ func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
case schemaMigrationStateLegacyBootstrap:
|
case schemaMigrationStateLegacyBootstrap:
|
||||||
if err := upgradeLegacyToLegacyTerminal(db, backend); err != nil {
|
if err := upgradeLegacyToTerminal(db, backend, ctx); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
case schemaMigrationStateMixed:
|
case schemaMigrationStateMixed:
|
||||||
@@ -171,8 +197,8 @@ func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if exists && legacyVersion != legacyMigrationTerminalVersion {
|
if exists && int64(legacyVersion) != LegacyBridgeVersion {
|
||||||
return fmt.Errorf("incomplete mixed migration state: legacy schema version %d does not match bridge terminal version %d", legacyVersion, legacyMigrationTerminalVersion)
|
return fmt.Errorf("incomplete mixed migration state: legacy schema version %d does not match bridge terminal version %d", legacyVersion, LegacyBridgeVersion)
|
||||||
}
|
}
|
||||||
if err := validateGooseBridgeState(db); err != nil {
|
if err := validateGooseBridgeState(db); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -181,14 +207,17 @@ func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
|||||||
return fmt.Errorf("unknown schema migration state: %d", state)
|
return fmt.Errorf("unknown schema migration state: %d", state)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := runGooseMigrations(db, backend); err != nil {
|
if err := runMigrations(db, backend, ctx); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := finalizeLegacyToGooseBridge(db); err != nil {
|
if err := finalizeLegacyToGooseBridge(db); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := repairCurrentSchemaState(db, backend); err != nil {
|
if err := ctx.RepairCurrentSchemaState(db, backend); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return validateCurrentDatabaseSchema(db, backend)
|
if err := ctx.ValidateCurrentDatabaseSchema(db, backend); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return ValidateRegisteredSchema(db)
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
package goose
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
presslygoose "github.com/pressly/goose/v3"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
const versionNodeCapabilitiesJSON int64 = 202606020001
|
||||||
|
|
||||||
|
// migration202606020001 adds a future-proof JSON field for node capability
|
||||||
|
// summaries after the legacy v17 migration bridge.
|
||||||
|
func migration202606020001(backend string, ctx Context) *presslygoose.Migration {
|
||||||
|
return newGORMMigration(
|
||||||
|
versionNodeCapabilitiesJSON,
|
||||||
|
"202606020001_add_node_capabilities_json.go",
|
||||||
|
backend,
|
||||||
|
ctx,
|
||||||
|
migrateNodeCapabilitiesJSON,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func migrateNodeCapabilitiesJSON(ctx Context, db *gorm.DB, backend string) error {
|
||||||
|
if err := ctx.ApplyCurrentSchema(db, backend); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
emptyJSON, err := json.Marshal([]string{})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("marshal default node capabilities: %w", err)
|
||||||
|
}
|
||||||
|
if err := db.Exec(
|
||||||
|
`UPDATE nodes SET capabilities_json = ? WHERE capabilities_json IS NULL OR TRIM(capabilities_json) = ''`,
|
||||||
|
string(emptyJSON),
|
||||||
|
).Error; err != nil {
|
||||||
|
return fmt.Errorf("backfill nodes.capabilities_json: %w", err)
|
||||||
|
}
|
||||||
|
return validateNodeCapabilitiesJSON(db)
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateNodeCapabilitiesJSON(db *gorm.DB) error {
|
||||||
|
if db == nil {
|
||||||
|
return fmt.Errorf("database handle is nil")
|
||||||
|
}
|
||||||
|
if !db.Migrator().HasColumn("nodes", "capabilities_json") {
|
||||||
|
return fmt.Errorf("column nodes.capabilities_json is missing")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
package goose
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
presslygoose "github.com/pressly/goose/v3"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
const LegacyBridgeVersion int64 = 17
|
||||||
|
|
||||||
|
type migrationFunc func(ctx Context, db *gorm.DB, backend string) error
|
||||||
|
|
||||||
|
func newBaselineMigration() *presslygoose.Migration {
|
||||||
|
migration := presslygoose.NewGoMigration(LegacyBridgeVersion, nil, nil)
|
||||||
|
migration.Source = fmt.Sprintf("%05d_legacy_terminal_baseline.go", LegacyBridgeVersion)
|
||||||
|
return migration
|
||||||
|
}
|
||||||
|
|
||||||
|
func newGORMMigration(version int64, source string, backend string, ctx Context, up migrationFunc) *presslygoose.Migration {
|
||||||
|
migration := presslygoose.NewGoMigration(version, &presslygoose.GoFunc{
|
||||||
|
RunDB: func(_ context.Context, sqlDB *sql.DB) error {
|
||||||
|
gormDB, err := openGORMDB(ctx, sqlDB, backend)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if backend == "postgres" {
|
||||||
|
return gormDB.Transaction(func(tx *gorm.DB) error {
|
||||||
|
return up(ctx, tx, backend)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return up(ctx, gormDB, backend)
|
||||||
|
},
|
||||||
|
}, nil)
|
||||||
|
migration.Source = source
|
||||||
|
return migration
|
||||||
|
}
|
||||||
|
|
||||||
|
func registeredMigrations(backend string, ctx Context) []*presslygoose.Migration {
|
||||||
|
return []*presslygoose.Migration{
|
||||||
|
migration202606020001(backend, ctx),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildMigrations(backend string, ctx Context) []*presslygoose.Migration {
|
||||||
|
migrations := []*presslygoose.Migration{newBaselineMigration()}
|
||||||
|
migrations = append(migrations, registeredMigrations(backend, ctx)...)
|
||||||
|
return migrations
|
||||||
|
}
|
||||||
|
|
||||||
|
func CurrentTargetVersion() int64 {
|
||||||
|
var maxVersion int64 = LegacyBridgeVersion
|
||||||
|
for _, migration := range buildMigrations("sqlite", noopContext{}) {
|
||||||
|
if migration.Version > maxVersion {
|
||||||
|
maxVersion = migration.Version
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return maxVersion
|
||||||
|
}
|
||||||
|
|
||||||
|
type noopContext struct{}
|
||||||
|
|
||||||
|
func (noopContext) ApplyCurrentSchema(db *gorm.DB, backend string) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (noopContext) RegisterSharding(db *gorm.DB, backend string) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
+19
-14
@@ -1,4 +1,4 @@
|
|||||||
package model
|
package goose
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -6,24 +6,29 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"github.com/glebarez/sqlite"
|
"github.com/glebarez/sqlite"
|
||||||
"github.com/pressly/goose/v3"
|
presslygoose "github.com/pressly/goose/v3"
|
||||||
"gorm.io/driver/postgres"
|
"gorm.io/driver/postgres"
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
"gorm.io/gorm/schema"
|
"gorm.io/gorm/schema"
|
||||||
)
|
)
|
||||||
|
|
||||||
func gooseDialectForBackend(backend string) (goose.Dialect, error) {
|
type Context interface {
|
||||||
|
ApplyCurrentSchema(db *gorm.DB, backend string) error
|
||||||
|
RegisterSharding(db *gorm.DB, backend string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
func dialectForBackend(backend string) (presslygoose.Dialect, error) {
|
||||||
switch backend {
|
switch backend {
|
||||||
case "postgres":
|
case "postgres":
|
||||||
return goose.DialectPostgres, nil
|
return presslygoose.DialectPostgres, nil
|
||||||
case "sqlite":
|
case "sqlite":
|
||||||
return goose.DialectSQLite3, nil
|
return presslygoose.DialectSQLite3, nil
|
||||||
default:
|
default:
|
||||||
return "", fmt.Errorf("unsupported database backend: %s", backend)
|
return "", fmt.Errorf("unsupported database backend: %s", backend)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func getGORMDBFromSQLDB(db *sql.DB, backend string) (*gorm.DB, error) {
|
func openGORMDB(ctx Context, db *sql.DB, backend string) (*gorm.DB, error) {
|
||||||
var dialector gorm.Dialector
|
var dialector gorm.Dialector
|
||||||
switch backend {
|
switch backend {
|
||||||
case "postgres":
|
case "postgres":
|
||||||
@@ -40,32 +45,32 @@ func getGORMDBFromSQLDB(db *sql.DB, backend string) (*gorm.DB, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if err := registerSharding(gormDB, backend); err != nil {
|
if err := ctx.RegisterSharding(gormDB, backend); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
return gormDB, nil
|
return gormDB, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildGooseProvider(db *gorm.DB, backend string) (*goose.Provider, error) {
|
func buildProvider(db *gorm.DB, backend string, ctx Context) (*presslygoose.Provider, error) {
|
||||||
sqlDB, err := db.DB()
|
sqlDB, err := db.DB()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
dialect, err := gooseDialectForBackend(backend)
|
dialect, err := dialectForBackend(backend)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
return goose.NewProvider(
|
return presslygoose.NewProvider(
|
||||||
dialect,
|
dialect,
|
||||||
sqlDB,
|
sqlDB,
|
||||||
nil,
|
nil,
|
||||||
goose.WithDisableGlobalRegistry(true),
|
presslygoose.WithDisableGlobalRegistry(true),
|
||||||
goose.WithGoMigrations(buildGooseMigrations(backend)...),
|
presslygoose.WithGoMigrations(buildMigrations(backend, ctx)...),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
func runGooseMigrations(db *gorm.DB, backend string) error {
|
func runMigrations(db *gorm.DB, backend string, ctx Context) error {
|
||||||
provider, err := buildGooseProvider(db, backend)
|
provider, err := buildProvider(db, backend, ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("build goose provider: %w", err)
|
return fmt.Errorf("build goose provider: %w", err)
|
||||||
}
|
}
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
package model
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"database/sql"
|
|
||||||
"fmt"
|
|
||||||
|
|
||||||
"github.com/pressly/goose/v3"
|
|
||||||
"gorm.io/gorm"
|
|
||||||
)
|
|
||||||
|
|
||||||
func newGooseBaselineMigration() *goose.Migration {
|
|
||||||
migration := goose.NewGoMigration(int64(legacyMigrationTerminalVersion), nil, nil)
|
|
||||||
migration.Source = fmt.Sprintf("%05d_legacy_terminal_baseline.go", legacyMigrationTerminalVersion)
|
|
||||||
return migration
|
|
||||||
}
|
|
||||||
|
|
||||||
func newGooseGORMMigration(
|
|
||||||
version int64,
|
|
||||||
source string,
|
|
||||||
backend string,
|
|
||||||
up func(ctx databaseSchemaMigrationContext, db *gorm.DB, backend string) error,
|
|
||||||
) *goose.Migration {
|
|
||||||
migration := goose.NewGoMigration(version, &goose.GoFunc{
|
|
||||||
RunDB: func(_ context.Context, sqlDB *sql.DB) error {
|
|
||||||
gormDB, err := getGORMDBFromSQLDB(sqlDB, backend)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
ctx := databaseSchemaMigrationContext{}
|
|
||||||
if backend == "postgres" {
|
|
||||||
return gormDB.Transaction(func(tx *gorm.DB) error {
|
|
||||||
return up(ctx, tx, backend)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return up(ctx, gormDB, backend)
|
|
||||||
},
|
|
||||||
}, nil)
|
|
||||||
migration.Source = source
|
|
||||||
return migration
|
|
||||||
}
|
|
||||||
|
|
||||||
func registeredGooseMigrations(backend string) []*goose.Migration {
|
|
||||||
_ = backend
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func buildGooseMigrations(backend string) []*goose.Migration {
|
|
||||||
migrations := []*goose.Migration{newGooseBaselineMigration()}
|
|
||||||
migrations = append(migrations, registeredGooseMigrations(backend)...)
|
|
||||||
return migrations
|
|
||||||
}
|
|
||||||
|
|
||||||
func currentGooseTargetVersion() int64 {
|
|
||||||
var maxVersion int64 = legacyMigrationTerminalVersion
|
|
||||||
for _, migration := range buildGooseMigrations("sqlite") {
|
|
||||||
if migration.Version > maxVersion {
|
|
||||||
maxVersion = migration.Version
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return maxVersion
|
|
||||||
}
|
|
||||||
@@ -89,6 +89,10 @@ func findDBModelByTableName(t *testing.T, tableName string) dbModel {
|
|||||||
return dbModel{}
|
return dbModel{}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func expectedCurrentDatabaseVersion() int {
|
||||||
|
return int(currentGooseTargetVersion())
|
||||||
|
}
|
||||||
|
|
||||||
func TestIsDatabaseEmpty(t *testing.T) {
|
func TestIsDatabaseEmpty(t *testing.T) {
|
||||||
db := openTestSQLiteDB(t, "empty.db")
|
db := openTestSQLiteDB(t, "empty.db")
|
||||||
|
|
||||||
@@ -303,8 +307,8 @@ func TestEnsureDatabaseSchemaUpToDateInitializesFreshDatabase(t *testing.T) {
|
|||||||
if !exists {
|
if !exists {
|
||||||
t.Fatal("expected database schema version to be recorded")
|
t.Fatal("expected database schema version to be recorded")
|
||||||
}
|
}
|
||||||
if version != currentDatabaseSchemaVersion {
|
if version != expectedCurrentDatabaseVersion() {
|
||||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||||
}
|
}
|
||||||
if db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
if db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
||||||
t.Fatal("expected fresh database to avoid legacy database_schema_versions table")
|
t.Fatal("expected fresh database to avoid legacy database_schema_versions table")
|
||||||
@@ -312,6 +316,9 @@ func TestEnsureDatabaseSchemaUpToDateInitializesFreshDatabase(t *testing.T) {
|
|||||||
if !db.Migrator().HasTable("goose_db_version") {
|
if !db.Migrator().HasTable("goose_db_version") {
|
||||||
t.Fatal("expected fresh database to initialize goose_db_version")
|
t.Fatal("expected fresh database to initialize goose_db_version")
|
||||||
}
|
}
|
||||||
|
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||||
|
t.Fatal("expected fresh database to apply goose migration nodes.capabilities_json")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
||||||
@@ -343,8 +350,8 @@ func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
|||||||
if !exists {
|
if !exists {
|
||||||
t.Fatal("expected legacy database to gain a schema version record")
|
t.Fatal("expected legacy database to gain a schema version record")
|
||||||
}
|
}
|
||||||
if version != currentDatabaseSchemaVersion {
|
if version != expectedCurrentDatabaseVersion() {
|
||||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||||
}
|
}
|
||||||
if db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
if db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
||||||
t.Fatal("expected legacy database_schema_versions table to be removed after bridging to goose")
|
t.Fatal("expected legacy database_schema_versions table to be removed after bridging to goose")
|
||||||
@@ -352,6 +359,9 @@ func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
|||||||
if !db.Migrator().HasTable("goose_db_version") {
|
if !db.Migrator().HasTable("goose_db_version") {
|
||||||
t.Fatal("expected legacy upgrade to initialize goose_db_version")
|
t.Fatal("expected legacy upgrade to initialize goose_db_version")
|
||||||
}
|
}
|
||||||
|
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||||
|
t.Fatal("expected legacy upgrade to apply goose migration nodes.capabilities_json")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMigrateOriginsSchemaBackfillsOrigins(t *testing.T) {
|
func TestMigrateOriginsSchemaBackfillsOrigins(t *testing.T) {
|
||||||
@@ -544,8 +554,11 @@ func TestEnsureDatabaseSchemaUpToDateAddsNodeIPManualOverride(t *testing.T) {
|
|||||||
if !exists {
|
if !exists {
|
||||||
t.Fatal("expected schema version record to exist")
|
t.Fatal("expected schema version record to exist")
|
||||||
}
|
}
|
||||||
if version != currentDatabaseSchemaVersion {
|
if version != expectedCurrentDatabaseVersion() {
|
||||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||||
|
}
|
||||||
|
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||||
|
t.Fatal("expected migration chain to include nodes.capabilities_json")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -628,8 +641,11 @@ func TestEnsureDatabaseSchemaUpToDateV16BackfillsNodeColumnsWhenNewColumnsAlread
|
|||||||
if !exists {
|
if !exists {
|
||||||
t.Fatal("expected schema version record to exist")
|
t.Fatal("expected schema version record to exist")
|
||||||
}
|
}
|
||||||
if version != currentDatabaseSchemaVersion {
|
if version != expectedCurrentDatabaseVersion() {
|
||||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||||
|
}
|
||||||
|
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||||
|
t.Fatal("expected v16 upgrade path to apply goose migration nodes.capabilities_json")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -673,6 +689,9 @@ func TestEnsureDatabaseSchemaUpToDateV16DropsLegacyNodeColumnsWhenAlreadyCurrent
|
|||||||
if !db.Migrator().HasTable("goose_db_version") {
|
if !db.Migrator().HasTable("goose_db_version") {
|
||||||
t.Fatal("expected current-schema goose_db_version table to exist")
|
t.Fatal("expected current-schema goose_db_version table to exist")
|
||||||
}
|
}
|
||||||
|
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||||
|
t.Fatal("expected current-schema repair to preserve goose column nodes.capabilities_json")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEnsureDatabaseSchemaUpToDateKeepsGooseOnlyDatabaseOnReentry(t *testing.T) {
|
func TestEnsureDatabaseSchemaUpToDateKeepsGooseOnlyDatabaseOnReentry(t *testing.T) {
|
||||||
@@ -702,8 +721,11 @@ func TestEnsureDatabaseSchemaUpToDateKeepsGooseOnlyDatabaseOnReentry(t *testing.
|
|||||||
if !exists {
|
if !exists {
|
||||||
t.Fatal("expected goose-only database to keep goose version record")
|
t.Fatal("expected goose-only database to keep goose version record")
|
||||||
}
|
}
|
||||||
if version != currentDatabaseSchemaVersion {
|
if version != expectedCurrentDatabaseVersion() {
|
||||||
t.Fatalf("unexpected goose version: got %d want %d", version, currentDatabaseSchemaVersion)
|
t.Fatalf("unexpected goose version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||||
|
}
|
||||||
|
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||||
|
t.Fatal("expected goose-only database to keep nodes.capabilities_json")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -37,6 +37,7 @@ type Node struct {
|
|||||||
RelayAgentAccessAddr string `json:"relay_agent_access_addr" gorm:"size:255"`
|
RelayAgentAccessAddr string `json:"relay_agent_access_addr" gorm:"size:255"`
|
||||||
RelayClientAccessAddr string `json:"relay_client_access_addr" gorm:"size:255"`
|
RelayClientAccessAddr string `json:"relay_client_access_addr" gorm:"size:255"`
|
||||||
RelayClientProxyURL string `json:"relay_client_proxy_url" gorm:"size:512"`
|
RelayClientProxyURL string `json:"relay_client_proxy_url" gorm:"size:512"`
|
||||||
|
CapabilitiesJSON string `json:"capabilities_json" gorm:"type:text;not null;default:'[]'"`
|
||||||
RelayStatus string `json:"relay_status" gorm:"size:16;not null;default:'unknown'"`
|
RelayStatus string `json:"relay_status" gorm:"size:16;not null;default:'unknown'"`
|
||||||
RelayWebServerEnabled bool `json:"relay_web_server_enabled" gorm:"not null;default:false"`
|
RelayWebServerEnabled bool `json:"relay_web_server_enabled" gorm:"not null;default:false"`
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
schemagoose "openflare/model/goose"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func currentGooseTargetVersion() int64 {
|
||||||
|
return schemagoose.CurrentTargetVersion()
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadGooseDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
||||||
|
return schemagoose.LoadDatabaseVersion(db)
|
||||||
|
}
|
||||||
|
|
||||||
|
func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
||||||
|
return schemagoose.EnsureDatabaseSchemaUpToDate(db, backend, databaseSchemaMigrationContext{})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) RegisterSharding(db *gorm.DB, backend string) error {
|
||||||
|
return registerSharding(db, backend)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) AutoMigrateLegacySchemaMetadata(db *gorm.DB) error {
|
||||||
|
return autoMigrateLegacySchemaMetadata(db)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) InitializeFreshDatabaseSchema(db *gorm.DB, backend string) error {
|
||||||
|
return initializeFreshDatabaseSchema(db, backend)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) IsDatabaseEmpty(db *gorm.DB) (bool, error) {
|
||||||
|
return isDatabaseEmpty(db)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) RepairCurrentSchemaState(db *gorm.DB, backend string) error {
|
||||||
|
if err := dropLegacyNodeColumns(db, backend); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := ensureDefaultGitHubAuthSource(db); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := ensureDefaultWAFRuleGroup(db); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) SaveLegacyDatabaseSchemaVersion(db *gorm.DB, version int) error {
|
||||||
|
return saveLegacyDatabaseSchemaVersion(db, version)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) UpgradeLegacyDatabaseSchema(db *gorm.DB, backend string, version int) error {
|
||||||
|
return upgradeLegacyDatabaseSchema(db, backend, version)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (databaseSchemaMigrationContext) ValidateCurrentDatabaseSchema(db *gorm.DB, backend string) error {
|
||||||
|
return validateCurrentDatabaseSchema(db, backend)
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user