mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-09-29 22:06:38 +08:00
[优化] 重构数据库迁移逻辑,整合遗留和Goose迁移处理
This commit is contained in:
@@ -179,4 +179,4 @@ go build -o openflare-agent ./cmd/agent
|
||||
4. 涉及配置、部署、API 或产品边界时同步更新文档。
|
||||
5. 风险较高的修改补充测试或等效联调验证。
|
||||
|
||||
数据库结构变更必须提升数据库版本号,并补充从上一版本到新版本的显式迁移方法和校验逻辑。
|
||||
数据库结构变更必须提升数据库版本号,并补充显式迁移方法和校验逻辑。v8-v17 保留为旧升级框架兼容链;v17 之后统一使用 goose,新的 goose 框架代码必须集中在 `openflare_server/model/goose` 包下;每次数据库升级都要在该包下新增独立的 `goose_<timestamp>_<description>.go` 文件,不得把具体迁移逻辑集中堆在 goose 注册入口中,也不得把新 goose 框架代码放回 `openflare_server/model` 根包。
|
||||
|
||||
@@ -112,17 +112,19 @@ Frontend:
|
||||
|
||||
每次提升数据库版本号时,必须补充从上一版本升级到新版本的显式迁移方法。迁移方法必须包含升级后的校验逻辑;只有校验通过,才能写入新的数据库版本记录。
|
||||
|
||||
v1-v7 视为历史初始基线,不再维护逐版本升级文件。从 v8 起,数据库迁移必须放在 `openflare_server/model/migrate` 目录中,并以目标版本命名文件,例如 `v16.go`。每个版本文件通过 `init()` 注册自己的迁移,当前数据库版本取已注册迁移的最大目标版本。不得为了整理文件而改变已发布 v8+ 迁移的语义。
|
||||
v1-v7 视为历史初始基线,不再维护逐版本升级文件。v8-v17 是旧升级框架的兼容迁移链,只保留在 `openflare_server/model/migrate` 目录中用于老库升级。旧库启动时必须先按旧框架升级到 `legacyMigrationTerminalVersion`,再桥接到 goose;不得为了整理文件而改变已发布 v8-v17 迁移的语义。
|
||||
|
||||
从 v17 之后,数据库升级统一使用 goose。新的 goose provider、桥接逻辑、注册入口和具体迁移文件必须全部放在 `openflare_server/model/goose` 包下,`openflare_server/model` 根包只保留纯净实体类、旧框架兼容适配和必要的上下文注入。每次新增数据库升级都必须新建一个单独的 Go 文件,文件名使用 `openflare_server/model/goose/goose_<timestamp>_<description>.go`,例如 `openflare_server/model/goose/goose_202606020001_add_node_capabilities_json.go`。迁移文件必须同时包含该版本的 goose migration 构造函数、升级逻辑和校验逻辑;`model/goose/migrations.go` 只能作为注册入口和公共构造工具,禁止把具体迁移逻辑集中堆放在该文件中。
|
||||
|
||||
执行数据库升级时必须按以下步骤完成:
|
||||
|
||||
1. 判断是否需要升级数据库版本:凡是新增/删除/重命名表、字段、索引、约束、列类型、分表规则,或改变持久化数据语义,都必须升级。
|
||||
2. 新增 `openflare_server/model/migrate/vN.go`,其中 `N` 为目标版本号。文件头部必须包含注释,说明本次升级了什么内容,以及为什么需要升级。
|
||||
3. 在 `vN.go` 中实现 `VN()`,并在 `init()` 中调用 `Register(VN())`。`FromVersion` 必须等于 `N-1`,`ToVersion` 必须等于 `N`。
|
||||
4. 在 `migrateVN` 中写入升级逻辑。可通过 `Context` 调用 `ApplyCurrentSchema`、历史 backfill、默认数据初始化等公共能力;复杂数据修复必须显式处理,不得只依赖 `AutoMigrate`。
|
||||
5. 在 `validateVN` 中写入升级后的校验逻辑。校验至少要覆盖新增表/字段/索引是否存在、关键默认数据是否存在、必要的数据回填是否成功。
|
||||
6. 如果新迁移需要新的公共 backfill 或校验辅助函数,将其放在 `openflare_server/model/migrations.go` 或更合适的 model 文件中,并通过 `Context` 暴露给 `model/migrate`,避免子包反向 import `model` 造成循环依赖。
|
||||
7. 补充迁移测试:至少覆盖从 `N-1` 老库升级到 `N` 后 schema version、字段/表结构、关键数据回填和校验结果。注册表连续性由 `model/migrate` 测试兜底,但具体业务迁移仍必须有测试。
|
||||
2. 新增 `openflare_server/model/goose/goose_<timestamp>_<description>.go`,其中 `<timestamp>` 为 goose 版本号。文件头部或迁移构造函数附近必须包含注释,说明本次升级了什么内容,以及为什么需要升级。
|
||||
3. 在该文件中实现独立迁移构造函数,并返回通过 `newGORMMigration(...)` 创建的 migration;随后只在 `openflare_server/model/goose/migrations.go` 的 `registeredMigrations(...)` 中新增一条注册项。
|
||||
4. 在同一个单独迁移文件中写入升级逻辑。可通过 goose `Context` 调用 `ApplyCurrentSchema`、历史 backfill、默认数据初始化等公共能力;复杂数据修复必须显式处理,不得只依赖 `AutoMigrate`。
|
||||
5. 在同一个单独迁移文件中写入升级后的校验逻辑。校验至少要覆盖新增表/字段/索引是否存在、关键默认数据是否存在、必要的数据回填是否成功。
|
||||
6. 如果新迁移需要新的公共 backfill 或校验辅助函数,优先放在该迁移文件中;只有多个迁移共同复用时,才放到 `openflare_server/model/goose` 包内的公共文件中。不要把新 goose 框架代码放回 `openflare_server/model` 根包。
|
||||
7. 补充迁移测试:至少覆盖从旧框架终点或上一 goose 版本升级后 schema version、字段/表结构、关键数据回填和校验结果。还应保留旧库从 v15/v17 桥接到 goose 的回归覆盖。
|
||||
8. 同步更新设计/开发文档;如果管理端 API、配置项或用户可见行为变化,还要同步更新对应指南、配置参考和 Swagger 文档。
|
||||
|
||||
新包启动后必须先检查数据库当前版本,再按顺序逐步升级到目标版本;禁止跳过中间升级步骤直接写目标版本。
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package model
|
||||
package goose
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
@@ -9,6 +9,17 @@ import (
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type BridgeContext interface {
|
||||
Context
|
||||
AutoMigrateLegacySchemaMetadata(db *gorm.DB) error
|
||||
InitializeFreshDatabaseSchema(db *gorm.DB, backend string) error
|
||||
IsDatabaseEmpty(db *gorm.DB) (bool, error)
|
||||
RepairCurrentSchemaState(db *gorm.DB, backend string) error
|
||||
SaveLegacyDatabaseSchemaVersion(db *gorm.DB, version int) error
|
||||
UpgradeLegacyDatabaseSchema(db *gorm.DB, backend string, version int) error
|
||||
ValidateCurrentDatabaseSchema(db *gorm.DB, backend string) error
|
||||
}
|
||||
|
||||
type schemaMigrationState int
|
||||
|
||||
const (
|
||||
@@ -19,8 +30,8 @@ const (
|
||||
schemaMigrationStateMixed
|
||||
)
|
||||
|
||||
func detectLegacySchemaState(db *gorm.DB) (schemaMigrationState, error) {
|
||||
hasLegacyTable := db.Migrator().HasTable(&DatabaseSchemaVersion{})
|
||||
func detectSchemaState(db *gorm.DB, ctx BridgeContext) (schemaMigrationState, error) {
|
||||
hasLegacyTable := db.Migrator().HasTable("database_schema_versions")
|
||||
hasGooseTable := db.Migrator().HasTable("goose_db_version")
|
||||
|
||||
switch {
|
||||
@@ -32,7 +43,7 @@ func detectLegacySchemaState(db *gorm.DB) (schemaMigrationState, error) {
|
||||
return schemaMigrationStateGooseOnly, nil
|
||||
}
|
||||
|
||||
empty, err := isDatabaseEmpty(db)
|
||||
empty, err := ctx.IsDatabaseEmpty(db)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
@@ -42,7 +53,7 @@ func detectLegacySchemaState(db *gorm.DB) (schemaMigrationState, error) {
|
||||
return schemaMigrationStateLegacyBootstrap, nil
|
||||
}
|
||||
|
||||
func loadGooseDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
||||
func LoadDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
||||
if db == nil || !db.Migrator().HasTable("goose_db_version") {
|
||||
return 0, false, nil
|
||||
}
|
||||
@@ -64,8 +75,29 @@ func loadGooseDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
||||
return int(version), true, nil
|
||||
}
|
||||
|
||||
func bootstrapLegacySchemaVersion(db *gorm.DB) error {
|
||||
if err := autoMigrateLegacySchemaMetadata(db); err != nil {
|
||||
func loadLegacyDatabaseSchemaVersion(db *gorm.DB) (int, bool, error) {
|
||||
if db == nil || !db.Migrator().HasTable("database_schema_versions") {
|
||||
return 0, false, nil
|
||||
}
|
||||
|
||||
var version int
|
||||
err := db.Table("database_schema_versions").
|
||||
Where("id = ?", 1).
|
||||
Select("version").
|
||||
Limit(1).
|
||||
Row().
|
||||
Scan(&version)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return 0, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return 0, false, err
|
||||
}
|
||||
return version, true, nil
|
||||
}
|
||||
|
||||
func bootstrapLegacySchemaVersion(db *gorm.DB, ctx BridgeContext) error {
|
||||
if err := ctx.AutoMigrateLegacySchemaMetadata(db); err != nil {
|
||||
return err
|
||||
}
|
||||
version, exists, err := loadLegacyDatabaseSchemaVersion(db)
|
||||
@@ -73,16 +105,16 @@ func bootstrapLegacySchemaVersion(db *gorm.DB) error {
|
||||
return err
|
||||
}
|
||||
if exists {
|
||||
if version > legacyMigrationTerminalVersion {
|
||||
return fmt.Errorf("legacy schema version %d is newer than supported terminal version %d", version, legacyMigrationTerminalVersion)
|
||||
if int64(version) > LegacyBridgeVersion {
|
||||
return fmt.Errorf("legacy schema version %d is newer than supported terminal version %d", version, LegacyBridgeVersion)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return saveLegacyDatabaseSchemaVersion(db, legacyDatabaseSchemaVersion)
|
||||
return ctx.SaveLegacyDatabaseSchemaVersion(db, 7)
|
||||
}
|
||||
|
||||
func upgradeLegacyToLegacyTerminal(db *gorm.DB, backend string) error {
|
||||
if err := bootstrapLegacySchemaVersion(db); err != nil {
|
||||
func upgradeLegacyToTerminal(db *gorm.DB, backend string, ctx BridgeContext) error {
|
||||
if err := bootstrapLegacySchemaVersion(db, ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
version, exists, err := loadLegacyDatabaseSchemaVersion(db)
|
||||
@@ -92,70 +124,64 @@ func upgradeLegacyToLegacyTerminal(db *gorm.DB, backend string) error {
|
||||
if !exists {
|
||||
return fmt.Errorf("legacy schema version record is missing after bootstrap")
|
||||
}
|
||||
return upgradeLegacyDatabaseSchema(db, backend, version)
|
||||
return ctx.UpgradeLegacyDatabaseSchema(db, backend, version)
|
||||
}
|
||||
|
||||
func validateGooseBridgeState(db *gorm.DB) error {
|
||||
version, exists, err := loadGooseDatabaseVersion(db)
|
||||
version, exists, err := LoadDatabaseVersion(db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !exists {
|
||||
return nil
|
||||
}
|
||||
if version < legacyMigrationTerminalVersion {
|
||||
return fmt.Errorf("goose schema version %d is below legacy bridge baseline %d", version, legacyMigrationTerminalVersion)
|
||||
if int64(version) < LegacyBridgeVersion {
|
||||
return fmt.Errorf("goose schema version %d is below legacy bridge baseline %d", version, LegacyBridgeVersion)
|
||||
}
|
||||
if int64(version) > currentGooseTargetVersion() {
|
||||
return fmt.Errorf("goose schema version %d is newer than application target version %d", version, currentGooseTargetVersion())
|
||||
if int64(version) > CurrentTargetVersion() {
|
||||
return fmt.Errorf("goose schema version %d is newer than application target version %d", version, CurrentTargetVersion())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func finalizeLegacyToGooseBridge(db *gorm.DB) error {
|
||||
gooseVersion, exists, err := loadGooseDatabaseVersion(db)
|
||||
gooseVersion, exists, err := LoadDatabaseVersion(db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !exists || gooseVersion < legacyMigrationTerminalVersion {
|
||||
if !exists || int64(gooseVersion) < LegacyBridgeVersion {
|
||||
return nil
|
||||
}
|
||||
if !db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
||||
if !db.Migrator().HasTable("database_schema_versions") {
|
||||
return nil
|
||||
}
|
||||
if err := db.Migrator().DropTable(&DatabaseSchemaVersion{}); err != nil {
|
||||
if err := db.Exec("DROP TABLE IF EXISTS database_schema_versions").Error; err != nil {
|
||||
return fmt.Errorf("drop legacy schema versions table failed: %w", err)
|
||||
}
|
||||
slog.Info("completed legacy-to-goose migration bridge", "goose_version", gooseVersion)
|
||||
return nil
|
||||
}
|
||||
|
||||
func repairCurrentSchemaState(db *gorm.DB, backend string) error {
|
||||
if err := dropLegacyNodeColumns(db, backend); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureDefaultGitHubAuthSource(db); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureDefaultWAFRuleGroup(db); err != nil {
|
||||
func ValidateRegisteredSchema(db *gorm.DB) error {
|
||||
if err := validateNodeCapabilitiesJSON(db); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
||||
state, err := detectLegacySchemaState(db)
|
||||
func EnsureDatabaseSchemaUpToDate(db *gorm.DB, backend string, ctx BridgeContext) error {
|
||||
state, err := detectSchemaState(db, ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
switch state {
|
||||
case schemaMigrationStateFresh:
|
||||
if err := initializeFreshDatabaseSchema(db, backend); err != nil {
|
||||
if err := ctx.InitializeFreshDatabaseSchema(db, backend); err != nil {
|
||||
return err
|
||||
}
|
||||
case schemaMigrationStateLegacyOnly:
|
||||
if err := upgradeLegacyToLegacyTerminal(db, backend); err != nil {
|
||||
if err := upgradeLegacyToTerminal(db, backend, ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
case schemaMigrationStateGooseOnly:
|
||||
@@ -163,7 +189,7 @@ func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
||||
return err
|
||||
}
|
||||
case schemaMigrationStateLegacyBootstrap:
|
||||
if err := upgradeLegacyToLegacyTerminal(db, backend); err != nil {
|
||||
if err := upgradeLegacyToTerminal(db, backend, ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
case schemaMigrationStateMixed:
|
||||
@@ -171,8 +197,8 @@ func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if exists && legacyVersion != legacyMigrationTerminalVersion {
|
||||
return fmt.Errorf("incomplete mixed migration state: legacy schema version %d does not match bridge terminal version %d", legacyVersion, legacyMigrationTerminalVersion)
|
||||
if exists && int64(legacyVersion) != LegacyBridgeVersion {
|
||||
return fmt.Errorf("incomplete mixed migration state: legacy schema version %d does not match bridge terminal version %d", legacyVersion, LegacyBridgeVersion)
|
||||
}
|
||||
if err := validateGooseBridgeState(db); err != nil {
|
||||
return err
|
||||
@@ -181,14 +207,17 @@ func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
||||
return fmt.Errorf("unknown schema migration state: %d", state)
|
||||
}
|
||||
|
||||
if err := runGooseMigrations(db, backend); err != nil {
|
||||
if err := runMigrations(db, backend, ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := finalizeLegacyToGooseBridge(db); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := repairCurrentSchemaState(db, backend); err != nil {
|
||||
if err := ctx.RepairCurrentSchemaState(db, backend); err != nil {
|
||||
return err
|
||||
}
|
||||
return validateCurrentDatabaseSchema(db, backend)
|
||||
if err := ctx.ValidateCurrentDatabaseSchema(db, backend); err != nil {
|
||||
return err
|
||||
}
|
||||
return ValidateRegisteredSchema(db)
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package goose
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
presslygoose "github.com/pressly/goose/v3"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
const versionNodeCapabilitiesJSON int64 = 202606020001
|
||||
|
||||
// migration202606020001 adds a future-proof JSON field for node capability
|
||||
// summaries after the legacy v17 migration bridge.
|
||||
func migration202606020001(backend string, ctx Context) *presslygoose.Migration {
|
||||
return newGORMMigration(
|
||||
versionNodeCapabilitiesJSON,
|
||||
"202606020001_add_node_capabilities_json.go",
|
||||
backend,
|
||||
ctx,
|
||||
migrateNodeCapabilitiesJSON,
|
||||
)
|
||||
}
|
||||
|
||||
func migrateNodeCapabilitiesJSON(ctx Context, db *gorm.DB, backend string) error {
|
||||
if err := ctx.ApplyCurrentSchema(db, backend); err != nil {
|
||||
return err
|
||||
}
|
||||
emptyJSON, err := json.Marshal([]string{})
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal default node capabilities: %w", err)
|
||||
}
|
||||
if err := db.Exec(
|
||||
`UPDATE nodes SET capabilities_json = ? WHERE capabilities_json IS NULL OR TRIM(capabilities_json) = ''`,
|
||||
string(emptyJSON),
|
||||
).Error; err != nil {
|
||||
return fmt.Errorf("backfill nodes.capabilities_json: %w", err)
|
||||
}
|
||||
return validateNodeCapabilitiesJSON(db)
|
||||
}
|
||||
|
||||
func validateNodeCapabilitiesJSON(db *gorm.DB) error {
|
||||
if db == nil {
|
||||
return fmt.Errorf("database handle is nil")
|
||||
}
|
||||
if !db.Migrator().HasColumn("nodes", "capabilities_json") {
|
||||
return fmt.Errorf("column nodes.capabilities_json is missing")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package goose
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
|
||||
presslygoose "github.com/pressly/goose/v3"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
const LegacyBridgeVersion int64 = 17
|
||||
|
||||
type migrationFunc func(ctx Context, db *gorm.DB, backend string) error
|
||||
|
||||
func newBaselineMigration() *presslygoose.Migration {
|
||||
migration := presslygoose.NewGoMigration(LegacyBridgeVersion, nil, nil)
|
||||
migration.Source = fmt.Sprintf("%05d_legacy_terminal_baseline.go", LegacyBridgeVersion)
|
||||
return migration
|
||||
}
|
||||
|
||||
func newGORMMigration(version int64, source string, backend string, ctx Context, up migrationFunc) *presslygoose.Migration {
|
||||
migration := presslygoose.NewGoMigration(version, &presslygoose.GoFunc{
|
||||
RunDB: func(_ context.Context, sqlDB *sql.DB) error {
|
||||
gormDB, err := openGORMDB(ctx, sqlDB, backend)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if backend == "postgres" {
|
||||
return gormDB.Transaction(func(tx *gorm.DB) error {
|
||||
return up(ctx, tx, backend)
|
||||
})
|
||||
}
|
||||
return up(ctx, gormDB, backend)
|
||||
},
|
||||
}, nil)
|
||||
migration.Source = source
|
||||
return migration
|
||||
}
|
||||
|
||||
func registeredMigrations(backend string, ctx Context) []*presslygoose.Migration {
|
||||
return []*presslygoose.Migration{
|
||||
migration202606020001(backend, ctx),
|
||||
}
|
||||
}
|
||||
|
||||
func buildMigrations(backend string, ctx Context) []*presslygoose.Migration {
|
||||
migrations := []*presslygoose.Migration{newBaselineMigration()}
|
||||
migrations = append(migrations, registeredMigrations(backend, ctx)...)
|
||||
return migrations
|
||||
}
|
||||
|
||||
func CurrentTargetVersion() int64 {
|
||||
var maxVersion int64 = LegacyBridgeVersion
|
||||
for _, migration := range buildMigrations("sqlite", noopContext{}) {
|
||||
if migration.Version > maxVersion {
|
||||
maxVersion = migration.Version
|
||||
}
|
||||
}
|
||||
return maxVersion
|
||||
}
|
||||
|
||||
type noopContext struct{}
|
||||
|
||||
func (noopContext) ApplyCurrentSchema(db *gorm.DB, backend string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (noopContext) RegisterSharding(db *gorm.DB, backend string) error {
|
||||
return nil
|
||||
}
|
||||
+19
-14
@@ -1,4 +1,4 @@
|
||||
package model
|
||||
package goose
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -6,24 +6,29 @@ import (
|
||||
"fmt"
|
||||
|
||||
"github.com/glebarez/sqlite"
|
||||
"github.com/pressly/goose/v3"
|
||||
presslygoose "github.com/pressly/goose/v3"
|
||||
"gorm.io/driver/postgres"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/schema"
|
||||
)
|
||||
|
||||
func gooseDialectForBackend(backend string) (goose.Dialect, error) {
|
||||
type Context interface {
|
||||
ApplyCurrentSchema(db *gorm.DB, backend string) error
|
||||
RegisterSharding(db *gorm.DB, backend string) error
|
||||
}
|
||||
|
||||
func dialectForBackend(backend string) (presslygoose.Dialect, error) {
|
||||
switch backend {
|
||||
case "postgres":
|
||||
return goose.DialectPostgres, nil
|
||||
return presslygoose.DialectPostgres, nil
|
||||
case "sqlite":
|
||||
return goose.DialectSQLite3, nil
|
||||
return presslygoose.DialectSQLite3, nil
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported database backend: %s", backend)
|
||||
}
|
||||
}
|
||||
|
||||
func getGORMDBFromSQLDB(db *sql.DB, backend string) (*gorm.DB, error) {
|
||||
func openGORMDB(ctx Context, db *sql.DB, backend string) (*gorm.DB, error) {
|
||||
var dialector gorm.Dialector
|
||||
switch backend {
|
||||
case "postgres":
|
||||
@@ -40,32 +45,32 @@ func getGORMDBFromSQLDB(db *sql.DB, backend string) (*gorm.DB, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := registerSharding(gormDB, backend); err != nil {
|
||||
if err := ctx.RegisterSharding(gormDB, backend); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return gormDB, nil
|
||||
}
|
||||
|
||||
func buildGooseProvider(db *gorm.DB, backend string) (*goose.Provider, error) {
|
||||
func buildProvider(db *gorm.DB, backend string, ctx Context) (*presslygoose.Provider, error) {
|
||||
sqlDB, err := db.DB()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
dialect, err := gooseDialectForBackend(backend)
|
||||
dialect, err := dialectForBackend(backend)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return goose.NewProvider(
|
||||
return presslygoose.NewProvider(
|
||||
dialect,
|
||||
sqlDB,
|
||||
nil,
|
||||
goose.WithDisableGlobalRegistry(true),
|
||||
goose.WithGoMigrations(buildGooseMigrations(backend)...),
|
||||
presslygoose.WithDisableGlobalRegistry(true),
|
||||
presslygoose.WithGoMigrations(buildMigrations(backend, ctx)...),
|
||||
)
|
||||
}
|
||||
|
||||
func runGooseMigrations(db *gorm.DB, backend string) error {
|
||||
provider, err := buildGooseProvider(db, backend)
|
||||
func runMigrations(db *gorm.DB, backend string, ctx Context) error {
|
||||
provider, err := buildProvider(db, backend, ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("build goose provider: %w", err)
|
||||
}
|
||||
@@ -1,62 +0,0 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
|
||||
"github.com/pressly/goose/v3"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
func newGooseBaselineMigration() *goose.Migration {
|
||||
migration := goose.NewGoMigration(int64(legacyMigrationTerminalVersion), nil, nil)
|
||||
migration.Source = fmt.Sprintf("%05d_legacy_terminal_baseline.go", legacyMigrationTerminalVersion)
|
||||
return migration
|
||||
}
|
||||
|
||||
func newGooseGORMMigration(
|
||||
version int64,
|
||||
source string,
|
||||
backend string,
|
||||
up func(ctx databaseSchemaMigrationContext, db *gorm.DB, backend string) error,
|
||||
) *goose.Migration {
|
||||
migration := goose.NewGoMigration(version, &goose.GoFunc{
|
||||
RunDB: func(_ context.Context, sqlDB *sql.DB) error {
|
||||
gormDB, err := getGORMDBFromSQLDB(sqlDB, backend)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx := databaseSchemaMigrationContext{}
|
||||
if backend == "postgres" {
|
||||
return gormDB.Transaction(func(tx *gorm.DB) error {
|
||||
return up(ctx, tx, backend)
|
||||
})
|
||||
}
|
||||
return up(ctx, gormDB, backend)
|
||||
},
|
||||
}, nil)
|
||||
migration.Source = source
|
||||
return migration
|
||||
}
|
||||
|
||||
func registeredGooseMigrations(backend string) []*goose.Migration {
|
||||
_ = backend
|
||||
return nil
|
||||
}
|
||||
|
||||
func buildGooseMigrations(backend string) []*goose.Migration {
|
||||
migrations := []*goose.Migration{newGooseBaselineMigration()}
|
||||
migrations = append(migrations, registeredGooseMigrations(backend)...)
|
||||
return migrations
|
||||
}
|
||||
|
||||
func currentGooseTargetVersion() int64 {
|
||||
var maxVersion int64 = legacyMigrationTerminalVersion
|
||||
for _, migration := range buildGooseMigrations("sqlite") {
|
||||
if migration.Version > maxVersion {
|
||||
maxVersion = migration.Version
|
||||
}
|
||||
}
|
||||
return maxVersion
|
||||
}
|
||||
@@ -89,6 +89,10 @@ func findDBModelByTableName(t *testing.T, tableName string) dbModel {
|
||||
return dbModel{}
|
||||
}
|
||||
|
||||
func expectedCurrentDatabaseVersion() int {
|
||||
return int(currentGooseTargetVersion())
|
||||
}
|
||||
|
||||
func TestIsDatabaseEmpty(t *testing.T) {
|
||||
db := openTestSQLiteDB(t, "empty.db")
|
||||
|
||||
@@ -303,8 +307,8 @@ func TestEnsureDatabaseSchemaUpToDateInitializesFreshDatabase(t *testing.T) {
|
||||
if !exists {
|
||||
t.Fatal("expected database schema version to be recorded")
|
||||
}
|
||||
if version != currentDatabaseSchemaVersion {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
||||
if version != expectedCurrentDatabaseVersion() {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||
}
|
||||
if db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
||||
t.Fatal("expected fresh database to avoid legacy database_schema_versions table")
|
||||
@@ -312,6 +316,9 @@ func TestEnsureDatabaseSchemaUpToDateInitializesFreshDatabase(t *testing.T) {
|
||||
if !db.Migrator().HasTable("goose_db_version") {
|
||||
t.Fatal("expected fresh database to initialize goose_db_version")
|
||||
}
|
||||
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||
t.Fatal("expected fresh database to apply goose migration nodes.capabilities_json")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
||||
@@ -343,8 +350,8 @@ func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
||||
if !exists {
|
||||
t.Fatal("expected legacy database to gain a schema version record")
|
||||
}
|
||||
if version != currentDatabaseSchemaVersion {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
||||
if version != expectedCurrentDatabaseVersion() {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||
}
|
||||
if db.Migrator().HasTable(&DatabaseSchemaVersion{}) {
|
||||
t.Fatal("expected legacy database_schema_versions table to be removed after bridging to goose")
|
||||
@@ -352,6 +359,9 @@ func TestEnsureDatabaseSchemaUpToDateUpgradesLegacyDatabase(t *testing.T) {
|
||||
if !db.Migrator().HasTable("goose_db_version") {
|
||||
t.Fatal("expected legacy upgrade to initialize goose_db_version")
|
||||
}
|
||||
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||
t.Fatal("expected legacy upgrade to apply goose migration nodes.capabilities_json")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrateOriginsSchemaBackfillsOrigins(t *testing.T) {
|
||||
@@ -544,8 +554,11 @@ func TestEnsureDatabaseSchemaUpToDateAddsNodeIPManualOverride(t *testing.T) {
|
||||
if !exists {
|
||||
t.Fatal("expected schema version record to exist")
|
||||
}
|
||||
if version != currentDatabaseSchemaVersion {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
||||
if version != expectedCurrentDatabaseVersion() {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||
}
|
||||
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||
t.Fatal("expected migration chain to include nodes.capabilities_json")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -628,8 +641,11 @@ func TestEnsureDatabaseSchemaUpToDateV16BackfillsNodeColumnsWhenNewColumnsAlread
|
||||
if !exists {
|
||||
t.Fatal("expected schema version record to exist")
|
||||
}
|
||||
if version != currentDatabaseSchemaVersion {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, currentDatabaseSchemaVersion)
|
||||
if version != expectedCurrentDatabaseVersion() {
|
||||
t.Fatalf("unexpected schema version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||
}
|
||||
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||
t.Fatal("expected v16 upgrade path to apply goose migration nodes.capabilities_json")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -673,6 +689,9 @@ func TestEnsureDatabaseSchemaUpToDateV16DropsLegacyNodeColumnsWhenAlreadyCurrent
|
||||
if !db.Migrator().HasTable("goose_db_version") {
|
||||
t.Fatal("expected current-schema goose_db_version table to exist")
|
||||
}
|
||||
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||
t.Fatal("expected current-schema repair to preserve goose column nodes.capabilities_json")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureDatabaseSchemaUpToDateKeepsGooseOnlyDatabaseOnReentry(t *testing.T) {
|
||||
@@ -702,8 +721,11 @@ func TestEnsureDatabaseSchemaUpToDateKeepsGooseOnlyDatabaseOnReentry(t *testing.
|
||||
if !exists {
|
||||
t.Fatal("expected goose-only database to keep goose version record")
|
||||
}
|
||||
if version != currentDatabaseSchemaVersion {
|
||||
t.Fatalf("unexpected goose version: got %d want %d", version, currentDatabaseSchemaVersion)
|
||||
if version != expectedCurrentDatabaseVersion() {
|
||||
t.Fatalf("unexpected goose version: got %d want %d", version, expectedCurrentDatabaseVersion())
|
||||
}
|
||||
if !db.Migrator().HasColumn(&Node{}, "capabilities_json") {
|
||||
t.Fatal("expected goose-only database to keep nodes.capabilities_json")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@ type Node struct {
|
||||
RelayAgentAccessAddr string `json:"relay_agent_access_addr" gorm:"size:255"`
|
||||
RelayClientAccessAddr string `json:"relay_client_access_addr" gorm:"size:255"`
|
||||
RelayClientProxyURL string `json:"relay_client_proxy_url" gorm:"size:512"`
|
||||
CapabilitiesJSON string `json:"capabilities_json" gorm:"type:text;not null;default:'[]'"`
|
||||
RelayStatus string `json:"relay_status" gorm:"size:16;not null;default:'unknown'"`
|
||||
RelayWebServerEnabled bool `json:"relay_web_server_enabled" gorm:"not null;default:false"`
|
||||
}
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
schemagoose "openflare/model/goose"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
func currentGooseTargetVersion() int64 {
|
||||
return schemagoose.CurrentTargetVersion()
|
||||
}
|
||||
|
||||
func loadGooseDatabaseVersion(db *gorm.DB) (int, bool, error) {
|
||||
return schemagoose.LoadDatabaseVersion(db)
|
||||
}
|
||||
|
||||
func ensureDatabaseSchemaUpToDate(db *gorm.DB, backend string) error {
|
||||
return schemagoose.EnsureDatabaseSchemaUpToDate(db, backend, databaseSchemaMigrationContext{})
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) RegisterSharding(db *gorm.DB, backend string) error {
|
||||
return registerSharding(db, backend)
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) AutoMigrateLegacySchemaMetadata(db *gorm.DB) error {
|
||||
return autoMigrateLegacySchemaMetadata(db)
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) InitializeFreshDatabaseSchema(db *gorm.DB, backend string) error {
|
||||
return initializeFreshDatabaseSchema(db, backend)
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) IsDatabaseEmpty(db *gorm.DB) (bool, error) {
|
||||
return isDatabaseEmpty(db)
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) RepairCurrentSchemaState(db *gorm.DB, backend string) error {
|
||||
if err := dropLegacyNodeColumns(db, backend); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureDefaultGitHubAuthSource(db); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureDefaultWAFRuleGroup(db); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) SaveLegacyDatabaseSchemaVersion(db *gorm.DB, version int) error {
|
||||
return saveLegacyDatabaseSchemaVersion(db, version)
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) UpgradeLegacyDatabaseSchema(db *gorm.DB, backend string, version int) error {
|
||||
return upgradeLegacyDatabaseSchema(db, backend, version)
|
||||
}
|
||||
|
||||
func (databaseSchemaMigrationContext) ValidateCurrentDatabaseSchema(db *gorm.DB, backend string) error {
|
||||
return validateCurrentDatabaseSchema(db, backend)
|
||||
}
|
||||
Reference in New Issue
Block a user