mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-10-03 07:06:36 +08:00
feat(cordis): add OpenFlare Cordis 架构改造设计
docs(changelog): 修正表述笔误
refactor(cordis): 磁盘缓存改用上上游能力并清理本地副本
按上游/下游归属规约:类型断言守卫已回流 Wavelet(f3d85d5,附回归用例),
本仓库删除 OpenFlare/plugins/server/pkg/cache 整包并改 import 到
Wavelet/pkg/cache/disk,同步后与上游零漂移。
验证:go build 通过;go test ./... exit 0(137 包 ok);256 条路由对拍与
232 条 swagger 操作均零差异;make build-all 四进制;前端零改动。
docs(cordis): 记录 T1 清理结果与五个复用阻塞点
refactor(cordis): server 复用上游 pkg 能力并删除等价本地副本
按上游/下游归属规约清理重复实现,删除 7 个与上游等价的本地包并改 import:
shared/response→pkg/response、pkg/{logger,mail,trace,httppool,cache/ram}→
上游同名包、infra/persistence/batchwriter→pkg/batchwriter。逐项核过差异:
httppool 逐字节相同;logger 的 Config 字段完全一致;response 的 7 个 Abort*
一致;cache/ram 换过去顺带把裸 go 变回带 panic 恢复的 util.Go。
两处非等价差异按语义处理:
- batchwriter.Stats 与 status DTO 原为类型别名,改为消费侧逐字段转换,
避免 model 反向依赖基础设施类型;
- 上游 pkg/idgen 要求显式 Init(本地副本为懒加载自动初始化),本次保留本地
副本,待与 infra 初始化一并迁移(已登记在清理计划)。
验证:go build 通过;go test ./... exit 0(138 包 ok);256 条路由对拍零差异;
make swagger 232 条操作零增减,且归一化后与旧文档深度相等——差异仅为
response.Any / logger.LogEntry 两个定义名随包路径改名,接口形状未变。
chore(cordis): 回流内核与 pkg/util 通用能力并清理 vendoring 污染
按新增的上游/下游归属规约:HandleRaw/BasePath 与版本比较、网络、格式化助手
属通用能力,已提交到 Wavelet 分支 feat/cordis-router-raw-routes,本仓库改为
纯同步获取(pkg/util 已零漂移),补丁登记保留至上游合并。
同时修掉我此前 git add -A 造成的污染:首次 vendoring 把上游工作区里被
gitignore 的运行期产物一起提交进来(upload 的 diskcache 缓存块 650 个与
driver_http/dist 前端构建物 380 个,共 12872 行/1030 文件)。sync-upstream.sh
现显式排除 uploads/dist/data/*.db,.gitignore 补上对应兜底规则。
AGENTS.md 增加上游/下游改动归属规约,并把仍指向前 Cordis 布局的硬性约束
(internal/router + Serve、internal/repository/logstore、internal/platform/bootstrap、
internal/cmd)改到当前插件路径。
验证:go build 通过;go test ./... exit 0(144 包 ok);make swagger 232 条
操作与基线逐条一致;make build-all 四进制;gofmt 干净。
feat(cordis): server 插件化并改由内核挂载控制面路由
新增 plugins/server/plugin.go:Apply 以 ctx.Router().Group(app.api_prefix)
声明根级与 /v1 全部路由;33 个注册函数由 *gin.RouterGroup 改为
core.RouterExtension,RegisterCollection 改用内核新增的 HandleRaw 保留
尾部斜杠变体,AdminMiddlewares 返回 []any(Go 不允许把 []T 展开为 ...any)。
删除 router.Serve 与 registerRoutes,装配根改为 core.App +
driver_http.New(WithEngine(router.BuildEngine())),监听、信号与优雅退出归内核;
前端 SPA 的 NoRoute 兜底因内核暂无贡献点而保留在引擎层。
路由保真证据:plugin_parity_test 对拍 baseline/routes-engine.txt 的 256 条
(方法 路径) 零差异;go test ./... exit 0(144 包 ok,含真实 handler 的
openflare/integration 用例走同一条挂载路径);make swagger 232 条操作与基线
逐条一致;golangci-lint 0 issues;make build-all 四进制;embed_frontend
标签编译通过;前端零改动。
已知待补:带 Redis 的实机 HTTP 冒烟(本机 6379 未启动,session store 与
改造前一样在建店阶段即 fatal),以及 bootstrap 的任务/设置/迁移注册迁入 Apply。
feat(core): RouterExtension 增加 HandleRaw 与 BasePath 以保真尾部斜杠路由
server 插件化的前置:Handle 经 cleanPath 会剥掉尾部斜杠,无法表达
/resource 与 /resource/ 两条不同路由,而 OpenFlare 有 20 个历史 list
端点两者都注册且部署关闭了 RedirectTrailingSlash,缺失即 404。新增
HandleRaw 与 BasePath(作用域包装器同样登记反注册),补 extpoints 用例;
并把 router.Serve 拆出 BuildEngine 以便交给 driver_http.WithEngine 复用,
新增路由表导出 harness,固化 256 条 (方法 路径) 基线供插件化对拍。
上游补丁登记于 backend/OpenFlare/upstream-patches.md,同步脚本改为按目录
前缀输出差异并在同步后提醒确认补丁是否仍在。
验证:go build 通过;go test ./... exit 0(143 包 ok);gofmt 干净。
docs(cordis): 记录 server 插件接入内核的可行路径与内核能力缺口
feat(cordis): agent/relay/flared 落地为内核驱动插件
三个边缘守护进程各新增 plugin.go,实现 core.Plugin + core.Driver
(自定义 DriverType 与同名 profile),装配与生命周期从 main 迁入
Apply/Start/Stop:Apply 负责 JSON 配置加载、运行环境与用户确保、
openresty/frps/frpc 管理器与各服务装配;Start 以 util.Go 拉起阻塞式
runner 与 GeoIP 周期更新;Stop 收敛主循环结果并在超时时报错而非静默。
入口改为 core.NewApp(core.WithProfile(...)) + Prepare/Run,保持
-config 旗标、默认路径、退出码与启动/停止日志不变。
验证:go build 通过;go test ./... exit 0(143 包 ok,含 3 个插件身份
与配置失败路径测试);make build-all 四进制产出;三进制实跑缺失配置
均 exit 1 且错误链保留 load {agent,relay,flared} config 原因;gofmt 干净。
refactor(cordis): 按功能职责拆分为 4 个插件与 share 共享层
backend/OpenFlare 不再平铺遗留分层,改为 plugins/{server,agent,relay,flared}
加 share/:控制面业务(openflare/admin/oauth/user/upload/cap/config/health 与
repository/model/infra/router 等支撑层)归 server;三个边缘守护进程各自成插件;
被两个以上插件消费的 protocol/geoip/wsclient/render/pagesarchive/edge 归 share。
同时把 pkg/util 与 buildinfo 合并回上游 pkg(上游已覆盖全部符号,仅 8 个函数与
2 个类型为 OpenFlare 独有,已一并迁入),装配根统一到 backend/cmd(含三个 daemon
入口),Dockerfile 与 release 工作流的构建路径和 -X 注入路径同步更新。
验证:go build 通过;go test ./... exit 0(141 包 ok);make swagger exit 0 且
232 条 API 操作与基线逐条一致;make build-all 产出 4 进制;-X 注入经二进制
strings 实测生效;日志后端直连门禁改写为按 server 插件业务域扫描并在扫描数为 0
时报错(防门禁静默失效);前端零改动。
feat(cordis): 落地 backend/share 共享层与上游同步脚本
跨插件共享资源(控制消息协议、GeoIP+iputil、边缘守护进程日志)从下游包
移入 backend/share,并声明其只能依赖 core/pkg 与标准/第三方库,禁止反向
引用下游业务与具体插件实现;新增 scripts/sync-upstream.sh 只覆盖
backend/{core,pkg,plugins},同步后 --check 报告零差异,证明与上游逐字一致。
go build 通过,go test ./... exit 0(142 包 ok),前端零改动。
refactor(cordis): 采用与 Wavelet 同构的单模块布局并引入上游内核
按上游结构落位:backend/{core,pkg,plugins} 为 Wavelet 上游拷贝,OpenFlare
全部业务收拢到上游 downstream 所对应的位置 backend/OpenFlare/,模块名保持
Wavelet 以保证上游 import 路径逐字一致、同步零改写;三个 daemon 入口移至
backend/OpenFlare/cmd,backend/cmd 与 main.go 作为控制面装配根。
行为不变:go build 通过,142 个测试包全绿(含上游插件测试),232 条 API
操作与改造前逐条一致,四进制产物正常,前端零改动。swagger 暂只扫描下游代码,
待 P4 挂载上游路由后再纳入 plugins/。
style: 修正模块路径改写导致的 import 分组排序漂移
refactor(layout): Go 代码迁入 backend/ 并将模块名简化为 OpenFlare
对齐上游 Wavelet 的仓库布局,为以第二 module 形态 vendoring Cordis 内核与
平台插件做准备:模块路径整体改写为 OpenFlare,Go 目标加 cd backend,
swaggo 产物移至 backend/docs 并把 json/yaml 复制回 docs/ 供站点消费,
Dockerfile 与 release 工作流的构建目录、ldflags 模块路径同步更新。
行为保持不变:232 条路由与改造前逐条一致,95 个测试包全绿,
四进制产物正常,前端零改动。
chore(cordis): 落地改造计划与 schema/路由基线
新增 legacy_dump_test 迁移快照 harness:在临时 sqlite 库上按生产顺序
(goose.UpTo → zone 导入 → goose.Up)跑完 76 个历史迁移并导出 schema 与
版本序列,作为改造前后一致性门禁的唯一事实来源。同时记录 232 条路由清单
与 foundation 实施计划。
docs(cordis): add OpenFlare Cordis 架构改造设计
明确上游以第二 module 形态 vendoring 进 backend/Wavelet、4 个插件
(server/agent/relay/flared) 全部装载内核,并规定保留 76 个历史 goose
迁移 + 一次性版本 stamp 桥接的迁移方案,配套三方 schema 一致性门禁,
确保已部署库不重跑历史、不丢数据。
This commit is contained in:
@@ -0,0 +1,150 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Package database 提供数据库连接与基础设施
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
clickhouseDriver "gorm.io/driver/clickhouse"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/plugin/opentelemetry/tracing"
|
||||
)
|
||||
|
||||
const (
|
||||
clickhouseMaxExecTime = 60 // ClickHouse 最大执行时间(秒)
|
||||
clickhouseReadTimeoutFactor = 2 // ReadTimeout 为 DialTimeout 的倍数
|
||||
)
|
||||
|
||||
var (
|
||||
// ChConn ClickHouse 原生连接实例,用于批量写入
|
||||
ChConn driver.Conn
|
||||
|
||||
chDB *gorm.DB
|
||||
)
|
||||
|
||||
// InitClickHouseWithConfig initializes the ClickHouse connection using the provided configuration.
|
||||
func InitClickHouseWithConfig(cfg ClickHouseConfig) error {
|
||||
if !cfg.Enabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
if cfg.Database == "" {
|
||||
return fmt.Errorf("[ClickHouse] database name is required (expected: wavelet)")
|
||||
}
|
||||
|
||||
opts := buildClickHouseOptions(cfg)
|
||||
|
||||
var err error
|
||||
ChConn, err = clickhouse.Open(opts)
|
||||
if err != nil {
|
||||
return fmt.Errorf("[ClickHouse] init connection failed: %w", err)
|
||||
}
|
||||
|
||||
if err = ChConn.Ping(context.Background()); err != nil {
|
||||
return fmt.Errorf("[ClickHouse] ping failed: %w", err)
|
||||
}
|
||||
|
||||
chDB, err = gorm.Open(clickhouseDriver.New(clickhouseDriver.Config{
|
||||
DSN: buildClickHouseDSN(cfg),
|
||||
}), &gorm.Config{
|
||||
SkipDefaultTransaction: true,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("[ClickHouse] init gorm connection failed: %w", err)
|
||||
}
|
||||
|
||||
if err = chDB.Use(
|
||||
tracing.NewPlugin(
|
||||
tracing.WithoutMetrics(),
|
||||
tracing.WithAttributes(
|
||||
attribute.String("db.instance", cfg.Database),
|
||||
attribute.String("db.system", "ClickHouse"),
|
||||
),
|
||||
),
|
||||
); err != nil {
|
||||
return fmt.Errorf("[ClickHouse] init trace failed: %w", err)
|
||||
}
|
||||
|
||||
sqlDB, err := chDB.DB()
|
||||
if err != nil {
|
||||
return fmt.Errorf("[ClickHouse] load sql db failed: %w", err)
|
||||
}
|
||||
|
||||
sqlDB.SetMaxIdleConns(cfg.MaxIdleConn)
|
||||
sqlDB.SetMaxOpenConns(cfg.MaxOpenConn)
|
||||
sqlDB.SetConnMaxLifetime(time.Duration(cfg.ConnMaxLifetime) * time.Second)
|
||||
|
||||
log.Println("[ClickHouse] connection established successfully")
|
||||
return nil
|
||||
}
|
||||
|
||||
func buildClickHouseOptions(cfg ClickHouseConfig) *clickhouse.Options {
|
||||
return &clickhouse.Options{
|
||||
Addr: cfg.Hosts,
|
||||
Auth: clickhouse.Auth{
|
||||
Database: cfg.Database,
|
||||
Username: cfg.Username,
|
||||
Password: cfg.Password,
|
||||
},
|
||||
Settings: clickhouse.Settings{
|
||||
"max_execution_time": clickhouseMaxExecTime,
|
||||
},
|
||||
Compression: &clickhouse.Compression{
|
||||
Method: clickhouse.CompressionLZ4,
|
||||
},
|
||||
DialTimeout: time.Duration(cfg.DialTimeout) * time.Second,
|
||||
MaxOpenConns: cfg.MaxOpenConn,
|
||||
MaxIdleConns: cfg.MaxIdleConn,
|
||||
ConnMaxLifetime: time.Duration(cfg.ConnMaxLifetime) * time.Second,
|
||||
ReadTimeout: time.Duration(cfg.DialTimeout*clickhouseReadTimeoutFactor) * time.Second,
|
||||
BlockBufferSize: cfg.BlockBufferSize,
|
||||
}
|
||||
}
|
||||
|
||||
func buildClickHouseDSN(cfg ClickHouseConfig) string {
|
||||
chURL := &url.URL{
|
||||
Scheme: "clickhouse",
|
||||
Host: strings.Join(cfg.Hosts, ","),
|
||||
Path: "/" + cfg.Database,
|
||||
}
|
||||
if cfg.Username != "" || cfg.Password != "" {
|
||||
chURL.User = url.UserPassword(cfg.Username, cfg.Password)
|
||||
}
|
||||
|
||||
query := chURL.Query()
|
||||
query.Set("dial_timeout", fmt.Sprintf("%ds", cfg.DialTimeout))
|
||||
query.Set("read_timeout", fmt.Sprintf("%ds", cfg.DialTimeout*clickhouseReadTimeoutFactor))
|
||||
query.Set("max_execution_time", strconv.Itoa(clickhouseMaxExecTime))
|
||||
chURL.RawQuery = query.Encode()
|
||||
|
||||
return chURL.String()
|
||||
}
|
||||
|
||||
// ChDB returns a context-aware GORM ClickHouse instance.
|
||||
func ChDB(ctx context.Context) *gorm.DB {
|
||||
if chDB == nil {
|
||||
return nil
|
||||
}
|
||||
return chDB.WithContext(ctx)
|
||||
}
|
||||
|
||||
// SetChDBForTest sets the package-level ClickHouse GORM instance for testing.
|
||||
func SetChDBForTest(d *gorm.DB) {
|
||||
chDB = d
|
||||
}
|
||||
|
||||
// SetChConnForTest sets the package-level native ClickHouse connection for testing.
|
||||
func SetChConnForTest(c driver.Conn) {
|
||||
ChConn = c
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package database
|
||||
|
||||
import "time"
|
||||
|
||||
// ReplicaConfig declares a single read-replica configuration for PostgreSQL.
|
||||
type ReplicaConfig struct {
|
||||
Host string `config:"host"`
|
||||
Port int `config:"port"`
|
||||
Username string `config:"username"`
|
||||
Password string `config:"password" secret:"true"`
|
||||
}
|
||||
|
||||
// Config declares PostgreSQL and SQLite database configuration.
|
||||
type Config struct {
|
||||
Enabled bool `config:"enabled" env:"DB_ENABLED" default:"false" autoEnable:"DB_HOST"`
|
||||
SQLitePath string `config:"sqlite_path" env:"SQLITE_PATH"`
|
||||
Host string `config:"host" env:"DB_HOST"`
|
||||
Port int `config:"port" env:"DB_PORT" default:"5432"`
|
||||
Username string `config:"username" env:"DB_USERNAME"`
|
||||
Password string `config:"password" env:"DB_PASSWORD" secret:"true"`
|
||||
Database string `config:"database" env:"DB_NAME" default:"wavelet"`
|
||||
MaxIdleConn int `config:"max_idle_conn" env:"DB_MAX_IDLE_CONN" default:"10"`
|
||||
MaxOpenConn int `config:"max_open_conn" env:"DB_MAX_OPEN_CONN" default:"100"`
|
||||
ConnMaxLifetime int `config:"conn_max_lifetime" env:"DB_CONN_MAX_LIFETIME" default:"3600"`
|
||||
ConnMaxIdleTime int `config:"conn_max_idle_time" env:"DB_CONN_MAX_IDLE_TIME" default:"600"`
|
||||
LogLevel string `config:"log_level" env:"DB_LOG_LEVEL" default:"warn"`
|
||||
SSLMode string `config:"ssl_mode" env:"DB_SSL_MODE" default:"disable"`
|
||||
TimeZone string `config:"time_zone" env:"DB_TIMEZONE" default:"UTC"`
|
||||
ApplicationName string `config:"application_name" env:"DB_APPLICATION_NAME" default:"wavelet"`
|
||||
SearchPath string `config:"search_path" env:"DB_SEARCH_PATH" default:"public"`
|
||||
PreferSimpleProtocol bool `config:"prefer_simple_protocol" env:"DB_PREFER_SIMPLE_PROTOCOL"`
|
||||
StatementCacheCapacity int `config:"statement_cache_capacity" env:"DB_STATEMENT_CACHE_CAPACITY"`
|
||||
DefaultQueryExecMode string `config:"default_query_exec_mode" env:"DB_DEFAULT_QUERY_EXEC_MODE"`
|
||||
Replicas []ReplicaConfig `config:"replicas"`
|
||||
SlowThreshold time.Duration `config:"slow_threshold" env:"DB_SLOW_THRESHOLD" default:"200ms"`
|
||||
}
|
||||
|
||||
// ClickHouseConfig declares the configuration for ClickHouse analytical storage.
|
||||
type ClickHouseConfig struct {
|
||||
Enabled bool `config:"enabled" env:"CLICKHOUSE_ENABLED" default:"false" autoEnable:"CLICKHOUSE_HOST"`
|
||||
Hosts []string `config:"hosts" env:"CLICKHOUSE_HOST"`
|
||||
Username string `config:"username" env:"CLICKHOUSE_USERNAME"`
|
||||
Password string `config:"password" env:"CLICKHOUSE_PASSWORD" secret:"true"`
|
||||
Database string `config:"database" env:"CLICKHOUSE_NAME" default:"wavelet"`
|
||||
MaxIdleConn int `config:"max_idle_conn" env:"CLICKHOUSE_MAX_IDLE_CONN" default:"10"`
|
||||
MaxOpenConn int `config:"max_open_conn" env:"CLICKHOUSE_MAX_OPEN_CONN" default:"50"`
|
||||
ConnMaxLifetime int `config:"conn_max_lifetime" env:"CLICKHOUSE_CONN_MAX_LIFETIME" default:"3600"`
|
||||
DialTimeout int `config:"dial_timeout" env:"CLICKHOUSE_DIAL_TIMEOUT" default:"10"`
|
||||
BlockBufferSize uint8 `config:"block_buffer_size" env:"CLICKHOUSE_BLOCK_BUFFER_SIZE" default:"10"`
|
||||
}
|
||||
|
||||
type appEnvConfig struct {
|
||||
Env string `config:"env" env:"APP_ENV" default:"development"`
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Package database provides the relational database infrastructure plugin for Cordis.
|
||||
package database
|
||||
|
||||
import (
|
||||
"Wavelet/core"
|
||||
"Wavelet/core/contracts"
|
||||
"context"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// Option configures the database plugin.
|
||||
type Option func(*Plugin)
|
||||
|
||||
// WithDB configures an explicit *gorm.DB instance for the plugin.
|
||||
func WithDB(d *gorm.DB) Option {
|
||||
return func(p *Plugin) {
|
||||
p.db = d
|
||||
}
|
||||
}
|
||||
|
||||
// WithNamedDB registers a named secondary database connection.
|
||||
func WithNamedDB(name string, d *gorm.DB) Option {
|
||||
return func(p *Plugin) {
|
||||
if p.namedDBs == nil {
|
||||
p.namedDBs = make(map[string]*gorm.DB)
|
||||
}
|
||||
p.namedDBs[name] = d
|
||||
}
|
||||
}
|
||||
|
||||
// Plugin implements core.Plugin to provide contracts.DBService into the Cordis micro-kernel.
|
||||
type Plugin struct {
|
||||
db *gorm.DB
|
||||
namedDBs map[string]*gorm.DB
|
||||
}
|
||||
|
||||
// New creates a new database infrastructure plugin.
|
||||
func New(opts ...Option) *Plugin {
|
||||
p := &Plugin{
|
||||
namedDBs: make(map[string]*gorm.DB),
|
||||
}
|
||||
for _, opt := range opts {
|
||||
if opt != nil {
|
||||
opt(p)
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Name returns the unique identifier of the database plugin.
|
||||
func (p *Plugin) Name() string {
|
||||
return "database"
|
||||
}
|
||||
|
||||
// DeclareConfig declares database configuration keys.
|
||||
func (p *Plugin) DeclareConfig() []core.ConfigBinding {
|
||||
return []core.ConfigBinding{
|
||||
{Prefix: "database", Target: &Config{}},
|
||||
{Prefix: "clickhouse", Target: &ClickHouseConfig{}},
|
||||
{Prefix: "app", Target: &appEnvConfig{}},
|
||||
}
|
||||
}
|
||||
|
||||
// Apply mounts the database service into the Context.
|
||||
func (p *Plugin) Apply(ctx *core.Context) error {
|
||||
var dbCfg Config
|
||||
if err := ctx.Config().Bind("database", &dbCfg); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var chCfg ClickHouseConfig
|
||||
if err := ctx.Config().Bind("clickhouse", &chCfg); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var appCfg appEnvConfig
|
||||
_ = ctx.Config().Bind("app", &appCfg)
|
||||
|
||||
targetDB := p.db
|
||||
if targetDB == nil {
|
||||
var err error
|
||||
targetDB, err = InitDBWithConfig(dbCfg, appCfg.Env == "production" || appCfg.Env == "prod")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if chCfg.Enabled {
|
||||
if err := InitClickHouseWithConfig(chCfg); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
svc := &dbServiceImpl{
|
||||
primary: targetDB,
|
||||
namedDBs: p.namedDBs,
|
||||
}
|
||||
|
||||
if sqlDB, err := targetDB.DB(); err == nil && sqlDB != nil {
|
||||
ctx.OnDispose(func() error {
|
||||
return sqlDB.Close()
|
||||
})
|
||||
}
|
||||
|
||||
core.Provide[contracts.DBService](ctx, svc)
|
||||
return nil
|
||||
}
|
||||
|
||||
// NewService wraps a GORM DB instance into a contracts.DBService.
|
||||
func NewService(primary *gorm.DB) contracts.DBService {
|
||||
return &dbServiceImpl{primary: primary}
|
||||
}
|
||||
|
||||
type dbServiceImpl struct {
|
||||
primary *gorm.DB
|
||||
namedDBs map[string]*gorm.DB
|
||||
}
|
||||
|
||||
func (s *dbServiceImpl) GORM() *gorm.DB {
|
||||
if s.primary != nil {
|
||||
return s.primary
|
||||
}
|
||||
return DB(context.Background())
|
||||
}
|
||||
|
||||
func (s *dbServiceImpl) DB(ctx context.Context) *gorm.DB {
|
||||
if s.primary != nil {
|
||||
return s.primary.WithContext(ctx)
|
||||
}
|
||||
return DB(ctx)
|
||||
}
|
||||
|
||||
func (s *dbServiceImpl) Named(name string) *gorm.DB {
|
||||
if s.namedDBs != nil {
|
||||
if d, ok := s.namedDBs[name]; ok && d != nil {
|
||||
return d
|
||||
}
|
||||
}
|
||||
return s.GORM()
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package database_test
|
||||
|
||||
import (
|
||||
"Wavelet/core"
|
||||
"Wavelet/core/contracts"
|
||||
"Wavelet/plugins/infra/database"
|
||||
"context"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/glebarez/sqlite"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type Item struct {
|
||||
ID uint64 `gorm:"primaryKey"`
|
||||
Name string
|
||||
}
|
||||
|
||||
func TestDatabasePlugin(t *testing.T) {
|
||||
dbPath := filepath.Join(t.TempDir(), "db_test.db")
|
||||
gdb, err := gorm.Open(sqlite.Open(dbPath), &gorm.Config{})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, gdb.AutoMigrate(&Item{}))
|
||||
|
||||
namedPath := filepath.Join(t.TempDir(), "named_test.db")
|
||||
namedDB, err := gorm.Open(sqlite.Open(namedPath), &gorm.Config{})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, namedDB.AutoMigrate(&Item{}))
|
||||
|
||||
p := database.New(
|
||||
database.WithDB(gdb),
|
||||
database.WithNamedDB("analytics", namedDB),
|
||||
)
|
||||
ctx := core.NewContext(context.Background())
|
||||
ctx.Config().SetSource(core.NewMapSource(map[string]any{
|
||||
"database.enabled": false,
|
||||
}))
|
||||
require.NoError(t, ctx.Config().Resolve())
|
||||
require.NoError(t, p.Apply(ctx))
|
||||
|
||||
svc, err := core.Inject[contracts.DBService](ctx)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, svc)
|
||||
|
||||
assert.Equal(t, gdb, svc.GORM())
|
||||
assert.NotNil(t, svc.DB(context.Background()))
|
||||
assert.Equal(t, namedDB, svc.Named("analytics"))
|
||||
assert.Equal(t, gdb, svc.Named("non_existent"))
|
||||
|
||||
// Verify DB write
|
||||
item := Item{ID: 1, Name: "TestItem"}
|
||||
require.NoError(t, svc.DB(context.Background()).Create(&item).Error)
|
||||
|
||||
var retrieved Item
|
||||
require.NoError(t, svc.GORM().First(&retrieved, 1).Error)
|
||||
assert.Equal(t, "TestItem", retrieved.Name)
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"net"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/glebarez/sqlite"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"gorm.io/driver/postgres"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/plugin/dbresolver"
|
||||
"gorm.io/plugin/opentelemetry/tracing"
|
||||
)
|
||||
|
||||
var db *gorm.DB
|
||||
|
||||
const sqliteDirMode = 0o750
|
||||
|
||||
// InitDB 初始化主数据库实例(支持 PostgreSQL / SQLite)
|
||||
func InitDB() (*gorm.DB, error) {
|
||||
return InitDBWithConfig(Config{}, false)
|
||||
}
|
||||
|
||||
// InitDBWithConfig initializes the main database with the provided config.
|
||||
func InitDBWithConfig(cfg Config, isProd bool) (*gorm.DB, error) {
|
||||
if !cfg.Enabled {
|
||||
return initSQLiteWithConfig(cfg, isProd)
|
||||
}
|
||||
return initPostgresWithConfig(cfg, isProd)
|
||||
}
|
||||
|
||||
func initSQLiteWithConfig(cfg Config, isProd bool) (*gorm.DB, error) {
|
||||
sqlitePath := cfg.SQLitePath
|
||||
if sqlitePath == "" {
|
||||
sqlitePath = "./data/wavelet.db"
|
||||
}
|
||||
|
||||
if sqlitePath != ":memory:" && !strings.HasPrefix(sqlitePath, "file:") {
|
||||
if dir := filepath.Dir(sqlitePath); dir != "" && dir != "." {
|
||||
if err := os.MkdirAll(dir, sqliteDirMode); err != nil {
|
||||
return nil, fmt.Errorf("create sqlite directory %q failed: %w", dir, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
targetDB, err := gorm.Open(sqlite.Open(sqlitePath), &gorm.Config{
|
||||
DisableForeignKeyConstraintWhenMigrating: true,
|
||||
Logger: &gormZapLogger{
|
||||
logLevel: parseLogLevel(cfg.LogLevel),
|
||||
slowThreshold: cfg.SlowThreshold,
|
||||
ignoreRecordNotFoundError: isProd,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Trace 注入
|
||||
if err = targetDB.Use(
|
||||
tracing.NewPlugin(
|
||||
tracing.WithoutMetrics(),
|
||||
tracing.WithAttributes(
|
||||
attribute.String("db.instance", sqlitePath),
|
||||
attribute.String("db.system", "SQLite"),
|
||||
),
|
||||
),
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
db = targetDB
|
||||
log.Printf("[SQLite] initialized (path: %s)\n", sqlitePath)
|
||||
return targetDB, nil
|
||||
}
|
||||
|
||||
func initPostgresWithConfig(cfg Config, isProd bool) (*gorm.DB, error) {
|
||||
// 构建主库 DSN 并连接
|
||||
primaryDSN := buildDSN(cfg, cfg.Host, cfg.Port, cfg.Username, cfg.Password)
|
||||
|
||||
pgConfig := postgres.Config{
|
||||
DSN: primaryDSN,
|
||||
PreferSimpleProtocol: cfg.PreferSimpleProtocol,
|
||||
}
|
||||
|
||||
targetDB, err := gorm.Open(postgres.New(pgConfig), &gorm.Config{
|
||||
DisableForeignKeyConstraintWhenMigrating: true,
|
||||
Logger: &gormZapLogger{
|
||||
logLevel: parseLogLevel(cfg.LogLevel),
|
||||
slowThreshold: cfg.SlowThreshold,
|
||||
ignoreRecordNotFoundError: isProd,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Trace 注入
|
||||
if err = targetDB.Use(
|
||||
tracing.NewPlugin(
|
||||
tracing.WithoutMetrics(),
|
||||
tracing.WithAttributes(
|
||||
attribute.String("db.instance", cfg.Database),
|
||||
attribute.String("db.ip", cfg.Host),
|
||||
attribute.String("server.address", net.JoinHostPort(cfg.Host, strconv.Itoa(cfg.Port))),
|
||||
attribute.String("db.system", "PostgreSQL"),
|
||||
),
|
||||
),
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(cfg.Replicas) > 0 {
|
||||
var replicaDialectors []gorm.Dialector
|
||||
for _, replica := range cfg.Replicas {
|
||||
username := replica.Username
|
||||
if username == "" {
|
||||
username = cfg.Username
|
||||
}
|
||||
password := replica.Password
|
||||
if password == "" {
|
||||
password = cfg.Password
|
||||
}
|
||||
replicaDSN := buildDSN(cfg, replica.Host, replica.Port, username, password)
|
||||
replicaDialectors = append(replicaDialectors, postgres.New(postgres.Config{
|
||||
DSN: replicaDSN,
|
||||
PreferSimpleProtocol: cfg.PreferSimpleProtocol,
|
||||
}))
|
||||
}
|
||||
|
||||
resolver := dbresolver.Register(dbresolver.Config{
|
||||
Replicas: replicaDialectors,
|
||||
Policy: dbresolver.RandomPolicy{},
|
||||
})
|
||||
|
||||
resolver.SetMaxIdleConns(cfg.MaxIdleConn).
|
||||
SetMaxOpenConns(cfg.MaxOpenConn).
|
||||
SetConnMaxLifetime(time.Duration(cfg.ConnMaxLifetime) * time.Second).
|
||||
SetConnMaxIdleTime(time.Duration(cfg.ConnMaxIdleTime) * time.Second)
|
||||
|
||||
if err = targetDB.Use(resolver); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
log.Printf("[PostgreSQL] initialized in Primary-Replica mode (%d replicas)\n", len(cfg.Replicas))
|
||||
} else {
|
||||
log.Println("[PostgreSQL] initialized in Standalone mode")
|
||||
}
|
||||
|
||||
// 获取通用数据库对象设置连接池
|
||||
sqlDB, err := targetDB.DB()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
sqlDB.SetMaxIdleConns(cfg.MaxIdleConn)
|
||||
sqlDB.SetMaxOpenConns(cfg.MaxOpenConn)
|
||||
sqlDB.SetConnMaxLifetime(time.Duration(cfg.ConnMaxLifetime) * time.Second)
|
||||
sqlDB.SetConnMaxIdleTime(time.Duration(cfg.ConnMaxIdleTime) * time.Second)
|
||||
|
||||
db = targetDB
|
||||
return targetDB, nil
|
||||
}
|
||||
|
||||
// buildDSN 构建 PostgreSQL DSN
|
||||
func buildDSN(cfg Config, host string, port int, username, password string) string {
|
||||
pqURL := &url.URL{
|
||||
Scheme: "postgres",
|
||||
Host: net.JoinHostPort(host, strconv.Itoa(port)),
|
||||
Path: cfg.Database,
|
||||
}
|
||||
if username != "" {
|
||||
pqURL.User = url.UserPassword(username, password)
|
||||
}
|
||||
|
||||
query := pqURL.Query()
|
||||
sslMode := cfg.SSLMode
|
||||
if sslMode == "" {
|
||||
sslMode = "disable"
|
||||
}
|
||||
query.Set("sslmode", sslMode)
|
||||
if cfg.ApplicationName != "" {
|
||||
query.Set("application_name", cfg.ApplicationName)
|
||||
}
|
||||
if cfg.SearchPath != "" {
|
||||
query.Set("search_path", cfg.SearchPath)
|
||||
}
|
||||
if cfg.DefaultQueryExecMode != "" {
|
||||
query.Set("default_query_exec_mode", cfg.DefaultQueryExecMode)
|
||||
}
|
||||
if cfg.StatementCacheCapacity > 0 {
|
||||
query.Set("statement_cache_capacity", strconv.Itoa(cfg.StatementCacheCapacity))
|
||||
}
|
||||
|
||||
rawQuery := query.Encode()
|
||||
if cfg.TimeZone != "" {
|
||||
if rawQuery != "" {
|
||||
rawQuery += "&"
|
||||
}
|
||||
rawQuery += "TimeZone=" + cfg.TimeZone
|
||||
}
|
||||
pqURL.RawQuery = rawQuery
|
||||
|
||||
return pqURL.String()
|
||||
}
|
||||
|
||||
// DB 返回带上下文追踪的 GORM 数据库实例
|
||||
func DB(ctx context.Context) *gorm.DB {
|
||||
if db == nil {
|
||||
return nil
|
||||
}
|
||||
return db.WithContext(ctx)
|
||||
}
|
||||
|
||||
// SetDB sets the package-level database instance for testing.
|
||||
func SetDB(d *gorm.DB) {
|
||||
db = d
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package database
|
||||
|
||||
import (
|
||||
"Wavelet/pkg/logger"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
gormLogger "gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
// nanoToMilli 纳秒转毫秒的除数
|
||||
const nanoToMilli = 1e6
|
||||
|
||||
type gormZapLogger struct {
|
||||
logLevel gormLogger.LogLevel
|
||||
ignoreRecordNotFoundError bool
|
||||
slowThreshold time.Duration
|
||||
}
|
||||
|
||||
func (l *gormZapLogger) LogMode(level gormLogger.LogLevel) gormLogger.Interface {
|
||||
clone := *l
|
||||
clone.logLevel = level
|
||||
return &clone
|
||||
}
|
||||
|
||||
func (l *gormZapLogger) Info(ctx context.Context, fmt string, args ...interface{}) {
|
||||
if l.logLevel >= gormLogger.Info {
|
||||
logger.InfoF(ctx, fmt, args...)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *gormZapLogger) Warn(ctx context.Context, fmt string, args ...interface{}) {
|
||||
if l.logLevel >= gormLogger.Warn {
|
||||
logger.WarnF(ctx, fmt, args...)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *gormZapLogger) Error(ctx context.Context, fmt string, args ...interface{}) {
|
||||
if l.logLevel >= gormLogger.Error {
|
||||
logger.ErrorF(ctx, fmt, args...)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *gormZapLogger) Trace(ctx context.Context, begin time.Time, fc func() (sql string, rowsAffected int64), err error) {
|
||||
elapsed := time.Since(begin)
|
||||
switch {
|
||||
case err != nil && l.logLevel >= gormLogger.Error && (!errors.Is(err, gorm.ErrRecordNotFound) || !l.ignoreRecordNotFoundError):
|
||||
_, rows := fc()
|
||||
logger.ErrorF(ctx, "database query failed: %s [%.3fms] [rows:%v]", err, float64(elapsed.Nanoseconds())/nanoToMilli, formatRows(rows))
|
||||
case elapsed > l.slowThreshold && l.slowThreshold != 0 && l.logLevel >= gormLogger.Warn:
|
||||
_, rows := fc()
|
||||
slowLog := fmt.Sprintf("SLOW SQL >= %v", l.slowThreshold)
|
||||
logger.WarnF(ctx, "%s [%.3fms] [rows:%v]", slowLog, float64(elapsed.Nanoseconds())/nanoToMilli, formatRows(rows))
|
||||
case l.logLevel == gormLogger.Info:
|
||||
sql, rows := fc()
|
||||
logger.DebugF(ctx, "[%.3fms] [rows:%v] %s", float64(elapsed.Nanoseconds())/nanoToMilli, formatRows(rows), sql)
|
||||
}
|
||||
}
|
||||
|
||||
func formatRows(rows int64) interface{} {
|
||||
if rows == -1 {
|
||||
return "-"
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func parseLogLevel(level string) gormLogger.LogLevel {
|
||||
level = strings.ToLower(level)
|
||||
switch level {
|
||||
case "silent":
|
||||
return gormLogger.Silent
|
||||
case "error":
|
||||
return gormLogger.Error
|
||||
case "warn":
|
||||
return gormLogger.Warn
|
||||
case "info":
|
||||
return gormLogger.Info
|
||||
case "debug":
|
||||
return gormLogger.Info
|
||||
default:
|
||||
return gormLogger.Info
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package database
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
gormLogger "gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
func TestParseLogLevel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
configuredLevel string
|
||||
want gormLogger.LogLevel
|
||||
}{
|
||||
{
|
||||
name: "debug enables SQL trace processing",
|
||||
configuredLevel: "debug",
|
||||
want: gormLogger.Info,
|
||||
},
|
||||
{
|
||||
name: "development preserves configured level",
|
||||
configuredLevel: "warn",
|
||||
want: gormLogger.Warn,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := parseLogLevel(tt.configuredLevel); got != tt.want {
|
||||
t.Fatalf("parseLogLevel() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user