mirror of
https://github.com/Rain-kl/OpenFlare.git
synced 2026-10-02 06:56:36 +08:00
feat(cordis): add OpenFlare Cordis 架构改造设计
docs(changelog): 修正表述笔误
refactor(cordis): 磁盘缓存改用上上游能力并清理本地副本
按上游/下游归属规约:类型断言守卫已回流 Wavelet(f3d85d5,附回归用例),
本仓库删除 OpenFlare/plugins/server/pkg/cache 整包并改 import 到
Wavelet/pkg/cache/disk,同步后与上游零漂移。
验证:go build 通过;go test ./... exit 0(137 包 ok);256 条路由对拍与
232 条 swagger 操作均零差异;make build-all 四进制;前端零改动。
docs(cordis): 记录 T1 清理结果与五个复用阻塞点
refactor(cordis): server 复用上游 pkg 能力并删除等价本地副本
按上游/下游归属规约清理重复实现,删除 7 个与上游等价的本地包并改 import:
shared/response→pkg/response、pkg/{logger,mail,trace,httppool,cache/ram}→
上游同名包、infra/persistence/batchwriter→pkg/batchwriter。逐项核过差异:
httppool 逐字节相同;logger 的 Config 字段完全一致;response 的 7 个 Abort*
一致;cache/ram 换过去顺带把裸 go 变回带 panic 恢复的 util.Go。
两处非等价差异按语义处理:
- batchwriter.Stats 与 status DTO 原为类型别名,改为消费侧逐字段转换,
避免 model 反向依赖基础设施类型;
- 上游 pkg/idgen 要求显式 Init(本地副本为懒加载自动初始化),本次保留本地
副本,待与 infra 初始化一并迁移(已登记在清理计划)。
验证:go build 通过;go test ./... exit 0(138 包 ok);256 条路由对拍零差异;
make swagger 232 条操作零增减,且归一化后与旧文档深度相等——差异仅为
response.Any / logger.LogEntry 两个定义名随包路径改名,接口形状未变。
chore(cordis): 回流内核与 pkg/util 通用能力并清理 vendoring 污染
按新增的上游/下游归属规约:HandleRaw/BasePath 与版本比较、网络、格式化助手
属通用能力,已提交到 Wavelet 分支 feat/cordis-router-raw-routes,本仓库改为
纯同步获取(pkg/util 已零漂移),补丁登记保留至上游合并。
同时修掉我此前 git add -A 造成的污染:首次 vendoring 把上游工作区里被
gitignore 的运行期产物一起提交进来(upload 的 diskcache 缓存块 650 个与
driver_http/dist 前端构建物 380 个,共 12872 行/1030 文件)。sync-upstream.sh
现显式排除 uploads/dist/data/*.db,.gitignore 补上对应兜底规则。
AGENTS.md 增加上游/下游改动归属规约,并把仍指向前 Cordis 布局的硬性约束
(internal/router + Serve、internal/repository/logstore、internal/platform/bootstrap、
internal/cmd)改到当前插件路径。
验证:go build 通过;go test ./... exit 0(144 包 ok);make swagger 232 条
操作与基线逐条一致;make build-all 四进制;gofmt 干净。
feat(cordis): server 插件化并改由内核挂载控制面路由
新增 plugins/server/plugin.go:Apply 以 ctx.Router().Group(app.api_prefix)
声明根级与 /v1 全部路由;33 个注册函数由 *gin.RouterGroup 改为
core.RouterExtension,RegisterCollection 改用内核新增的 HandleRaw 保留
尾部斜杠变体,AdminMiddlewares 返回 []any(Go 不允许把 []T 展开为 ...any)。
删除 router.Serve 与 registerRoutes,装配根改为 core.App +
driver_http.New(WithEngine(router.BuildEngine())),监听、信号与优雅退出归内核;
前端 SPA 的 NoRoute 兜底因内核暂无贡献点而保留在引擎层。
路由保真证据:plugin_parity_test 对拍 baseline/routes-engine.txt 的 256 条
(方法 路径) 零差异;go test ./... exit 0(144 包 ok,含真实 handler 的
openflare/integration 用例走同一条挂载路径);make swagger 232 条操作与基线
逐条一致;golangci-lint 0 issues;make build-all 四进制;embed_frontend
标签编译通过;前端零改动。
已知待补:带 Redis 的实机 HTTP 冒烟(本机 6379 未启动,session store 与
改造前一样在建店阶段即 fatal),以及 bootstrap 的任务/设置/迁移注册迁入 Apply。
feat(core): RouterExtension 增加 HandleRaw 与 BasePath 以保真尾部斜杠路由
server 插件化的前置:Handle 经 cleanPath 会剥掉尾部斜杠,无法表达
/resource 与 /resource/ 两条不同路由,而 OpenFlare 有 20 个历史 list
端点两者都注册且部署关闭了 RedirectTrailingSlash,缺失即 404。新增
HandleRaw 与 BasePath(作用域包装器同样登记反注册),补 extpoints 用例;
并把 router.Serve 拆出 BuildEngine 以便交给 driver_http.WithEngine 复用,
新增路由表导出 harness,固化 256 条 (方法 路径) 基线供插件化对拍。
上游补丁登记于 backend/OpenFlare/upstream-patches.md,同步脚本改为按目录
前缀输出差异并在同步后提醒确认补丁是否仍在。
验证:go build 通过;go test ./... exit 0(143 包 ok);gofmt 干净。
docs(cordis): 记录 server 插件接入内核的可行路径与内核能力缺口
feat(cordis): agent/relay/flared 落地为内核驱动插件
三个边缘守护进程各新增 plugin.go,实现 core.Plugin + core.Driver
(自定义 DriverType 与同名 profile),装配与生命周期从 main 迁入
Apply/Start/Stop:Apply 负责 JSON 配置加载、运行环境与用户确保、
openresty/frps/frpc 管理器与各服务装配;Start 以 util.Go 拉起阻塞式
runner 与 GeoIP 周期更新;Stop 收敛主循环结果并在超时时报错而非静默。
入口改为 core.NewApp(core.WithProfile(...)) + Prepare/Run,保持
-config 旗标、默认路径、退出码与启动/停止日志不变。
验证:go build 通过;go test ./... exit 0(143 包 ok,含 3 个插件身份
与配置失败路径测试);make build-all 四进制产出;三进制实跑缺失配置
均 exit 1 且错误链保留 load {agent,relay,flared} config 原因;gofmt 干净。
refactor(cordis): 按功能职责拆分为 4 个插件与 share 共享层
backend/OpenFlare 不再平铺遗留分层,改为 plugins/{server,agent,relay,flared}
加 share/:控制面业务(openflare/admin/oauth/user/upload/cap/config/health 与
repository/model/infra/router 等支撑层)归 server;三个边缘守护进程各自成插件;
被两个以上插件消费的 protocol/geoip/wsclient/render/pagesarchive/edge 归 share。
同时把 pkg/util 与 buildinfo 合并回上游 pkg(上游已覆盖全部符号,仅 8 个函数与
2 个类型为 OpenFlare 独有,已一并迁入),装配根统一到 backend/cmd(含三个 daemon
入口),Dockerfile 与 release 工作流的构建路径和 -X 注入路径同步更新。
验证:go build 通过;go test ./... exit 0(141 包 ok);make swagger exit 0 且
232 条 API 操作与基线逐条一致;make build-all 产出 4 进制;-X 注入经二进制
strings 实测生效;日志后端直连门禁改写为按 server 插件业务域扫描并在扫描数为 0
时报错(防门禁静默失效);前端零改动。
feat(cordis): 落地 backend/share 共享层与上游同步脚本
跨插件共享资源(控制消息协议、GeoIP+iputil、边缘守护进程日志)从下游包
移入 backend/share,并声明其只能依赖 core/pkg 与标准/第三方库,禁止反向
引用下游业务与具体插件实现;新增 scripts/sync-upstream.sh 只覆盖
backend/{core,pkg,plugins},同步后 --check 报告零差异,证明与上游逐字一致。
go build 通过,go test ./... exit 0(142 包 ok),前端零改动。
refactor(cordis): 采用与 Wavelet 同构的单模块布局并引入上游内核
按上游结构落位:backend/{core,pkg,plugins} 为 Wavelet 上游拷贝,OpenFlare
全部业务收拢到上游 downstream 所对应的位置 backend/OpenFlare/,模块名保持
Wavelet 以保证上游 import 路径逐字一致、同步零改写;三个 daemon 入口移至
backend/OpenFlare/cmd,backend/cmd 与 main.go 作为控制面装配根。
行为不变:go build 通过,142 个测试包全绿(含上游插件测试),232 条 API
操作与改造前逐条一致,四进制产物正常,前端零改动。swagger 暂只扫描下游代码,
待 P4 挂载上游路由后再纳入 plugins/。
style: 修正模块路径改写导致的 import 分组排序漂移
refactor(layout): Go 代码迁入 backend/ 并将模块名简化为 OpenFlare
对齐上游 Wavelet 的仓库布局,为以第二 module 形态 vendoring Cordis 内核与
平台插件做准备:模块路径整体改写为 OpenFlare,Go 目标加 cd backend,
swaggo 产物移至 backend/docs 并把 json/yaml 复制回 docs/ 供站点消费,
Dockerfile 与 release 工作流的构建目录、ldflags 模块路径同步更新。
行为保持不变:232 条路由与改造前逐条一致,95 个测试包全绿,
四进制产物正常,前端零改动。
chore(cordis): 落地改造计划与 schema/路由基线
新增 legacy_dump_test 迁移快照 harness:在临时 sqlite 库上按生产顺序
(goose.UpTo → zone 导入 → goose.Up)跑完 76 个历史迁移并导出 schema 与
版本序列,作为改造前后一致性门禁的唯一事实来源。同时记录 232 条路由清单
与 foundation 实施计划。
docs(cordis): add OpenFlare Cordis 架构改造设计
明确上游以第二 module 形态 vendoring 进 backend/Wavelet、4 个插件
(server/agent/relay/flared) 全部装载内核,并规定保留 76 个历史 goose
迁移 + 一次性版本 stamp 桥接的迁移方案,配套三方 schema 一致性门禁,
确保已部署库不重跑历史、不丢数据。
This commit is contained in:
Vendored
+432
@@ -0,0 +1,432 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Package disk implements a platform-level disk-backed cache with size limit, TTL, and LRU eviction.
|
||||
package disk
|
||||
|
||||
import (
|
||||
"Wavelet/pkg/util"
|
||||
"container/list"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/peterbourgon/diskv/v3"
|
||||
)
|
||||
|
||||
// ErrCacheMiss represents a cache miss.
|
||||
var ErrCacheMiss = errors.New("cache miss")
|
||||
|
||||
// Constants for disk cache configuration and sizing
|
||||
const (
|
||||
headerSize = 8 // 8 bytes metadata prefix for expiration UnixNano timestamp
|
||||
defaultMaxSizeMB = 100
|
||||
defaultTTLMinutes = 60
|
||||
cacheDirPerm = 0o750
|
||||
|
||||
// DefaultExpiration applies the cache-wide default TTL.
|
||||
DefaultExpiration time.Duration = 0
|
||||
// NoExpiration stores the item without a TTL. Size limits and LRU eviction still apply.
|
||||
NoExpiration time.Duration = -1
|
||||
)
|
||||
|
||||
// Status represents the runtime cache statistics.
|
||||
type Status struct {
|
||||
TotalSize int64 `json:"total_size"`
|
||||
KeysCount int `json:"keys_count"`
|
||||
MaxSizeMB int64 `json:"max_size_mb"`
|
||||
TTLMinutes int64 `json:"ttl_minutes"`
|
||||
LRUEnabled bool `json:"lru_enabled"`
|
||||
BasePath string `json:"base_path"`
|
||||
}
|
||||
|
||||
// Cache implements the disk-backed cache with size limits, TTL, and LRU eviction.
|
||||
type Cache struct {
|
||||
mu sync.RWMutex
|
||||
d *diskv.Diskv
|
||||
basePath string
|
||||
maxSize int64 // in bytes
|
||||
defaultTTL time.Duration
|
||||
lruEnabled bool
|
||||
|
||||
// LRU and Size tracking
|
||||
currentSize int64
|
||||
items map[string]*list.Element
|
||||
evictList *list.List
|
||||
}
|
||||
|
||||
type cacheItem struct {
|
||||
key string
|
||||
size int64
|
||||
expiredAt time.Time
|
||||
}
|
||||
|
||||
// New creates a new Cache instance.
|
||||
func New(basePath string) *Cache {
|
||||
d := diskv.New(diskv.Options{
|
||||
BasePath: basePath,
|
||||
Transform: func(_ string) []string { return []string{} }, // flat structure for easy walk
|
||||
CacheSizeMax: 1024 * 1024, // 1MB in-memory cache size for diskv itself
|
||||
})
|
||||
|
||||
c := &Cache{
|
||||
d: d,
|
||||
basePath: basePath,
|
||||
maxSize: defaultMaxSizeMB * 1024 * 1024, // 100MB default
|
||||
defaultTTL: defaultTTLMinutes * time.Minute, // 60 minutes default
|
||||
lruEnabled: true,
|
||||
items: make(map[string]*list.Element),
|
||||
evictList: list.New(),
|
||||
}
|
||||
|
||||
// Scan directory on startup to rebuild LRU and size tracking
|
||||
_ = c.loadTracker()
|
||||
return c
|
||||
}
|
||||
|
||||
var (
|
||||
defaultCache *Cache
|
||||
defaultCacheOnce sync.Once
|
||||
|
||||
// defaultCleanupInterval 全局磁盘缓存默认的过期清理巡检周期
|
||||
defaultCleanupInterval = 10 * time.Minute
|
||||
)
|
||||
|
||||
// Default returns the default global disk cache instance.
|
||||
func Default() *Cache {
|
||||
defaultCacheOnce.Do(func() {
|
||||
defaultCache = New("uploads/diskcache")
|
||||
util.Go(func() {
|
||||
defaultCache.StartCleanupWorker(defaultCleanupInterval)
|
||||
})
|
||||
})
|
||||
return defaultCache
|
||||
}
|
||||
|
||||
// Set stores a key-value pair in the cache.
|
||||
// Use DefaultExpiration for the configured default TTL, NoExpiration for no
|
||||
// TTL, or a positive duration for a business-specific TTL.
|
||||
func (c *Cache) Set(key string, value []byte, ttl time.Duration) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
if ttl == DefaultExpiration {
|
||||
ttl = c.defaultTTL
|
||||
}
|
||||
|
||||
var expiredAt time.Time
|
||||
if ttl > 0 {
|
||||
expiredAt = time.Now().Add(ttl)
|
||||
}
|
||||
|
||||
// Prepare data layout: 8 bytes expiration timestamp + raw payload
|
||||
buf := make([]byte, headerSize+len(value))
|
||||
var expNano int64
|
||||
if !expiredAt.IsZero() {
|
||||
expNano = expiredAt.UnixNano()
|
||||
}
|
||||
binary.BigEndian.PutUint64(buf[0:headerSize], uint64(expNano))
|
||||
copy(buf[headerSize:], value)
|
||||
|
||||
// Write to diskv
|
||||
if err := c.d.Write(key, buf); err != nil {
|
||||
return fmt.Errorf("failed to write key to disk: %w", err)
|
||||
}
|
||||
|
||||
// Get file size on disk (approximate)
|
||||
size := int64(len(buf))
|
||||
|
||||
// Update memory tracker
|
||||
if elem, ok := c.items[key]; ok {
|
||||
item, ok := elem.Value.(*cacheItem)
|
||||
if !ok {
|
||||
return fmt.Errorf("cache: evict list entry for %q has invalid type %T", key, elem.Value)
|
||||
}
|
||||
c.currentSize += size - item.size
|
||||
item.size = size
|
||||
item.expiredAt = expiredAt
|
||||
c.evictList.MoveToFront(elem)
|
||||
} else {
|
||||
item := &cacheItem{
|
||||
key: key,
|
||||
size: size,
|
||||
expiredAt: expiredAt,
|
||||
}
|
||||
elem := c.evictList.PushFront(item)
|
||||
c.items[key] = elem
|
||||
c.currentSize += size
|
||||
}
|
||||
|
||||
// Evict items if size limit exceeded and LRU is enabled
|
||||
c.evict()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get retrieves a key's value from the cache.
|
||||
func (c *Cache) Get(key string) ([]byte, error) {
|
||||
c.mu.RLock()
|
||||
elem, ok := c.items[key]
|
||||
if !ok {
|
||||
c.mu.RUnlock()
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
item, ok := elem.Value.(*cacheItem)
|
||||
if !ok {
|
||||
c.mu.RUnlock()
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
if !item.expiredAt.IsZero() && time.Now().After(item.expiredAt) {
|
||||
c.mu.RUnlock()
|
||||
return c.getAndDeleteIfExpired(key)
|
||||
}
|
||||
c.mu.RUnlock()
|
||||
|
||||
// Read from disk outside the lock so concurrent cache hits do not serialize on I/O.
|
||||
data, err := c.d.Read(key)
|
||||
if err != nil {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if _, stillExists := c.items[key]; stillExists {
|
||||
_ = c.deleteUnlocked(key)
|
||||
}
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
if len(data) < headerSize {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if _, stillExists := c.items[key]; stillExists {
|
||||
_ = c.deleteUnlocked(key)
|
||||
}
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
payload := data[headerSize:]
|
||||
|
||||
// Brief write lock only for LRU bookkeeping.
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
elem, ok = c.items[key]
|
||||
if !ok {
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
c.evictList.MoveToFront(elem)
|
||||
|
||||
return payload, nil
|
||||
}
|
||||
|
||||
func (c *Cache) getAndDeleteIfExpired(key string) ([]byte, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
elem, ok := c.items[key]
|
||||
if !ok {
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
item, ok := elem.Value.(*cacheItem)
|
||||
if !ok {
|
||||
_ = c.deleteUnlocked(key)
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
if !item.expiredAt.IsZero() && time.Now().After(item.expiredAt) {
|
||||
_ = c.deleteUnlocked(key)
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
data, err := c.d.Read(key)
|
||||
if err != nil {
|
||||
_ = c.deleteUnlocked(key)
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
if len(data) < headerSize {
|
||||
_ = c.deleteUnlocked(key)
|
||||
return nil, ErrCacheMiss
|
||||
}
|
||||
|
||||
c.evictList.MoveToFront(elem)
|
||||
return data[headerSize:], nil
|
||||
}
|
||||
|
||||
// Delete removes a key-value pair from the cache.
|
||||
func (c *Cache) Delete(key string) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.deleteUnlocked(key)
|
||||
}
|
||||
|
||||
func (c *Cache) deleteUnlocked(key string) error {
|
||||
if elem, ok := c.items[key]; ok {
|
||||
if item, ok := elem.Value.(*cacheItem); ok {
|
||||
c.currentSize -= item.size
|
||||
}
|
||||
c.evictList.Remove(elem)
|
||||
delete(c.items, key)
|
||||
}
|
||||
return c.d.Erase(key)
|
||||
}
|
||||
|
||||
// Clear flushes all cached elements.
|
||||
func (c *Cache) Clear() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.currentSize = 0
|
||||
c.items = make(map[string]*list.Element)
|
||||
c.evictList.Init()
|
||||
|
||||
return c.d.EraseAll()
|
||||
}
|
||||
|
||||
// Status returns the cache status.
|
||||
func (c *Cache) Status() Status {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
|
||||
return Status{
|
||||
TotalSize: c.currentSize,
|
||||
KeysCount: len(c.items),
|
||||
MaxSizeMB: c.maxSize / (1024 * 1024),
|
||||
TTLMinutes: int64(c.defaultTTL.Minutes()),
|
||||
LRUEnabled: c.lruEnabled,
|
||||
BasePath: c.basePath,
|
||||
}
|
||||
}
|
||||
|
||||
// UpdatePolicy dynamically updates policies.
|
||||
func (c *Cache) UpdatePolicy(maxSizeMB, ttlMinutes int64, lruEnabled bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.maxSize = maxSizeMB * 1024 * 1024
|
||||
c.defaultTTL = time.Duration(ttlMinutes) * time.Minute
|
||||
c.lruEnabled = lruEnabled
|
||||
c.evict()
|
||||
}
|
||||
|
||||
// evict evicts oldest items if current size exceeds maxSize and LRU is enabled.
|
||||
func (c *Cache) evict() {
|
||||
if !c.lruEnabled {
|
||||
return
|
||||
}
|
||||
|
||||
for c.currentSize > c.maxSize && c.evictList.Len() > 0 {
|
||||
elem := c.evictList.Back()
|
||||
item, ok := elem.Value.(*cacheItem)
|
||||
if !ok {
|
||||
c.evictList.Remove(elem)
|
||||
continue
|
||||
}
|
||||
c.currentSize -= item.size
|
||||
c.evictList.Remove(elem)
|
||||
delete(c.items, item.key)
|
||||
_ = c.d.Erase(item.key)
|
||||
}
|
||||
}
|
||||
|
||||
// loadTracker scans the cache directory on startup to rebuild memory state.
|
||||
func (c *Cache) loadTracker() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
// Ensure directory exists
|
||||
if err := os.MkdirAll(c.basePath, cacheDirPerm); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
type loadedItem struct {
|
||||
key string
|
||||
size int64
|
||||
expiredAt time.Time
|
||||
modTime time.Time
|
||||
}
|
||||
var loadedItems []loadedItem
|
||||
|
||||
// Walk keys through diskv
|
||||
keysChan := c.d.Keys(nil)
|
||||
for key := range keysChan {
|
||||
// Read raw bytes to parse expiration prefix
|
||||
data, err := c.d.Read(key)
|
||||
if err != nil || len(data) < headerSize {
|
||||
_ = c.d.Erase(key) // corrupted file, wipe
|
||||
continue
|
||||
}
|
||||
|
||||
expNano := int64(binary.BigEndian.Uint64(data[0:headerSize])) //nolint:gosec // false positive: UnixNano fits within int64
|
||||
var expiredAt time.Time
|
||||
if expNano > 0 {
|
||||
expiredAt = time.Unix(0, expNano)
|
||||
}
|
||||
|
||||
// Check mod time for ordering
|
||||
path := filepath.Join(c.basePath, key)
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
loadedItems = append(loadedItems, loadedItem{
|
||||
key: key,
|
||||
size: int64(len(data)),
|
||||
expiredAt: expiredAt,
|
||||
modTime: info.ModTime(),
|
||||
})
|
||||
}
|
||||
|
||||
// Sort by ModTime ascending (oldest first) so we rebuild LRU correctly
|
||||
sort.Slice(loadedItems, func(i, j int) bool {
|
||||
return loadedItems[i].modTime.Before(loadedItems[j].modTime)
|
||||
})
|
||||
|
||||
// Populate LRU (PushFront so that newest items are at the front, oldest at the back)
|
||||
for _, item := range loadedItems {
|
||||
entry := &cacheItem{
|
||||
key: item.key,
|
||||
size: item.size,
|
||||
expiredAt: item.expiredAt,
|
||||
}
|
||||
element := c.evictList.PushFront(entry)
|
||||
c.items[item.key] = element
|
||||
c.currentSize += item.size
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// StartCleanupWorker periodically cleans up expired cache items.
|
||||
func (c *Cache) StartCleanupWorker(interval time.Duration) {
|
||||
ticker := time.NewTicker(interval)
|
||||
for range ticker.C {
|
||||
c.cleanExpired()
|
||||
}
|
||||
}
|
||||
|
||||
// cleanExpired scans memory for expired items and removes them.
|
||||
func (c *Cache) cleanExpired() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
for key, elem := range c.items {
|
||||
item, ok := elem.Value.(*cacheItem)
|
||||
if !ok {
|
||||
c.evictList.Remove(elem)
|
||||
delete(c.items, key)
|
||||
continue
|
||||
}
|
||||
if !item.expiredAt.IsZero() && now.After(item.expiredAt) {
|
||||
c.currentSize -= item.size
|
||||
c.evictList.Remove(elem)
|
||||
delete(c.items, key)
|
||||
_ = c.d.Erase(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package disk
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// 这些用例锁住「LRU 链表节点被污染时不得 panic」的行为:一旦 items 与 evictList
|
||||
// 的不变量被破坏(例如后续改动误写节点),缓存必须降级为未命中/跳过,
|
||||
// 而不是在读、写、删除与淘汰路径上崩掉整个进程。
|
||||
|
||||
// corruptEntry 写入一个键后把其链表节点值换成非法类型,返回缓存。
|
||||
func corruptEntry(t *testing.T, key string) *Cache {
|
||||
t.Helper()
|
||||
|
||||
c := New(t.TempDir())
|
||||
require.NoError(t, c.Set(key, []byte("payload"), time.Minute))
|
||||
|
||||
elem, ok := c.items[key]
|
||||
require.True(t, ok, "entry must be tracked after Set")
|
||||
elem.Value = "not-a-cacheItem"
|
||||
return c
|
||||
}
|
||||
|
||||
func TestGetToleratesCorruptEvictEntry(t *testing.T) {
|
||||
c := corruptEntry(t, "k")
|
||||
|
||||
got, err := c.Get("k")
|
||||
require.ErrorIs(t, err, ErrCacheMiss)
|
||||
require.Nil(t, got)
|
||||
}
|
||||
|
||||
func TestSetOverCorruptEvictEntryReportsError(t *testing.T) {
|
||||
c := corruptEntry(t, "k")
|
||||
|
||||
err := c.Set("k", []byte("second"), time.Minute)
|
||||
require.Error(t, err, "Set must report the corrupted tracker entry instead of panicking")
|
||||
require.Contains(t, err.Error(), "invalid type")
|
||||
}
|
||||
|
||||
func TestDeleteToleratesCorruptEvictEntry(t *testing.T) {
|
||||
c := corruptEntry(t, "k")
|
||||
|
||||
require.NotPanics(t, func() { _ = c.Delete("k") })
|
||||
require.NotContains(t, c.items, "k")
|
||||
}
|
||||
|
||||
func TestEvictToleratesCorruptEvictEntry(t *testing.T) {
|
||||
c := corruptEntry(t, "k")
|
||||
// 让任意写入都触发淘汰扫描:扫到被污染的节点必须跳过而非 panic。
|
||||
c.UpdatePolicy(0, 0, true)
|
||||
|
||||
require.NotPanics(t, func() {
|
||||
for i := range 4 {
|
||||
_ = c.Set(string(rune('a'+i)), []byte("x"), time.Minute)
|
||||
}
|
||||
})
|
||||
}
|
||||
Vendored
+199
@@ -0,0 +1,199 @@
|
||||
// Copyright 2026 Arctel.net
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
package disk
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDiskCacheBasic(t *testing.T) {
|
||||
testDir := t.TempDir()
|
||||
|
||||
c := New(testDir)
|
||||
defer func() { _ = c.Clear() }()
|
||||
|
||||
key := "key1"
|
||||
val := []byte("value1")
|
||||
|
||||
// Get non-existent
|
||||
_, err := c.Get(key)
|
||||
if err != ErrCacheMiss {
|
||||
t.Fatalf("expected ErrCacheMiss, got %v", err)
|
||||
}
|
||||
|
||||
// Set & Get
|
||||
err = c.Set(key, val, 10*time.Second)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set cache: %v", err)
|
||||
}
|
||||
|
||||
got, err := c.Get(key)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to get cache: %v", err)
|
||||
}
|
||||
|
||||
if !bytes.Equal(got, val) {
|
||||
t.Errorf("expected %s, got %s", val, got)
|
||||
}
|
||||
|
||||
// Delete
|
||||
err = c.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to delete: %v", err)
|
||||
}
|
||||
|
||||
_, err = c.Get(key)
|
||||
if err != ErrCacheMiss {
|
||||
t.Errorf("expected ErrCacheMiss after delete, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskCacheTTL(t *testing.T) {
|
||||
testDir := t.TempDir()
|
||||
|
||||
c := New(testDir)
|
||||
defer func() { _ = c.Clear() }()
|
||||
|
||||
key := "ttlkey"
|
||||
val := []byte("ttlval")
|
||||
|
||||
// Set with 200ms TTL
|
||||
err := c.Set(key, val, 200*time.Millisecond)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set: %v", err)
|
||||
}
|
||||
|
||||
// Immediate Get should succeed
|
||||
got, err := c.Get(key)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to get: %v", err)
|
||||
}
|
||||
if !bytes.Equal(got, val) {
|
||||
t.Errorf("expected %s, got %s", val, got)
|
||||
}
|
||||
|
||||
// Sleep 250ms to expire
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
|
||||
// Get should fail with cache miss
|
||||
_, err = c.Get(key)
|
||||
if err != ErrCacheMiss {
|
||||
t.Errorf("expected ErrCacheMiss after TTL expiration, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskCacheExpirationPolicies(t *testing.T) {
|
||||
testDir := t.TempDir()
|
||||
|
||||
c := New(testDir)
|
||||
defer func() { _ = c.Clear() }()
|
||||
c.defaultTTL = 50 * time.Millisecond
|
||||
|
||||
if err := c.Set("default", []byte("default"), DefaultExpiration); err != nil {
|
||||
t.Fatalf("Set(default, DefaultExpiration) returned error: %v", err)
|
||||
}
|
||||
if err := c.Set("custom", []byte("custom"), 150*time.Millisecond); err != nil {
|
||||
t.Fatalf("Set(custom, 150ms) returned error: %v", err)
|
||||
}
|
||||
if err := c.Set("permanent", []byte("permanent"), NoExpiration); err != nil {
|
||||
t.Fatalf("Set(permanent, NoExpiration) returned error: %v", err)
|
||||
}
|
||||
|
||||
time.Sleep(80 * time.Millisecond)
|
||||
|
||||
if _, err := c.Get("default"); err != ErrCacheMiss {
|
||||
t.Errorf("Get(default) error = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
if _, err := c.Get("custom"); err != nil {
|
||||
t.Errorf("Get(custom) returned error before custom TTL elapsed: %v", err)
|
||||
}
|
||||
if _, err := c.Get("permanent"); err != nil {
|
||||
t.Errorf("Get(permanent) returned error: %v", err)
|
||||
}
|
||||
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
if _, err := c.Get("custom"); err != ErrCacheMiss {
|
||||
t.Errorf("Get(custom) error = %v, want ErrCacheMiss", err)
|
||||
}
|
||||
if _, err := c.Get("permanent"); err != nil {
|
||||
t.Errorf("Get(permanent) returned error after other entries expired: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskCacheNoExpirationSurvivesReload(t *testing.T) {
|
||||
testDir := t.TempDir()
|
||||
|
||||
c := New(testDir)
|
||||
if err := c.Set("permanent", []byte("value"), NoExpiration); err != nil {
|
||||
t.Fatalf("Set(permanent, NoExpiration) returned error: %v", err)
|
||||
}
|
||||
|
||||
reloaded := New(testDir)
|
||||
defer func() { _ = reloaded.Clear() }()
|
||||
|
||||
got, err := reloaded.Get("permanent")
|
||||
if err != nil {
|
||||
t.Fatalf("reloaded Get(permanent) returned error: %v", err)
|
||||
}
|
||||
if !bytes.Equal(got, []byte("value")) {
|
||||
t.Errorf("reloaded Get(permanent) = %q, want %q", got, "value")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiskCacheLRUEviction(t *testing.T) {
|
||||
testDir := t.TempDir()
|
||||
|
||||
c := New(testDir)
|
||||
defer func() { _ = c.Clear() }()
|
||||
|
||||
// Force a very small max size of 20 bytes for testing (8 bytes header + payload)
|
||||
// So 2 items of 2 bytes payload = 2 * (8 + 2) = 20 bytes max.
|
||||
c.maxSize = 20
|
||||
c.lruEnabled = true
|
||||
|
||||
// Write item 1: 8 + 2 = 10 bytes
|
||||
err := c.Set("k1", []byte("v1"), DefaultExpiration)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set k1: %v", err)
|
||||
}
|
||||
|
||||
// Write item 2: 8 + 2 = 10 bytes
|
||||
err = c.Set("k2", []byte("v2"), DefaultExpiration)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set k2: %v", err)
|
||||
}
|
||||
|
||||
// Both should exist
|
||||
if _, err := c.Get("k1"); err != nil {
|
||||
t.Errorf("k1 should exist: %v", err)
|
||||
}
|
||||
if _, err := c.Get("k2"); err != nil {
|
||||
t.Errorf("k2 should exist: %v", err)
|
||||
}
|
||||
|
||||
// Access k1 again to make it MRU, k2 becomes LRU
|
||||
_, _ = c.Get("k1")
|
||||
|
||||
err = c.Set("k3", []byte("v3"), DefaultExpiration)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set k3: %v", err)
|
||||
}
|
||||
|
||||
// k2 should be evicted, k1 and k3 should exist
|
||||
_, err = c.Get("k2")
|
||||
if err != ErrCacheMiss {
|
||||
t.Errorf("expected k2 to be evicted, got error %v", err)
|
||||
}
|
||||
|
||||
if _, err := c.Get("k1"); err != nil {
|
||||
t.Errorf("k1 should still exist: %v", err)
|
||||
}
|
||||
|
||||
if _, err := c.Get("k3"); err != nil {
|
||||
t.Errorf("k3 should exist: %v", err)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user