Compare commits

..

3 Commits

Author SHA1 Message Date
truewhile a7b24e1bf1 Merge pull request #47 from truewhile/main
拉齐代码
2026-10-20 11:33:06 +08:00
truewhile cd6b37c6a0 Merge pull request #35 from truewhile/feature/reading
Feature/reading
2026-10-01 21:59:11 +08:00
truewhile ef8b017c63 Merge pull request #19 from truewhile/main
拉取最新代码
2026-09-02 17:13:21 +08:00
135 changed files with 725 additions and 7303 deletions
-3
View File
@@ -79,9 +79,6 @@ func (a *application) start() error {
}
}
if err := database.ApplyPendingSQLiteRestore(context.Background(), a.cfg, a.logger); err != nil {
return fmt.Errorf("apply pending database restore: %w", err)
}
db, err := database.Open(a.cfg, a.logger)
if err != nil {
a.logger.Error("database open failed", zap.Error(err))
BIN
View File
Binary file not shown.
@@ -1,46 +0,0 @@
# Playback Correctness and Isolation Implementation Plan
**Goal:** Fix the five playback defects from the 2026-10-21 review and make playback capability queries side-effect-free, without replacing the player or adding dependencies.
**Scope:** The user approved implementing the reviewed fixes. Work in the current clean checkout; do not commit, push, deploy, or alter unrelated files.
## Design and invariants
- Keep existing media IDs, stream routes, direct playback and external-player URLs compatible.
- Add optional `session_id` to local HLS requests. Authenticated handlers supply the actual user identity; a client parameter never supplies the owner. New web playback sessions use random IDs. Legacy clients are isolated by user and, where available, Emby play-session/device identity.
- Keep transcoder media lookup/logging separate from its internal task key. A task key incorporates user, media and playback session. Each actual ffmpeg run writes to a fresh generation directory, so delayed segment requests never read a replacement run's output.
- Stop only the authenticated caller's playback session. Check media visibility and scoped playback tokens on stop, just as on playlist/segment requests. Segment and playlist queries must match the running generation.
- Cloud child requests resolve their real session media first, compare query media, enforce token scope and visibility against the real media, and reject mismatches before upstream fetch.
- Treat player business positions as absolute source seconds. Only local HLS converts to/from a relative ffmpeg timeline.
- Invalidate outstanding cloud-transcode requests on media/unmount, explicit source/quality changes, and newer requests. Guard both fulfilled and rejected responses.
- Select local fallbacks only from available local qualities. Keep the existing 115-first preference when a cloud quality is available.
- `PlaybackInfo` only reads capabilities/status. Explicit `StartTranscode` requests enqueue cloud work. Retain pending/ready response contracts, polling, and retries.
## Tasks
1. Add regression tests for local-scope isolation, stale generations, stop authorization, cloud media binding, pure capability reads, timeline conversion, available-local selection and stale async results. Run tests before implementation and verify the intended failures.
2. Add tested playback helpers; route skip/undo/progress/quality/subtitle calculations through the common absolute timeline; guard cloud transcode results and fix local fallback choice.
3. Extend existing transcoder/stream methods with an optional scope, preserve legacy internal calls, isolate task keys/output directories, and pass caller scope from web and Emby handlers. Bind web local HLS URLs and stop requests to captured session IDs.
4. Enforce real-media authorization on cloud child requests and authenticated ownership on local stop.
5. Split read-only cloud capability lookup from explicit cloud enqueue. Update existing cloud tests to use the explicit start operation and test no push during reads.
6. Run focused and full Go tests, frontend tests, TypeScript/Vite build, lint diagnostics and review the diff for security, lifecycle and API compatibility. Fix regressions before reporting completion.
## Verification commands
- `node --experimental-strip-types --test src/pages/hlsPlayback.test.ts src/pages/directPlaySeek.test.ts src/pages/playerPageModel.test.ts src/pages/playbackQuality.test.ts src/pages/playbackRequestGuard.test.ts src/utils/skipSegments.test.ts`
- `go test ./internal/service ./internal/handler ./internal/middleware -count=1`
- `go test ./...`
- `npm run build` and targeted ESLint for changed frontend files.
## Out of scope
New remux/audio-only ffmpeg pipelines, speculative browser codec policy changes, player replacement, broad UI refactors, performance improvement claims without measurements, and production deployment.
## Verification outcome
- Six frontend playback regression scripts pass; targeted ESLint and the TypeScript/Vite production build pass. The existing bundle-size warning remains.
- Playback/stream/transcode/progress tests in service, handler and middleware pass, including scoped output reads, stale generation rejection, scoped stops, device-to-segment propagation, read-only capability queries and concurrent cloud start deduplication.
- Whole-repository tests pass with only `TestQueueLibraryRootScanImportsNewLibraryMedia` and `TestSQLiteRestoreStagesWithoutChangingLiveDatabase` excluded. Both were run independently and reproduce Windows file-lock / SQLite errors in unmodified scan/database code. They are not silently disabled in source.
- FFmpeg is not installed in this environment, so the existing real-transcode lifecycle test skips. Actual video, upstream 115 and browser end-to-end playback remain unverified.
- Clients with no playback-session or device identifier retain per-user legacy isolation; new web requests always carry an independent session ID.
- No commit, push or deployment was performed.
+2 -1
View File
@@ -47,11 +47,12 @@ func setDefaults(v *viper.Viper) {
v.SetDefault("cache.cache_dir", "./cache")
v.SetDefault("cache.images_max_size_mb", 2000)
v.SetDefault("cache.images_originals_max_size_mb", 256)
v.SetDefault("cache.images_originals_ttl_hours", 168)
v.SetDefault("cache.memory_max_size_mb", DefaultCacheMemoryMaxSizeMB)
v.SetDefault("cache.redis_url", "")
v.SetDefault("cache.redis_prefix", "mebox")
v.SetDefault("cache.media_ttl_seconds", 90)
v.SetDefault("cache.emby_latest_ttl_seconds", 900)
v.SetDefault("cache.emby_latest_ttl_seconds", 300)
v.SetDefault("cache.reader_content_max_size_mb", 1024)
v.SetDefault("cache.reader_content_ttl_hours", 168)
v.SetDefault("cache.reader_files_ttl_hours", 720)
+3
View File
@@ -50,6 +50,9 @@ func (c *Config) normalize() error {
if c.Cache.ImagesOriginalsMaxSizeMB < 0 {
c.Cache.ImagesOriginalsMaxSizeMB = 0
}
if c.Cache.ImagesOriginalsTTLHours < 0 {
c.Cache.ImagesOriginalsTTLHours = 0
}
if c.Cache.MemoryMaxSizeMB <= 0 {
c.Cache.MemoryMaxSizeMB = DefaultCacheMemoryMaxSizeMB
}
+4 -2
View File
@@ -125,6 +125,9 @@ type CacheConfig struct {
// ImagesOriginalsMaxSizeMB 限制 cache/images 下原图子配额(MB)。原图只是
// 生成缩略图的原料,丢了可以重新回源;0 表示不单独限制(只受总量约束)。
ImagesOriginalsMaxSizeMB int `mapstructure:"images_originals_max_size_mb"`
// ImagesOriginalsTTLHours 是原图的保留时长(小时),超过即删除;0 表示
// 不按时间淘汰。缩短它能让原图缓存保持在一个滚动窗口内。
ImagesOriginalsTTLHours int `mapstructure:"images_originals_ttl_hours"`
// MemoryMaxSizeMB 限制进程内 L1 缓存总字节数,JSON/对象缓存共用该预算。
MemoryMaxSizeMB int `mapstructure:"memory_max_size_mb"`
RedisURL string `mapstructure:"redis_url"`
@@ -133,8 +136,7 @@ type CacheConfig struct {
// EmbyLatestTTLSeconds 是 Emby「最新添加」(Items/Latest) 的缓存时长。
// 客户端刷新首页时会并发请求全部媒体库的 Latest(生产环境观察到 73 个
// 并发),缓存过短会让这批请求同时穿透并各自重建 payload,在低配主机
// 上造成秒级延迟。默认 900 秒;过期后仍会先返回陈旧副本并在后台重建,
// 新入库内容最迟 15 分钟后出现在最新列表(扫描完成会主动失效缓存)。
// 上造成秒级延迟。默认 300 秒,新入库内容最迟 5 分钟后出现在最新列表。
EmbyLatestTTLSeconds int `mapstructure:"emby_latest_ttl_seconds"`
// ReaderContentMaxSizeMB 阅读正文持久缓存的总容量(MB),0 表示不限制。
ReaderContentMaxSizeMB int `mapstructure:"reader_content_max_size_mb"`
-5
View File
@@ -376,11 +376,6 @@ func ensurePerformanceIndexes(db *gorm.DB) error {
`DROP INDEX IF EXISTS idx_media_library_release_active`,
`CREATE INDEX IF NOT EXISTS idx_media_library_created_active ON media(library_id, created_at DESC) WHERE deleted_at IS NULL`,
`CREATE INDEX IF NOT EXISTS idx_media_library_episode_active ON media(library_id, season_num, episode_num, created_at DESC) WHERE deleted_at IS NULL`,
// 虚拟剧集分组索引:series_id 为空的剧集行要按
// (season_num, episode_num, episode_fraction, created_at) 排序后做内存
// 分组。没有它 SQLite 只能全表扫描再建临时 B-tree,而 Emby 的
// 卡片/海报查询每次只关心其中一部剧(生产环境实测单次 1.3s)。
`CREATE INDEX IF NOT EXISTS idx_media_virtual_series_group ON media(season_num, episode_num, episode_fraction, created_at) WHERE deleted_at IS NULL AND (series_id IS NULL OR series_id = '')`,
`CREATE INDEX IF NOT EXISTS idx_media_library_root_active ON media(library_id, library_root_id) WHERE deleted_at IS NULL`,
`CREATE INDEX IF NOT EXISTS idx_media_series_active ON media(series_id, season_num, episode_num) WHERE deleted_at IS NULL`,
`CREATE INDEX IF NOT EXISTS idx_media_library_series_active ON media(library_id, series_id, season_num, episode_num) WHERE deleted_at IS NULL`,
-189
View File
@@ -1,189 +0,0 @@
package database
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/config"
)
var sqliteRestoreMu sync.Mutex
func sqliteRestorePath(cfg *config.Config) (string, error) {
if cfg == nil || strings.TrimSpace(cfg.Database.DBPath) == "" || cfg.Database.DBPath == ":memory:" || strings.HasPrefix(cfg.Database.DBPath, "file:") {
return "", errors.New("恢复需要配置有效的 SQLite 数据库文件路径")
}
return filepath.Abs(cfg.Database.DBPath)
}
func sqliteRestoreDSN(path, mode string) string {
u := url.URL{Scheme: "file", Path: filepath.ToSlash(path)}
q := url.Values{"mode": {mode}, "_pragma": {"busy_timeout(5000)"}}
u.RawQuery = q.Encode()
return u.String()
}
func validateSQLiteRestore(ctx context.Context, path string) error {
info, err := os.Lstat(path)
if err != nil {
return err
}
if !info.Mode().IsRegular() {
return errors.New("备份必须是常规文件")
}
db, err := sql.Open("sqlite", sqliteRestoreDSN(path, "ro"))
if err != nil {
return err
}
defer db.Close()
var result string
if err := db.QueryRowContext(ctx, "PRAGMA quick_check").Scan(&result); err != nil {
return fmt.Errorf("SQLite 备份校验失败: %w", err)
}
if result != "ok" {
return fmt.Errorf("SQLite 备份校验失败: %s", result)
}
var tables int
if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name IN ('users', 'media', 'settings')").Scan(&tables); err != nil {
return err
}
if tables == 0 {
return errors.New("备份不包含 MeBox 数据表")
}
return nil
}
// StageSQLiteRestore leaves the live database and its WAL untouched. The
// validated copy is applied on the next startup, before any database is opened.
func StageSQLiteRestore(ctx context.Context, cfg *config.Config, source string) error {
sqliteRestoreMu.Lock()
defer sqliteRestoreMu.Unlock()
dbPath, err := sqliteRestorePath(cfg)
if err != nil {
return err
}
pending := dbPath + ".restore-pending"
if _, err := os.Lstat(pending); err == nil {
return errors.New("已有等待恢复的备份,请重启 MeBox 后再操作")
} else if !errors.Is(err, os.ErrNotExist) {
return err
}
if err := validateSQLiteRestore(ctx, source); err != nil {
return err
}
in, err := os.Open(source)
if err != nil {
return err
}
defer in.Close()
out, err := os.CreateTemp(filepath.Dir(dbPath), ".mebox-restore-*")
if err != nil {
return err
}
tmp := out.Name()
defer os.Remove(tmp)
if _, err := io.Copy(out, in); err != nil {
_ = out.Close()
return err
}
if err := out.Sync(); err != nil {
_ = out.Close()
return err
}
if err := out.Close(); err != nil {
return err
}
if err := validateSQLiteRestore(ctx, tmp); err != nil {
return err
}
return os.Rename(tmp, pending)
}
// ApplyPendingSQLiteRestore must run before database.Open and before workers or
// HTTP listeners start. The previous database is retained alongside the file.
func ApplyPendingSQLiteRestore(ctx context.Context, cfg *config.Config, log *zap.Logger) error {
if cfg == nil {
return nil
}
dialect := normalizeDatabaseType(cfg.Database.Type)
if dialect == "auto" {
dialect = effectiveAutoDatabaseType(cfg)
}
if dialect != "sqlite" {
return nil
}
sqliteRestoreMu.Lock()
defer sqliteRestoreMu.Unlock()
dbPath, err := sqliteRestorePath(cfg)
if err != nil {
return err
}
pending := dbPath + ".restore-pending"
if _, err := os.Lstat(pending); errors.Is(err, os.ErrNotExist) {
return nil
} else if err != nil {
return err
}
if err := validateSQLiteRestore(ctx, pending); err != nil {
return err
}
previous := ""
if _, err := os.Lstat(dbPath); err == nil {
// A clean checkpoint prevents an old WAL from being replayed into the
// replacement database. A busy checkpoint leaves all files untouched.
live, err := sql.Open("sqlite", sqliteRestoreDSN(dbPath, "rw"))
if err != nil {
return err
}
var busy, pages, copied int
err = live.QueryRowContext(ctx, "PRAGMA wal_checkpoint(TRUNCATE)").Scan(&busy, &pages, &copied)
closeErr := live.Close()
if err != nil {
return err
}
if closeErr != nil {
return closeErr
}
if busy != 0 {
return errors.New("SQLite 数据库仍被占用,恢复已保留等待下次启动")
}
old, err := os.CreateTemp(filepath.Dir(dbPath), filepath.Base(dbPath)+".before-restore-*")
if err != nil {
return err
}
previous = old.Name()
if err := old.Close(); err != nil {
return err
}
if err := os.Remove(previous); err != nil {
return err
}
if err := os.Rename(dbPath, previous); err != nil {
return err
}
} else if !errors.Is(err, os.ErrNotExist) {
return err
}
if err := os.Rename(pending, dbPath); err != nil {
if previous != "" {
if rollbackErr := os.Rename(previous, dbPath); rollbackErr != nil {
return fmt.Errorf("恢复失败: %w;回退失败: %v,原数据库保留于 %s", err, rollbackErr, previous)
}
}
return err
}
if log != nil {
log.Warn("SQLite backup restored before startup", zap.String("previous_database", previous))
}
return nil
}
-104
View File
@@ -1,104 +0,0 @@
package database
import (
"database/sql"
"os"
"path/filepath"
"testing"
"github.com/truewhile/MeBox/internal/config"
)
func TestSQLiteRestoreStagesWithoutChangingLiveDatabase(t *testing.T) {
root := t.TempDir()
cfg := &config.Config{}
cfg.Database.Type = "sqlite"
cfg.Database.DBPath = filepath.Join(root, "live.db")
live, err := sql.Open("sqlite", sqliteRestoreDSN(cfg.Database.DBPath, "rwc"))
if err != nil {
t.Fatal(err)
}
defer live.Close()
for _, statement := range []string{
"PRAGMA journal_mode=WAL", "CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT)",
"INSERT INTO settings VALUES ('state', 'backup')",
} {
if _, err := live.Exec(statement); err != nil {
t.Fatal(err)
}
}
source := filepath.Join(root, "backup.db")
if _, err := live.Exec("VACUUM INTO ?", source); err != nil {
t.Fatal(err)
}
if _, err := live.Exec("UPDATE settings SET value = 'live'"); err != nil {
t.Fatal(err)
}
if err := StageSQLiteRestore(t.Context(), cfg, source); err != nil {
t.Fatal(err)
}
var value string
if err := live.QueryRow("SELECT value FROM settings").Scan(&value); err != nil || value != "live" {
t.Fatalf("staging changed live database: value=%q err=%v", value, err)
}
if _, err := live.Exec("UPDATE settings SET value = 'after-stage'"); err != nil {
t.Fatalf("database could not write after staging: %v", err)
}
if err := StageSQLiteRestore(t.Context(), cfg, source); err == nil {
t.Fatal("second staging should not replace the pending restore")
}
if err := os.Remove(source); err != nil {
t.Fatal(err)
}
if err := live.Close(); err != nil {
t.Fatal(err)
}
if err := ApplyPendingSQLiteRestore(t.Context(), cfg, nil); err != nil {
t.Fatal(err)
}
restored, err := sql.Open("sqlite", sqliteRestoreDSN(cfg.Database.DBPath, "ro"))
if err != nil {
t.Fatal(err)
}
defer restored.Close()
if err := restored.QueryRow("SELECT value FROM settings").Scan(&value); err != nil || value != "backup" {
t.Fatalf("restore failed: value=%q err=%v", value, err)
}
previous, err := filepath.Glob(cfg.Database.DBPath + ".before-restore-*")
if err != nil || len(previous) != 1 {
t.Fatalf("previous database not preserved: files=%v err=%v", previous, err)
}
old, err := sql.Open("sqlite", sqliteRestoreDSN(previous[0], "ro"))
if err != nil {
t.Fatal(err)
}
defer old.Close()
if err := old.QueryRow("SELECT value FROM settings").Scan(&value); err != nil || value != "after-stage" {
t.Fatalf("old database lost latest writes: value=%q err=%v", value, err)
}
if err := ApplyPendingSQLiteRestore(t.Context(), cfg, nil); err != nil {
t.Fatalf("repeat startup: %v", err)
}
}
func TestSQLiteRestoreRejectsCorruptAndSymlinkBackups(t *testing.T) {
root := t.TempDir()
cfg := &config.Config{}
cfg.Database.DBPath = filepath.Join(root, "live.db")
source := filepath.Join(root, "corrupt.db")
if err := os.WriteFile(source, []byte("not a SQLite database"), 0o600); err != nil {
t.Fatal(err)
}
if err := StageSQLiteRestore(t.Context(), cfg, source); err == nil {
t.Fatal("corrupt backup accepted")
}
link := filepath.Join(root, "link.db")
if err := os.Symlink(source, link); err == nil {
if err := StageSQLiteRestore(t.Context(), cfg, link); err == nil {
t.Fatal("symlink backup accepted")
}
}
if _, err := os.Stat(cfg.Database.DBPath + ".restore-pending"); !os.IsNotExist(err) {
t.Fatalf("failed validation left pending data: %v", err)
}
}
+1 -1
View File
@@ -87,7 +87,7 @@ func updateSettingHandler(svc *service.Container) gin.HandlerFunc {
if req.Key == "transcode.hw_enabled" || req.Key == "transcode.hw_accel" || req.Key == "transcoder.hardware_accel" || req.Key == "transcoder.encoder" {
svc.Transcoder.StopAll()
}
if req.Key == "cache.images_max_size_mb" || req.Key == "cache.images_originals_max_size_mb" {
if req.Key == "cache.images_max_size_mb" || req.Key == "cache.images_originals_max_size_mb" || req.Key == "cache.images_originals_ttl_hours" {
if svc.Scheduler != nil {
_ = svc.Scheduler.RunNowAsync(c.Request.Context(), "image_cache_cleanup")
}
-18
View File
@@ -1,18 +0,0 @@
package handler
import (
"net/http"
"github.com/gin-gonic/gin"
"github.com/truewhile/MeBox/internal/service"
)
func backgroundTasksHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
if svc == nil || svc.Tasks == nil {
c.JSON(http.StatusOK, service.TaskSnapshot{Active: []service.BackgroundTask{}, Recent: []service.BackgroundTask{}})
return
}
c.JSON(http.StatusOK, svc.Tasks.Snapshot())
}
}
+1 -18
View File
@@ -3,7 +3,6 @@ package handler
import (
"net/http"
"path/filepath"
"github.com/gin-gonic/gin"
@@ -42,28 +41,12 @@ func deleteBackupHandler(svc *service.Container) gin.HandlerFunc {
}
}
func downloadBackupHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
filename := c.Query("filename")
path, err := svc.Backup.ResolvePath(filename)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.FileAttachment(path, filepath.Base(path))
}
}
func restoreBackupHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
if err := svc.Backup.Restore(c.Request.Context(), c.Query("filename")); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
message := "数据库已恢复,请重启 MeBox 以刷新运行时配置"
if svc.Repo.DB.Dialector.Name() == "sqlite" {
message = "备份已校验并暂存,请重启 MeBox 后应用恢复"
}
c.JSON(http.StatusOK, gin.H{"message": message, "restart_required": true})
c.JSON(http.StatusOK, gin.H{"message": "restored — restart the server to apply"})
}
}
+2 -24
View File
@@ -124,32 +124,10 @@ func cloud115HLSSessionHandler(svc *service.Container) gin.HandlerFunc {
c.JSON(http.StatusBadRequest, gin.H{"error": "missing media_id"})
return
}
if c.Request.Context().Err() != nil {
c.AbortWithStatus(499)
if !enforceScopedPlaybackToken(c, mediaID) {
return
}
actualMediaID, err := svc.Cloud115.HLSProxy().SessionMediaID(c.Param("session"), c.Param("key"))
if err != nil {
c.JSON(http.StatusNotFound, gin.H{"code": "hls_session_not_found", "error": "hls session not found"})
return
}
if mediaID != actualMediaID {
c.JSON(http.StatusForbidden, gin.H{"error": "playback session media mismatch"})
return
}
if !enforceScopedPlaybackToken(c, actualMediaID) {
return
}
if svc.Repo == nil || svc.Repo.Media == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "media not found"})
return
}
m, findErr := svc.Repo.Media.FindByID(c.Request.Context(), actualMediaID)
if findErr != nil || m == nil || !mediaVisibleForRequest(c, svc, m) {
c.JSON(http.StatusNotFound, gin.H{"error": "media not found"})
return
}
err = svc.Cloud115.HLSProxy().ServeChild(
err := svc.Cloud115.HLSProxy().ServeChild(
c.Request.Context(),
c.Writer,
c.Request,
-19
View File
@@ -50,25 +50,6 @@ func embyServeImage(c *gin.Context, svc *service.Container, id, imageType, tag s
embyServePlaceholderImage(c)
return
}
// 媒体库主图:按 Emby 的做法,用库内几张成员海报拼一张封面(folder
// image)。没有自定义封面时才生成;自定义封面继续走下面的代理路径。
if svc.ImageProxy != nil && strings.EqualFold(strings.TrimSpace(imageType), "Primary") {
if sources, lib := service.LibraryCoverSourcesForLibrary(ctx, svc.Repo, id); lib != nil &&
strings.TrimSpace(lib.CoverURL) == "" && len(sources) > 0 {
width, height := libraryCoverRequestSize(c)
if data, err := svc.ImageProxy.LibraryCoverImage(ctx, id, sources, width, height); err == nil && len(data) > 0 {
c.Header("Cache-Control", "public, max-age=300, stale-while-revalidate=86400")
c.Header("Content-Type", "image/jpeg")
c.Header("Content-Length", strconv.Itoa(len(data)))
if c.Request.Method == http.MethodHead {
c.Status(http.StatusOK)
return
}
c.Data(http.StatusOK, "image/jpeg", data)
return
}
}
}
// PersonImageURL handles TMDb/remote people first and falls back to regular
// media/library artwork, so the official /Items/{personId}/Images route
// works for synthetic person IDs as well as normal item IDs.
+2 -10
View File
@@ -418,11 +418,7 @@ func embyVideoHLSPlaylistHandler(svc *service.Container) gin.HandlerFunc {
c.Status(http.StatusNotFound)
return
}
scope, ok := localHLSPlaybackScope(c)
if !ok {
return
}
err = svc.Stream.ServeHLSPlaylist(c.Writer, c.Request, c.Param("id"), scope)
err = svc.Stream.ServeHLSPlaylist(c.Writer, c.Request, c.Param("id"))
if errors.Is(err, service.ErrTranscodeDisabled) {
c.JSON(http.StatusConflict, gin.H{"error": "transcode disabled"})
return
@@ -448,11 +444,7 @@ func embyVideoHLSSegmentHandler(svc *service.Container) gin.HandlerFunc {
c.Status(http.StatusNotFound)
return
}
scope, ok := localHLSPlaybackScope(c)
if !ok {
return
}
if err := svc.Stream.ServeHLSSegment(c.Writer, c.Request, c.Param("id"), c.Param("seg"), scope); err != nil {
if err := svc.Stream.ServeHLSSegment(c.Writer, c.Request, c.Param("id"), c.Param("seg")); err != nil {
c.Status(http.StatusNotFound)
}
}
-37
View File
@@ -1,37 +0,0 @@
package handler
import (
"net/http"
"net/http/httptest"
"testing"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/service"
)
func TestStopHLSChecksMediaVisibility(t *testing.T) {
router, svc, secret := newPlaybackScopeTestRouter(t)
svc.Transcoder = service.NewTranscoderService(svc.Cfg, zap.NewNop(), svc.Repo, nil)
router.DELETE("/api/hls/:id", middleware.AuthRequired(secret), stopTranscodeHandler(svc))
req := httptest.NewRequest(http.MethodDelete, "/api/hls/nonexistent?session_id=player-a", nil)
req.Header.Set("Authorization", "Bearer "+signedTestToken(t, secret))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusNotFound {
t.Fatalf("stop of a nonexistent/invisible media: status=%d body=%s, want 404", rec.Code, rec.Body.String())
}
profile := &model.PlayProfile{UserID: "user-1", Name: "restricted", IsDefault: true, AllowedLibraryIDs: `["other-library"]`}
if err := svc.Repo.DB.Create(profile).Error; err != nil {
t.Fatal(err)
}
req = httptest.NewRequest(http.MethodDelete, "/api/hls/media-1?session_id=player-a", nil)
req.Header.Set("Authorization", "Bearer "+signedTestToken(t, secret))
rec = httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusNotFound {
t.Fatalf("invisible existing media can be stopped: %d %s", rec.Code, rec.Body.String())
}
}
-100
View File
@@ -1,100 +0,0 @@
package handler
import (
"context"
"crypto/sha256"
"encoding/hex"
"net/http"
"strconv"
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/truewhile/MeBox/internal/service"
)
// libraryCoverHandler 输出媒体库封面图(Emby 风格的成员海报拼图)。
//
// 之前 Web 端是拿两张成员海报在前端用 CSS 拼格子,Emby 客户端则直接拿一张
// 评分最高的海报,结果同一个库在不同客户端长得完全不一样。现在服务端生成
// 一张 16:9 的拼图(Emby 的 folder image 做法),Emby 客户端、Web 和 App
// 都指向这一份,外观统一。
func libraryCoverHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
id := strings.TrimSpace(c.Param("id"))
if id == "" || svc == nil || svc.ImageProxy == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "cover unavailable"})
return
}
ctx, cancel := context.WithTimeout(c.Request.Context(), 25*time.Second)
defer cancel()
sources, lib := service.LibraryCoverSourcesForLibrary(ctx, svc.Repo, id)
if lib == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "library not found"})
return
}
if strings.TrimSpace(lib.CoverURL) != "" {
// 管理员自定义封面优先,保持既有行为(代理该地址)。
if err := svc.ImageProxy.Serve(ctx, c.Writer, c.Request, lib.CoverURL); err != nil {
c.JSON(http.StatusNotFound, gin.H{"error": "custom cover unavailable"})
}
return
}
if len(sources) == 0 {
c.JSON(http.StatusNotFound, gin.H{"error": "no library items with artwork"})
return
}
width, height := libraryCoverRequestSize(c)
data, err := svc.ImageProxy.LibraryCoverImage(ctx, id, sources, width, height)
if err != nil || len(data) == 0 {
c.JSON(http.StatusNotFound, gin.H{"error": "cover generation failed"})
return
}
writeLibraryCoverImage(c, data, http.StatusOK)
}
}
// libraryCoverRequestSize 解析请求的封面尺寸:Web 用 w/h,Emby 用
// maxWidth/maxHeight,两种都兼容。服务端统一输出 16:9 画布(与该库在 Emby
// 里上报的 PrimaryImageAspectRatio 一致),客户端传的尺寸只当作「不超过」的
// 边界使用。
func libraryCoverRequestSize(c *gin.Context) (int, int) {
return firstPositiveQueryIntParam(c, "maxWidth", "maxwidth", "width", "w"),
firstPositiveQueryIntParam(c, "maxHeight", "maxheight", "height", "h")
}
func firstPositiveQueryIntParam(c *gin.Context, names ...string) int {
for _, name := range names {
raw := strings.TrimSpace(c.Query(name))
if raw == "" {
continue
}
if n, err := strconv.Atoi(raw); err == nil && n > 0 {
return n
}
if f, err := strconv.ParseFloat(raw, 64); err == nil && f > 0 {
return int(f)
}
}
return 0
}
func writeLibraryCoverImage(c *gin.Context, data []byte, status int) {
sum := sha256.Sum256(data)
etag := `"` + hex.EncodeToString(sum[:8]) + `"`
c.Header("ETag", etag)
c.Header("Cache-Control", "public, max-age=300, stale-while-revalidate=86400")
if match := strings.TrimSpace(c.GetHeader("If-None-Match")); match != "" && match == etag {
c.Status(http.StatusNotModified)
return
}
c.Header("Content-Type", "image/jpeg")
c.Header("Content-Length", strconv.Itoa(len(data)))
if c.Request.Method == http.MethodHead {
c.Status(status)
return
}
c.Data(status, "image/jpeg", data)
}
@@ -1,118 +0,0 @@
package handler
import (
"bytes"
"image/color"
"image/jpeg"
"net/http"
"net/http/httptest"
"net/url"
"path/filepath"
"testing"
"time"
"github.com/gin-gonic/gin"
"github.com/golang-jwt/jwt/v5"
"github.com/glebarez/sqlite"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service"
)
func coverTestToken(t *testing.T, secret, userID string) string {
t.Helper()
token := jwt.NewWithClaims(jwt.SigningMethodHS256, middleware.Claims{
UserID: userID,
Role: "admin",
RegisteredClaims: jwt.RegisteredClaims{
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour)),
IssuedAt: jwt.NewNumericDate(time.Now()),
Subject: userID,
},
})
signed, err := token.SignedString([]byte(secret))
if err != nil {
t.Fatalf("sign token: %v", err)
}
return signed
}
// TestLibraryCoverRouteServesImageThroughFullRouter 走完整的 handler.Register
// 路由表请求封面:既验证路由挂载正确,也验证鉴权链(AuthRequired +
// activeUserRequired)不会把它挡成 404。
func TestLibraryCoverRouteServesImageThroughFullRouter(t *testing.T) {
gin.SetMode(gin.TestMode)
const secret = "test-secret"
poster := coverTestJPEG(t, color.RGBA{R: 200, G: 120, B: 60, A: 255})
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/jpeg")
_, _ = w.Write(poster)
}))
defer upstream.Close()
parsed, _ := url.Parse(upstream.URL)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
t.Fatalf("open db: %v", err)
}
if err := db.AutoMigrate(&model.Library{}, &model.Media{}, &model.User{}); err != nil {
t.Fatalf("migrate: %v", err)
}
repos := repository.New(db)
user := model.User{Base: model.Base{ID: "user-1"}, Username: "viewer", IsActive: true}
if err := db.Create(&user).Error; err != nil {
t.Fatalf("create user: %v", err)
}
lib := model.Library{Name: "电影", Path: "/media/movies", Type: "movie", Enabled: true}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
for i := 0; i < 4; i++ {
row := model.Media{
Base: model.Base{ID: "m" + string(rune('a'+i))},
LibraryID: lib.ID,
Title: "Movie " + string(rune('A'+i)),
Path: "/media/movies/m" + string(rune('a'+i)) + ".mkv",
PosterURL: upstream.URL + "/p" + string(rune('1'+i)) + ".jpg",
}
if err := db.Create(&row).Error; err != nil {
t.Fatalf("create media: %v", err)
}
}
cfg := &config.Config{
Cache: config.CacheConfig{CacheDir: filepath.Join(t.TempDir(), "cache")},
Secrets: config.SecretsConfig{JWTSecret: secret},
}
proxy := service.NewImageProxy(cfg, zap.NewNop())
proxy.SetAllowedRemoteHostsProvider(func() []string { return []string{parsed.Host} })
svc := &service.Container{Repo: repos, ImageProxy: proxy}
engine := gin.New()
Register(engine, cfg, zap.NewNop(), svc)
token := coverTestToken(t, secret, user.ID)
req := httptest.NewRequest(http.MethodGet, "/api/libraries/"+lib.ID+"/cover?w=480&h=270&api_key="+token, nil)
w := httptest.NewRecorder()
engine.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("status = %d body=%s", w.Code, w.Body.String())
}
if ct := w.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Fatalf("content type = %q, want image/jpeg", ct)
}
img, err := jpeg.Decode(bytes.NewReader(w.Body.Bytes()))
if err != nil {
t.Fatalf("decode cover: %v", err)
}
if img.Bounds().Dx() != 480 || img.Bounds().Dy() != 270 {
t.Fatalf("cover size = %v, want 480x270", img.Bounds())
}
}
-228
View File
@@ -1,228 +0,0 @@
package handler
import (
"bytes"
"image"
"image/color"
"image/jpeg"
"net/http"
"net/http/httptest"
"net/url"
"path/filepath"
"testing"
"github.com/gin-gonic/gin"
"github.com/glebarez/sqlite"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service"
)
func coverTestJPEG(t *testing.T, c color.RGBA) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, 160, 240))
for y := 0; y < 240; y++ {
for x := 0; x < 160; x++ {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 90}); err != nil {
t.Fatalf("encode jpeg: %v", err)
}
return buf.Bytes()
}
// newLibraryCoverTestRouter 构造一个带真实图片代理的路由:成员海报来自
// 本地 httptest 服务,走的是与生产一致的 ImageProxy 抓取路径。
func newLibraryCoverTestRouter(t *testing.T) (*gin.Engine, *repository.Container, *httptest.Server) {
t.Helper()
gin.SetMode(gin.TestMode)
poster := coverTestJPEG(t, color.RGBA{R: 200, G: 120, B: 60, A: 255})
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/jpeg")
_, _ = w.Write(poster)
}))
t.Cleanup(upstream.Close)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
t.Fatalf("open db: %v", err)
}
if err := db.AutoMigrate(&model.Library{}, &model.Media{}); err != nil {
t.Fatalf("migrate: %v", err)
}
repos := repository.New(db)
parsed, err := url.Parse(upstream.URL)
if err != nil {
t.Fatalf("parse upstream: %v", err)
}
cfg := &config.Config{Cache: config.CacheConfig{CacheDir: filepath.Join(t.TempDir(), "cache")}}
proxy := service.NewImageProxy(cfg, zap.NewNop())
proxy.SetAllowedRemoteHostsProvider(func() []string { return []string{parsed.Host} })
svc := &service.Container{Repo: repos, ImageProxy: proxy}
router := gin.New()
router.GET("/libraries/:id/cover", libraryCoverHandler(svc))
router.HEAD("/libraries/:id/cover", libraryCoverHandler(svc))
return router, repos, upstream
}
func seedCoverLibrary(t *testing.T, repos *repository.Container, posterBase string, coverURL string) model.Library {
t.Helper()
lib := model.Library{Name: "电影", Path: "/media/movies", Type: "movie", Enabled: true, CoverURL: coverURL}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
for i := 0; i < 4; i++ {
row := model.Media{
Base: model.Base{ID: "m" + string(rune('a'+i))},
LibraryID: lib.ID,
Title: "Movie " + string(rune('A'+i)),
Path: "/media/movies/m" + string(rune('a'+i)) + ".mkv",
PosterURL: posterBase + "/p" + string(rune('1'+i)) + ".jpg",
}
if err := repos.DB.Create(&row).Error; err != nil {
t.Fatalf("create media: %v", err)
}
}
return lib
}
// TestLibraryCoverHandlerServesGeneratedCollage 覆盖 Web/App 的封面接口:
// 返回的是服务端拼好的 16:9 图片,而不是单张成员海报。
func TestLibraryCoverHandlerServesGeneratedCollage(t *testing.T) {
router, repos, upstream := newLibraryCoverTestRouter(t)
lib := seedCoverLibrary(t, repos, upstream.URL, "")
req := httptest.NewRequest(http.MethodGet, "/libraries/"+lib.ID+"/cover?w=480&h=270", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("status = %d body=%s", w.Code, w.Body.String())
}
if ct := w.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Fatalf("content type = %q, want image/jpeg", ct)
}
img, err := jpeg.Decode(bytes.NewReader(w.Body.Bytes()))
if err != nil {
t.Fatalf("decode cover: %v", err)
}
if img.Bounds().Dx() != 480 || img.Bounds().Dy() != 270 {
t.Fatalf("cover size = %v, want 480x270", img.Bounds())
}
if etag := w.Header().Get("ETag"); etag == "" {
t.Fatal("expected an ETag so clients can revalidate")
}
// 条件请求应拿到 304。
req = httptest.NewRequest(http.MethodGet, "/libraries/"+lib.ID+"/cover?w=480&h=270", nil)
req.Header.Set("If-None-Match", w.Header().Get("ETag"))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusNotModified {
t.Fatalf("conditional request status = %d, want 304", rec.Code)
}
}
func TestLibraryCoverHandlerUnknownLibraryReturns404(t *testing.T) {
router, _, _ := newLibraryCoverTestRouter(t)
req := httptest.NewRequest(http.MethodGet, "/libraries/missing/cover", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404", w.Code)
}
}
// TestLibraryCoverHandlerServesCustomCover 管理员填了 cover_url 时必须继续
// 用自定义封面,不能在不知不觉中换成拼图。
func TestLibraryCoverHandlerServesCustomCover(t *testing.T) {
router, repos, upstream := newLibraryCoverTestRouter(t)
lib := seedCoverLibrary(t, repos, upstream.URL, upstream.URL+"/custom.jpg")
req := httptest.NewRequest(http.MethodGet, "/libraries/"+lib.ID+"/cover", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("status = %d body=%s", w.Code, w.Body.String())
}
if _, err := jpeg.Decode(bytes.NewReader(w.Body.Bytes())); err != nil {
t.Fatalf("custom cover should be an image: %v", err)
}
}
// TestEmbyLibraryPrimaryUsesGeneratedCollage 保证 Emby 客户端请求库主图时
// 拿到的也是拼图(与 Web/App 同一份生成逻辑)。
func TestEmbyLibraryPrimaryUsesGeneratedCollage(t *testing.T) {
gin.SetMode(gin.TestMode)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
t.Fatalf("open db: %v", err)
}
if err := db.AutoMigrate(&model.Library{}, &model.Media{}); err != nil {
t.Fatalf("migrate: %v", err)
}
poster := coverTestJPEG(t, color.RGBA{G: 180, R: 90, B: 60, A: 255})
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/jpeg")
_, _ = w.Write(poster)
}))
defer upstream.Close()
parsed, _ := url.Parse(upstream.URL)
repos := repository.New(db)
lib := model.Library{Name: "剧集", Path: "/media/tv", Type: "tv", Enabled: true}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
for i := 0; i < 4; i++ {
row := model.Media{
Base: model.Base{ID: "e" + string(rune('a'+i))},
LibraryID: lib.ID,
Title: "Show " + string(rune('A'+i)),
Path: "/media/tv/show" + string(rune('a'+i)) + "/S01E01.mkv",
SeasonNum: 1, EpisodeNum: 1,
PosterURL: upstream.URL + "/p" + string(rune('1'+i)) + ".jpg",
}
if err := repos.DB.Create(&row).Error; err != nil {
t.Fatalf("create media: %v", err)
}
}
cfg := &config.Config{Cache: config.CacheConfig{CacheDir: filepath.Join(t.TempDir(), "cache")}}
proxy := service.NewImageProxy(cfg, zap.NewNop())
proxy.SetAllowedRemoteHostsProvider(func() []string { return []string{parsed.Host} })
svc := &service.Container{
Repo: repos,
Emby: service.NewEmbyService(cfg, zap.NewNop(), repos),
ImageProxy: proxy,
}
router := gin.New()
registerEmbyRoutes(router, "test-secret", svc)
req := httptest.NewRequest(http.MethodGet, "/Items/"+lib.ID+"/Images/Primary?maxWidth=400&maxHeight=300", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("status = %d body=%s", w.Code, w.Body.String())
}
img, err := jpeg.Decode(bytes.NewReader(w.Body.Bytes()))
if err != nil {
t.Fatalf("decode emby library cover: %v", err)
}
// 请求的是 400×300 的盒子,拼图按 16:9 装进去 → 400×225。
if img.Bounds().Dx() != 400 || img.Bounds().Dy() != 225 {
t.Fatalf("cover size = %v, want 400x225 (16:9 inside the requested box)", img.Bounds())
}
}
+119 -284
View File
@@ -4,11 +4,8 @@ package handler
import (
"context"
"errors"
"fmt"
"math"
"net/http"
"net/url"
"sort"
"strconv"
"strings"
"sync"
@@ -41,32 +38,8 @@ type webLibraryPayload struct {
RemoteSource string `json:"remote_source,omitempty"`
Total int64 `json:"total,omitempty"`
Cards []service.SeriesCardView `json:"cards,omitempty"`
// CoverImageURL 是客户端可直接使用的「有效封面」:管理员自定义的
// cover_url 优先,否则指向服务端生成的成员海报拼图接口。Web / App 都
// 读这一个字段,避免各自用卡片在前端拼格子、长得还不一样。
CoverImageURL string `json:"cover_image_url,omitempty"`
}
// effectiveLibraryCoverURL 计算库封面的有效地址。远程挂载库的封面由上游
// Emby 决定,这里返回空串,调用方继续用卡片拼图兜底。
func effectiveLibraryCoverURL(lib model.Library, isRemote bool) string {
if isRemote {
return ""
}
if raw := strings.TrimSpace(lib.CoverURL); raw != "" {
return raw
}
if strings.TrimSpace(lib.ID) == "" {
return ""
}
return "/api/libraries/" + url.PathEscape(lib.ID) + "/cover"
}
// remoteLibraryPreviewBudget 限制 /api/libraries?with_preview=1 里远程 Emby
// 库预览(总条数 + 最新卡片)的总耗时。这些预览只用于首页卡片,值得为它
// 排队等待的量很小;此前它跟随请求上下文,上游超时会让整页等 15 秒以上。
var remoteLibraryPreviewBudget = 3 * time.Second
// remoteLibraryItemTypes 远程库内容拉取时按 CollectionType 过滤直属条目,
// 避免电影库里的合集文件夹(Folder) 漏出为电影卡片。
func remoteLibraryItemTypes(collectionType string) string {
@@ -144,10 +117,9 @@ func listLibrariesHandler(svc *service.Container) gin.HandlerFunc {
}
for _, p := range previews {
out = append(out, webLibraryPayload{
Library: p.Library,
Total: p.Total,
Cards: service.NewSeriesCardViews(p.Cards),
CoverImageURL: effectiveLibraryCoverURL(p.Library, false),
Library: p.Library,
Total: p.Total,
Cards: service.NewSeriesCardViews(p.Cards),
})
}
} else {
@@ -166,7 +138,7 @@ func listLibrariesHandler(svc *service.Container) gin.HandlerFunc {
if counts != nil {
total = counts[l.ID]
}
out = append(out, webLibraryPayload{Library: l, Total: total, CoverImageURL: effectiveLibraryCoverURL(l, false)})
out = append(out, webLibraryPayload{Library: l, Total: total})
}
}
// 精确指定目标库时,如果目标全是本地库,就不必枚举远程挂载。
@@ -204,12 +176,6 @@ func listLibrariesHandler(svc *service.Container) gin.HandlerFunc {
remotePayloads[i] = webLibraryPayload{Library: v.Library, IsRemoteEmby: true, RemoteSource: v.AccountName}
}
if withPreview && len(allowedViews) > 0 {
// 远程库预览只是首页卡片上的「最新几条」,绝不能拖住整个
// 库列表:上游 Emby 一旦超时,每个远程库都要等满
// embyRemoteHTTPTimeout(15s),串起来就是 17–30 秒。
// 给整段预览一个总预算,超时就只返回库名,卡片留给下一次刷新。
previewCtx, cancelPreview := context.WithTimeout(ctx, remoteLibraryPreviewBudget)
defer cancelPreview()
const maxRemotePreviewWorkers = 6
sem := make(chan struct{}, maxRemotePreviewWorkers)
var wg sync.WaitGroup
@@ -221,11 +187,11 @@ func listLibrariesHandler(svc *service.Container) gin.HandlerFunc {
select {
case sem <- struct{}{}:
defer func() { <-sem }()
case <-previewCtx.Done():
case <-ctx.Done():
return
}
helper.Run(svc.Log, "media.remotePreview", func() {
acct := svc.EmbyRemote.AccountByID(previewCtx, v.AccountID)
acct := svc.EmbyRemote.AccountByID(ctx, v.AccountID)
if acct == nil {
return
}
@@ -237,10 +203,10 @@ func listLibrariesHandler(svc *service.Container) gin.HandlerFunc {
Name: v.Library.Name,
}
itemTypes := remoteLibraryItemTypes(v.CollectionType)
if _, total, err := svc.EmbyRemote.RemoteLibraryMedia(previewCtx, tmpMount, acct, v.RemoteID, itemTypes, 0, 1); err == nil {
if _, total, err := svc.EmbyRemote.RemoteLibraryMedia(ctx, tmpMount, acct, v.RemoteID, itemTypes, 0, 1); err == nil {
remotePayloads[i].Total = total
}
if cards, err := svc.EmbyRemote.RemoteLatestCards(previewCtx, tmpMount, acct, v.RemoteID, limit); err == nil {
if cards, err := svc.EmbyRemote.RemoteLatestCards(ctx, tmpMount, acct, v.RemoteID, limit); err == nil {
remotePayloads[i].Cards = service.NewSeriesCardViews(cards)
}
})
@@ -293,9 +259,9 @@ func getLibraryHandler(svc *service.Container) gin.HandlerFunc {
c.JSON(http.StatusNotFound, gin.H{"error": "not found"})
return
}
c.JSON(http.StatusOK, webLibraryPayload{Library: libs[0], CoverImageURL: effectiveLibraryCoverURL(libs[0], false)})
c.JSON(http.StatusOK, webLibraryPayload{Library: libs[0]})
} else {
c.JSON(http.StatusOK, webLibraryPayload{Library: *lib, CoverImageURL: effectiveLibraryCoverURL(*lib, false)})
c.JSON(http.StatusOK, webLibraryPayload{Library: *lib})
}
}
}
@@ -590,17 +556,19 @@ func listMediaHandler(svc *service.Container) gin.HandlerFunc {
})
return
}
items, total, err := svc.Media.ListMediaVisibleSortedPage(ctx, id, page, size, mediaVisibilityForRequest(c, svc), filters, sortSpec, history)
grouped, err := svc.Media.GroupedMediaVisibleFiltered(ctx, id, mediaVisibilityForRequest(c, svc), filters)
if err != nil {
writeInternalOrCanceled(c, err)
return
}
grouped = service.SortMediaItems(grouped, sortSpec.Field, sortSpec.Order, history)
items := service.PaginateMediaItems(grouped, page, size)
if items == nil {
items = []service.MediaItem{}
}
c.JSON(http.StatusOK, gin.H{
"items": items,
"total": total,
"total": len(grouped),
"page": page,
"page_size": size,
})
@@ -688,9 +656,6 @@ func paginateSlice[T any](items []T, page, size int) []T {
if len(items) == 0 {
return []T{}
}
if page-1 > len(items)/size {
return []T{}
}
start := (page - 1) * size
if start >= len(items) {
return []T{}
@@ -705,257 +670,127 @@ func paginateSlice[T any](items []T, page, size int) []T {
func searchMediaHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
ctx := c.Request.Context()
query := strings.TrimSpace(c.Query("q"))
q := c.Query("q")
visibility := mediaVisibilityForRequest(c, svc)
filter := service.MediaSearchFilter{
LibraryID: strings.TrimSpace(c.Query("library_id")),
YearMin: boundedSearchYear(c.Query("year_min")),
YearMax: boundedSearchYear(c.Query("year_max")),
}
for _, genre := range c.QueryArray("genre") {
if genre = strings.TrimSpace(genre); genre != "" && len(filter.Genres) < 8 {
filter.Genres = append(filter.Genres, genre)
groupVersions := c.DefaultQuery("group_versions", "1") != "0"
fetchRemote := func(limit int) []model.Media {
if svc.EmbyRemote == nil || strings.TrimSpace(q) == "" {
return nil
}
remoteItems, _ := svc.EmbyRemote.RemoteSearchMedia(ctx, q, limit, visibility)
return remoteItems
}
if filter.YearMin > 0 && filter.YearMax > 0 && filter.YearMin > filter.YearMax {
filter.YearMin, filter.YearMax = filter.YearMax, filter.YearMin
if c.DefaultQuery("group_series", "0") != "0" {
localItems, err := svc.Media.SearchMediaVisible(ctx, q, 50000, visibility)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
remoteItems := fetchRemote(50000)
all := service.GroupMediaSeriesItems(append(localItems, remoteItems...))
if c.Query("page") != "" || c.Query("page_size") != "" {
page, _ := strconv.Atoi(c.DefaultQuery("page", "1"))
size, _ := strconv.Atoi(c.DefaultQuery("page_size", "50"))
paged := paginateSlice(all, page, size)
c.JSON(http.StatusOK, gin.H{
"items": paged,
"total": len(all),
"page": page,
"page_size": size,
})
return
}
limit, _ := strconv.Atoi(c.DefaultQuery("limit", "50"))
if limit <= 0 {
limit = 50
}
if len(all) > limit {
all = all[:limit]
}
c.JSON(http.StatusOK, gin.H{"items": all})
return
}
page, _ := strconv.Atoi(c.DefaultQuery("page", "1"))
if page < 1 {
page = 1
}
size, _ := strconv.Atoi(c.DefaultQuery("page_size", "50"))
if size < 1 {
size = 50
} else if size > 2000 {
size = 2000
if c.Query("page") != "" || c.Query("page_size") != "" {
page, _ := strconv.Atoi(c.DefaultQuery("page", "1"))
size, _ := strconv.Atoi(c.DefaultQuery("page_size", "50"))
if !groupVersions {
localItems, _, err := svc.Media.SearchMediaVisiblePage(ctx, q, 1, 50000, visibility)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
remoteItems := fetchRemote(size * 2)
all := append(localItems, remoteItems...)
paged := paginateSlice(all, page, size)
c.JSON(http.StatusOK, gin.H{
"items": paged,
"total": len(all),
"page": page,
"page_size": size,
})
return
}
localItems, err := svc.Media.SearchMediaVisible(ctx, q, 50000, visibility)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
remoteItems := fetchRemote(size * 2)
all := append(localItems, remoteItems...)
grouped := service.GroupMediaVersions(all)
paged := service.PaginateMediaItems(grouped, page, size)
c.JSON(http.StatusOK, gin.H{
"items": paged,
"total": len(grouped),
"page": page,
"page_size": size,
})
return
}
limit, _ := strconv.Atoi(c.DefaultQuery("limit", "50"))
if limit < 1 {
if limit <= 0 {
limit = 50
} else if limit > 50000 {
limit = 50000
}
paginated := c.Query("page") != "" || c.Query("page_size") != ""
sortBy := normalizeMediaSearchSort(c.Query("sort"))
// 远程搜索与本地查询并行发起:本地结果不再排在远程超时之后才返回。
remoteCh := startRemoteMediaSearch(ctx, svc, query, visibility, filter)
// Sorting and grouping must operate on the same candidate set before
// either a page size or the caller's output limit is applied.
items, err := svc.Media.SearchMediaVisibleFiltered(ctx, query, 50000, visibility, filter)
if !groupVersions {
localItems, err := svc.Media.SearchMediaVisible(ctx, q, limit, visibility)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
remoteItems := fetchRemote(limit)
all := append(localItems, remoteItems...)
if len(all) > limit {
all = all[:limit]
}
if all == nil {
all = []model.Media{}
}
c.JSON(http.StatusOK, gin.H{"items": all})
return
}
localItems, err := svc.Media.SearchMediaVisible(ctx, q, 50000, visibility)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
if remoteCh != nil {
for _, item := range <-remoteCh {
items = append(items, item)
}
remoteItems := fetchRemote(limit)
all := append(localItems, remoteItems...)
grouped := service.GroupMediaVersions(all)
items := service.FirstMediaItems(grouped, limit)
if items == nil {
items = []service.MediaItem{}
}
if c.DefaultQuery("group_series", "0") != "0" {
items = service.GroupMediaSeriesItems(items)
sortSearchMedia(items, sortBy)
writeMediaSearchResults(c, items, page, size, limit, paginated)
return
}
if c.DefaultQuery("group_versions", "1") != "0" {
grouped := service.GroupMediaVersions(items)
if sortBy != "" {
sort.SliceStable(grouped, func(i, j int) bool {
return mediaSearchSortLess(grouped[i].Media, grouped[j].Media, sortBy)
})
}
writeMediaSearchResults(c, grouped, page, size, limit, paginated)
return
}
sortSearchMedia(items, sortBy)
writeMediaSearchResults(c, items, page, size, limit, paginated)
c.JSON(http.StatusOK, gin.H{"items": items})
}
}
const (
// remoteSearchItemLimit 是每个远程 Emby 挂载点返回的条目上限。此前这里传
// 50000(等于让上游做一次全库查询),上游稍慢就会一路跑到 5 秒超时,把每次
// 搜索都拖成秒级;搜索场景只需要少量候选,多出来的结果也会被分页丢掉。
remoteSearchItemLimit = 100
// remoteSearchBudget 限制远程结果最多给一次请求增加多少延迟。
remoteSearchBudget = 2 * time.Second
// remoteSearchCacheTTL 让同一关键词的翻页与重复搜索复用远程结果,避免每翻
// 一页都重新打一遍远程挂载点。
remoteSearchCacheTTL = 30 * time.Second
// remoteSearchFailureCacheTTL 用于上游超时/失败时的短缓存:部分结果只保留
// 几秒,避免反复吃满 remoteSearchBudget,又不至于长期压住后续重试。
remoteSearchFailureCacheTTL = 5 * time.Second
)
// startRemoteMediaSearch 并发发起远程搜索,返回一个只投递一次结果的通道。
// 不满足远程搜索条件时返回 nil。通道带缓冲,调用方即使提前返回也不会泄漏
// goroutine;请求结束时请求上下文被取消,远程调用会随之退出。
func startRemoteMediaSearch(
ctx context.Context,
svc *service.Container,
query string,
visibility service.MediaVisibility,
filter service.MediaSearchFilter,
) <-chan []model.Media {
if svc == nil || svc.EmbyRemote == nil || query == "" {
return nil
}
if filter.LibraryID != "" && !service.IsEmbyRemoteID(filter.LibraryID) {
return nil
}
results := make(chan []model.Media, 1)
go func() {
results <- remoteMediaSearchItems(ctx, svc, query, visibility, filter)
}()
return results
}
func remoteMediaSearchItems(
ctx context.Context,
svc *service.Container,
query string,
visibility service.MediaVisibility,
filter service.MediaSearchFilter,
) []model.Media {
cacheKey := remoteSearchCacheKey(query, visibility)
if svc.Cache != nil {
if cached, ok := svc.Cache.GetObject(cacheKey); ok {
if rows, ok := cached.([]model.Media); ok {
return filterRemoteSearchItems(rows, filter)
}
}
}
searchCtx, cancel := context.WithTimeout(ctx, remoteSearchBudget)
defer cancel()
remote, _ := svc.EmbyRemote.RemoteSearchMedia(searchCtx, query, remoteSearchItemLimit, visibility)
if svc.Cache != nil {
// 空结果也要缓存:否则查不到片名时每次都会重新等到 remoteSearchBudget。
if remote == nil {
remote = []model.Media{}
}
ttl := remoteSearchCacheTTL
if searchCtx.Err() != nil {
ttl = remoteSearchFailureCacheTTL
}
svc.Cache.SetObject(cacheKey, remote, ttl)
}
return filterRemoteSearchItems(remote, filter)
}
func filterRemoteSearchItems(remote []model.Media, filter service.MediaSearchFilter) []model.Media {
if len(remote) == 0 {
return nil
}
out := make([]model.Media, 0, len(remote))
for _, item := range remote {
if remoteSearchMatchesFilter(item, filter) {
out = append(out, item)
}
}
return out
}
// remoteSearchCacheKey 按可见范围隔离远程搜索结果,避免把某个用户可见挂载
// 点的结果复用给可见范围不同的用户。
func remoteSearchCacheKey(query string, visibility service.MediaVisibility) string {
allowed := append([]string(nil), visibility.AllowedLibraryIDs...)
hidden := append([]string(nil), visibility.HiddenLibraryIDs...)
sort.Strings(allowed)
sort.Strings(hidden)
return fmt.Sprintf("media:obj:remote-search:%t|%s|%s|%s",
visibility.IncludeNSFW, strings.Join(allowed, ","), strings.Join(hidden, ","), query)
}
func writeMediaSearchResults[T any](c *gin.Context, items []T, page, size, limit int, paginated bool) {
if items == nil {
items = []T{}
}
if paginated {
c.JSON(http.StatusOK, gin.H{
"items": paginateSlice(items, page, size),
"total": len(items), "page": page, "page_size": size,
})
return
}
if len(items) > limit {
items = items[:limit]
}
c.JSON(http.StatusOK, gin.H{"items": items})
}
func remoteSearchMatchesFilter(item model.Media, filter service.MediaSearchFilter) bool {
if filter.LibraryID != "" && item.LibraryID != filter.LibraryID && item.DisplayLibraryID != filter.LibraryID {
return false
}
if filter.YearMin > 0 && item.Year < filter.YearMin || filter.YearMax > 0 && item.Year > filter.YearMax {
return false
}
if len(filter.Genres) == 0 {
return true
}
normalize := func(value string) string {
return strings.ToLower(strings.ReplaceAll(strings.TrimSpace(value), " ", ""))
}
for _, genre := range strings.Split(strings.ReplaceAll(item.Genres, ",", ","), ",") {
for _, wanted := range filter.Genres {
if normalize(genre) == normalize(wanted) {
return true
}
}
}
return false
}
func normalizeMediaSearchSort(value string) string {
value = strings.TrimSpace(value)
switch value {
case "title_asc", "year_desc", "rating_desc":
return value
default:
return ""
}
}
func sortSearchMedia(items []model.Media, sortBy string) {
if sortBy == "" {
return
}
sort.SliceStable(items, func(i, j int) bool {
return mediaSearchSortLess(items[i], items[j], sortBy)
})
}
func mediaSearchSortLess(a, b model.Media, sortBy string) bool {
switch sortBy {
case "title_asc":
left, right := strings.ToLower(strings.TrimSpace(a.Title)), strings.ToLower(strings.TrimSpace(b.Title))
if left != right {
return left < right
}
case "year_desc":
if a.Year != b.Year {
return a.Year > b.Year
}
case "rating_desc":
if a.Rating != b.Rating {
return a.Rating > b.Rating
}
}
if !a.CreatedAt.Equal(b.CreatedAt) {
return a.CreatedAt.After(b.CreatedAt)
}
return a.ID < b.ID
}
func boundedSearchYear(value string) int {
year, err := strconv.Atoi(strings.TrimSpace(value))
if err != nil || year < 1870 || year > 2200 {
return 0
}
return year
}
func streamHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
ctx := c.Request.Context()
@@ -1,136 +0,0 @@
package handler
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/gin-gonic/gin"
"github.com/glebarez/sqlite"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service"
)
// TestListLibrariesRemotePreviewIsBounded 锁定远程库预览的总预算:
// 上游 Emby 卡住时,/api/libraries?with_preview=1 必须按时返回远程库条目
// (只是没有卡片),而不是跟着上游一起等十几秒。线上实测这个接口因此出现过
// 17.4 秒、30 秒(前端放弃)的响应,设置页也跟着整页空转。
func TestListLibrariesRemotePreviewIsBounded(t *testing.T) {
gin.SetMode(gin.TestMode)
oldBudget := remoteLibraryPreviewBudget
remoteLibraryPreviewBudget = 200 * time.Millisecond
t.Cleanup(func() { remoteLibraryPreviewBudget = oldBudget })
// Views / 用户列表正常返回,Preview(Items 查询)永远不返回:模拟上游半挂。
release := make(chan struct{})
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
path := r.URL.Path
switch {
case strings.HasSuffix(path, "/Views"):
_ = json.NewEncoder(w).Encode(map[string]any{
"Items": []map[string]any{{
"Id": "view-1", "Name": "远程动画", "CollectionType": "tvshows",
}},
})
return
case strings.HasSuffix(path, "/Users"):
// api_key 账号要先解析出远程用户 Id,否则请求路径会落到兜底 "0"。
_ = json.NewEncoder(w).Encode([]map[string]any{{"Id": "uid-1", "Name": "admin"}})
return
case r.Method == http.MethodPost:
_ = json.NewEncoder(w).Encode(map[string]any{
"AccessToken": "fake-token",
"User": map[string]any{"Id": "uid-1"},
})
return
}
select {
case <-release:
case <-r.Context().Done():
case <-time.After(30 * time.Second):
}
w.WriteHeader(http.StatusGatewayTimeout)
}))
defer upstream.Close()
defer close(release)
// 唯一库名 + cache=shared:预览协程会并发读库(账号解析)。
dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", strings.ReplaceAll(t.Name(), "/", "_"))
db, err := gorm.Open(sqlite.Open(dsn), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
if err := db.AutoMigrate(&model.Library{}, &model.Media{}, &model.StrmAccount{}, &model.EmbyMount{}, &model.Setting{}, &model.User{}, &model.PlayProfile{}); err != nil {
t.Fatal(err)
}
repos := repository.New(db)
if err := db.Create(&model.User{Base: model.Base{ID: "admin-1"}, Username: "admin", Role: "admin"}).Error; err != nil {
t.Fatal(err)
}
rawConfig, _ := json.Marshal(map[string]string{"url": upstream.URL, "token": "fake-token"})
acct := model.StrmAccount{
Base: model.Base{ID: "acct-1"}, Name: "远程Emby", Provider: model.StrmProviderEmbyRemote,
Config: string(rawConfig), Enabled: true,
}
if err := repos.StrmAccount.Create(t.Context(), &acct); err != nil {
t.Fatal(err)
}
mount := model.EmbyMount{
Base: model.Base{ID: "mount-1"}, AccountID: acct.ID, RemoteViewID: "view-1",
RemoteViewName: "远程动画", CollectionType: "tvshows", Enabled: true,
}
if err := repos.EmbyMount.Create(t.Context(), &mount); err != nil {
t.Fatal(err)
}
svc := &service.Container{
Repo: repos,
Media: service.NewMediaService(&config.Config{}, zap.NewNop(), repos),
EmbyRemote: service.NewEmbyRemoteService(&config.Config{}, zap.NewNop(), repos, service.NewCryptoService("", zap.NewNop())),
Cache: service.NewRuntimeCacheService(&config.Config{}, zap.NewNop()),
}
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Set(middleware.CtxUserID, "admin-1")
c.Set(middleware.CtxUserRole, "admin")
c.Request = httptest.NewRequest(http.MethodGet, "/api/libraries?with_preview=1&include_hidden=1", nil)
start := time.Now()
listLibrariesHandler(svc)(c)
elapsed := time.Since(start)
if w.Code != http.StatusOK {
t.Fatalf("status=%d body=%s", w.Code, w.Body.String())
}
if elapsed > 2*time.Second {
t.Fatalf("request took %v with a hung upstream; the preview budget must bound it", elapsed)
}
var payload []webLibraryPayload
if err := json.Unmarshal(w.Body.Bytes(), &payload); err != nil {
t.Fatalf("decode response: %v (%s)", err, w.Body.String())
}
var remote *webLibraryPayload
for i := range payload {
if payload[i].IsRemoteEmby {
remote = &payload[i]
break
}
}
if remote == nil {
t.Fatalf("remote library missing from the bounded response: %#v", payload)
}
if len(remote.Cards) != 0 || remote.Total != 0 {
t.Fatalf("expected the preview to be skipped on timeout, got total=%d cards=%d", remote.Total, len(remote.Cards))
}
}
-37
View File
@@ -1,37 +0,0 @@
package handler
import (
"net/http"
"strconv"
"github.com/gin-gonic/gin"
"github.com/truewhile/MeBox/internal/service"
)
func mediaQualityHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
page, _ := strconv.Atoi(c.DefaultQuery("page", "1"))
pageSize, _ := strconv.Atoi(c.DefaultQuery("page_size", "50"))
if page < 1 {
page = 1
} else if page > 1000000 {
page = 1000000
}
if pageSize < 1 {
pageSize = 50
} else if pageSize > 100 {
pageSize = 100
}
issue := c.DefaultQuery("issue", "unmatched")
if issue != "unmatched" && issue != "poster" && issue != "missing" {
c.JSON(http.StatusBadRequest, gin.H{"error": "未知检查分类"})
return
}
items, total, err := svc.Media.QualityIssues(c.Request.Context(), issue, page, pageSize)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"items": items, "total": total, "page": page, "page_size": pageSize})
}
}
@@ -1,88 +0,0 @@
package handler
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"github.com/gin-gonic/gin"
"github.com/glebarez/sqlite"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service"
)
func TestSearchMediaSortLimitFiltersAndDeepPagination(t *testing.T) {
gin.SetMode(gin.TestMode)
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
sqlDB, _ := db.DB()
defer sqlDB.Close()
if err := db.AutoMigrate(&model.Library{}, &model.Media{}, &model.User{}, &model.Setting{}, &model.PlayProfile{}); err != nil {
t.Fatal(err)
}
repos := repository.New(db)
lib := model.Library{Name: "Review", Path: "/review", Type: "movie", Enabled: true}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatal(err)
}
rows := make([]model.Media, 2001)
for i := range rows {
rows[i] = model.Media{
Base: model.Base{ID: fmt.Sprintf("review-%04d", i)}, LibraryID: lib.ID,
Title: fmt.Sprintf("Review %04d", i), Path: fmt.Sprintf("/review/%04d.mp4", i), Year: 2020, Genres: "Comedy",
}
}
rows[2000].Year = 2025
rows[2000].Genres = "Drama"
if err := db.CreateInBatches(rows, 100).Error; err != nil {
t.Fatal(err)
}
svc := &service.Container{Repo: repos, Media: service.NewMediaService(&config.Config{}, zap.NewNop(), repos)}
cases := []struct {
name, query, expectedID string
count, total, page, size int
}{
{name: "sorted output limit", query: "sort=year_desc&limit=1", expectedID: "review-2000", count: 1},
{name: "page past old 2000 cap", query: "sort=title_asc&page=2001&page_size=1", expectedID: "review-2000", count: 1, total: 2001, page: 2001, size: 1},
{name: "genre filtering", query: "genre=Drama&library_id=" + lib.ID, expectedID: "review-2000", count: 1},
{name: "normalized page", query: "page=0&page_size=0", count: 50, total: 2001, page: 1, size: 50},
{name: "bounded page size", query: "page=1&page_size=99999", count: 2000, total: 2001, page: 1, size: 2000},
{name: "huge page does not overflow", query: "page=" + strconv.Itoa(int(^uint(0)>>1)) + "&page_size=2000", count: 0, total: 2001, page: int(^uint(0) >> 1), size: 2000},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
w := httptest.NewRecorder()
ctx, _ := gin.CreateTestContext(w)
ctx.Set(middleware.CtxUserID, "admin")
ctx.Set(middleware.CtxUserRole, "admin")
ctx.Request = httptest.NewRequest(http.MethodGet, "/api/media?q=Review&group_versions=0&"+tc.query, nil)
searchMediaHandler(svc)(ctx)
var response struct {
Items []model.Media `json:"items"`
Total int `json:"total"`
Page int `json:"page"`
Size int `json:"page_size"`
}
if w.Code != http.StatusOK || json.Unmarshal(w.Body.Bytes(), &response) != nil {
t.Fatalf("search failed: status=%d body=%s", w.Code, w.Body.String())
}
if len(response.Items) != tc.count || response.Total != tc.total || response.Page != tc.page || response.Size != tc.size {
t.Fatalf("response count=%d total=%d page=%d size=%d, want %#v", len(response.Items), response.Total, response.Page, response.Size, tc)
}
if tc.expectedID != "" && response.Items[0].ID != tc.expectedID {
t.Fatalf("first ID=%q want %q", response.Items[0].ID, tc.expectedID)
}
})
}
}
@@ -1,153 +0,0 @@
package handler
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"github.com/gin-gonic/gin"
"github.com/glebarez/sqlite"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service"
)
// TestSearchMediaRemoteSearchBoundedAndCached 防止远程搜索再次退化:
// 1. 每个挂载点的 Limit 必须是 remoteSearchItemLimit,而不是历史全库查询的 50000;
// 2. 同一关键词的重复搜索应命中缓存,不再打上游;
// 3. 指定本地媒体库时不应枚举远程挂载点。
func TestSearchMediaRemoteSearchBoundedAndCached(t *testing.T) {
gin.SetMode(gin.TestMode)
var mu sync.Mutex
searchCalls := 0
searchLimits := make([]string, 0, 4)
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if term := r.URL.Query().Get("SearchTerm"); term != "" {
mu.Lock()
searchCalls++
searchLimits = append(searchLimits, r.URL.Query().Get("Limit"))
mu.Unlock()
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{
"TotalRecordCount": 1,
"Items": []map[string]any{{
"Id": "remote-1",
"Name": "远方剧集",
"Type": "Series",
"ProductionYear": 2024,
}},
})
}))
defer upstream.Close()
// 唯一库名 + cache=shared:远程搜索与本地查询会并发读库,
// 而 :memory: 下每个连接各自一个空库,并发连接会读到「no such table」。
dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", strings.ReplaceAll(t.Name(), "/", "_"))
db, err := gorm.Open(sqlite.Open(dsn), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
if err := db.AutoMigrate(&model.Library{}, &model.Media{}, &model.StrmAccount{}, &model.EmbyMount{}, &model.Setting{}, &model.User{}, &model.PlayProfile{}); err != nil {
t.Fatal(err)
}
repos := repository.New(db)
if err := repos.DB.Create(&model.User{Base: model.Base{ID: "user-1"}, Username: "admin", Role: "admin"}).Error; err != nil {
t.Fatal(err)
}
lib := model.Library{Name: "本地电影", Path: "/media/movies", Type: "movie", Enabled: true}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatal(err)
}
if err := db.Create(&model.Media{
Base: model.Base{ID: "local-1"}, LibraryID: lib.ID, Title: "流浪地球", Path: "/media/movies/流浪地球.mkv",
}).Error; err != nil {
t.Fatal(err)
}
rawConfig, _ := json.Marshal(map[string]string{"url": upstream.URL, "token": "fake-token"})
acct := model.StrmAccount{
Base: model.Base{ID: "acct"}, Name: "远程Emby", Provider: model.StrmProviderEmbyRemote,
Config: string(rawConfig), Enabled: true,
}
if err := repos.StrmAccount.Create(t.Context(), &acct); err != nil {
t.Fatal(err)
}
mount := model.EmbyMount{
Base: model.Base{ID: "mount-1"}, AccountID: acct.ID, RemoteViewID: "view-1",
RemoteViewName: "远程剧集", CollectionType: "tvshows", Enabled: true,
}
if err := repos.EmbyMount.Create(t.Context(), &mount); err != nil {
t.Fatal(err)
}
svc := &service.Container{
Repo: repos,
Media: service.NewMediaService(&config.Config{}, zap.NewNop(), repos),
EmbyRemote: service.NewEmbyRemoteService(&config.Config{}, zap.NewNop(), repos, service.NewCryptoService("", zap.NewNop())),
Cache: service.NewRuntimeCacheService(&config.Config{}, zap.NewNop()),
}
search := func(params string) []service.MediaItem {
t.Helper()
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Set(middleware.CtxUserID, "user-1")
c.Set(middleware.CtxUserRole, "admin")
c.Request = httptest.NewRequest(http.MethodGet, "/api/media?"+params, nil)
searchMediaHandler(svc)(c)
if w.Code != http.StatusOK {
t.Fatalf("search status=%d, body=%s", w.Code, w.Body.String())
}
var response struct {
Items []service.MediaItem `json:"items"`
}
if err := json.Unmarshal(w.Body.Bytes(), &response); err != nil {
t.Fatal(err)
}
return response.Items
}
callCount := func() int {
mu.Lock()
defer mu.Unlock()
return searchCalls
}
if items := search("q=远方&page=1&page_size=48"); len(items) != 1 || items[0].Title != "远方剧集" {
t.Fatalf("expected remote result, got %#v", items)
}
if calls := callCount(); calls != 1 {
t.Fatalf("expected exactly one upstream search, got %d", calls)
}
mu.Lock()
limit := searchLimits[0]
mu.Unlock()
if limit != "100" {
t.Fatalf("remote search Limit=%q, want bounded 100 (50000 means a full-library query)", limit)
}
// 同一关键词再来一次(例如刷新/重复搜索)应复用缓存,不再请求上游。
if items := search("q=远方&page=1&page_size=48"); len(items) != 1 || items[0].Title != "远方剧集" {
t.Fatalf("cached search lost the remote result: %#v", items)
}
if calls := callCount(); calls != 1 {
t.Fatalf("expected cached remote search to skip upstream, got %d calls", calls)
}
// 指定本地媒体库时不应枚举远程挂载点。
if items := search("q=流浪&page=1&page_size=48&library_id=" + lib.ID); len(items) != 1 || items[0].ID != "local-1" {
t.Fatalf("expected local-only result, got %#v", items)
}
if calls := callCount(); calls != 1 {
t.Fatalf("local library filter must not hit upstream, got %d calls", calls)
}
}
+2 -13
View File
@@ -5,8 +5,6 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/gin-gonic/gin"
@@ -29,17 +27,13 @@ func TestSearchMediaSeriesKeepsRemoteLibrariesSeparate(t *testing.T) {
"TotalRecordCount": 1,
"Items": []map[string]any{{
"Id": "same-series", "Name": "龙珠改", "Type": "Series",
"ProviderIds": map[string]string{"Tmdb": "61709"},
"ProductionYear": 2024, "Genres": []string{"Science Fiction"},
"ProviderIds": map[string]string{"Tmdb": "61709"},
}},
})
}))
defer upstream.Close()
// 唯一库名 + cache=shared:远程搜索与本地查询会并发读库,
// 而 :memory: 下每个连接各自一个空库,并发连接会读到「no such table」。
dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", strings.ReplaceAll(t.Name(), "/", "_"))
db, err := gorm.Open(sqlite.Open(dsn), &gorm.Config{})
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
@@ -114,9 +108,4 @@ func TestSearchMediaSeriesKeepsRemoteLibrariesSeparate(t *testing.T) {
}
seen[item.DisplayLibraryID] = true
}
remoteLibraryID := service.EncodeEmbyRemoteID("mount-a", "view-a")
items, total := request("page=1&page_size=8&library_id=" + url.QueryEscape(remoteLibraryID) + "&year_min=2020&genre=Science+Fiction")
if total != 1 || len(items) != 1 || items[0].DisplayLibraryID != remoteLibraryID {
t.Fatalf("remote library/year/genre filters lost results: total=%d items=%#v", total, items)
}
}
+1 -4
View File
@@ -559,10 +559,7 @@ func TestSearchMediaHandlerIncludesEmbyRemote(t *testing.T) {
}))
defer server.Close()
// 唯一库名 + cache=shared:远程搜索与本地查询会并发读库,
// 而 :memory: 下每个连接各自一个空库,并发连接会读到「no such table」。
dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", strings.ReplaceAll(t.Name(), "/", "_"))
db, err := gorm.Open(sqlite.Open(dsn), &gorm.Config{})
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{})
if err != nil {
t.Fatal(err)
}
+3 -3
View File
@@ -149,10 +149,10 @@ func readerDeleteSourceHandler(svc *service.Container) gin.HandlerFunc {
}
func readerDebugSourceHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Key string `json:"key" binding:"required"`
}
return func(c *gin.Context) {
var body struct {
Key string `json:"key" binding:"required"`
}
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
-3
View File
@@ -13,8 +13,6 @@ func registerAdminRoutes(api *gin.RouterGroup, cfg *config.Config, svc *service.
admin := api.Group("/admin")
admin.Use(middleware.AuthRequired(cfg.Secrets.JWTSecret), middleware.AdminRequired())
registerAdminUserRoutes(admin, svc)
admin.GET("/tasks", backgroundTasksHandler(svc))
admin.GET("/quality", mediaQualityHandler(svc))
registerAdminPermissionRoutes(admin, svc)
registerAdminSystemRoutes(admin, svc)
registerAdminBackupRoutes(admin, svc)
@@ -145,7 +143,6 @@ func registerAdminSystemRoutes(admin *gin.RouterGroup, svc *service.Container) {
func registerAdminBackupRoutes(admin *gin.RouterGroup, svc *service.Container) {
admin.GET("/backups", listBackupsHandler(svc))
admin.POST("/backups", createBackupHandler(svc))
admin.GET("/backups/download", downloadBackupHandler(svc))
admin.DELETE("/backups", deleteBackupHandler(svc))
admin.POST("/backups/restore", restoreBackupHandler(svc))
}
@@ -36,9 +36,6 @@ func registerAuthedLibraryRoutes(authed *gin.RouterGroup, svc *service.Container
authed.GET("/libraries", listLibrariesHandler(svc))
authed.POST("/libraries", middleware.AdminRequired(), createLibraryHandler(svc))
authed.GET("/libraries/:id", getLibraryHandler(svc))
// 服务端生成的媒体库封面(成员海报拼图),Web / App / Emby 共用同一份。
authed.GET("/libraries/:id/cover", libraryCoverHandler(svc))
authed.HEAD("/libraries/:id/cover", libraryCoverHandler(svc))
authed.PATCH("/libraries/:id", middleware.AdminRequired(), updateLibraryHandler(svc))
authed.PUT("/libraries/reorder", middleware.AdminRequired(), reorderLibrariesHandler(svc))
authed.DELETE("/libraries/:id", middleware.AdminRequired(), deleteLibraryHandler(svc))
@@ -21,8 +21,6 @@ func registerAuthedDLNARoutes(authed *gin.RouterGroup, svc *service.Container) {
}
func registerAuthedRealtimeRoutes(authed *gin.RouterGroup, svc *service.Container) {
// These streams carry operator tasks, including filesystem paths and scan
// errors. Their payloads are not scoped to an individual user's libraries.
authed.GET("/ws", middleware.AdminRequired(), wsHandler(svc))
authed.GET("/events", middleware.AdminRequired(), sseHandler(svc))
authed.GET("/ws", wsHandler(svc))
authed.GET("/events", sseHandler(svc))
}
+3 -43
View File
@@ -27,15 +27,7 @@ func hlsPlaylistHandler(svc *service.Container) gin.HandlerFunc {
if !enforceScopedPlaybackToken(c, m.ID) {
return
}
scope, ok := localHLSPlaybackScope(c)
if !ok {
return
}
err = svc.Stream.ServeHLSPlaylist(c.Writer, c.Request, c.Param("id"), scope)
if errors.Is(err, service.ErrHLSGenerationSuperseded) {
c.JSON(http.StatusConflict, gin.H{"code": "hls_generation_superseded", "error": err.Error()})
return
}
err = svc.Stream.ServeHLSPlaylist(c.Writer, c.Request, c.Param("id"))
if errors.Is(err, service.ErrMediaNotFound) {
c.JSON(http.StatusNotFound, gin.H{"error": "not found"})
return
@@ -70,11 +62,7 @@ func hlsSegmentHandler(svc *service.Container) gin.HandlerFunc {
if !enforceScopedPlaybackToken(c, m.ID) {
return
}
scope, ok := localHLSPlaybackScope(c)
if !ok {
return
}
err = svc.Stream.ServeHLSSegment(c.Writer, c.Request, c.Param("id"), c.Param("seg"), scope)
err = svc.Stream.ServeHLSSegment(c.Writer, c.Request, c.Param("id"), c.Param("seg"))
if err != nil {
c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
return
@@ -84,39 +72,11 @@ func hlsSegmentHandler(svc *service.Container) gin.HandlerFunc {
func stopTranscodeHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
m, err := findMediaForPlaybackEndpoint(c, svc, c.Param("id"))
if err != nil || m == nil || !mediaVisibleForRequest(c, svc, m) {
c.JSON(http.StatusNotFound, gin.H{"error": "media not found"})
return
}
if !enforceScopedPlaybackToken(c, m.ID) {
return
}
scope, ok := localHLSPlaybackScope(c)
if !ok {
return
}
if svc.Transcoder != nil {
svc.Transcoder.StopPlayback(m.ID, scope)
}
svc.Transcoder.StopJob(c.Param("id"))
c.Status(http.StatusNoContent)
}
}
func localHLSPlaybackScope(c *gin.Context) (service.HLSPlaybackScope, bool) {
userID := currentUserID(c)
if userID == "" {
c.JSON(http.StatusUnauthorized, gin.H{"error": "missing playback owner"})
return service.HLSPlaybackScope{}, false
}
scope, err := service.HLSPlaybackScopeForRequest(c.Request, userID)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return service.HLSPlaybackScope{}, false
}
return scope, true
}
func imageProxyHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
raw := c.Query("url")
+5 -8
View File
@@ -1,7 +1,9 @@
// Package handler — WebSocket endpoint.
//
// Administrators connect to /api/ws?token=... with their REST access token.
// Operator events arrive as {"topic":"...","payload":{...}}.
// Clients connect to /api/ws?token=... (the token is the same JWT used for
// REST calls). The first message they send is a JSON {"action":"subscribe",
// "topics":["scan","scrape","transcode"]}. Subsequent server-pushed events
// arrive as {"topic":"...","payload":{...}}.
package handler
import (
@@ -50,11 +52,8 @@ func wsHandler(svc *service.Container) gin.HandlerFunc {
sub := svc.WSHub.Subscribe(id, nil)
defer svc.WSHub.Unsubscribe(id)
// Reading processes WebSocket control frames and detects disconnects.
done := make(chan struct{})
conn.SetReadLimit(64 << 10)
// Reader: accept ping + subscription updates.
go func() {
defer close(done)
for {
if _, _, err := conn.NextReader(); err != nil {
_ = conn.Close()
@@ -68,8 +67,6 @@ func wsHandler(svc *service.Container) gin.HandlerFunc {
defer ticker.Stop()
for {
select {
case <-done:
return
case ev, ok := <-sub.Out:
if !ok {
return
-93
View File
@@ -1,93 +0,0 @@
package handler
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/gin-gonic/gin"
"github.com/golang-jwt/jwt/v5"
"github.com/gorilla/websocket"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/service"
)
func realtimeTestRouter(hub *service.Hub) *gin.Engine {
gin.SetMode(gin.TestMode)
router := gin.New()
authed := router.Group("/api")
authed.Use(middleware.AuthRequired("realtime-test"))
registerAuthedRealtimeRoutes(authed, &service.Container{WSHub: hub})
return router
}
func realtimeTestToken(t *testing.T, role string) string {
t.Helper()
token, err := jwt.NewWithClaims(jwt.SigningMethodHS256, middleware.Claims{
UserID: "viewer", Role: role,
RegisteredClaims: jwt.RegisteredClaims{ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour))},
}).SignedString([]byte("realtime-test"))
if err != nil {
t.Fatal(err)
}
return token
}
func TestRealtimeStreamsRejectOrdinaryUsers(t *testing.T) {
router := realtimeTestRouter(nil)
for _, path := range []string{"/api/ws", "/api/events"} {
for _, role := range []string{"", "user"} {
req := httptest.NewRequest(http.MethodGet, path, nil)
want := http.StatusUnauthorized
if role != "" {
req.Header.Set("Authorization", "Bearer "+realtimeTestToken(t, role))
want = http.StatusForbidden
}
recorder := httptest.NewRecorder()
router.ServeHTTP(recorder, req)
if recorder.Code != want {
t.Fatalf("%s role %q: status %d, want %d", path, role, recorder.Code, want)
}
}
}
}
func TestAdminWebSocketReceivesTaskEvents(t *testing.T) {
hub := service.NewHub(zap.NewNop())
go hub.Run()
defer hub.Stop()
server := httptest.NewServer(realtimeTestRouter(hub))
defer server.Close()
url := "ws" + strings.TrimPrefix(server.URL, "http") + "/api/ws?token=" + realtimeTestToken(t, "admin")
conn, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
stop := make(chan struct{})
defer close(stop)
go func() {
ticker := time.NewTicker(10 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-stop:
return
case <-ticker.C:
hub.Publish("task", service.BackgroundTask{ID: "task-1", SourcePath: "/media/private"})
}
}
}()
_ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))
var event service.Event
if err := conn.ReadJSON(&event); err != nil {
t.Fatal(err)
}
if event.Topic != "task" || event.Payload.(map[string]any)["source_path"] != "/media/private" {
t.Fatalf("event = %#v", event)
}
}
-23
View File
@@ -1,23 +0,0 @@
package model
import "time"
// BackgroundTaskRecord persists the operator-facing lifecycle of scan,
// organize, and other long-running application tasks.
type BackgroundTaskRecord struct {
ID string `gorm:"primaryKey;size:36" json:"id"`
Kind string `gorm:"index;size:32;not null" json:"kind"`
Name string `gorm:"size:255;not null" json:"name"`
Status string `gorm:"index;size:24;not null" json:"status"`
Stage string `gorm:"size:64" json:"stage"`
SourcePath string `gorm:"size:2048" json:"source_path"`
DestPath string `gorm:"size:2048" json:"dest_path"`
Message string `gorm:"type:text" json:"message"`
Error string `gorm:"type:text" json:"error"`
Details string `gorm:"type:text" json:"details"`
Metrics string `gorm:"type:text" json:"metrics"`
Items string `gorm:"type:text" json:"items"`
StartedAt time.Time `gorm:"index" json:"started_at"`
UpdatedAt time.Time `json:"updated_at"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
}
+6 -11
View File
@@ -1,7 +1,5 @@
package model
import "time"
// Library 表示一个逻辑媒体库。Path 保留为兼容字段,指向第一条 LibraryRoot。
type Library struct {
Base
@@ -28,15 +26,12 @@ type LibraryRoot struct {
// Media 是单个可播放项。剧集链接到 SeriesID;电影 SeriesID == ""。
type Media struct {
Base
AvailabilityState string `gorm:"index;size:16;not null;default:available" json:"availability_state"`
MissingSince *time.Time `json:"missing_since,omitempty"`
LastSeenAt *time.Time `json:"last_seen_at,omitempty"`
LibraryID string `gorm:"index;size:36;index:idx_media_library_release,priority:1" json:"library_id"`
LibraryRootID string `gorm:"index;size:36" json:"library_root_id,omitempty"`
SeriesID string `gorm:"index;size:128" json:"series_id,omitempty"`
Title string `gorm:"size:255;not null" json:"title"`
OriginalName string `gorm:"size:255" json:"original_name,omitempty"`
EpisodeTitle string `gorm:"size:255" json:"episode_title,omitempty"`
LibraryID string `gorm:"index;size:36;index:idx_media_library_release,priority:1" json:"library_id"`
LibraryRootID string `gorm:"index;size:36" json:"library_root_id,omitempty"`
SeriesID string `gorm:"index;size:128" json:"series_id,omitempty"`
Title string `gorm:"size:255;not null" json:"title"`
OriginalName string `gorm:"size:255" json:"original_name,omitempty"`
EpisodeTitle string `gorm:"size:255" json:"episode_title,omitempty"`
// Path 的唯一性由 database 包按方言显式建索引,这里刻意不用 uniqueIndex:
// PostgreSQL 的 btree 单条索引项上限 2704 字节,而 varchar(1024) 在中文路径
// 下可达 3072 字节,直接对整列建唯一索引会让长路径的扫描插入直接失败
-1
View File
@@ -61,7 +61,6 @@ func AllModels() []interface{} {
&StrmUploadTask{},
&StrmDirCache{},
&ScrapeTask{},
&BackgroundTaskRecord{},
&EmbyMount{},
&ReaderBookSource{},
&ReaderSourceState{},
+8
View File
@@ -178,6 +178,14 @@ func (r *MediaRepository) YearRange(ctx context.Context, filter MediaQueryFilter
return min, max, nil
}
func (r *MediaRepository) indexMediaBestEffort(ctx context.Context, media model.Media) {
backend, ok := r.searchBackend.(MediaSearchSyncBackend)
if !ok {
return
}
_ = backend.IndexMedia(ctx, []model.Media{media})
}
// FindByID returns the media row or (nil, nil).
func (r *MediaRepository) FindByID(ctx context.Context, id string) (*model.Media, error) {
var m model.Media
+3 -38
View File
@@ -5,7 +5,6 @@ import (
"errors"
"path/filepath"
"strings"
"time"
"gorm.io/gorm"
"gorm.io/gorm/clause"
@@ -97,34 +96,15 @@ func (r *MediaRepository) UpsertBatchWithAliases(ctx context.Context, items []Me
// indexByIDBestEffort 在事务提交后按 ID 回读最新行并同步搜索索引。
func (r *MediaRepository) indexByIDBestEffort(ctx context.Context, ids []string) {
backend, ok := r.searchBackend.(MediaSearchSyncBackend)
if !ok {
if _, ok := r.searchBackend.(MediaSearchSyncBackend); !ok {
return
}
unique := make([]string, 0, len(ids))
seen := make(map[string]struct{}, len(ids))
for _, id := range ids {
if id == "" {
continue
}
if _, exists := seen[id]; !exists {
seen[id] = struct{}{}
unique = append(unique, id)
}
}
// Read committed rows and submit one bulk per bounded batch. Keep network
// work outside the transaction and below SQLite's parameter limit.
const batchSize = 500
for start := 0; start < len(unique); start += batchSize {
batch := unique[start:min(start+batchSize, len(unique))]
rows, err := r.ListMediaByIDs(ctx, batch)
if err == nil {
err = backend.IndexMedia(ctx, rows)
}
if err != nil {
// The primary write has committed; do not report a failed upsert and
// make scanners repeat it. Surface the separate index failure instead.
r.db.Logger.Error(ctx, "media saved but search index sync failed for %d items: %v", len(batch), err)
if fresh, err := r.FindByID(ctx, id); err == nil && fresh != nil {
r.indexMediaBestEffort(ctx, *fresh)
}
}
}
@@ -164,12 +144,6 @@ func (r *MediaRepository) upsertWithDB(ctx context.Context, db *gorm.DB, m *mode
existing.Path = m.Path
existing.DeletedAt = gorm.DeletedAt{}
}
if _, ok := updates["availability_state"]; ok {
existing.AvailabilityState = "available"
existing.MissingSince = nil
now := updates["last_seen_at"].(time.Time)
existing.LastSeenAt = &now
}
*m = existing
return existing.ID, nil
}
@@ -223,10 +197,6 @@ func (r *MediaRepository) findOrCreateMediaByPath(ctx context.Context, db *gorm.
if m.ScrapeStatus == "" {
m.ScrapeStatus = "pending"
}
now := time.Now()
m.AvailabilityState = "available"
m.MissingSince = nil
m.LastSeenAt = &now
// PostgreSQL 唯一键异常会中止整个事务,之后重查也会失败。
// DO NOTHING 保持事务可用,并兼容 path 的表达式唯一索引。
result := db.WithContext(ctx).Clauses(clause.OnConflict{DoNothing: true}).Create(m)
@@ -276,11 +246,6 @@ func (r *MediaRepository) findMediaByAlias(ctx context.Context, db *gorm.DB, cur
func mediaUpsertUpdates(existing, incoming model.Media, reclaimLibrary bool) map[string]any {
updates := map[string]any{}
if existing.AvailabilityState != "available" || existing.MissingSince != nil {
updates["availability_state"] = "available"
updates["missing_since"] = nil
updates["last_seen_at"] = time.Now()
}
addMediaFileScanUpdates(updates, existing, incoming)
addMediaTitleUpdates(updates, existing, incoming)
addMediaExternalIDUpdates(updates, existing, incoming)
+4 -119
View File
@@ -4,10 +4,8 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strings"
@@ -137,113 +135,14 @@ func (b *OpenSearchMediaBackend) EnsureIndex(ctx context.Context) error {
}
func (b *OpenSearchMediaBackend) IndexMedia(ctx context.Context, rows []model.Media) error {
const batchSize = 500
var failures []error
for start := 0; start < len(rows); start += batchSize {
if err := b.indexMediaBatch(ctx, rows[start:min(start+batchSize, len(rows))]); err != nil {
failures = append(failures, err)
if ctx.Err() != nil {
break
}
}
}
return errors.Join(failures...)
}
type openSearchBulkResponse struct {
Errors bool `json:"errors"`
Items []struct {
Index struct {
ID string `json:"_id"`
Status int `json:"status"`
Error *struct {
Type string `json:"type"`
} `json:"error"`
} `json:"index"`
} `json:"items"`
}
func (b *OpenSearchMediaBackend) indexMediaBatch(ctx context.Context, rows []model.Media) error {
if len(rows) == 0 {
return nil
}
pending := rows
var failures []error
const attempts = 3
for attempt := 0; attempt < attempts; attempt++ {
if err := ctx.Err(); err != nil {
return errors.Join(append(failures, err)...)
}
if attempt > 0 {
timer := time.NewTimer(time.Duration(100*(1<<(attempt-1))) * time.Millisecond)
select {
case <-ctx.Done():
timer.Stop()
return errors.Join(append(failures, ctx.Err())...)
case <-timer.C:
}
}
response, err := b.submitMediaBulk(ctx, pending)
if err != nil {
if retryableOpenSearchError(err) && attempt < attempts-1 {
continue
}
return errors.Join(append(failures, err)...)
}
if len(response.Items) != len(pending) {
return errors.Join(append(failures, fmt.Errorf("opensearch bulk returned %d item results for %d documents", len(response.Items), len(pending)))...)
}
var retry []model.Media
failed := false
for i, item := range response.Items {
result := item.Index
if result.Status >= 200 && result.Status < 300 && result.Error == nil {
continue
}
failed = true
if retryableOpenSearchStatus(result.Status) && attempt < attempts-1 {
retry = append(retry, pending[i])
continue
}
errorType := "unknown"
if result.Error != nil {
errorType = result.Error.Type
}
failures = append(failures, fmt.Errorf("opensearch bulk document %q failed: status %d (%s)", pending[i].ID, result.Status, errorType))
}
if response.Errors && !failed {
failures = append(failures, errors.New("opensearch bulk reported errors without failed item results"))
}
if len(retry) == 0 {
break
}
pending = retry
}
return errors.Join(failures...)
}
func retryableOpenSearchStatus(status int) bool {
return status == http.StatusRequestTimeout || status == http.StatusTooManyRequests || status >= 500
}
func retryableOpenSearchError(err error) bool {
var statusError *openSearchHTTPError
if errors.As(err, &statusError) {
return retryableOpenSearchStatus(statusError.status)
}
var networkError net.Error
return errors.As(err, &networkError)
}
func (b *OpenSearchMediaBackend) submitMediaBulk(ctx context.Context, rows []model.Media) (*openSearchBulkResponse, error) {
if len(rows) == 0 {
return &openSearchBulkResponse{}, nil
}
var bulk bytes.Buffer
enc := json.NewEncoder(&bulk)
for _, row := range rows {
if err := enc.Encode(map[string]any{"index": map[string]any{"_index": b.index, "_id": row.ID}}); err != nil {
return nil, err
return err
}
if err := enc.Encode(map[string]any{
"id": row.ID,
@@ -256,24 +155,10 @@ func (b *OpenSearchMediaBackend) submitMediaBulk(ctx context.Context, rows []mod
"deleted": row.DeletedAt.Valid,
"created_at": row.CreatedAt,
}); err != nil {
return nil, err
return err
}
}
var response openSearchBulkResponse
if err := b.do(ctx, http.MethodPost, "/_bulk", &bulk, "application/x-ndjson", &response); err != nil {
return nil, err
}
return &response, nil
}
type openSearchHTTPError struct {
method string
path string
status int
}
func (e *openSearchHTTPError) Error() string {
return fmt.Sprintf("opensearch %s %s returned %d", e.method, e.path, e.status)
return b.do(ctx, http.MethodPost, "/_bulk", &bulk, "application/x-ndjson", nil)
}
func (b *OpenSearchMediaBackend) doJSON(ctx context.Context, method, path string, body any, out any) error {
@@ -305,7 +190,7 @@ func (b *OpenSearchMediaBackend) do(ctx context.Context, method, path string, bo
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return &openSearchHTTPError{method: method, path: path, status: resp.StatusCode}
return fmt.Errorf("opensearch %s %s returned %d", method, path, resp.StatusCode)
}
if out == nil {
_, _ = io.Copy(io.Discard, resp.Body)
-164
View File
@@ -1,23 +1,12 @@
package repository
import (
"bufio"
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
)
func TestOpenSearchMediaBackendSearchesIDs(t *testing.T) {
@@ -66,156 +55,3 @@ func TestOpenSearchMediaBackendSearchesIDs(t *testing.T) {
t.Fatalf("ids=%#v total=%d", ids, total)
}
}
func bulkRequestIDs(r *http.Request) []string {
var ids []string
scanner := bufio.NewScanner(r.Body)
for scanner.Scan() {
var action struct {
Index struct {
ID string `json:"_id"`
} `json:"index"`
}
_ = json.Unmarshal(scanner.Bytes(), &action)
ids = append(ids, action.Index.ID)
if !scanner.Scan() {
break
}
}
return ids
}
func writeBulkResults(w http.ResponseWriter, ids []string, statuses map[string]int) {
items := make([]any, 0, len(ids))
failed := false
for _, id := range ids {
status := statuses[id]
if status == 0 {
status = http.StatusCreated
}
result := map[string]any{"_id": id, "status": status}
if status >= 400 {
failed = true
result["error"] = map[string]any{"type": "test_failure"}
}
items = append(items, map[string]any{"index": result})
}
_ = json.NewEncoder(w).Encode(map[string]any{"errors": failed, "items": items})
}
func TestOpenSearchBulkRetriesOnlyTransientFailedDocuments(t *testing.T) {
var requests [][]string
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ids := bulkRequestIDs(r)
requests = append(requests, ids)
statuses := map[string]int{}
if len(requests) == 1 {
statuses["retry"] = http.StatusTooManyRequests
statuses["invalid"] = http.StatusBadRequest
}
writeBulkResults(w, ids, statuses)
}))
defer upstream.Close()
backend := NewOpenSearchMediaBackend(config.SearchConfig{Backend: "opensearch", OpenSearchURL: upstream.URL})
rows := []model.Media{{Base: model.Base{ID: "ok"}}, {Base: model.Base{ID: "retry"}}, {Base: model.Base{ID: "invalid"}}}
err := backend.IndexMedia(t.Context(), rows)
if err == nil || !strings.Contains(err.Error(), `"invalid"`) {
t.Fatalf("error = %v, want permanent document failure", err)
}
if len(requests) != 2 || strings.Join(requests[1], ",") != "retry" {
t.Fatalf("requests = %#v; successful/permanent documents must not be retried", requests)
}
}
func TestOpenSearchBulkBoundsRetriesAndRejectsMalformedResponses(t *testing.T) {
for _, scenario := range []string{"transient-item", "transient-http", "invalid-http", "missing-results"} {
t.Run(scenario, func(t *testing.T) {
calls := 0
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
switch scenario {
case "transient-item":
writeBulkResults(w, bulkRequestIDs(r), map[string]int{"m1": http.StatusServiceUnavailable})
case "transient-http":
w.WriteHeader(http.StatusServiceUnavailable)
case "invalid-http":
w.WriteHeader(http.StatusBadRequest)
case "missing-results":
fmt.Fprint(w, `{"errors":false,"items":[]}`)
}
}))
defer upstream.Close()
backend := NewOpenSearchMediaBackend(config.SearchConfig{Backend: "opensearch", OpenSearchURL: upstream.URL})
if err := backend.IndexMedia(t.Context(), []model.Media{{Base: model.Base{ID: "m1"}}}); err == nil {
t.Fatal("failed indexing was reported as successful")
}
want := 1
if strings.HasPrefix(scenario, "transient") {
want = 3
}
if calls != want {
t.Fatalf("calls = %d, want %d", calls, want)
}
})
}
}
func TestOpenSearchBulkHonorsCancellationDuringRetry(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
calls := 0
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
writeBulkResults(w, bulkRequestIDs(r), map[string]int{"m1": http.StatusTooManyRequests})
cancel()
}))
defer upstream.Close()
backend := NewOpenSearchMediaBackend(config.SearchConfig{Backend: "opensearch", OpenSearchURL: upstream.URL})
err := backend.IndexMedia(ctx, []model.Media{{Base: model.Base{ID: "m1"}}})
if !errors.Is(err, context.Canceled) || calls != 1 {
t.Fatalf("error = %v, calls = %d", err, calls)
}
}
func TestMediaUpsertBatchIndexesCommittedRowsTogether(t *testing.T) {
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)})
if err != nil {
t.Fatal(err)
}
sqlDB, _ := db.DB()
sqlDB.SetMaxOpenConns(1)
defer sqlDB.Close()
if err := db.AutoMigrate(&model.Media{}); err != nil {
t.Fatal(err)
}
repos := New(db)
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
calls := 0
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
ids := bulkRequestIDs(r)
if len(ids) != 5 {
t.Errorf("bulk contains %d documents, want 5", len(ids))
}
// With only one DB connection this also detects network work inside the
// transaction: a separate read could not observe the committed batch.
var count int64
if err := db.WithContext(ctx).Model(&model.Media{}).Count(&count).Error; err != nil || count != 5 {
t.Errorf("committed rows = %d, error = %v", count, err)
}
writeBulkResults(w, ids, nil)
}))
defer upstream.Close()
repos.Media.SetSearchBackend(NewOpenSearchMediaBackend(config.SearchConfig{Backend: "opensearch", OpenSearchURL: upstream.URL}))
var rows []*model.Media
for i := 0; i < 5; i++ {
rows = append(rows, &model.Media{LibraryID: "lib", Path: fmt.Sprintf("/media/%d.mkv", i)})
}
if err := repos.Media.UpsertBatch(ctx, rows); err != nil {
t.Fatal(err)
}
if calls != 1 {
t.Fatalf("index requests = %d, want one bulk", calls)
}
}
@@ -24,14 +24,6 @@ func (s *ApiConfigService) TestConnection(ctx context.Context, provider string)
switch provider {
case "tmdb":
return s.testTMDb(cfg)
case "bangumi":
return s.testBangumi(ctx, cfg)
case "thetvdb":
return s.testTheTVDB(ctx, cfg)
case "fanart":
return s.testFanart(ctx, cfg)
case "douban":
return s.testDouban(ctx, cfg)
case "openai":
return s.testOpenAI(cfg)
case "deepseek":
@@ -1,191 +0,0 @@
package service
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
"github.com/truewhile/MeBox/internal/model"
)
// Bangumi's public metadata endpoints work without a token. If one was saved,
// also check /v0/me: public endpoints alone may accept an expired token.
func (s *ApiConfigService) testBangumi(ctx context.Context, cfg *model.APIConfig) (string, error) {
req, err := newMetadataProbeRequest(ctx, cfg, "https://api.bgm.tv", "/v0/subjects/253", http.MethodGet, nil)
if err != nil {
return "error", err
}
if token := strings.TrimSpace(cfg.APIKey); token != "" {
req.Header.Set("Authorization", "Bearer "+token)
identityReq := req.Clone(ctx)
identityURL := *req.URL
identityURL.Path = strings.TrimSuffix(identityURL.Path, "/subjects/253") + "/me"
identityReq.URL = &identityURL
var identity struct {
ID int `json:"id"`
}
if result, err := testMetadataProbeJSON(identityReq, "Bangumi", &identity); err != nil {
return result, err
}
if identity.ID <= 0 {
return "error", errors.New("Bangumi returned an invalid user response")
}
}
var subject struct {
ID int `json:"id"`
Name string `json:"name"`
}
if result, err := testMetadataProbeJSON(req, "Bangumi", &subject); err != nil {
return result, err
}
if subject.ID != 253 || strings.TrimSpace(subject.Name) == "" {
return "error", errors.New("Bangumi returned an invalid subject response")
}
return "success", nil
}
func (s *ApiConfigService) testTheTVDB(ctx context.Context, cfg *model.APIConfig) (string, error) {
key := strings.TrimSpace(cfg.APIKey)
if key == "" {
return "error", errors.New("TheTVDB API key is required")
}
body, err := json.Marshal(map[string]string{"apikey": key})
if err != nil {
return "error", err
}
req, err := newMetadataProbeRequest(ctx, cfg, "https://api4.thetvdb.com/v4", "/login", http.MethodPost, strings.NewReader(string(body)))
if err != nil {
return "error", err
}
req.Header.Set("Content-Type", "application/json")
var response struct {
Status string `json:"status"`
Data struct {
Token string `json:"token"`
} `json:"data"`
}
if result, err := testMetadataProbeJSON(req, "TheTVDB", &response); err != nil {
return result, err
}
if response.Status != "success" || strings.TrimSpace(response.Data.Token) == "" {
return "error", errors.New("TheTVDB login did not return a valid token")
}
return "success", nil
}
func (s *ApiConfigService) testFanart(ctx context.Context, cfg *model.APIConfig) (string, error) {
key := strings.TrimSpace(cfg.APIKey)
if key == "" {
return "error", errors.New("Fanart API key is required")
}
req, err := newMetadataProbeRequest(ctx, cfg, "https://webservice.fanart.tv/v3", "/movies/550", http.MethodGet, nil)
if err != nil {
return "error", err
}
query := req.URL.Query()
query.Set("api_key", key)
req.URL.RawQuery = query.Encode()
var response struct {
TMDbID json.Number `json:"tmdb_id"`
Name string `json:"name"`
}
if result, err := testMetadataProbeJSON(req, "Fanart", &response); err != nil {
return result, err
}
if response.TMDbID.String() != "550" || strings.TrimSpace(response.Name) == "" {
return "error", errors.New("Fanart returned an invalid movie response")
}
return "success", nil
}
func (s *ApiConfigService) testDouban(ctx context.Context, cfg *model.APIConfig) (string, error) {
req, err := newMetadataProbeRequest(ctx, cfg, "https://movie.douban.com", "/j/subject_suggest", http.MethodGet, nil)
if err != nil {
return "error", err
}
query := req.URL.Query()
query.Set("q", "Inception")
req.URL.RawQuery = query.Encode()
req.Header.Set("User-Agent", userAgents[0])
req.Header.Set("Referer", "https://movie.douban.com/")
req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8")
if cookie := strings.TrimSpace(cfg.APIKey); cookie != "" {
req.Header.Set("Cookie", cookie)
}
var suggestions []struct {
ID string `json:"id"`
Title string `json:"title"`
}
if result, err := testMetadataProbeJSON(req, "Douban", &suggestions); err != nil {
return result, err
}
// An empty array is a valid search result; null or an error object is not.
if suggestions == nil {
return "error", errors.New("Douban returned an invalid search response (login or verification may be required)")
}
for _, suggestion := range suggestions {
if suggestion.ID == "" || strings.TrimSpace(suggestion.Title) == "" {
return "error", errors.New("Douban returned an invalid search suggestion")
}
}
return "success", nil
}
func newMetadataProbeRequest(ctx context.Context, cfg *model.APIConfig, defaultBase, path, method string, body io.Reader) (*http.Request, error) {
base := strings.TrimRight(strings.TrimSpace(cfg.BaseURL), "/")
if base == "" {
base = defaultBase
}
req, err := http.NewRequestWithContext(ctx, method, base+path, body)
if err != nil {
// The saved base URL may contain credentials. Never echo it to the UI.
return nil, fmt.Errorf("%s: invalid base URL", cfg.Provider)
}
req.Header.Set("User-Agent", "MeBox/0.1 (https://github.com/truewhile/MeBox)")
req.Header.Set("Accept", "application/json")
return req, nil
}
// Decode a bounded, complete JSON document rather than accepting any HTTP 200
// (reverse proxies and anti-bot/login pages often return 200 as well).
func testMetadataProbeJSON(req *http.Request, provider string, out any) (string, error) {
client := NewExternalHTTPClient(10 * time.Second)
defer client.CloseIdleConnections()
resp, err := client.Do(req)
if err != nil {
// url.Error includes the full request URL, including Fanart's API key.
var urlErr *url.Error
if errors.As(err, &urlErr) {
err = urlErr.Err
}
return "error", fmt.Errorf("%s connection failed: %w", provider, err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusUnauthorized {
return "invalid", fmt.Errorf("%s authentication failed (HTTP 401): check configured credentials", provider)
}
if resp.StatusCode == http.StatusForbidden {
return "error", fmt.Errorf("%s access denied (HTTP 403): check credentials, permissions or anti-bot verification", provider)
}
if resp.StatusCode != http.StatusOK {
return "error", fmt.Errorf("%s API returned status %d", provider, resp.StatusCode)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, (1<<20)+1))
if err != nil {
return "error", fmt.Errorf("%s response read failed: %w", provider, err)
}
if len(body) > 1<<20 {
return "error", fmt.Errorf("%s response exceeds 1 MiB", provider)
}
if err := json.Unmarshal(body, out); err != nil {
// Do not include the response body: it may echo credentials.
return "error", fmt.Errorf("%s returned invalid JSON (login or verification page may have intercepted the request)", provider)
}
return "success", nil
}
@@ -1,252 +0,0 @@
package service
import (
"context"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/repository"
)
type metadataProbeFixture struct {
provider string
path string
key string
body string
}
var metadataProbeFixtures = []metadataProbeFixture{
{"bangumi", "/v0/subjects/253", "", `{"id":253,"name":"カウボーイビバップ","type":2}`},
{"thetvdb", "/login", "tvdb-test-secret", `{"status":"success","data":{"token":"test-jwt"}}`},
{"fanart", "/movies/550", "fanart+test&secret", `{"name":"Fight Club","tmdb_id":"550","imdb_id":"tt0137523","movieposter":[]}`},
{"douban", "/j/subject_suggest", "bid=test-cookie", `[{"id":"3541415","title":"盗梦空间","year":"2010","type":"movie"}]`},
}
func newMetadataConnectionTestService(t *testing.T) *ApiConfigService {
t.Helper()
log := zap.NewNop()
return NewApiConfigService(&config.Config{}, log, repository.New(newServiceTestDB(t)), NewCryptoService("metadata-test-secret", log))
}
func saveMetadataProbeConfig(t *testing.T, svc *ApiConfigService, provider, baseURL, key string) {
t.Helper()
if _, err := svc.Upsert(t.Context(), provider, key, baseURL, "", true); err != nil {
t.Fatal(err)
}
}
// Removing a dispatch branch, ignoring the saved URL/key, or using the wrong
// endpoint must fail this integration test (real database and HTTP server).
func TestMetadataConnectionUsesSavedConfig(t *testing.T) {
svc := newMetadataConnectionTestService(t)
for _, fixture := range metadataProbeFixtures {
t.Run(fixture.provider, func(t *testing.T) {
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if r.URL.Path != "/proxy"+fixture.path {
t.Errorf("request path = %q", r.URL.Path)
}
if r.Header.Get("User-Agent") == "" {
t.Error("missing User-Agent")
}
if fixture.provider == "thetvdb" {
var body map[string]string
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Error(err)
}
if r.Method != http.MethodPost || body["apikey"] != fixture.key || r.Header.Get("Content-Type") != "application/json" {
t.Error("TheTVDB login must POST the saved, decrypted API key as JSON")
}
} else if r.Method != http.MethodGet {
t.Errorf("method = %s, want GET", r.Method)
}
if fixture.provider == "fanart" && r.URL.Query().Get("api_key") != fixture.key {
t.Error("Fanart API key was not escaped/passed correctly")
}
if fixture.provider == "douban" {
if r.Header.Get("Cookie") != fixture.key || r.Header.Get("Referer") != "https://movie.douban.com/" || r.URL.Query().Get("q") == "" {
t.Error("Douban search must carry the configured cookie, referer and query")
}
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(fixture.body))
}))
defer server.Close()
saveMetadataProbeConfig(t, svc, fixture.provider, " "+server.URL+"/proxy/ ", fixture.key)
result, err := svc.TestConnection(t.Context(), fixture.provider)
if err != nil || result != "success" {
t.Fatalf("TestConnection = %q, %v, want success", result, err)
}
if calls != 1 {
t.Errorf("HTTP calls = %d, want 1", calls)
}
})
}
}
func TestMetadataConnectionRejectsBadResponses(t *testing.T) {
svc := newMetadataConnectionTestService(t)
for _, fixture := range metadataProbeFixtures {
for _, response := range []struct {
name string
status int
body string
result string
}{
{"unauthorized", 401, `{"error":"denied"}`, "invalid"},
{"forbidden", 403, `<html>Access denied</html>`, "error"},
{"rate limited", 429, `{}`, "error"},
{"upstream error", 503, `{}`, "error"},
{"challenge page", 200, `<html>Please log in or complete verification</html>`, "error"},
{"empty body", 200, ``, "error"},
{"wrong schema", 200, `{"error":"intercepted"}`, "error"},
{"null", 200, `null`, "error"},
{"trailing garbage", 200, fixture.body + `<html>blocked</html>`, "error"},
} {
t.Run(fixture.provider+"/"+response.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(response.status)
_, _ = w.Write([]byte(response.body))
}))
defer server.Close()
saveMetadataProbeConfig(t, svc, fixture.provider, server.URL, fixture.key)
result, err := svc.TestConnection(t.Context(), fixture.provider)
if err == nil || result != response.result {
t.Fatalf("TestConnection = %q, %v, want %q and error", result, err, response.result)
}
if fixture.key != "" && strings.Contains(err.Error(), fixture.key) {
t.Error("error exposes configured credentials")
}
})
}
}
}
func TestMetadataConnectionRequiresAPIKeys(t *testing.T) {
svc := newMetadataConnectionTestService(t)
for _, provider := range []string{"thetvdb", "fanart"} {
t.Run(provider, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Error("must reject missing key before making a request")
}))
defer server.Close()
saveMetadataProbeConfig(t, svc, provider, server.URL, " ")
result, err := svc.TestConnection(t.Context(), provider)
if result != "error" || err == nil || !strings.Contains(err.Error(), "API key") {
t.Fatalf("TestConnection = %q, %v, want missing API key error", result, err)
}
})
}
}
func TestMetadataConnectionBangumiValidatesConfiguredToken(t *testing.T) {
svc := newMetadataConnectionTestService(t)
for _, valid := range []bool{true, false} {
name := "valid"
if !valid {
name = "invalid"
}
t.Run(name, func(t *testing.T) {
meCalls, subjectCalls := 0, 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer bangumi-test-token" {
t.Error("missing saved Bangumi bearer token")
}
switch r.URL.Path {
case "/v0/me":
meCalls++
if !valid {
w.WriteHeader(http.StatusUnauthorized)
return
}
_, _ = w.Write([]byte(`{"id":42,"username":"test-user"}`))
case "/v0/subjects/253":
subjectCalls++
_, _ = w.Write([]byte(`{"id":253,"name":"カウボーイビバップ"}`))
default:
t.Errorf("unexpected path: %s", r.URL.Path)
w.WriteHeader(http.StatusNotFound)
}
}))
defer server.Close()
saveMetadataProbeConfig(t, svc, "bangumi", server.URL, "bangumi-test-token")
result, err := svc.TestConnection(t.Context(), "bangumi")
if valid && (result != "success" || err != nil || subjectCalls != 1) {
t.Fatalf("valid token: result=%q err=%v subjectCalls=%d", result, err, subjectCalls)
}
if !valid && (result != "invalid" || err == nil || subjectCalls != 0) {
t.Fatalf("invalid token: result=%q err=%v subjectCalls=%d", result, err, subjectCalls)
}
if meCalls != 1 {
t.Errorf("identity requests = %d, want 1", meCalls)
}
})
}
}
func TestMetadataConnectionDoubanDoesNotRequireCookie(t *testing.T) {
svc := newMetadataConnectionTestService(t)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Cookie") != "" {
t.Error("unexpected Cookie for public search")
}
_, _ = w.Write([]byte(`[]`))
}))
defer server.Close()
saveMetadataProbeConfig(t, svc, "douban", server.URL, "")
result, err := svc.TestConnection(t.Context(), "douban")
if result != "success" || err != nil {
t.Fatalf("public search with empty results = %q, %v", result, err)
}
}
func TestMetadataConnectionHonorsContext(t *testing.T) {
svc := newMetadataConnectionTestService(t)
for _, fixture := range metadataProbeFixtures {
t.Run(fixture.provider, func(t *testing.T) {
started := make(chan struct{})
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body)
close(started)
<-r.Context().Done()
}))
defer server.Close()
saveMetadataProbeConfig(t, svc, fixture.provider, server.URL, fixture.key)
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
defer cancel()
go func() {
select {
case <-started:
cancel()
case <-ctx.Done():
}
}()
result, err := svc.TestConnection(ctx, fixture.provider)
if result != "error" || !errors.Is(err, context.Canceled) {
t.Fatalf("cancelled request = %q, %v, want context.Canceled", result, err)
}
})
}
}
func TestMetadataConnectionTransportErrorsHideKeys(t *testing.T) {
svc := newMetadataConnectionTestService(t)
server := httptest.NewServer(http.NotFoundHandler())
server.Close()
key := "sensitive-fanart-key"
saveMetadataProbeConfig(t, svc, "fanart", server.URL, key)
result, err := svc.TestConnection(t.Context(), "fanart")
if result != "error" || err == nil || strings.Contains(err.Error(), key) {
t.Fatalf("transport failure = %q, %v, want error without credentials", result, err)
}
}
+37 -81
View File
@@ -5,7 +5,7 @@
//
// 两种方言的备份形态不同:
// - SQLite:用 "VACUUM INTO"(3.27.0 起可用)产出一个一致的库文件,
// 扩展名 .db;恢复先暂存,在下次启动打开数据库前替换文件。
// 扩展名 .db;恢复即「替换库文件 + 重启」。
// - PostgreSQL:没有单文件语义,用逻辑备份(见 backup_postgres.go),
// 扩展名 .json.gz;恢复是在单个事务内重建全部表。
package service
@@ -24,7 +24,6 @@ import (
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/database"
)
// BackupService manages database backups.
@@ -45,7 +44,6 @@ type BackupInfo struct {
Path string `json:"path"`
Size int64 `json:"size"`
CreatedAt time.Time `json:"created_at"`
Automatic bool `json:"automatic"`
}
func (b *BackupService) backupDir() string {
@@ -71,23 +69,6 @@ func (b *BackupService) backupFilePath(filename string) (string, error) {
return path, nil
}
// ResolvePath returns the absolute path of a listed backup after applying the
// same filename and directory checks used by restore and delete operations.
func (b *BackupService) ResolvePath(filename string) (string, error) {
path, err := b.backupFilePath(filename)
if err != nil {
return "", err
}
info, err := os.Lstat(path)
if err != nil {
return "", err
}
if !info.Mode().IsRegular() {
return "", errors.New("backup is not a regular file")
}
return path, nil
}
// isPostgres reports whether the service is backed by PostgreSQL.
//
// PostgreSQL 不支持 SQLite 的 "VACUUM INTO",按文件替换恢复也会写错目标,
@@ -99,23 +80,11 @@ func (b *BackupService) isPostgres() bool {
// Create produces a new hot backup. SQLite uses "VACUUM INTO"; PostgreSQL uses
// a logical dump.
func (b *BackupService) Create(ctx context.Context) (*BackupInfo, error) {
return b.create(ctx, false)
}
func (b *BackupService) CreateAutomatic(ctx context.Context) (*BackupInfo, error) {
return b.create(ctx, true)
}
func (b *BackupService) create(ctx context.Context, automatic bool) (*BackupInfo, error) {
dir := b.backupDir()
if err := os.MkdirAll(dir, 0o750); err != nil {
return nil, err
}
now := time.Now()
ts := fmt.Sprintf("%s_%09d", now.Format("20060102_150405"), now.Nanosecond())
if automatic {
ts = "auto_" + ts
}
ts := time.Now().Format("20060102_150405")
var (
dst string
@@ -139,24 +108,15 @@ func (b *BackupService) create(ctx context.Context, automatic bool) (*BackupInfo
Path: dst,
Size: stat.Size(),
CreatedAt: stat.ModTime(),
Automatic: automatic,
}, nil
}
func (b *BackupService) createSQLiteBackup(ctx context.Context, stamp string) (string, error) {
dst := filepath.Join(b.backupDir(), fmt.Sprintf("mebox_%s.db", stamp))
tmp := dst + ".tmp"
defer os.Remove(tmp)
// Publish only completed backups, with credentials protected by file mode.
if err := b.db.WithContext(ctx).Exec("VACUUM INTO ?", tmp).Error; err != nil {
// VACUUM INTO creates a clean, non-WAL copy atomically.
if err := b.db.WithContext(ctx).Exec("VACUUM INTO ?", dst).Error; err != nil {
return "", fmt.Errorf("VACUUM INTO failed: %w", err)
}
if err := os.Chmod(tmp, 0o600); err != nil {
return "", err
}
if err := os.Rename(tmp, dst); err != nil {
return "", err
}
return dst, nil
}
@@ -172,7 +132,7 @@ func (b *BackupService) List() ([]BackupInfo, error) {
}
out := make([]BackupInfo, 0, len(entries))
for _, e := range entries {
if !e.Type().IsRegular() || !isBackupFilename(e.Name()) {
if e.IsDir() || !isBackupFilename(e.Name()) {
continue
}
info, err := e.Info()
@@ -184,7 +144,6 @@ func (b *BackupService) List() ([]BackupInfo, error) {
Path: filepath.Join(dir, e.Name()),
Size: info.Size(),
CreatedAt: info.ModTime(),
Automatic: strings.HasPrefix(e.Name(), "mebox_auto_"),
})
}
sort.Slice(out, func(i, j int) bool {
@@ -193,36 +152,6 @@ func (b *BackupService) List() ([]BackupInfo, error) {
return out, nil
}
// Prune keeps maxCount automatic backups for the current database dialect.
// Manual backups and backups from another dialect are never pruned.
// A value below one disables pruning so an invalid setting cannot erase all
// recoverable copies.
func (b *BackupService) Prune(maxCount int) error {
if maxCount < 1 {
return nil
}
items, err := b.List()
if err != nil {
return err
}
eligible := items[:0]
for _, item := range items {
if item.Automatic && strings.HasSuffix(item.Filename, postgresBackupSuffix) == b.isPostgres() {
eligible = append(eligible, item)
}
}
items = eligible
if maxCount >= len(items) {
return nil
}
for _, item := range items[maxCount:] {
if err := b.Delete(item.Filename); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
}
return nil
}
// Delete removes a single backup file.
func (b *BackupService) Delete(filename string) error {
path, err := b.backupFilePath(filename)
@@ -232,10 +161,12 @@ func (b *BackupService) Delete(filename string) error {
return os.Remove(path)
}
// Restore stages SQLite backups for the next startup. PostgreSQL restores
// rebuild every table inside a single transaction.
// Restore replaces the live database with the given backup.
// WARNING: this is destructive. SQLite restores by swapping the database file
// (the caller should shut down the server afterwards); PostgreSQL restores by
// rebuilding every table inside a single transaction.
func (b *BackupService) Restore(ctx context.Context, filename string) error {
src, err := b.ResolvePath(filename)
src, err := b.backupFilePath(filename)
if err != nil {
return err
}
@@ -251,10 +182,35 @@ func (b *BackupService) Restore(ctx context.Context, filename string) error {
if !strings.HasSuffix(filename, ".db") {
return errors.New("该备份是 PostgreSQL 逻辑备份,当前运行在 SQLite 上,无法恢复")
}
if err := database.StageSQLiteRestore(ctx, b.cfg, src); err != nil {
return b.restoreSQLiteBackup(filename, src)
}
func (b *BackupService) restoreSQLiteBackup(filename, src string) error {
dbPath, err := filepath.Abs(b.cfg.Database.DBPath)
if err != nil || dbPath == "" {
return errors.New("invalid database path")
}
// Strategy: rename live → live.old, copy backup → live, delete old.
old := dbPath + ".before_restore"
if err := os.Rename(dbPath, old); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
b.log.Warn("SQLite backup staged; restart to apply", zap.String("backup", filename))
data, err := os.ReadFile(src) // #nosec G304 -- src is constrained by backupFilePath to the configured backups directory.
if err != nil {
// Revert
_ = os.Rename(old, dbPath)
return err
}
if err := os.WriteFile(dbPath, data, 0o600); err != nil { // #nosec G703 -- database path is trusted process config and normalized before restore.
_ = os.Rename(old, dbPath)
return err
}
_ = os.Remove(old)
// Also remove WAL/SHM so SQLite opens the fresh file cleanly.
_ = os.Remove(dbPath + "-wal")
_ = os.Remove(dbPath + "-shm")
b.log.Warn("database restored from backup — restart the server",
zap.String("backup", filename))
return nil
}
+12 -33
View File
@@ -3,7 +3,6 @@ package service
import (
"compress/gzip"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
@@ -88,8 +87,7 @@ func postgresBackupModelSchemas(db *gorm.DB) (map[string]*schema.Schema, map[str
// createPostgresBackup 把全库导出成一个 gzip 压缩的 JSON Lines 文件。
func (b *BackupService) createPostgresBackup(ctx context.Context, stamp string) (string, error) {
dst := filepath.Join(b.backupDir(), "mebox_"+stamp+postgresBackupSuffix)
tmp := dst + ".tmp"
file, err := os.OpenFile(tmp, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
file, err := os.OpenFile(dst, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
if err != nil {
return "", err
}
@@ -98,7 +96,7 @@ func (b *BackupService) createPostgresBackup(ctx context.Context, stamp string)
_ = file.Close()
if !complete {
// 半截备份文件比没有备份更危险:留着会让人以为有可用的还原点。
_ = os.Remove(tmp)
_ = os.Remove(dst)
}
}()
@@ -111,30 +109,17 @@ func (b *BackupService) createPostgresBackup(ctx context.Context, stamp string)
}); err != nil {
return "", fmt.Errorf("写入备份头失败: %w", err)
}
var options []*sql.TxOptions
if b.isPostgres() {
options = append(options, &sql.TxOptions{Isolation: sql.LevelRepeatableRead, ReadOnly: true})
}
// Every table must come from the same snapshot, including while background
// workers are inserting related rows during an automatic backup.
if err := b.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
snapshot := *b
snapshot.db = tx
for _, m := range model.AllModels() {
table, sch, err := parseModelSchema(tx, m)
if err != nil {
return err
}
if !tx.Migrator().HasTable(table) {
continue
}
if err := snapshot.dumpPostgresTable(ctx, m, sch, table, enc); err != nil {
return err
}
for _, m := range model.AllModels() {
table, sch, err := parseModelSchema(b.db, m)
if err != nil {
return "", err
}
if !b.db.Migrator().HasTable(table) {
continue
}
if err := b.dumpPostgresTable(ctx, m, sch, table, enc); err != nil {
return "", err
}
return nil
}, options...); err != nil {
return "", err
}
if err := zw.Close(); err != nil {
return "", fmt.Errorf("压缩备份失败: %w", err)
@@ -142,12 +127,6 @@ func (b *BackupService) createPostgresBackup(ctx context.Context, stamp string)
if err := file.Sync(); err != nil {
return "", fmt.Errorf("写入备份失败: %w", err)
}
if err := file.Close(); err != nil {
return "", err
}
if err := os.Rename(tmp, dst); err != nil {
return "", err
}
complete = true
return dst, nil
}
-65
View File
@@ -1,65 +0,0 @@
package service
import (
"os"
"path/filepath"
"runtime"
"testing"
"time"
)
func TestBackupRetentionPreservesManualAndOtherDialectBackups(t *testing.T) {
svc, _ := newBackupTestService(t)
names := []string{"mebox_auto_new.db", "mebox_auto_old.db", "mebox_manual.db", "mebox_auto_postgres.json.gz"}
for i, name := range names {
path := filepath.Join(svc.backupDir(), name)
if err := os.WriteFile(path, []byte("fixture"), 0o600); err != nil {
t.Fatal(err)
}
when := time.Now().Add(-time.Duration(i) * time.Hour)
if err := os.Chtimes(path, when, when); err != nil {
t.Fatal(err)
}
}
if err := svc.Prune(1); err != nil {
t.Fatal(err)
}
for _, name := range names {
_, err := os.Stat(filepath.Join(svc.backupDir(), name))
if name == "mebox_auto_old.db" {
if !os.IsNotExist(err) {
t.Fatalf("old automatic backup was retained: %v", err)
}
} else if err != nil {
t.Fatalf("backup %s should have been preserved: %v", name, err)
}
}
}
func TestBackupCreateUsesUniquePrivateCompletedFiles(t *testing.T) {
svc, _ := newBackupTestService(t)
manual, err := svc.Create(t.Context())
if err != nil {
t.Fatal(err)
}
automatic, err := svc.CreateAutomatic(t.Context())
if err != nil {
t.Fatal(err)
}
if manual.Filename == automatic.Filename || manual.Automatic || !automatic.Automatic {
t.Fatalf("invalid backup identities: manual=%#v automatic=%#v", manual, automatic)
}
items, err := svc.List()
if err != nil || len(items) != 2 {
t.Fatalf("completed backup list: items=%#v err=%v", items, err)
}
for _, item := range items {
info, err := os.Stat(item.Path)
if err != nil {
t.Fatal(err)
}
if runtime.GOOS != "windows" && info.Mode().Perm() != 0o600 {
t.Fatalf("backup permissions are not private: %s err=%v", item.Path, err)
}
}
}
+34 -22
View File
@@ -74,20 +74,22 @@ type ImageCacheScope struct {
DropUnknownResizeVariants bool
}
// ImageCachePool 描述一个图片缓存池:范围 + 自身配额。
// ImageCachePool 描述一个图片缓存池:范围 + 自身配额 + 保留时长。
//
// 分池的意义在于两类缓存的“可再生成本”完全不同:原图只是生成缩略图的原料,
// 丢了可以重新回源;派生缩略图是客户端热路径真正读取的成品,重新生成代价高。
// 因此原图配小配额,总量超限时也优先淘汰原图。
//
// 淘汰只看占用(配额头寸),不按文件年龄删除:缩略图生成成本高,只要磁盘还
// 放得下就该留着,让再次浏览直接命中缓存。
// 因此原图配小配额、短保留,总量超限时也优先淘汰原图。
type ImageCachePool struct {
Name string
Scopes []ImageCacheScope
MaxBytes int64 // 0 = 不单独限制
MaxBytes int64 // 0 = 不单独限制
MaxAge time.Duration // 0 = 不按时间淘汰
}
// imageFailMarkerMaxAge 是失败标记文件的保留时长:标记只用于观测/重试,
// 过期即视为噪音清掉(不会因为标记残留而阻止重新抓取)。
const imageFailMarkerMaxAge = 24 * time.Hour
// tempImageCacheFile 报告文件名是否是下载/缩放过程中的临时文件。它们正在被
// 并发写入,清理时必须跳过。
func tempImageCacheFile(name string) bool {
@@ -180,14 +182,12 @@ func removeImageCacheEntry(entry imageCacheFileEntry, result *PruneImageCacheRes
_ = os.Remove(entry.path + ".fail")
}
// PruneImageCachePools 按池清理图片缓存,只做占用淘汰:
// 1. 池自身超过 MaxBytes 时按 mtime 淘汰到 80%(留水位,避免连续写入即触发);
// 2. 全部池合计超过 totalBytes 时,仍按“先原图、后派生”的顺序淘汰最旧文件,
// PruneImageCachePools 按池清理图片缓存:
// 1. 删除超过池 MaxAge 的文件(原图按保留时长淘汰);
// 2. 池自身超过 MaxBytes 时按 mtime 淘汰到 80%(留水位,避免连续写入即触发);
// 3. 全部池合计超过 totalBytes 时,仍按“先原图、后派生”的顺序淘汰最旧文件,
// 使总量上限始终是硬保证,同时让客户端热路径的缩略图活得更久。
//
// 不按文件年龄删除:过期即删会让一张几天前看过的海报重新解码(2 核机器上
// 每张约 0.2 秒),代价远高于占着那几十 KB 磁盘。
//
// 传入的池顺序即总量超限时的淘汰优先级(排在前面的先被淘汰)。
func PruneImageCachePools(pools []ImageCachePool, totalBytes int64) (PruneImageCacheResult, error) {
var result PruneImageCacheResult
@@ -207,18 +207,16 @@ func PruneImageCachePools(pools []ImageCachePool, totalBytes int64) (PruneImageC
}
result.RemainingBytes = result.TotalBytesBefore
// 失败标记不计入容量:只清掉对应缓存文件已经不存在的孤儿标记(它们只是
// 观测/重试用的噪音,留着没有任何作用)。
// 过期的失败标记直接清掉:它们不计容量,只用于观测与重试。
cutoffMarkers := time.Now().Add(-imageFailMarkerMaxAge)
for _, marker := range failMarkers {
base := strings.TrimSuffix(marker.path, ".fail")
if _, err := os.Stat(base); err == nil {
continue
if marker.modTime.Before(cutoffMarkers) {
_ = os.Remove(marker.path)
}
_ = os.Remove(marker.path)
}
now := time.Now()
deleted := make(map[string]bool, len(entries))
deleteEntry := func(entry imageCacheFileEntry) {
if deleted[entry.path] {
return
@@ -227,7 +225,20 @@ func PruneImageCachePools(pools []ImageCachePool, totalBytes int64) (PruneImageC
removeImageCacheEntry(entry, &result)
}
// 1. 池自身配额。
// 1. 按池保留时长淘汰(原图池)。
for i := range pools {
if pools[i].MaxAge <= 0 {
continue
}
cutoff := now.Add(-pools[i].MaxAge)
for _, entry := range entries {
if entry.pool == i && entry.modTime.Before(cutoff) {
deleteEntry(entry)
}
}
}
// 2. 池自身配额。
for i := range pools {
maxBytes := pools[i].MaxBytes
if maxBytes <= 0 {
@@ -252,7 +263,7 @@ func PruneImageCachePools(pools []ImageCachePool, totalBytes int64) (PruneImageC
}
}
// 2. 总量硬上限:跨池按淘汰优先级(池顺序)再按 mtime 淘汰。
// 3. 总量硬上限:跨池按淘汰优先级(池顺序)再按 mtime 淘汰。
if totalBytes > 0 && result.RemainingBytes > totalBytes {
target := totalBytes * 80 / 100
for _, entry := range evictionOrder(entries, deleted) {
@@ -346,10 +357,11 @@ func PruneImageCache(imagesDir string, maxSizeBytes int64) (PruneImageCacheResul
// 第一个池是原图:历史版本平铺在 images/ 下的旧缓存也归入此池,升级后会被
// 逐步淘汰;第二个池是派生成品(本地缩放结果 + 挂载 Emby 按尺寸返回的成品)。
// 池顺序即总量超限时的淘汰优先级。
func ImageCachePools(imagesDir string, originalsMaxBytes int64) []ImageCachePool {
func ImageCachePools(imagesDir string, originalsMaxBytes int64, originalsMaxAge time.Duration) []ImageCachePool {
originals := ImageCachePool{
Name: "originals",
MaxBytes: originalsMaxBytes,
MaxAge: originalsMaxAge,
Scopes: []ImageCacheScope{
{Root: imagesDir, Recursive: false}, // 旧版平铺布局
{Root: filepath.Join(imagesDir, imageOriginalCacheSubdir), Recursive: true},
+44 -44
View File
@@ -9,7 +9,7 @@ import (
"time"
)
// writePoolFile ??????????????? mtime ???????
// writePoolFile ?????????????????????????
func writePoolFile(t *testing.T, path string, size int, modTime time.Time) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
@@ -39,38 +39,42 @@ func mustExist(t *testing.T, path string) {
}
}
// currentVariantPath ???????????????????
// <64 ????>.<?>x<?>q<??>.img
// currentVariantPath ????????????????.<??x<??q<??>.img??
// ??????????????????????????
func currentVariantPath(dir, keyChar string, width, height, quality int) string {
key := strings.Repeat(keyChar, sha256HexLength)
return filepath.Join(dir, fmt.Sprintf("%s.%dx%dq%d.img", key, width, height, quality))
}
// TestPruneImageCachePoolsKeepsOldFilesUnderQuota ????????????
// ???????????????????????? / TTL ????
func TestPruneImageCachePoolsKeepsOldFilesUnderQuota(t *testing.T) {
// TestPruneImageCachePoolsOriginalTTL ????????????????????
// ?? TTL ??????????????????????
func TestPruneImageCachePoolsOriginalTTL(t *testing.T) {
imagesDir := t.TempDir()
now := time.Now()
oldOriginal := filepath.Join(imagesDir, imageOriginalCacheSubdir, "old-original")
oldThumb := currentVariantPath(filepath.Join(imagesDir, imageResizeCacheSubdir), "b", 480, 600, 80)
writePoolFile(t, oldOriginal, 400, now.Add(-90*24*time.Hour))
writePoolFile(t, oldThumb, 400, now.Add(-90*24*time.Hour))
stale := filepath.Join(imagesDir, imageOriginalCacheSubdir, "stale-original")
fresh := filepath.Join(imagesDir, imageOriginalCacheSubdir, "fresh-original")
thumb := currentVariantPath(filepath.Join(imagesDir, imageResizeCacheSubdir), "b", 480, 600, 80)
writePoolFile(t, stale, 400, now.Add(-8*24*time.Hour))
writePoolFile(t, fresh, 400, now.Add(-1*time.Hour))
// ??????????TTL ????
writePoolFile(t, thumb, 400, now.Add(-8*24*time.Hour))
pools := ImageCachePools(imagesDir, 0)
pools := ImageCachePools(imagesDir, 0, 7*24*time.Hour)
res, err := PruneImageCachePools(pools, 100*1024*1024)
if err != nil {
t.Fatalf("prune failed: %v", err)
}
if res.DeletedFiles != 0 {
t.Fatalf("deleted files = %d, want 0 (age must not trigger deletion)", res.DeletedFiles)
if res.DeletedFiles != 1 {
t.Fatalf("deleted files = %d, want 1", res.DeletedFiles)
}
mustExist(t, oldOriginal)
mustExist(t, oldThumb)
mustNotExist(t, stale)
mustExist(t, fresh)
mustExist(t, thumb)
}
// TestPruneImageCachePoolsEvictsOriginalsBeforeDerived ????????????
// ????????????????
// ????????????????????
func TestPruneImageCachePoolsEvictsOriginalsBeforeDerived(t *testing.T) {
imagesDir := t.TempDir()
now := time.Now()
@@ -78,13 +82,13 @@ func TestPruneImageCachePoolsEvictsOriginalsBeforeDerived(t *testing.T) {
original := filepath.Join(imagesDir, imageOriginalCacheSubdir, "original")
rendition := filepath.Join(imagesDir, imageRenditionCacheSubdir, "rendition")
thumb := currentVariantPath(filepath.Join(imagesDir, imageResizeCacheSubdir), "c", 480, 600, 80)
// ?????????????????????????
// ?????????????????
writePoolFile(t, original, 1024*1024, now.Add(-72*time.Hour))
writePoolFile(t, rendition, 1024*1024, now.Add(-2*time.Hour))
writePoolFile(t, thumb, 1024*1024, now.Add(-1*time.Hour))
pools := ImageCachePools(imagesDir, 0)
// ?? 3MB??? 2.4MB -> ??? 1.92MB ???
pools := ImageCachePools(imagesDir, 0, 0)
// ?? 3MB????2.4MB -> ????? 1.92MB??
res, err := PruneImageCachePools(pools, 2*1024*1024+400*1024)
if err != nil {
t.Fatalf("prune failed: %v", err)
@@ -100,8 +104,8 @@ func TestPruneImageCachePoolsEvictsOriginalsBeforeDerived(t *testing.T) {
}
}
// TestPruneImageCachePoolsOriginalQuotaKeepsDerived ???????????
// ????????????
// TestPruneImageCachePoolsOriginalQuotaKeepsDerived ????????????
// ??????????????
func TestPruneImageCachePoolsOriginalQuotaKeepsDerived(t *testing.T) {
imagesDir := t.TempDir()
now := time.Now()
@@ -113,8 +117,8 @@ func TestPruneImageCachePoolsOriginalQuotaKeepsDerived(t *testing.T) {
writePoolFile(t, newOriginal, 800, now.Add(-1*time.Hour))
writePoolFile(t, thumb, 800, now.Add(-2*time.Hour))
// ???? 1000??? 1600 ?? -> ??? 800??? 1 ???????
pools := ImageCachePools(imagesDir, 1000)
// ???? 1000 ??????????1600 -> ????800????? 1 ????
pools := ImageCachePools(imagesDir, 1000, 0)
res, err := PruneImageCachePools(pools, 0)
if err != nil {
t.Fatalf("prune failed: %v", err)
@@ -127,26 +131,24 @@ func TestPruneImageCachePoolsOriginalQuotaKeepsDerived(t *testing.T) {
mustExist(t, thumb)
}
// TestPruneImageCachePoolsCleansFailMarkers ???????????????
// ???????????????????????????????????
// TestPruneImageCachePoolsCleansFailMarkers ????????????????
// ????????????????
func TestPruneImageCachePoolsCleansFailMarkers(t *testing.T) {
imagesDir := t.TempDir()
now := time.Now()
entry := filepath.Join(imagesDir, imageOriginalCacheSubdir, "entry")
entryMarker := entry + ".fail"
kept := filepath.Join(imagesDir, imageOriginalCacheSubdir, "kept")
keptMarker := kept + ".fail"
orphanMarker := filepath.Join(imagesDir, imageOriginalCacheSubdir, "orphan.fail")
freshMarker := filepath.Join(imagesDir, imageOriginalCacheSubdir, "fresh.fail")
writePoolFile(t, entry, 900, now.Add(-2*time.Hour))
writePoolFile(t, entryMarker, 35, now.Add(-2*time.Hour))
writePoolFile(t, kept, 300, now.Add(-1*time.Hour))
writePoolFile(t, keptMarker, 35, now.Add(-1*time.Hour))
writePoolFile(t, entry, 900, now.Add(-10*time.Hour))
writePoolFile(t, entryMarker, 35, now.Add(-10*time.Hour))
writePoolFile(t, orphanMarker, 35, now.Add(-48*time.Hour))
writePoolFile(t, freshMarker, 35, now.Add(-1*time.Minute))
// ???? 500??? 900+300=1200 ???? mtime ????? entry ? 400?
pools := ImageCachePools(imagesDir, 500)
// ???? 500????900 ?????????????????
pools := ImageCachePools(imagesDir, 500, 0)
res, err := PruneImageCachePools(pools, 0)
if err != nil {
t.Fatalf("prune failed: %v", err)
@@ -156,27 +158,26 @@ func TestPruneImageCachePoolsCleansFailMarkers(t *testing.T) {
}
mustNotExist(t, entry)
mustNotExist(t, entryMarker)
mustExist(t, kept)
mustExist(t, keptMarker)
mustNotExist(t, orphanMarker)
mustExist(t, freshMarker)
}
// TestPruneImageCachePoolsSkipsTemporaryFiles ??????????????
// TestPruneImageCachePoolsSkipsTemporaryFiles ????????????????
// ?????????????????
func TestPruneImageCachePoolsSkipsTemporaryFiles(t *testing.T) {
imagesDir := t.TempDir()
tmp := filepath.Join(imagesDir, imageOriginalCacheSubdir, "img-1234.tmp")
writePoolFile(t, tmp, 4096, time.Now().Add(-48*time.Hour))
pools := ImageCachePools(imagesDir, 1)
pools := ImageCachePools(imagesDir, 1, 0)
if _, err := PruneImageCachePools(pools, 1); err != nil {
t.Fatalf("prune failed: %v", err)
}
mustExist(t, tmp)
}
// TestPruneImageCachePoolsDropsLegacyResizeVariants ?????? .img ??
// ???????????????????????????
// TestPruneImageCachePoolsDropsLegacyResizeVariants ????????.img????
// ?????????????????????????????????????
func TestPruneImageCachePoolsDropsLegacyResizeVariants(t *testing.T) {
imagesDir := t.TempDir()
resized := filepath.Join(imagesDir, imageResizeCacheSubdir)
@@ -187,7 +188,7 @@ func TestPruneImageCachePoolsDropsLegacyResizeVariants(t *testing.T) {
writePoolFile(t, current, 400, time.Now())
writePoolFile(t, other, 400, time.Now())
pools := ImageCachePools(imagesDir, 0)
pools := ImageCachePools(imagesDir, 0, 0)
res, err := PruneImageCachePools(pools, 0)
if err != nil {
t.Fatalf("prune failed: %v", err)
@@ -200,8 +201,8 @@ func TestPruneImageCachePoolsDropsLegacyResizeVariants(t *testing.T) {
}
}
// TestPruneImageCachePoolsCountsLegacyFlatLayout ???????images/ ???
// ???????????????????????
// TestPruneImageCachePoolsCountsLegacyFlatLayout ??????images/ ????
// ?????????????????
func TestPruneImageCachePoolsCountsLegacyFlatLayout(t *testing.T) {
imagesDir := t.TempDir()
legacy := filepath.Join(imagesDir, "legacy-flat-cache")
@@ -209,8 +210,7 @@ func TestPruneImageCachePoolsCountsLegacyFlatLayout(t *testing.T) {
writePoolFile(t, legacy, 900, time.Now().Add(-30*24*time.Hour))
writePoolFile(t, thumb, 900, time.Now().Add(-30*24*time.Hour))
// ???? 500??????????????????????????
pools := ImageCachePools(imagesDir, 500)
pools := ImageCachePools(imagesDir, 0, 24*time.Hour)
res, err := PruneImageCachePools(pools, 0)
if err != nil {
t.Fatalf("prune failed: %v", err)
@@ -1,33 +0,0 @@
package service
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestCloud115HLSChildRejectsMismatchedSessionMediaBeforeFetch(t *testing.T) {
fetched := false
proxy := &Cloud115HLSProxy{sessions: map[string]*cloud115HLSSession{}}
proxy.client = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
fetched = true
return &http.Response{StatusCode: http.StatusOK, Header: http.Header{"Content-Type": []string{"video/mp2t"}}, Body: io.NopCloser(strings.NewReader("private-segment")), Request: req}, nil
})}
proxy.sessions["session-b"] = &cloud115HLSSession{
ID: "session-b", MediaID: "media-b", ExpiresAt: time.Now().Add(time.Hour),
entries: map[string]string{"segment": "https://cpats01.115.com/seg.ts"},
}
for _, query := range []string{"?media_id=media-a", ""} {
fetched = false
req := httptest.NewRequest(http.MethodGet, "/api/cloud115/hls/session-b/segment"+query, nil)
rec := httptest.NewRecorder()
err := proxy.ServeChild(context.Background(), rec, req, "session-b", "segment")
if err == nil || fetched {
t.Errorf("query %q must reject the wrong/missing media before fetch; err=%v fetched=%v", query, err, fetched)
}
}
}
+2 -15
View File
@@ -20,9 +20,8 @@ import (
)
var (
ErrCloud115HLSSessionNotFound = errors.New("115 hls session not found")
ErrCloud115HLSUpstreamExpired = errors.New("115 hls upstream expired")
ErrCloud115HLSSessionMediaMismatch = errors.New("115 hls session media mismatch")
ErrCloud115HLSSessionNotFound = errors.New("115 hls session not found")
ErrCloud115HLSUpstreamExpired = errors.New("115 hls upstream expired")
)
const (
@@ -132,9 +131,6 @@ func (p *Cloud115HLSProxy) ServeChild(ctx context.Context, w http.ResponseWriter
if !ok {
return ErrCloud115HLSSessionNotFound
}
if r == nil || strings.TrimSpace(r.URL.Query().Get("media_id")) != session.MediaID {
return ErrCloud115HLSSessionMediaMismatch
}
resp, err := p.fetchUpstream(ctx, upstream, r.Header.Get("Range"))
if err != nil {
return err
@@ -190,15 +186,6 @@ func (p *Cloud115HLSProxy) ServeChild(ctx context.Context, w http.ResponseWriter
return err
}
// SessionMediaID resolves the server-owned media identity before handler authorization.
func (p *Cloud115HLSProxy) SessionMediaID(sessionID, key string) (string, error) {
session, _, ok := p.lookup(sessionID, key)
if !ok {
return "", ErrCloud115HLSSessionNotFound
}
return session.MediaID, nil
}
func (p *Cloud115HLSProxy) newSession(mediaID string, definition int) *cloud115HLSSession {
now := time.Now()
return &cloud115HLSSession{
+6 -24
View File
@@ -104,13 +104,11 @@ func (s *Cloud115PlaybackService) HLSProxy() *Cloud115HLSProxy {
return s.proxy
}
// PlaybackInfo only reads direct/115-cloud/local playback capabilities.
// Cloud work is queued exclusively by StartTranscode, never by polling.
// PlaybackInfo 返回媒体在 direct/115-cloud/local 三种模式下的能力描述。
//
// requestedDefinition 为空或 0 时默认选择 1080P;该档位不可用时返回
// transcoding 状态(并只触发一次加速转码,避免轮询风暴)。
func (s *Cloud115PlaybackService) PlaybackInfo(ctx context.Context, mediaID string, requestedDefinition int) (*PlaybackInfo, error) {
return s.playbackInfo(ctx, mediaID, requestedDefinition, false)
}
func (s *Cloud115PlaybackService) playbackInfo(ctx context.Context, mediaID string, requestedDefinition int, startTranscode bool) (*PlaybackInfo, error) {
if s == nil || s.repo == nil || s.repo.Media == nil {
return nil, ErrMediaNotFound
}
@@ -181,26 +179,13 @@ func (s *Cloud115PlaybackService) playbackInfo(ctx context.Context, mediaID stri
return info, nil
}
if startTranscode {
info.Transcode = s.ensureCloudTranscode(ctx, account.ID, pickCode, selected, client)
} else {
info.Transcode = Cloud115TranscodeState{
State: "idle", Definition: selected.ID, Message: "该清晰度尚未就绪,可请求 115 云端转码", RetryAfterSec: 5,
}
s.mu.Lock()
if last, ok := s.pushState[account.ID+":"+pickCode]; ok {
info.Transcode.State = "transcoding"
info.Transcode.Message = last.message
info.Transcode.StartedAt = last.at.Unix()
}
s.mu.Unlock()
}
info.Transcode = s.ensureCloudTranscode(ctx, account.ID, pickCode, selected, client)
return info, nil
}
// StartTranscode 显式触发某个清晰度的云端转码,返回最新状态。
func (s *Cloud115PlaybackService) StartTranscode(ctx context.Context, mediaID string, definition int) (*PlaybackInfo, error) {
return s.playbackInfo(ctx, mediaID, definition, true)
return s.PlaybackInfo(ctx, mediaID, definition)
}
// ResolveCloud115URL 返回指定清晰度的 115 云端 m3u8 地址。
@@ -273,9 +258,6 @@ func (s *Cloud115PlaybackService) ensureCloudTranscode(
}
}
}
// Reserve the in-flight attempt before releasing the lock. Two simultaneous
// explicit starts must not both submit a push while the first awaits 115.
s.pushState[key] = cloud115PushAttempt{at: now, message: "正在提交 115 云端转码请求…"}
s.mu.Unlock()
err := client.SubmitVideoPush(ctx, pickCode, "vip_push")
+3 -10
View File
@@ -16,7 +16,7 @@ import (
"github.com/truewhile/MeBox/internal/service/cloud115"
)
func TestCloud115PlaybackInfoIsReadOnlyAndStartTriggersTranscodeOnce(t *testing.T) {
func TestCloud115PlaybackInfoTriggersTranscodeOnce(t *testing.T) {
var playCalls int32
var pushCalls int32
var ready int32
@@ -93,16 +93,9 @@ func TestCloud115PlaybackInfoIsReadOnlyAndStartTriggersTranscodeOnce(t *testing.
}
svc := NewCloud115PlaybackService(nil, zap.NewNop(), repos, strm)
if _, err := svc.PlaybackInfo(ctx, media.ID, 0); err != nil {
t.Fatalf("read playback info: %v", err)
}
if got := atomic.LoadInt32(&pushCalls); got != 0 {
t.Fatalf("read-only playback info triggered %d cloud transcode pushes", got)
}
info, err := svc.StartTranscode(ctx, media.ID, 0)
info, err := svc.PlaybackInfo(ctx, media.ID, 0)
if err != nil {
t.Fatalf("start transcode: %v", err)
t.Fatalf("first playback info: %v", err)
}
if info.Provider != model.StrmProvider115 || info.DefaultQuality != "4" {
t.Fatalf("unexpected playback info: %#v", info)
@@ -1,77 +0,0 @@
package service
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"sync"
"sync/atomic"
"testing"
"time"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/service/cloud115"
)
func TestCloud115ConcurrentStartsDoNotDuplicateInFlightPush(t *testing.T) {
started := make(chan struct{})
release := make(chan struct{})
var releaseOnce sync.Once
var pushCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if pushCalls.Add(1) == 1 {
close(started)
}
<-release
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"state": true, "code": 0, "data": []any{}})
}))
defer server.Close()
defer releaseOnce.Do(func() { close(release) })
oldBase := cloud115.ProAPIBase
cloud115.ProAPIBase = server.URL
defer func() { cloud115.ProAPIBase = oldBase }()
client := cloud115.NewOpenClient("review-app", "access", "refresh")
svc := NewCloud115PlaybackService(nil, zap.NewNop(), nil, nil)
quality := PlaybackQuality{ID: "4", Label: "1080P", Source: "cloud"}
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
first := make(chan struct{})
go func() { defer close(first); svc.ensureCloudTranscode(ctx, "account", "pick", quality, client) }()
select {
case <-started:
case <-ctx.Done():
t.Fatal("first push did not reach the local server")
}
second := make(chan Cloud115TranscodeState, 1)
go func() { second <- svc.ensureCloudTranscode(ctx, "account", "pick", quality, client) }()
secondReceived := false
select {
case state := <-second:
secondReceived = true
if state.State != "transcoding" {
t.Errorf("in-flight request should be reported as transcoding: %#v", state)
}
case <-time.After(250 * time.Millisecond):
t.Error("second start issued another request instead of reusing the in-flight push")
}
releaseOnce.Do(func() { close(release) })
select {
case <-first:
case <-ctx.Done():
t.Fatal("first push did not finish")
}
// The duplicate path is awaited before restoring the package-wide fake API.
if !secondReceived {
select {
case <-second:
case <-ctx.Done():
t.Fatal("second start did not finish")
}
}
if got := pushCalls.Load(); got != 1 {
t.Fatalf("pushes=%d, want 1", got)
}
}
@@ -1,254 +0,0 @@
package service
import (
"context"
"fmt"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/glebarez/sqlite"
"go.uber.org/zap"
"gorm.io/gorm"
"gorm.io/gorm/logger"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
)
// resetEmbyInvalThrottle 清掉全局失效节流,让测试能断言「这一次调用到底有
// 没有真的清缓存」。测试结束后恢复原值,避免影响同包其他用例。
func resetEmbyInvalThrottle(t *testing.T) {
t.Helper()
embyInvalMu.Lock()
previous := embyInvalLast
embyInvalLast = time.Time{}
embyInvalMu.Unlock()
t.Cleanup(func() {
embyInvalMu.Lock()
embyInvalLast = previous
embyInvalMu.Unlock()
})
}
func newCachedTestEmbyService(t *testing.T) *EmbyService {
t.Helper()
svc := newTestEmbyService(t)
cache := NewRuntimeCacheService(&config.Config{}, zap.NewNop())
t.Cleanup(func() { _ = cache.Close() })
return svc.SetRuntimeCache(cache)
}
// TestRecordProgressKeepsEmbyPayloadCache 覆盖首页加载慢的核心回归点:
// 播放期间客户端每 5-10 秒上报一次进度,如果每次都清空 media:emby:*,
// 客户端刷新首页时 73 个媒体库的 Latest 会全部回源重建(生产环境实测
// 首批 8-25s)。普通进度上报必须保留载荷缓存;只有「已看完」翻转才失效。
func TestRecordProgressKeepsEmbyPayloadCache(t *testing.T) {
svc := newCachedTestEmbyService(t)
const mediaID = "ep-1"
media := model.Media{
Base: model.Base{ID: mediaID},
Title: "第 1 集",
Path: `/media/tv/Show/Season 01/S01E01.mkv`,
DurationSec: 100,
}
if err := svc.repo.DB.Create(&media).Error; err != nil {
t.Fatalf("create media: %v", err)
}
key := svc.embyLatestCacheKey("user-1", "", 16)
svc.cache.SetJSONWithStale(t.Context(), key, embyLatestCacheValue{
Items: []map[string]any{{"Id": "cached-item"}},
}, time.Minute, time.Hour)
resetEmbyInvalThrottle(t)
// 60s / 100s:播放中途
if err := svc.RecordProgressWithSession(t.Context(), "user-1", mediaID, 600_000_000, 1_000_000_000, "session-1"); err != nil {
t.Fatalf("record progress: %v", err)
}
if _, stale, ok := svc.cachedLatestItemsWithStale(t.Context(), key); !ok || stale {
t.Fatal("普通进度上报不应让 Emby 列表载荷缓存失效")
}
resetEmbyInvalThrottle(t)
// 98s / 100s:进入片尾 30s 内的「已看完」区间
if err := svc.RecordProgressWithSession(t.Context(), "user-1", mediaID, 980_000_000, 1_000_000_000, "session-2"); err != nil {
t.Fatalf("record completed progress: %v", err)
}
if _, _, ok := svc.cachedLatestItemsWithStale(t.Context(), key); ok {
t.Fatal("看完状态翻转后应让 Emby 列表载荷缓存失效")
}
}
// TestFindSeriesGroupCachesSiblingVirtualSeries 覆盖 N+1 回归点:一次全表
// 扫描算出了整库的剧集分组,若只缓存命中的那一部,同一页的 16 张卡片和
// 16 张海报会各自重扫一遍 media 表。批量记忆后同批分组都应可命中。
func TestFindSeriesGroupCachesSiblingVirtualSeries(t *testing.T) {
svc := newTestEmbyService(t)
lib := model.Library{Name: "剧集", Path: `/media/tv`, Type: "tv", Enabled: true}
if err := svc.repo.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
shows := []string{"Alpha", "Bravo", "Charlie"}
rows := make([]model.Media, 0, len(shows)*2)
for _, show := range shows {
for ep := 1; ep <= 2; ep++ {
rows = append(rows, model.Media{
Base: model.Base{ID: show + "-" + strconv.Itoa(ep)},
LibraryID: lib.ID,
Title: show,
Path: fmt.Sprintf("/media/tv/%s/Season 01/S01E%02d.mkv", show, ep),
SeasonNum: 1,
EpisodeNum: ep,
})
}
}
for i := range rows {
if err := svc.repo.DB.Create(&rows[i]).Error; err != nil {
t.Fatalf("create media: %v", err)
}
}
seriesIDs := map[string]string{}
for i := range rows {
if rows[i].EpisodeNum == 1 {
seriesIDs[rows[i].Title] = svc.seriesIDForMedia(t.Context(), &rows[i])
}
}
target, ok := seriesIDs["Alpha"]
if !ok {
t.Fatal("missing fixture series")
}
if _, found, err := svc.findSeriesGroup(t.Context(), target, ""); err != nil || !found {
t.Fatalf("findSeriesGroup(%s) = (%v, %v), want found", target, found, err)
}
for show, id := range seriesIDs {
if show == "Alpha" {
continue
}
if _, cached := svc.cachedSeriesGroup(id); !cached {
t.Fatalf("同一次扫描算出的 %s 应一并进入缓存,否则每张卡片/海报都会重扫整库", show)
}
}
}
// TestLatestPayloadKeepsStaleCopyForIdleClients 保证「最新添加」载荷在新鲜期
// 之后仍有可用的陈旧副本:客户端空闲很久后再打开首页时,先拿陈旧结果并由
// 后台重建,而不是让全部媒体库同时回源。
func TestLatestPayloadKeepsStaleCopyForIdleClients(t *testing.T) {
svc := newCachedTestEmbyService(t)
key := svc.embyLatestCacheKey("user-1", "", 16)
svc.cache.SetJSONWithStale(t.Context(), key, embyLatestCacheValue{
Items: []map[string]any{{"Id": "stale-item"}},
}, time.Nanosecond, embyLatestStaleWindow)
time.Sleep(2 * time.Millisecond)
items, stale, ok := svc.cachedLatestItemsWithStale(t.Context(), key)
if !ok || !stale {
t.Fatalf("expected a stale hit, got ok=%v stale=%v", ok, stale)
}
if len(items) != 1 || items[0]["Id"] != "stale-item" {
t.Fatalf("unexpected stale payload: %#v", items)
}
}
// mediaScanCounter 统计真正落到 media 表上的查询次数,用来断言「整库数据
// 只扫一次」:GORM 的 Trace 回调里能拿到最终 SQL。
type mediaScanCounter struct {
logger.Interface
mu sync.Mutex
scans int
}
func (c *mediaScanCounter) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
sql, _ := fc()
if strings.Contains(sql, "FROM `media`") {
c.mu.Lock()
c.scans++
c.mu.Unlock()
}
}
func (c *mediaScanCounter) Scans() int {
c.mu.Lock()
defer c.mu.Unlock()
return c.scans
}
// TestVirtualSeriesLookupsShareOneScan 覆盖冷启动风暴的回归点:客户端刷新
// 首页时 73 个库的卡片和海报各自要一个虚拟 series 分组。修好之后,第一次
// 扫描把整库分组都写进缓存,同一页剩下的查询都不该再落 SQL;随后一个不存在
// 的虚拟 ID 也不该反复触发整库重扫。
func TestVirtualSeriesLookupsShareOneScan(t *testing.T) {
counter := &mediaScanCounter{Interface: logger.Discard}
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{Logger: counter})
if err != nil {
t.Fatalf("open db: %v", err)
}
if sqlDB, err := db.DB(); err == nil {
sqlDB.SetMaxOpenConns(1)
t.Cleanup(func() { _ = sqlDB.Close() })
}
if err := db.AutoMigrate(model.AllModels()...); err != nil {
t.Fatalf("migrate: %v", err)
}
svc := NewEmbyService(&config.Config{}, zap.NewNop(), repository.New(db))
lib := model.Library{Name: "剧集", Path: `/media/tv`, Type: "tv", Enabled: true}
if err := svc.repo.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
shows := []string{"Alpha", "Bravo", "Charlie"}
rows := make([]model.Media, 0, len(shows)*2)
for _, show := range shows {
for ep := 1; ep <= 2; ep++ {
rows = append(rows, model.Media{
Base: model.Base{ID: show + "-" + strconv.Itoa(ep)},
LibraryID: lib.ID,
Title: show,
Path: fmt.Sprintf("/media/tv/%s/Season 01/S01E%02d.mkv", show, ep),
SeasonNum: 1,
EpisodeNum: ep,
})
}
}
for i := range rows {
if err := svc.repo.DB.Create(&rows[i]).Error; err != nil {
t.Fatalf("create media: %v", err)
}
}
seriesIDs := map[string]string{}
for i := range rows {
if rows[i].EpisodeNum == 1 {
seriesIDs[rows[i].Title] = svc.seriesIDForMedia(t.Context(), &rows[i])
}
}
if _, found, err := svc.findSeriesGroup(t.Context(), seriesIDs["Alpha"], ""); err != nil || !found {
t.Fatalf("findSeriesGroup(Alpha) = (%v, %v), want found", found, err)
}
if got := counter.Scans(); got != 1 {
t.Fatalf("首次查询应恰好扫描 media 一次,实际 %d 次", got)
}
for _, show := range []string{"Bravo", "Charlie"} {
if _, found, err := svc.findSeriesGroup(t.Context(), seriesIDs[show], ""); err != nil || !found {
t.Fatalf("findSeriesGroup(%s) = (%v, %v), want found", show, found, err)
}
}
if got := counter.Scans(); got != 1 {
t.Fatalf("同批分组应命中缓存,不应再扫 media,实际累计 %d 次", got)
}
if _, found, err := svc.findSeriesGroup(t.Context(), embyVirtualSeriesPrefix+"missing", ""); err != nil || found {
t.Fatalf("未知虚拟 ID 应为 not found,实际 found=%v err=%v", found, err)
}
if got := counter.Scans(); got != 1 {
t.Fatalf("刚扫过整库后,未知 ID 不应重扫 media,实际累计 %d 次", got)
}
}
-20
View File
@@ -56,17 +56,6 @@ type EmbyService struct {
virtualSeries map[string]embySeriesCacheEntry
virtualSeasons map[string]embySeasonCacheEntry
virtualArtwork map[string]embyArtworkCacheEntry
// virtualGroups 把「整库虚拟剧集分组」的扫描折叠成一次。
//
// 客户端刷新首页时会同时请求 73 个库的 Latest,每个库的卡片和海报都要
// 各自的虚拟 series 分组。不同剧的分组缓存键不同,singleflight 键也不同,
// 于是第一波请求会并发重扫整张 media 表(生产环境实测冷启动 14-17s)。
// 按用户折叠成一次扫描后,同一波请求共享结果并整批进缓存。
virtualGroups singleflight.Group
// virtualScanAt 记录每个用户最近一次整库扫描的时间,用于避免为同一个
// 不存在的 id 反复重扫(必须按用户区分:可见性不同,别人的快照不能当
// 自己的答案)。由 virtualMu 保护。
virtualScanAt map[string]time.Time
visibilityMu sync.RWMutex
visibilityCache map[string]embyVisibilityCacheEntry
@@ -197,15 +186,6 @@ const (
embyVirtualSeriesCap = 8000
embyVirtualSeasonCap = 16000
embyVirtualArtworkCap = 24000
// embySeriesGroupBulkCacheEpisodes 限制一次批量记忆的剧集行数。
//
// 批量记忆把「一次扫描服务整页卡片 + 整页海报」的收益拿满,但同时让整库
// 剧集常驻内存。超过该量级时退回只缓存命中的那一部,避免大库吃光内存。
embySeriesGroupBulkCacheEpisodes = 8000
// embyVirtualScanGrace 是「刚做过整库虚拟剧集扫描」的保护窗口。窗口内再
// 请求一个不在缓存里的虚拟 series ID,直接按不存在处理,避免为同一个
// 未知 id 反复重扫整库。
embyVirtualScanGrace = 10 * time.Second
// Clients that already cached the 1x1 placeholder treat a stable tag as
// immutable. Virtual ids are the ones that served that placeholder, so
// only those tags get a suffix that forces a refetch.
+1 -10
View File
@@ -6,7 +6,6 @@ import (
"sort"
"strconv"
"strings"
"time"
)
type embyItemsCacheValue struct {
@@ -78,15 +77,7 @@ func (e *EmbyService) embySimilarCacheKey(mediaID, userID string, limit int) str
}
// defaultEmbyLatestCacheTTLSeconds 是 Emby「最新添加」缓存的兜底时长。
const defaultEmbyLatestCacheTTLSeconds = 900
// embyLatestStaleWindow 是「最新添加」载荷在进程内保留的陈旧副本时长。
//
// 客户端空闲很久之后再打开首页时,新鲜期早已过去:生产环境观察到这时
// 73 个媒体库的 Latest 同时回源,首批请求要 8-25s 才回来,客户端等不及
// 还会重试,进一步放大压力。保留陈旧副本后,这类请求立刻拿到上次的结果,
// 由后台异步重建(见 refreshLatestItemsAsync),用户不再等 SQL。
const embyLatestStaleWindow = 24 * time.Hour
const defaultEmbyLatestCacheTTLSeconds = 300
// embyLatestCacheTTLSeconds 返回「最新添加」列表的缓存时长。它刻意比通用
// 媒体缓存更长:客户端刷新首页时会同时请求全部媒体库的 Latest(生产环境
+1 -1
View File
@@ -153,7 +153,7 @@ func (e *EmbyService) loadLatestItemsCached(ctx context.Context, userID, parentI
}
if e.cache != nil {
freshTTL := time.Duration(e.embyLatestCacheTTLSeconds()) * time.Second
e.cache.SetJSONWithStale(loadCtx, cacheKey, value, freshTTL, freshTTL+embyLatestStaleWindow)
e.cache.SetJSONWithStale(loadCtx, cacheKey, value, freshTTL, freshTTL+30*time.Minute)
}
e.rememberArtworkRefs(value.Artwork)
return value, nil
+40 -130
View File
@@ -41,65 +41,47 @@ func (e *EmbyService) findSeriesGroup(ctx context.Context, id, userID string) (e
return embySeriesGroup{}, false, nil
}
if strings.HasPrefix(id, embyVirtualSeriesPrefix) {
return e.findVirtualSeriesGroup(ctx, id, userID)
if group, ok := e.cachedSeriesGroup(id); ok {
return group, true, nil
}
}
var rows []model.Media
q := e.repo.DB.WithContext(ctx).Model(&model.Media{}).Where("season_num > 0 OR episode_num > 0")
q = e.applyUserMediaVisibility(ctx, q, userID)
q = q.Where("series_id = ?", id)
if !strings.HasPrefix(id, embyVirtualSeriesPrefix) {
q = q.Where("series_id = ?", id)
} else {
// 虚拟 series ID 只可能来自 series_id 为空的媒体:
// 有 series_id 时分组 key 就是 series_id 本身(UUID,不带虚拟前缀)。
q = q.Where("series_id IS NULL OR series_id = ''")
}
if err := q.Order("media.season_num asc, media.episode_num asc, media.episode_fraction asc, media.created_at asc").Limit(embySeriesGroupingLimit).Find(&rows).Error; err != nil {
return embySeriesGroup{}, false, err
}
groups := e.seriesGroupsFromMedia(ctx, rows)
bulkCached := e.rememberScannedSeriesGroups(groups)
for _, group := range groups {
for _, group := range e.seriesGroupsFromMedia(ctx, rows) {
if group.ID == id {
if !bulkCached {
e.rememberSeriesGroup(group)
}
e.rememberSeriesGroup(group)
return group, true, nil
}
}
if series, err := e.repo.Series.FindByID(ctx, id); err != nil {
return embySeriesGroup{}, false, err
} else if series != nil {
return embySeriesGroup{
ID: series.ID,
LibraryID: series.LibraryID,
Name: series.Title,
PosterURL: series.PosterURL,
BackdropURL: series.BackdropURL,
Overview: series.Overview,
Rating: series.Rating,
Year: series.Year,
TMDbID: series.TMDbID,
BangumiID: series.BangumiID,
CreatedAt: series.CreatedAt,
DateLastMediaAdded: series.CreatedAt,
}, true, nil
}
return embySeriesGroup{}, false, nil
}
// findVirtualSeriesGroup 处理 series_id 为空的剧集分组(虚拟 series ID)。
//
// 虚拟 ID 是 hash(库名 + 剧名),无法反解,只能整库算出分组后按 ID 取用。
// 这里走 loadVirtualSeriesGroups,把同一用户的并发请求折叠成一次扫描:
// 首页一次刷新会同时要 73 个库的卡片和海报,各扫一遍在 2 核主机上要 14-17s。
func (e *EmbyService) findVirtualSeriesGroup(ctx context.Context, id, userID string) (embySeriesGroup, bool, error) {
if group, ok := e.cachedSeriesGroup(id); ok {
return group, true, nil
}
if e.virtualScanRecently(userID) {
return embySeriesGroup{}, false, nil
}
groups, err := e.loadVirtualSeriesGroups(ctx, userID)
if err != nil {
return embySeriesGroup{}, false, err
}
for _, group := range groups {
if group.ID == id {
return group, true, nil
if !strings.HasPrefix(id, embyVirtualSeriesPrefix) {
if series, err := e.repo.Series.FindByID(ctx, id); err != nil {
return embySeriesGroup{}, false, err
} else if series != nil {
return embySeriesGroup{
ID: series.ID,
LibraryID: series.LibraryID,
Name: series.Title,
PosterURL: series.PosterURL,
BackdropURL: series.BackdropURL,
Overview: series.Overview,
Rating: series.Rating,
Year: series.Year,
TMDbID: series.TMDbID,
BangumiID: series.BangumiID,
CreatedAt: series.CreatedAt,
DateLastMediaAdded: series.CreatedAt,
}, true, nil
}
}
return embySeriesGroup{}, false, nil
@@ -121,14 +103,20 @@ func (e *EmbyService) findSeasonGroup(ctx context.Context, id, userID string) (e
return season, true, nil
}
// 回退:未刮削(series_id 为空,虚拟 key 由库名+名称派生)的媒体只能全量分组。
// 复用共享扫描:首页并发刷新时整库只扫一次。
groups, err := e.loadVirtualSeriesGroups(ctx, userID)
if err != nil {
var rows []model.Media
q := e.repo.DB.WithContext(ctx).Model(&model.Media{}).
Where("(series_id IS NULL OR series_id = '') AND (season_num > 0 OR episode_num > 0)")
q = e.applyUserMediaVisibility(ctx, q, userID)
if err := q.
Order("media.season_num asc, media.episode_num asc, media.episode_fraction asc, media.created_at asc").
Limit(embySeriesGroupingLimit).
Find(&rows).Error; err != nil {
return embySeasonGroup{}, false, err
}
for _, series := range groups {
for _, series := range e.seriesGroupsFromMedia(ctx, rows) {
for _, season := range e.seasonsForSeries(series) {
if season.ID == id {
e.rememberSeriesGroup(series)
return season, true, nil
}
}
@@ -198,84 +186,6 @@ func (e *EmbyService) seasonGroupForSeries(ctx context.Context, id, seriesID, us
return embySeasonGroup{}, false, nil
}
// rememberScannedSeriesGroups 批量记住一次扫描得到的全部分组,返回是否真的
// 批量写入(超过剧集预算时不做,调用方需自行缓存命中的那一部)。
//
// 只缓存命中的一部时,同一部剧的多张卡片/海报仍要各扫一次;整批写入后,
// 同一次扫描算出来的其他剧也直接命中,于是整页 16 张卡片 + 16 张海报共用
// 一次全表扫描。
func (e *EmbyService) rememberScannedSeriesGroups(groups []embySeriesGroup) bool {
if e == nil || len(groups) == 0 {
return false
}
episodes := 0
for i := range groups {
episodes += len(groups[i].Episodes)
if episodes > embySeriesGroupBulkCacheEpisodes {
return false
}
}
e.rememberSeriesGroups(groups)
return true
}
// loadVirtualSeriesGroups 返回 series_id 为空的整库剧集分组。
//
// 同一用户的并发调用共享同一次扫描:客户端刷新首页时 73 个库的 Latest 同时
// 进来,每个库的卡片和海报又各自要自己的虚拟 series 分组。分组缓存键各不
// 相同,singleflight 也就不会合并,于是第一波请求把整张 media 表并发重算
// 了 73 次(生产环境实测 14-17s)。这里按用户折叠成一次,结果整批进缓存,
// 调用方拿到切片后按 ID 取用即可。
func (e *EmbyService) loadVirtualSeriesGroups(ctx context.Context, userID string) ([]embySeriesGroup, error) {
if e == nil || e.repo == nil {
return nil, nil
}
v, err, _ := e.virtualGroups.Do("virtual:"+userID, func() (any, error) {
// 与 loadLatestItemsCached 一致:一个客户端断开不应取消正在为其他
// 客户端填充的共享扫描。
loadCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
var rows []model.Media
q := e.repo.DB.WithContext(loadCtx).Model(&model.Media{}).
Where("(series_id IS NULL OR series_id = '') AND (season_num > 0 OR episode_num > 0)")
q = e.applyUserMediaVisibility(loadCtx, q, userID)
if err := q.
Order("media.season_num asc, media.episode_num asc, media.episode_fraction asc, media.created_at asc").
Limit(embySeriesGroupingLimit).
Find(&rows).Error; err != nil {
return nil, err
}
groups := e.seriesGroupsFromMedia(loadCtx, rows)
e.rememberSeriesGroups(groups)
e.markVirtualScan(userID)
return groups, nil
})
if err != nil {
return nil, err
}
groups, _ := v.([]embySeriesGroup)
return groups, nil
}
// markVirtualScan 记录该用户最近一次整库扫描的时间。
func (e *EmbyService) markVirtualScan(userID string) {
e.virtualMu.Lock()
if e.virtualScanAt == nil {
e.virtualScanAt = make(map[string]time.Time, 4)
}
e.virtualScanAt[userID] = time.Now()
e.virtualMu.Unlock()
}
// virtualScanRecently 报告该用户是否刚做过一次整库扫描。窗口内仍不在缓存里
// 的目标 ID 按不存在处理,避免为同一个未知 ID 反复重扫整库。
func (e *EmbyService) virtualScanRecently(userID string) bool {
e.virtualMu.RLock()
scannedAt := e.virtualScanAt[userID]
e.virtualMu.RUnlock()
return !scannedAt.IsZero() && time.Since(scannedAt) < embyVirtualScanGrace
}
func (e *EmbyService) seriesGroupsFromMedia(ctx context.Context, rows []model.Media) []embySeriesGroup {
byID := map[string]*embySeriesGroup{}
order := []string{}
+3 -29
View File
@@ -26,45 +26,19 @@ func (e *EmbyService) rememberSeriesGroup(group embySeriesGroup) {
if e == nil || strings.TrimSpace(group.ID) == "" {
return
}
e.rememberSeriesGroups([]embySeriesGroup{group})
}
// rememberSeriesGroups 批量记住一组剧集分组(含派生季与封面)。
//
// 一次全表扫描会同时算出整库的剧集分组,而调用方往往只关心其中一部。若只
// 逐个 rememberSeriesGroup,同一次首页刷新里 16 张卡片 + 16 张海报都要各自
// 命中自己的虚拟 series ID,未命中就重扫整张 media 表(生产环境实测单次
// 1.3s,73 个库并发时把连接池排满,首页要 8-25s)。批量写入后冷缓存只付
// 一次扫描,其余全部是 O(1) 命中。
func (e *EmbyService) rememberSeriesGroups(groups []embySeriesGroup) {
if e == nil || len(groups) == 0 {
return
}
expiresAt := time.Now().Add(embyVirtualCacheTTL)
e.virtualMu.Lock()
defer e.virtualMu.Unlock()
if e.virtualSeries == nil {
e.virtualSeries = make(map[string]embySeriesCacheEntry, len(groups))
e.virtualSeries = make(map[string]embySeriesCacheEntry)
}
if e.virtualSeasons == nil {
e.virtualSeasons = make(map[string]embySeasonCacheEntry, len(groups))
e.virtualSeasons = make(map[string]embySeasonCacheEntry)
}
if e.virtualArtwork == nil {
e.virtualArtwork = make(map[string]embyArtworkCacheEntry, len(groups)*2)
e.virtualArtwork = make(map[string]embyArtworkCacheEntry)
}
for _, group := range groups {
e.rememberSeriesGroupLocked(group, expiresAt)
}
// 写入后再裁剪:逐条裁剪会让整批写入退化成 O(n²),且中途裁剪可能把
// 同一批里刚写进去的分组又淘汰掉。
e.trimVirtualCachesLocked(time.Now())
}
// rememberSeriesGroupLocked 写入单个分组,调用方必须已持有 virtualMu。
func (e *EmbyService) rememberSeriesGroupLocked(group embySeriesGroup, expiresAt time.Time) {
if strings.TrimSpace(group.ID) == "" {
return
}
e.virtualSeries[group.ID] = embySeriesCacheEntry{group: group, expiresAt: expiresAt}
e.virtualArtwork[group.ID] = embyArtworkCacheEntry{primary: group.PosterURL, backdrop: group.BackdropURL, expiresAt: expiresAt}
e.virtualArtwork[group.ID+"-bd"] = embyArtworkCacheEntry{primary: group.PosterURL, backdrop: group.BackdropURL, expiresAt: expiresAt}
+2 -27
View File
@@ -2,9 +2,6 @@ package service
import (
"context"
"crypto/sha256"
"encoding/hex"
"io"
"sort"
"strings"
"time"
@@ -281,20 +278,6 @@ func isSupportedEmbyCollectionType(t string) bool {
return false
}
// libraryCoverTag 生成媒体库拼图封面的图片版本号。
//
// 第三方客户端把库主图按 (URL, ImageTag) 缓存,tag 不变就不会重新请求。
// 用成员海报列表的指纹做 tag,库内容变化或生成规则调整时客户端自然会刷新。
func libraryCoverTag(libraryID string, sources []string) string {
h := sha256.New()
_, _ = io.WriteString(h, libraryID)
for _, src := range sources {
_, _ = io.WriteString(h, src)
_, _ = h.Write([]byte{0})
}
return libraryID + "-" + hex.EncodeToString(h.Sum(nil))[:8]
}
func (e *EmbyService) libraryAsView(ctx context.Context, l *model.Library) map[string]any {
collectionType := "movies"
switch l.Type {
@@ -309,17 +292,9 @@ func (e *EmbyService) libraryAsView(ctx context.Context, l *model.Library) map[s
}
// 库封面:与剧集/电影一样,只有在能解析出封面时才广告 ImageTags.Primary,
// 否则客户端会认为该库无主图而不去请求 /Images/Primary。
//
// 拼图封面的 tag 带上「成员海报」的指纹:库内容变化或生成规则调整(例如
// 从 2×2 改成竖切)时 tag 会变,客户端才会丢缓存重新拉图;固定用库 ID
// 会让第三方客户端永远显示第一次缓存下来的封面。
imageTags := map[string]string{}
if sources, lib := LibraryCoverSourcesForLibrary(ctx, e.repo, l.ID); lib != nil {
if len(sources) > 0 {
imageTags["Primary"] = libraryCoverTag(l.ID, sources)
} else if strings.TrimSpace(lib.CoverURL) != "" {
imageTags["Primary"] = l.ID
}
if e.LibraryHasCover(ctx, l.ID) {
imageTags["Primary"] = l.ID
}
return map[string]any{
"Id": l.ID,
+5 -11
View File
@@ -136,12 +136,7 @@ func (e *EmbyService) RecordProgressWithSession(ctx context.Context, userID, med
SessionID: playSessionID,
SessionStartedAtMs: embyPlaySessionStartedAtMs(playSessionID),
})
// 普通进度上报不清列表载荷缓存。播放期间客户端每 5-10 秒上报一次,
// 每次清空 media:emby:* 会让「最新添加」整批 payload(客户端刷新首页时
// 一次并发 73 个库)反复回源 SQL,生产环境实测因此出现 8-25s 的加载。
// 续播位置由 /Items/Resume 与条目详情实时查库,列表里的进度条允许滞后;
// 只有「已看完」这类状态翻转才让缓存失效(position 到达片尾 30s 内)。
if err == nil && playbackProgressCompleted(pos, dur) {
if err == nil {
e.invalidateEmbyItemsCache(ctx)
}
return err
@@ -292,11 +287,10 @@ func applyMeBoxUserData(raw any, history *model.PlaybackHistory, favorite bool)
return userData
}
// embyInvalMu 节流全量缓存失效:收藏/标记已看这类显式操作会改变列表里的
// UserData,必须让 payload 失效。客户端连续点选时会短时间触发多次,节流到
// 30s 一次即可,避免每次都 SCAN+DEL 全部 media:emby:* 缓存。
// 播放进度上报不走这里(见 RecordProgressWithSession):续播位置实时查库,
// 列表里的进度条允许滞后,不值得为它把整批首页载荷打回源 SQL。
// embyInvalMu 节流全量缓存失效:播放期间客户端每 5-10s 上报一次进度,
// 每次都 SCAN+DEL 全部 media:emby:* 缓存会把缓存命中率持续打穿(其他
// 客户端每次翻页都回源 SQL)。条目载荷的 UserData 在请求时动态合并,
// 进度类变更做 30s 节流即可,不影响正确性观感。
var (
embyInvalMu sync.Mutex
embyInvalLast time.Time
-109
View File
@@ -1,109 +0,0 @@
package service
import (
"crypto/sha256"
"encoding/hex"
"errors"
"net/http"
"os"
"path/filepath"
"strings"
"time"
"unicode"
)
var ErrInvalidHLSPlaybackScope = errors.New("invalid HLS playback scope")
var ErrHLSGenerationSuperseded = errors.New("HLS playback generation superseded")
// HLSPlaybackScope separates a media's independent users and player instances.
// UserID must come from server authentication, never from a request parameter.
type HLSPlaybackScope struct {
UserID string
SessionID string
}
func HLSPlaybackScopeForRequest(r *http.Request, userID string) (HLSPlaybackScope, error) {
scope := HLSPlaybackScope{UserID: strings.TrimSpace(userID)}
if scope.UserID == "" || r == nil {
return scope, ErrInvalidHLSPlaybackScope
}
for _, key := range []string{"session_id", "PlaySessionId", "playSessionId", "DeviceId", "deviceId"} {
if value := strings.TrimSpace(r.URL.Query().Get(key)); value != "" {
scope.SessionID = value
break
}
}
if scope.SessionID == "" {
if device := strings.TrimSpace(r.Header.Get("X-Emby-Device-Id")); device != "" {
scope.SessionID = device
}
}
if len(scope.SessionID) > 256 || strings.IndexFunc(scope.SessionID, unicode.IsControl) >= 0 {
return scope, ErrInvalidHLSPlaybackScope
}
// Clients without a session/device identifier remain compatible but only
// share a legacy task with the same authenticated user, never other users.
return scope, nil
}
func (scope HLSPlaybackScope) jobKey(mediaID string) string {
if scope.UserID == "" && scope.SessionID == "" {
return mediaID // Existing internal callers/tests retain their legacy API.
}
sum := sha256.Sum256([]byte(scope.UserID + "\x00" + mediaID + "\x00" + scope.SessionID))
return "player-" + hex.EncodeToString(sum[:])
}
func hlsPlaybackJobKey(mediaID string, scopes []HLSPlaybackScope) string {
if len(scopes) == 0 {
return mediaID
}
return scopes[0].jobKey(mediaID)
}
func (job *hlsJob) key() string {
if job.jobKey != "" {
return job.jobKey
}
return job.mediaID
}
func (t *TranscoderService) jobPlaylistPath(job *hlsJob) string {
dir := job.outputDir
if dir == "" {
dir = t.HLSDir(job.key())
}
return filepath.Join(dir, "index.m3u8")
}
// hlsOutputDir captures one immutable run directory. Even if a new run starts
// after this check, the request cannot accidentally open its replacement files.
func (t *TranscoderService) hlsOutputDir(key string, generation int64) (string, error) {
t.mu.Lock()
defer t.mu.Unlock()
job := t.jobs[key]
if job == nil {
return "", ErrMediaNotFound
}
if job.seekGen != generation {
return "", ErrHLSGenerationSuperseded
}
t.touchJobLocked(key)
if job.outputDir != "" {
return job.outputDir, nil
}
return t.HLSDir(key), nil
}
func (t *TranscoderService) StopPlayback(mediaID string, scope HLSPlaybackScope) {
key := scope.jobKey(mediaID)
unlock := t.lockJobStart(key)
defer unlock()
t.mu.Lock()
job := t.detachJobLocked(key)
t.mu.Unlock()
waitJobExit(job, 12*time.Second)
// Keep cleanup under the same gate as start: a delayed stop must not wipe
// an output directory created by a concurrent replacement request.
_ = os.RemoveAll(t.HLSDir(key))
}
@@ -1,133 +0,0 @@
package service
import (
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
"time"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
)
func cachedScopedHLS(t *testing.T, scopes ...HLSPlaybackScope) (*StreamService, *TranscoderService, map[string]string) {
t.Helper()
cfg := &config.Config{}
cfg.Cache.CacheDir = t.TempDir()
cfg.Transcoder.Enabled = true
db := newServiceTestDB(t)
if err := db.Create(&model.Media{Base: model.Base{ID: "movie"}, Path: "unused-cached-source", DurationSec: 60}).Error; err != nil {
t.Fatal(err)
}
transcoder := NewTranscoderService(cfg, zap.NewNop(), repository.New(db), nil)
paths := map[string]string{}
for _, scope := range scopes {
key := scope.jobKey("movie")
dir := filepath.Join(transcoder.HLSDir(key), "run-cached")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
playlist := filepath.Join(dir, "index.m3u8")
if err := os.WriteFile(playlist, []byte("#EXTM3U\n#EXTINF:4,\nseg_00000.ts\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "seg_00000.ts"), []byte(scope.UserID+"/"+scope.SessionID), 0o600); err != nil {
t.Fatal(err)
}
transcoder.jobs[key] = &hlsJob{mediaID: "movie", jobKey: key, outputDir: dir, subtitleStream: -1, startedAt: time.Now(), cancel: func() {}}
paths[key] = playlist
}
return NewStreamService(cfg, zap.NewNop(), repository.New(db), transcoder), transcoder, paths
}
func TestHLSPlaybackEnsureReusesOnlyMatchingOwnerAndSession(t *testing.T) {
a := HLSPlaybackScope{UserID: "viewer", SessionID: "tab-a"}
b := HLSPlaybackScope{UserID: "viewer", SessionID: "tab-b"}
_, transcoder, paths := cachedScopedHLS(t, a, b)
for _, scope := range []HLSPlaybackScope{a, b} {
playlist, err := transcoder.EnsureJobFromSubtitleQuality(t.Context(), "movie", 0, 0, -1, "", scope)
if err != nil {
t.Fatal(err)
}
if playlist != paths[scope.jobKey("movie")] {
t.Fatalf("scope %#v reused another session's playlist %q", scope, playlist)
}
}
}
func TestHLSPlaybackStopRemovesOnlyItsSessionOutput(t *testing.T) {
a := HLSPlaybackScope{UserID: "viewer", SessionID: "tab-a"}
b := HLSPlaybackScope{UserID: "viewer", SessionID: "tab-b"}
_, transcoder, paths := cachedScopedHLS(t, a, b)
transcoder.StopPlayback("movie", a)
if _, err := os.Stat(paths[a.jobKey("movie")]); !os.IsNotExist(err) {
t.Fatalf("stopped session output must be cleaned up, err=%v", err)
}
if _, err := os.Stat(paths[b.jobKey("movie")]); err != nil {
t.Fatalf("another player's output was removed: %v", err)
}
}
func TestHLSPlaybackIdleCleanupRemovesExpiredSessionOutput(t *testing.T) {
a := HLSPlaybackScope{UserID: "viewer", SessionID: "tab-a"}
b := HLSPlaybackScope{UserID: "viewer", SessionID: "tab-b"}
_, transcoder, paths := cachedScopedHLS(t, a, b)
job := transcoder.jobs[a.jobKey("movie")]
job.lastAccess = time.Now().Add(-2 * time.Minute)
if !transcoder.stopIdleJob(job, time.Minute) {
t.Fatal("expired session was not stopped")
}
if _, err := os.Stat(paths[a.jobKey("movie")]); !os.IsNotExist(err) {
t.Fatalf("expired output was retained: %v", err)
}
if _, err := os.Stat(paths[b.jobKey("movie")]); err != nil {
t.Fatalf("active session output was removed: %v", err)
}
}
func TestHLSPlaybackPropagatesHeaderDeviceIdentityToSegments(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/api/hls/movie/index.m3u8?token=test", nil)
req.Header.Set("X-Emby-Device-Id", "legacy-device")
scope, err := HLSPlaybackScopeForRequest(req, "viewer")
if err != nil {
t.Fatal(err)
}
svc, _, _ := cachedScopedHLS(t, scope)
rec := httptest.NewRecorder()
if err := svc.ServeHLSPlaylist(rec, req, "movie", scope); err != nil {
t.Fatal(err)
}
var segment string
for _, line := range strings.Split(rec.Body.String(), "\n") {
if strings.HasPrefix(line, "seg_") {
segment = line
break
}
}
u, err := url.Parse(segment)
if err != nil || u.Query().Get("session_id") == "" {
t.Fatalf("playlist must carry the header-only device scope to child URLs: %q err=%v", segment, err)
}
child := httptest.NewRequest(http.MethodGet, "/api/hls/movie/"+segment, nil)
childScope, err := HLSPlaybackScopeForRequest(child, "viewer")
if err != nil {
t.Fatal(err)
}
if childScope.jobKey("movie") != scope.jobKey("movie") {
t.Fatal("child request changed its owner's device scope")
}
childRec := httptest.NewRecorder()
if err := svc.ServeHLSSegment(childRec, child, "movie", "seg_00000.ts", childScope); err != nil {
t.Fatal(err)
}
if childRec.Body.String() != "viewer/"+scope.SessionID {
t.Fatalf("wrong scoped segment: %s", childRec.Body.String())
}
}
-97
View File
@@ -1,97 +0,0 @@
package service
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/config"
)
func TestHLSPlaybackScopesIsolateSegmentReadsAndStop(t *testing.T) {
cfg := &config.Config{}
cfg.Cache.CacheDir = t.TempDir()
transcoder := NewTranscoderService(cfg, zap.NewNop(), nil, nil)
svc := NewStreamService(cfg, zap.NewNop(), nil, transcoder)
a := HLSPlaybackScope{UserID: "user-a", SessionID: "tab-a"}
b := HLSPlaybackScope{UserID: "user-b", SessionID: "tab-a"}
c := HLSPlaybackScope{UserID: "user-a", SessionID: "tab-c"}
for _, scope := range []HLSPlaybackScope{a, b, c} {
key := scope.jobKey("movie")
dir := filepath.Join(cfg.Cache.CacheDir, key, "run-current")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "seg_00000.ts"), []byte(scope.UserID+"/"+scope.SessionID), 0o600); err != nil {
t.Fatal(err)
}
transcoder.jobs[key] = &hlsJob{mediaID: "movie", jobKey: key, outputDir: dir, seekGen: 100, cancel: func() {}}
}
for _, scope := range []HLSPlaybackScope{a, b, c} {
req := httptest.NewRequest(http.MethodGet, "/api/hls/movie/seg_00000.ts?_seek=100", nil)
rec := httptest.NewRecorder()
if err := svc.ServeHLSSegment(rec, req, "movie", "seg_00000.ts", scope); err != nil {
t.Fatal(err)
}
if got, want := rec.Body.String(), scope.UserID+"/"+scope.SessionID; got != want {
t.Fatalf("segment=%q, want %q", got, want)
}
}
transcoder.StopPlayback("movie", a)
if transcoder.jobs[a.jobKey("movie")] != nil {
t.Fatal("stopped scope is still active")
}
if transcoder.jobs[b.jobKey("movie")] == nil || transcoder.jobs[c.jobKey("movie")] == nil {
t.Fatal("stopping one player affected another user/tab")
}
// A delayed stop from a previous local-playback session must not affect a replacement.
transcoder.StopPlayback("movie", a)
if transcoder.jobs[c.jobKey("movie")] == nil {
t.Fatal("old stop cancelled a new session")
}
}
func TestHLSPlaybackRejectsStaleSegmentGeneration(t *testing.T) {
cfg := &config.Config{}
cfg.Cache.CacheDir = t.TempDir()
transcoder := NewTranscoderService(cfg, zap.NewNop(), nil, nil)
svc := NewStreamService(cfg, zap.NewNop(), nil, transcoder)
scope := HLSPlaybackScope{UserID: "viewer", SessionID: "tab"}
key := scope.jobKey("movie")
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "seg_00000.ts"), []byte("new-generation"), 0o600); err != nil {
t.Fatal(err)
}
transcoder.jobs[key] = &hlsJob{mediaID: "movie", jobKey: key, outputDir: dir, seekGen: 200, cancel: func() {}}
req := httptest.NewRequest(http.MethodGet, "/api/hls/movie/seg_00000.ts?_seek=100", nil)
rec := httptest.NewRecorder()
if err := svc.ServeHLSSegment(rec, req, "movie", "seg_00000.ts", scope); err == nil {
t.Fatal("stale segment request read a replacement generation")
}
if rec.Body.Len() != 0 {
t.Fatal("stale request exposed replacement bytes")
}
}
func TestHLSPlaybackScopeRequiresServerOwner(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/?session_id=tab&user_id=attacker-controlled", nil)
if _, err := HLSPlaybackScopeForRequest(req, ""); err == nil {
t.Fatal("missing authenticated owner was accepted")
}
a, err := HLSPlaybackScopeForRequest(req, "real-owner")
if err != nil {
t.Fatal(err)
}
if a.UserID != "real-owner" {
t.Fatal("client parameter supplied the owner")
}
req = httptest.NewRequest(http.MethodGet, "/?PlaySessionId=emby-session", nil)
b, err := HLSPlaybackScopeForRequest(req, "real-owner")
if err != nil || b.SessionID == "" {
t.Fatalf("Emby session must stay scoped: %#v %v", b, err)
}
}
+6 -12
View File
@@ -49,10 +49,6 @@ type ImageProxy struct {
resizeSemMu sync.Mutex
resizeSem chan struct{}
// resizeQueueWait 是一次请求最多为一格解码槽排队的时长。为 0 时用
// defaultResizeQueueWait;测试会把它调小以免真的等待两秒。
resizeQueueWait time.Duration
// libraryRootsFn returns the configured media library roots so that
// sidecar poster/artwork files stored alongside media (under arbitrary
// per-library paths) are allowed by isAllowedLocalPath. It is provided
@@ -90,12 +86,6 @@ const (
// imageRenditionCacheSubdir 存放挂载 Emby 直接按尺寸产出的成品图:它们
// 已经是客户端要的最终尺寸,和派生缩略图一样属于长期保留的热路径缓存。
imageRenditionCacheSubdir = "renditions"
// defaultResizeQueueWait 是请求为一格解码槽排队的最长时间。一张 3MP
// 海报的解码+缩放约 0.15s,2 核机器两格槽位足以在几秒内消化一整屏
// 海报;等待远好于「槽位忙就回退多兆字节原图」(那样每张图在外部网络
// 下要十几秒)。超时后仍回退原图,保证请求不会被无限期挂住。
defaultResizeQueueWait = 3 * time.Second
)
// NewImageProxy is the constructor.
@@ -200,7 +190,7 @@ func imageResizeConcurrency() int {
}
// Prune removes oldest cached images until disk usage is within the configured limit.
// 原图池按独立配额优先淘汰,客户端热路径读取的成品图最后淘汰。
// 原图池按独立配额与保留时长优先淘汰,客户端热路径读取的成品图最后淘汰。
func (p *ImageProxy) Prune() (PruneImageCacheResult, error) {
if p.cfg == nil || p.cfg.Cache.ImagesMaxSizeMB <= 0 {
return PruneImageCacheResult{}, nil
@@ -209,7 +199,11 @@ func (p *ImageProxy) Prune() (PruneImageCacheResult, error) {
if p.cfg.Cache.ImagesOriginalsMaxSizeMB > 0 {
originalsMaxBytes = int64(p.cfg.Cache.ImagesOriginalsMaxSizeMB) * 1024 * 1024
}
var originalsMaxAge time.Duration
if p.cfg.Cache.ImagesOriginalsTTLHours > 0 {
originalsMaxAge = time.Duration(p.cfg.Cache.ImagesOriginalsTTLHours) * time.Hour
}
totalBytes := int64(p.cfg.Cache.ImagesMaxSizeMB) * 1024 * 1024
pools := ImageCachePools(p.cacheDir, originalsMaxBytes)
pools := ImageCachePools(p.cacheDir, originalsMaxBytes, originalsMaxAge)
return PruneImageCachePools(pools, totalBytes)
}
+14 -79
View File
@@ -17,7 +17,6 @@ import (
"path/filepath"
"strconv"
"strings"
"time"
"go.uber.org/zap"
@@ -139,8 +138,8 @@ func fitSize(srcW, srcH, maxW, maxH int) (int, int) {
var errImageResizeTooLarge = errors.New("image source exceeds resize pixel budget")
// resizeImageData 按选项缩放图片并重新编码。返回 unchanged=true 表示源图
// 已经是可直接下发的紧凑字节,调用方应直接输出原始字节(不重复编码)。
// resizeImageData 按选项缩放图片并重新编码。返回 unchanged=true 表示原图
// 本身已满足目标尺寸,调用方应直接输出原始字节(不重复编码、不损失质量)。
func resizeImageData(data []byte, o imageResizeOptions) (out []byte, ctype string, unchanged bool, err error) {
if len(data) == 0 || !o.active() {
return data, detectContentType(data), true, nil
@@ -152,21 +151,12 @@ func resizeImageData(data []byte, o imageResizeOptions) (out []byte, ctype strin
if cfg.Width <= 0 || cfg.Height <= 0 {
return nil, "", false, errors.New("invalid image dimensions")
}
// 先用 DecodeConfig 判断是否需要缩放。尺寸已在目标框内时仍要重编码:
// 刮削来源的海报常是 1000×1500、1MB 以上的 PNG,直接原样下发会让一处
// 海报墙就从「几十 KB 缩略图」变成「几十 MB 原图」——外部网络下每张
// 都要十几秒。只有本来就是紧凑 Web 格式(JPEG/WebP)且体积很小的源图
// 才值得零成本直出。
// 先用 DecodeConfig 判断是否需要解码整图:图已够小就零成本返回原字节。
dstW, dstH := fitSize(cfg.Width, cfg.Height, o.MaxWidth, o.MaxHeight)
unchangedSize := dstW == cfg.Width && dstH == cfg.Height
if unchangedSize && isCompactWebFormat(format) && len(data) <= compactImageSkipBytes {
if dstW == cfg.Width && dstH == cfg.Height {
return data, detectContentType(data), true, nil
}
if int64(cfg.Width)*int64(cfg.Height) > imageResizeMaxSourcePixels {
if unchangedSize {
// 尺寸已满足但像素量超出解码预算:宁可直出原图,也不冒 OOM 风险。
return data, detectContentType(data), true, nil
}
return nil, "", false, errImageResizeTooLarge
}
@@ -175,12 +165,7 @@ func resizeImageData(data []byte, o imageResizeOptions) (out []byte, ctype strin
return nil, "", false, err
}
dst := image.NewRGBA(image.Rect(0, 0, dstW, dstH))
if unchangedSize {
// 不缩放时省掉一次全图重采样,只做格式转换。
draw.Draw(dst, dst.Bounds(), src, src.Bounds().Min, draw.Src)
} else {
resizeScaled(dst, src, cfg.Width, cfg.Height)
}
resizeScaled(dst, src, cfg.Width, cfg.Height)
// 只有可能带透明的源格式才需要逐像素确认,避免 JPEG 的无谓遍历。
// 写实海报的 PNG 通常比等价 JPEG 大一个数量级,因此在确认不含透明
@@ -246,12 +231,9 @@ func isOpaqueImage(img *image.RGBA) bool {
// resizeSourceKey 生成源图身份键:只覆盖源文件身份(路径 + 大小 + mtime),不含
// 目标尺寸。同一源图的所有尺寸档位共用这个前缀,才能互相列举与派生(小图直接
// 从已缓存的大图缩小,而不是重新解码原图)。
//
// 前缀里的 v3 是缓存格式版本:v2 时期「尺寸已在目标框内」的 PNG 会原样直出、
// 不写缓存,而现在同样条件会重编码成 JPEG,旧缓存与新结果语义不同,必须换键。
func (o imageResizeOptions) resizeSourceKey(sourceID string, stat os.FileInfo) string {
h := sha256.New()
_, _ = fmt.Fprintf(h, "v3|%s", sourceID)
_, _ = fmt.Fprintf(h, "v2|%s", sourceID)
if stat != nil {
_, _ = fmt.Fprintf(h, "|%d|%d", stat.Size(), stat.ModTime().UnixNano())
}
@@ -374,53 +356,13 @@ func (p *ImageProxy) tryAcquireResizeSlot() (func(), bool) {
}
}
func (p *ImageProxy) resizeQueueTimeout() time.Duration {
if p != nil && p.resizeQueueWait > 0 {
return p.resizeQueueWait
}
return defaultResizeQueueWait
}
// acquireResizeSlotFor 对紧凑小图只尝试一次槽位,其余情况最多排队
// resizeQueueTimeout。
//
// 从前这里对 1.5MB 以下的图一律「槽位忙就跳过缩放」,结果是并发加载海报墙
// 时几乎每张图都回退成 1MB 上下的 PNG 原图:一页要传几十 MB,外部网络下单张
// 图就要二十多秒。缩略图本身只有几十 KB,排队几百毫秒明显更划算;只有真正
// 大到会拖慢页面的原图才值得在超时后直出。
// acquireResizeSlotFor 对中等体积的图只尝试一次槽位。忙则放弃缩放,
// 避免背景图墙在 2 核机器上排成数秒。超大原图仍等待,以免直出多兆字节。
func (p *ImageProxy) acquireResizeSlotFor(ctx context.Context, size int64) (func(), bool) {
if size > 0 && size <= compactImageSkipBytes {
if size > 0 && size <= resizeQueueBypassBytes {
return p.tryAcquireResizeSlot()
}
return p.acquireResizeSlotWithin(ctx, p.resizeQueueTimeout())
}
// acquireResizeSlotWithin 在 timeout 内等待一格解码槽;超时或请求取消返回
// false,调用方回退原图。
func (p *ImageProxy) acquireResizeSlotWithin(ctx context.Context, timeout time.Duration) (func(), bool) {
if p == nil {
return func() {}, true
}
if timeout <= 0 {
return p.tryAcquireResizeSlot()
}
p.resizeSemMu.Lock()
if p.resizeSem == nil {
p.resizeSem = make(chan struct{}, imageResizeConcurrency())
}
sem := p.resizeSem
p.resizeSemMu.Unlock()
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case sem <- struct{}{}:
return func() { <-sem }, true
case <-timer.C:
return nil, false
case <-ctx.Done():
return nil, false
}
return p.acquireResizeSlot(ctx)
}
// serveResizedFromFile 从 srcPath 读取图片,按选项缩放后写出,并把结果缓存
@@ -530,6 +472,10 @@ func (p *ImageProxy) ensureResizeCache(ctx context.Context, srcPath string, o im
// ponytail: 200KB 覆盖这台机器上的典型海报(平均约 100KB);更大的图仍尝试缩放。
const compactImageSkipBytes = 200 * 1024
// resizeQueueBypassBytes 是“槽位忙就放弃缩放”的上限。超过它的原图继续排队,
// 以免把未压缩的剧照直接发给客户端。
const resizeQueueBypassBytes = 1536 * 1024
// serveCompactOriginal 在源文件已经很小且是浏览器可直接显示的 JPEG/WebP 时
// 跳过解码。返回 false 表示仍应走缩放路径。
func serveCompactOriginal(w http.ResponseWriter, r *http.Request, srcPath string, stat os.FileInfo) bool {
@@ -558,17 +504,6 @@ func isCompactWebImage(header []byte) bool {
string(header[8:12]) == "WEBP"
}
// isCompactWebFormat 判断 image.DecodeConfig 报告的格式是否已经适合直接下发
// 给浏览器。PNG/BMP/GIF 即使尺寸很小也可能有几百 KB,需要重编码成 JPEG。
func isCompactWebFormat(format string) bool {
switch strings.ToLower(format) {
case "jpeg", "jpg", "webp":
return true
default:
return false
}
}
// writeResizeCache 原子写入缩放结果;失败只记日志,不影响本次响应。
// 与 writeImageCache 一样不再持有全局锁:临时文件名唯一、rename 原子,
// 持锁只会把缩略图的写盘和原图的写盘串成一条队。
+10 -120
View File
@@ -123,58 +123,17 @@ func TestResizeImageDataScalesDownAndReencodesAsJPEG(t *testing.T) {
}
}
// 尺寸已经在目标框内的 PNG 也必须转成 JPEG。刮削来的海报常是 1000×1500、
// 1MB 级的 PNG;此前这种情况原样直出,一处海报墙就要下载几十 MB,外部网络下
// 每张图要十几秒(线上实测单张 1.3MB / 20 多秒)。
func TestResizeImageDataReencodesPNGWithinBounds(t *testing.T) {
func TestResizeImageDataLeavesImagesWithinBoundsUntouched(t *testing.T) {
data := encodeTestPNG(t, 200, 300, 255)
out, ctype, unchanged, err := resizeImageData(data, imageResizeOptions{MaxWidth: 400, MaxHeight: 600, Quality: 80})
if err != nil {
t.Fatalf("resize: %v", err)
}
if unchanged {
t.Fatal("PNG within the target box must still be re-encoded as a compact web format")
}
if ctype != "image/jpeg" {
t.Fatalf("content type = %q, want image/jpeg", ctype)
}
cfg, format, err := image.DecodeConfig(bytes.NewReader(out))
if err != nil {
t.Fatalf("decode re-encoded image: %v", err)
}
if format != "jpeg" {
t.Fatalf("format = %q, want jpeg", format)
}
if cfg.Width != 200 || cfg.Height != 300 {
t.Fatalf("re-encoded to %dx%d, want the original 200x300 (no upscaling)", cfg.Width, cfg.Height)
}
}
// 已经是紧凑 Web 格式且体积很小的源图仍然零成本直出。
func TestResizeImageDataLeavesSmallCompactSourceUntouched(t *testing.T) {
var buf bytes.Buffer
img := image.NewRGBA(image.Rect(0, 0, 200, 300))
for y := 0; y < 300; y++ {
for x := 0; x < 200; x++ {
img.Set(x, y, color.RGBA{R: uint8(x % 16 * 16), G: uint8(y % 16 * 16), B: 200, A: 255})
}
}
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 80}); err != nil {
t.Fatalf("encode test jpeg: %v", err)
}
data := buf.Bytes()
if len(data) > compactImageSkipBytes {
t.Fatalf("test jpeg is %d bytes, expected a compact source", len(data))
}
out, ctype, unchanged, err := resizeImageData(data, imageResizeOptions{MaxWidth: 400, MaxHeight: 600})
out, _, unchanged, err := resizeImageData(data, imageResizeOptions{MaxWidth: 400, MaxHeight: 600})
if err != nil {
t.Fatalf("resize: %v", err)
}
if !unchanged {
t.Fatal("a compact JPEG already within bounds should be served unchanged")
t.Fatal("expected an image already within bounds to be returned unchanged")
}
if !bytes.Equal(out, data) || ctype != "image/jpeg" {
t.Fatalf("expected the original jpeg bytes back, got %d bytes of %q", len(out), ctype)
if !bytes.Equal(out, data) {
t.Fatal("expected the original bytes to be returned verbatim")
}
}
@@ -343,9 +302,7 @@ func TestServeResizedFromFileSkipsDecodeForCompactJPEG(t *testing.T) {
}
}
// 排队上限:槽位一直不空时,请求最多等 resizeQueueWait,然后回退原图,
// 不会被无限期挂住。这条约束保护的是请求本身,而不是「不等就直出原图」。
func TestServeResizedFromFileFallsBackWhenResizeQueueStaysBusy(t *testing.T) {
func TestServeResizedFromFileDoesNotQueueWhenResizeBusy(t *testing.T) {
dir := t.TempDir()
src := dir + string(os.PathSeparator) + "backdrop.jpg"
data := make([]byte, compactImageSkipBytes+1024)
@@ -353,7 +310,7 @@ func TestServeResizedFromFileFallsBackWhenResizeQueueStaysBusy(t *testing.T) {
if err := os.WriteFile(src, data, 0o644); err != nil {
t.Fatalf("write source: %v", err)
}
proxy := &ImageProxy{cacheDir: dir + string(os.PathSeparator) + "cache", resizeQueueWait: 50 * time.Millisecond}
proxy := &ImageProxy{cacheDir: dir + string(os.PathSeparator) + "cache"}
var releases []func()
for i := 0; i < imageResizeConcurrency(); i++ {
release, ok := proxy.acquireResizeSlot(context.Background())
@@ -370,77 +327,10 @@ func TestServeResizedFromFileFallsBackWhenResizeQueueStaysBusy(t *testing.T) {
start := time.Now()
handled := proxy.serveResizedFromFile(httptest.NewRecorder(), httptest.NewRequest("GET", "/x?maxWidth=480", nil), src, imageResizeOptions{MaxWidth: 480})
if elapsed := time.Since(start); elapsed > 500*time.Millisecond {
t.Fatalf("request waited %v with every slot held; the queue wait must be bounded", elapsed)
if time.Since(start) > 200*time.Millisecond {
t.Fatal("resize slot was busy but the request still waited in the queue")
}
if handled {
t.Fatal("expected a permanently busy queue to fall back so the caller can serve the original")
}
}
// 槽位只是短暂忙碌时必须排队等待并产出缩略图,而不是直出多兆字节原图。
// 这是线上「海报墙每张图 20 多秒」的根因:并发加载时槽位全忙,旧逻辑把
// 1MB 级 PNG 原样下发。
// encodeNoisyPNG 生成难以压缩的 PNG:规则色块会被压到 1KB 级别,测不出
// 「大体积 PNG 被原样下发」这条回归。
func encodeNoisyPNG(t *testing.T, w, h int) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, w, h))
seed := uint32(0x9e3779b9)
for y := 0; y < h; y++ {
for x := 0; x < w; x++ {
seed = seed*1664525 + 1013904223
img.Set(x, y, color.RGBA{R: uint8(seed >> 24), G: uint8(seed >> 16), B: uint8(seed >> 8), A: 255})
}
}
var buf bytes.Buffer
if err := png.Encode(&buf, img); err != nil {
t.Fatalf("encode noisy png: %v", err)
}
return buf.Bytes()
}
func TestServeResizedFromFileWaitsForSlotInsteadOfServingOriginal(t *testing.T) {
dir := t.TempDir()
src := dir + string(os.PathSeparator) + "poster.png"
original := encodeNoisyPNG(t, 378, 540)
if len(original) <= compactImageSkipBytes {
t.Fatalf("test poster is %d bytes, want a source above the compact threshold", len(original))
}
if err := os.WriteFile(src, original, 0o644); err != nil {
t.Fatalf("write source: %v", err)
}
proxy := &ImageProxy{cacheDir: dir + string(os.PathSeparator) + "cache"}
var releases []func()
for i := 0; i < imageResizeConcurrency(); i++ {
release, ok := proxy.acquireResizeSlot(context.Background())
if !ok {
t.Fatal("expected a resize slot")
}
releases = append(releases, release)
}
go func() {
time.Sleep(40 * time.Millisecond)
releases[0]()
}()
rec := httptest.NewRecorder()
if !proxy.serveResizedFromFile(rec, httptest.NewRequest("GET", "/x?maxWidth=480&maxHeight=600", nil), src, imageResizeOptions{MaxWidth: 480, MaxHeight: 600}) {
t.Fatal("expected the request to wait for the slot and serve a thumbnail")
}
body := rec.Body.Bytes()
if bytes.Equal(body, original) {
t.Fatal("served the multi-hundred-KB PNG original instead of a re-encoded thumbnail")
}
if rec.Header().Get("Content-Type") != "image/jpeg" {
t.Fatalf("content type = %q, want image/jpeg", rec.Header().Get("Content-Type"))
}
// 同一请求第二次必须直接命中缩放缓存。
rec2 := httptest.NewRecorder()
if !proxy.serveResizedFromFile(rec2, httptest.NewRequest("GET", "/x?maxWidth=480&maxHeight=600", nil), src, imageResizeOptions{MaxWidth: 480, MaxHeight: 600}) {
t.Fatal("expected the cached thumbnail to be reused")
}
if !bytes.Equal(rec2.Body.Bytes(), body) {
t.Fatal("expected the cached thumbnail bytes to be identical")
t.Fatal("expected a busy resize slot to fall back so the caller can serve the original")
}
}
-564
View File
@@ -1,564 +0,0 @@
package service
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"image"
"image/jpeg"
"io"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"golang.org/x/image/draw"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
)
// 媒体库封面拼图(Emby 风格的「封面集」)。
//
// Emby / Jellyfin 在没有管理员自定义封面时,会用库内几张海报拼成一张
// 2×2 的合成图作为库主图(folder image)。MeBox 早先只是挑评分最高的
// 那一张单图,于是每个库的封面看起来都像「随便选了一张」。这里按同样的
// 思路生成拼图,供 Emby 客户端、Web 和 App 共用同一份图片。
const (
// libraryCoverDefaultWidth / Height 是拼图输出的默认画布。
//
// 库条目的 PrimaryImageAspectRatio 报的是 16:9(见 emby_system.go),
// 客户端按这个比例摆卡片;生成同样比例的图可以避免客户端再裁一刀。
libraryCoverDefaultWidth = 480
libraryCoverDefaultHeight = 270
libraryCoverMinWidth = 96
libraryCoverMaxWidth = 1920
libraryCoverAspect = 16.0 / 9.0
// libraryCoverSourceLimit 是拼图实际取用的成员海报数上限。
//
// 封面是竖切分栏:三栏在 16:9 画布上每栏接近 3:5,既保留海报主体,又能
// 让一个库有多部作品可辨识;再多就会把海报裁成很窄的竖条。
libraryCoverSourceLimit = 3
// libraryCoverMaxPanels 是合成函数能接受的最多分栏数(更长的来源列表
// 只取前几张)。留 4 是给「按库类型调栏数」留的余量。
libraryCoverMaxPanels = 4
// libraryCoverCandidatePool 是挑选海报时的候选池大小。只看最近的一批
// 成员,避免为一个库封面扫全库;对「最新入库」的库来说这批就是它最好
// 的那几张。
libraryCoverCandidatePool = 200
libraryCoverQuality = 85
libraryCoverSubdir = "library-covers"
)
// libraryCoverSourceCache 缓存「库 → 成员海报来源」。Emby 客户端进入首页
// 时会对每个库各请求一次主图,把这个查询压到每库 10 分钟一次,避免几十个
// 200 行的列表查询同时打进来。
var libraryCoverSourceCache = struct {
mu sync.Mutex
items map[string]libraryCoverSourceEntry
}{items: make(map[string]libraryCoverSourceEntry)}
type libraryCoverSourceEntry struct {
sources []string
expiresAt time.Time
}
const libraryCoverSourceTTL = 10 * time.Minute
func cachedLibraryCoverSources(libraryID string) ([]string, bool) {
libraryCoverSourceCache.mu.Lock()
defer libraryCoverSourceCache.mu.Unlock()
entry, ok := libraryCoverSourceCache.items[libraryID]
if !ok || time.Now().After(entry.expiresAt) {
if ok {
delete(libraryCoverSourceCache.items, libraryID)
}
return nil, false
}
return entry.sources, true
}
func rememberLibraryCoverSources(libraryID string, sources []string) {
libraryCoverSourceCache.mu.Lock()
defer libraryCoverSourceCache.mu.Unlock()
if len(libraryCoverSourceCache.items) > 4000 {
libraryCoverSourceCache.items = make(map[string]libraryCoverSourceEntry)
}
libraryCoverSourceCache.items[libraryID] = libraryCoverSourceEntry{
sources: sources,
expiresAt: time.Now().Add(libraryCoverSourceTTL),
}
}
// LibraryCoverSources 返回媒体库封面拼图要用的海报来源,最多 limit 张。
//
// 选法与网页旧逻辑一致(库内最近一批成员,按 seriesArtworkScore 取质量
// 最高者),额外按剧集去重,避免四张海报全是同一部剧的不同集。
func LibraryCoverSources(ctx context.Context, repos *repository.Container, libraryID string, limit int) []string {
if repos == nil || repos.Media == nil || strings.TrimSpace(libraryID) == "" {
return nil
}
if limit <= 0 || limit > libraryCoverSourceLimit {
limit = libraryCoverSourceLimit
}
if cached, ok := cachedLibraryCoverSources(libraryID); ok {
if len(cached) > limit {
return cached[:limit]
}
return cached
}
rows, err := repos.Media.ListByLibrariesFilteredNoCount(ctx, []string{libraryID}, 0, libraryCoverCandidatePool,
repository.MediaQueryFilter{IncludeNSFW: true})
if err != nil || len(rows) == 0 {
return nil
}
type candidate struct {
url string
key string
score int
}
candidates := make([]candidate, 0, len(rows))
for i := range rows {
row := rows[i]
raw := strings.TrimSpace(row.PosterURL)
if raw == "" {
raw = strings.TrimSpace(row.BackdropURL)
}
if raw == "" {
continue
}
key := mediaSeriesKey(row)
if key == "" {
key = "id:" + row.ID
}
candidates = append(candidates, candidate{url: raw, key: key, score: seriesArtworkScore(row)})
}
// 评分高的优先;同分保持数据库顺序(最近的在前),封面更贴近「最新入库」。
sort.SliceStable(candidates, func(i, j int) bool { return candidates[i].score > candidates[j].score })
out := make([]string, 0, limit)
seenKey := make(map[string]struct{}, limit)
seenURL := make(map[string]struct{}, limit)
for _, item := range candidates {
if len(out) >= limit {
break
}
if _, ok := seenKey[item.key]; ok {
continue
}
if _, ok := seenURL[item.url]; ok {
continue
}
seenKey[item.key] = struct{}{}
seenURL[item.url] = struct{}{}
out = append(out, item.url)
}
if len(out) == 0 {
rememberLibraryCoverSources(libraryID, nil)
return nil
}
rememberLibraryCoverSources(libraryID, out)
return out
}
// LibraryCoverCacheDir 返回拼图缓存目录(与图片代理的 images/ 分开存放,
// 不会被图片缓存的配额清理误删)。
func LibraryCoverCacheDir(cfgCacheDir string) string {
base := strings.TrimSpace(cfgCacheDir)
if base == "" {
base = "./cache"
}
return filepath.Join(base, libraryCoverSubdir)
}
// LibraryCoverImage 生成(或命中缓存)媒体库封面拼图。
//
// width/height 是期望的输出画布;服务端会按 16:9 收敛,返回的图片不会
// 超过该边界。sources 由调用方提供(见 LibraryCoverSources),这样 Emby
// 路由和 Web API 可以共用同一套生成与缓存逻辑。
func (p *ImageProxy) LibraryCoverImage(ctx context.Context, libraryID string, sources []string, width, height int) ([]byte, error) {
if p == nil {
return nil, errors.New("image proxy unavailable")
}
libraryID = strings.TrimSpace(libraryID)
sources = normalizeCoverSources(sources)
if libraryID == "" || len(sources) == 0 {
return nil, errors.New("no library cover sources")
}
outW, outH := libraryCoverCanvas(width, height)
cachePath, sourceKey := p.libraryCoverCachePath(libraryID, sources, outW, outH)
if data, err := os.ReadFile(cachePath); err == nil && len(data) > 0 { // #nosec G304 -- cachePath is derived from a content hash.
return data, nil
}
key := fmt.Sprintf("library-cover:%s-%dx%d-%s", libraryID, outW, outH, sourceKey)
v, err, _ := p.fetchGroup.Do(key, func() (any, error) {
if data, err := os.ReadFile(cachePath); err == nil && len(data) > 0 { // #nosec G304 -- cachePath is derived from a content hash.
return data, nil
}
data, buildErr := p.buildLibraryCover(ctx, libraryID, sources, outW, outH, cachePath, sourceKey)
if buildErr != nil {
return nil, buildErr
}
return data, nil
})
if err != nil {
return nil, err
}
data, _ := v.([]byte)
if len(data) == 0 {
return nil, errors.New("empty library cover")
}
return data, nil
}
func (p *ImageProxy) buildLibraryCover(ctx context.Context, libraryID string, sources []string, width, height int, cachePath, sourceKey string) ([]byte, error) {
images := p.fetchCoverSources(ctx, sources)
if len(images) == 0 {
return nil, errors.New("no library cover image fetched")
}
// 解码与合成是 CPU 密集操作:和缩略图共用同一组解码槽,避免一次首页
// 刷新把 2 核主机打满。
release, ok := p.acquireResizeSlotWithin(ctx, p.resizeQueueTimeout())
if ok {
defer release()
}
data, err := composeLibraryCover(images, width, height)
if err != nil {
return nil, err
}
if err := os.MkdirAll(filepath.Dir(cachePath), 0o750); err != nil {
return data, nil
}
if err := writeFileAtomic(cachePath, data); err != nil {
p.warn("write library cover cache failed", err)
return data, nil
}
p.pruneLibraryCoverVariants(filepath.Dir(cachePath), libraryID, sourceKey)
return data, nil
}
// fetchCoverSources 并发拉取成员海报(受图片代理自身的缓存/鉴权/SSRF 规则
// 保护)。单张失败只丢弃该张,其余仍然可以拼图。
func (p *ImageProxy) fetchCoverSources(ctx context.Context, sources []string) [][]byte {
out := make([][]byte, len(sources))
var wg sync.WaitGroup
sem := make(chan struct{}, libraryCoverMaxPanels)
fetchCtx, cancel := context.WithTimeout(ctx, 15*time.Second)
defer cancel()
for i, raw := range sources {
i, raw := i, raw
wg.Add(1)
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
data, err := p.coverSourceBytes(fetchCtx, raw)
if err != nil || len(data) == 0 {
return
}
out[i] = data
}()
}
wg.Wait()
compacted := make([][]byte, 0, len(out))
for _, data := range out {
if len(data) > 0 {
compacted = append(compacted, data)
}
}
return compacted
}
// coverSourceBytes 读取一张海报来源。
//
// 刮削会把 sidecar 海报存成 /media/... 这类绝对路径,而 fetch 路径只接受
// http(s)(validateURL 要求有 host),直接丢给 Fetch 会全部失败。本地路径
// 按图片代理同一套 isAllowedLocalPath 白名单读盘,远程地址才走 Fetch,
// 这样鉴权、SSRF 校验与磁盘缓存的行为与 /api/img 完全一致。
func (p *ImageProxy) coverSourceBytes(ctx context.Context, raw string) ([]byte, error) {
raw = strings.TrimSpace(raw)
if raw == "" {
return nil, errors.New("empty cover source")
}
if isLocalImagePath(raw) {
abs, err := filepath.Abs(filepath.Clean(raw))
if err != nil {
return nil, err
}
if !p.isAllowedLocalPath(abs) {
return nil, errors.New("local cover path not allowed")
}
return os.ReadFile(abs) // #nosec G304 -- path validated against the library roots above.
}
data, _, err := p.Fetch(ctx, raw)
if err != nil {
return nil, err
}
return data, nil
}
// libraryCoverCachePath 生成缓存路径。
//
// 文件名里的 sourceKey 只由成员海报列表决定(不含尺寸):成员变化时换一批
// 文件,同一批成员的多个尺寸变体可以共存,不会被互相删掉。size 单独进文件名,
// 让 Emby(按客户端 maxWidth/maxHeight)与 Web/App(固定档位)各用自己的成品。
func (p *ImageProxy) libraryCoverCachePath(libraryID string, sources []string, width, height int) (string, string) {
h := sha256.New()
for _, src := range sources {
_, _ = io.WriteString(h, src)
_, _ = h.Write([]byte{0})
}
sourceKey := hex.EncodeToString(h.Sum(nil))[:16]
name := fmt.Sprintf("%s-%dx%d-%s.jpg", safeCoverFileID(libraryID), width, height, sourceKey)
return filepath.Join(LibraryCoverCacheDir(p.cacheDirBase()), name), sourceKey
}
func (p *ImageProxy) cacheDirBase() string {
if p != nil && p.cfg != nil {
return p.cfg.Cache.CacheDir
}
return ""
}
// pruneLibraryCoverVariants 清掉同一个库、旧成员列表留下的封面文件(含各
// 尺寸变体),当前成员列表对应的成品全部保留。
func (p *ImageProxy) pruneLibraryCoverVariants(dir, libraryID, keepSourceKey string) {
entries, err := os.ReadDir(dir)
if err != nil {
return
}
prefix := safeCoverFileID(libraryID) + "-"
for _, entry := range entries {
name := entry.Name()
if entry.IsDir() || !strings.HasPrefix(name, prefix) || !strings.HasSuffix(name, ".jpg") {
continue
}
if strings.Contains(name, "-"+keepSourceKey+".jpg") {
continue
}
_ = os.Remove(filepath.Join(dir, name)) // #nosec G304 -- name comes from ReadDir of our own cache dir.
}
}
// safeCoverFileID 让文件名只包含安全字符(库 ID 来自数据库,但仍做一次
// 白名单过滤,避免任何路径分隔符进入文件名)。
func safeCoverFileID(libraryID string) string {
var b strings.Builder
for _, r := range libraryID {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_':
b.WriteRune(r)
}
}
if b.Len() == 0 {
return "library"
}
return b.String()
}
func normalizeCoverSources(sources []string) []string {
seen := make(map[string]struct{}, len(sources))
out := make([]string, 0, len(sources))
for _, raw := range sources {
raw = strings.TrimSpace(raw)
if raw == "" {
continue
}
if _, ok := seen[raw]; ok {
continue
}
seen[raw] = struct{}{}
out = append(out, raw)
if len(out) >= libraryCoverMaxPanels {
break
}
}
return out
}
// libraryCoverCanvas 把请求的尺寸收敛到 16:9 画布,并夹在合理区间内。
func libraryCoverCanvas(width, height int) (int, int) {
if width <= 0 && height <= 0 {
return libraryCoverDefaultWidth, libraryCoverDefaultHeight
}
if width <= 0 {
width = int(float64(height)*libraryCoverAspect + 0.5)
}
if height <= 0 {
height = int(float64(width)/libraryCoverAspect + 0.5)
}
// 客户端给的是一个「不超过」的盒子:按 16:9 装进去。
if float64(width)/float64(height) > libraryCoverAspect {
width = int(float64(height)*libraryCoverAspect + 0.5)
} else {
height = int(float64(width)/libraryCoverAspect + 0.5)
}
if width < libraryCoverMinWidth {
width = libraryCoverMinWidth
height = int(float64(width)/libraryCoverAspect + 0.5)
}
if width > libraryCoverMaxWidth {
width = libraryCoverMaxWidth
height = int(float64(width)/libraryCoverAspect + 0.5)
}
return width, height
}
// composeLibraryCover 把若干张海报拼成库封面:竖切成若干等宽竖条。
//
// - 1 张:铺满整块画布;
// - 2 张:左右各一半(默认形态,对应网页/客户端卡片的分屏观感);
// - 3/4 张:三等分 / 四等分竖条。
//
// 每格按中心裁切铺满(crop-to-fill),这样不同比例的海报都能对齐分栏。
func composeLibraryCover(images [][]byte, width, height int) ([]byte, error) {
if len(images) == 0 {
return nil, errors.New("no images to compose")
}
if width <= 0 || height <= 0 {
return nil, errors.New("invalid cover size")
}
decoded := make([]image.Image, 0, len(images))
for _, data := range images {
img, ok := decodeCoverImage(data)
if ok {
decoded = append(decoded, img)
}
if len(decoded) >= libraryCoverMaxPanels {
break
}
}
if len(decoded) == 0 {
return nil, errors.New("no decodable cover images")
}
canvas := image.NewRGBA(image.Rect(0, 0, width, height))
cells := make([]image.Rectangle, 0, len(decoded))
if len(decoded) == 1 {
cells = append(cells, canvas.Bounds())
} else {
for i := range decoded {
x0 := width * i / len(decoded)
x1 := width * (i + 1) / len(decoded)
cells = append(cells, image.Rect(x0, 0, x1, height))
}
}
for i, cell := range cells {
drawCoverCell(canvas, cell, decoded[i])
}
var buf bytes.Buffer
if err := jpeg.Encode(&buf, canvas, &jpeg.Options{Quality: libraryCoverQuality}); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// decodeCoverImage 解码一张海报,并对异常大的图做像素上限保护:一次拼图
// 会同时解码 4 张,2 核 2G 的机器上不能任由超大原图把内存吃光。
func decodeCoverImage(data []byte) (image.Image, bool) {
if len(data) == 0 {
return nil, false
}
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 {
return nil, false
}
if cfg.Width*cfg.Height > imageResizeMaxSourcePixels {
return nil, false
}
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return nil, false
}
return img, true
}
// drawCoverCell 把一张图按中心裁切铺满目标格子。
func drawCoverCell(dst *image.RGBA, cell image.Rectangle, src image.Image) {
if src == nil || cell.Dx() <= 0 || cell.Dy() <= 0 {
return
}
bounds := src.Bounds()
srcW, srcH := bounds.Dx(), bounds.Dy()
if srcW <= 0 || srcH <= 0 {
return
}
targetAspect := float64(cell.Dx()) / float64(cell.Dy())
srcAspect := float64(srcW) / float64(srcH)
crop := bounds
if srcAspect > targetAspect {
cropW := int(float64(srcH)*targetAspect + 0.5)
if cropW < 1 {
cropW = 1
}
x0 := bounds.Min.X + (srcW-cropW)/2
crop = image.Rect(x0, bounds.Min.Y, x0+cropW, bounds.Max.Y)
} else if srcAspect < targetAspect {
cropH := int(float64(srcW)/targetAspect + 0.5)
if cropH < 1 {
cropH = 1
}
y0 := bounds.Min.Y + (srcH-cropH)/2
crop = image.Rect(bounds.Min.X, y0, bounds.Max.X, y0+cropH)
}
scale := float64(cell.Dx()) / float64(crop.Dx())
var scaler draw.Scaler = draw.CatmullRom
if scale < imageResizeCheapScaleRatio {
scaler = draw.ApproxBiLinear
}
scaler.Scale(dst, cell, src, crop, draw.Src, nil)
}
// writeFileAtomic 先写临时文件再 rename,避免并发的读取方读到半个文件。
func writeFileAtomic(path string, data []byte) error {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o750); err != nil {
return err
}
tmp, err := os.CreateTemp(dir, ".cover-*")
if err != nil {
return err
}
if _, err := tmp.Write(data); err != nil {
_ = tmp.Close()
_ = os.Remove(tmp.Name())
return err
}
if err := tmp.Close(); err != nil {
_ = os.Remove(tmp.Name())
return err
}
if err := os.Rename(tmp.Name(), path); err != nil {
_ = os.Remove(tmp.Name())
return err
}
return nil
}
// LibraryCoverSourcesForLibrary 是给 handler 的便捷封装:库存在且没有自定
// 义封面时返回成员海报来源;远程挂载库由各自的上游负责封面,不在此生成。
func LibraryCoverSourcesForLibrary(ctx context.Context, repos *repository.Container, libraryID string) ([]string, *model.Library) {
if repos == nil || repos.Library == nil {
return nil, nil
}
libraryID = strings.TrimSpace(libraryID)
if libraryID == "" {
return nil, nil
}
lib, err := repos.Library.FindByID(ctx, libraryID)
if err != nil || lib == nil {
return nil, nil
}
if strings.TrimSpace(lib.CoverURL) != "" {
return nil, lib
}
return LibraryCoverSources(ctx, repos, libraryID, libraryCoverSourceLimit), lib
}
-335
View File
@@ -1,335 +0,0 @@
package service
import (
"bytes"
"image"
"image/color"
"image/jpeg"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"sync/atomic"
"testing"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
)
func solidJPEG(t *testing.T, c color.Color, w, h int) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, w, h))
for y := 0; y < h; y++ {
for x := 0; x < w; x++ {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 95}); err != nil {
t.Fatalf("encode jpeg: %v", err)
}
return buf.Bytes()
}
func decodeJPEG(t *testing.T, data []byte) image.Image {
t.Helper()
img, err := jpeg.Decode(bytes.NewReader(data))
if err != nil {
t.Fatalf("decode composed cover: %v", err)
}
return img
}
// TestComposeLibraryCoverSplitsIntoSideBySideColumns 保证多张海报是竖切成
// 左右分栏(左→右顺序),而不是 2×2 田字格。
func TestComposeLibraryCoverSplitsIntoSideBySideColumns(t *testing.T) {
images := [][]byte{
solidJPEG(t, color.RGBA{R: 220, A: 255}, 200, 300),
solidJPEG(t, color.RGBA{G: 220, A: 255}, 200, 300),
solidJPEG(t, color.RGBA{B: 220, A: 255}, 200, 300),
solidJPEG(t, color.RGBA{R: 220, G: 220, A: 255}, 200, 300),
}
out, err := composeLibraryCover(images, 480, 270)
if err != nil {
t.Fatalf("compose: %v", err)
}
img := decodeJPEG(t, out)
if img.Bounds().Dx() != 480 || img.Bounds().Dy() != 270 {
t.Fatalf("size = %v, want 480x270", img.Bounds())
}
// 四张图 → 四条竖栏;同一栏从顶部到底部颜色一致(竖切,不是上下分割)。
y := img.Bounds().Dy() / 2
checks := []struct {
name string
x int
dominant func(color.RGBA) bool
}{
{"column 1 red", 60, func(c color.RGBA) bool { return c.R > c.G+40 && c.R > c.B+40 }},
{"column 2 green", 180, func(c color.RGBA) bool { return c.G > c.R+40 && c.G > c.B+40 }},
{"column 3 blue", 300, func(c color.RGBA) bool { return c.B > c.R+40 && c.B > c.G+40 }},
{"column 4 yellow", 420, func(c color.RGBA) bool { return c.R > c.B+40 && c.G > c.B+40 }},
}
for _, check := range checks {
for _, yy := range []int{2, y, img.Bounds().Dy() - 3} {
r, g, b, _ := img.At(check.x, yy).RGBA()
got := color.RGBA{R: uint8(r >> 8), G: uint8(g >> 8), B: uint8(b >> 8), A: 255}
if !check.dominant(got) {
t.Fatalf("%s at y=%d: pixel = %v", check.name, yy, got)
}
}
}
}
// TestComposeLibraryCoverHandlesFewerPosters 覆盖 1/2/3 张成员的布局。
func TestComposeLibraryCoverHandlesFewerPosters(t *testing.T) {
cases := []struct {
name string
count int
}{
{"single", 1},
{"pair", 2},
{"triple", 3},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
images := make([][]byte, 0, tc.count)
for i := 0; i < tc.count; i++ {
images = append(images, solidJPEG(t, color.RGBA{R: 200, G: 120, A: 255}, 300, 450))
}
out, err := composeLibraryCover(images, 320, 180)
if err != nil {
t.Fatalf("compose: %v", err)
}
img := decodeJPEG(t, out)
if img.Bounds().Dx() != 320 || img.Bounds().Dy() != 180 {
t.Fatalf("size = %v, want 320x180", img.Bounds())
}
})
}
}
// TestComposeLibraryCoverSkipsUndecodableSources 保证一张坏图不会让整张
// 封面失败:其余可解码的成员仍然会拼出来。
func TestComposeLibraryCoverSkipsUndecodableSources(t *testing.T) {
images := [][]byte{
[]byte("not an image"),
solidJPEG(t, color.RGBA{G: 200, A: 255}, 200, 300),
}
out, err := composeLibraryCover(images, 240, 135)
if err != nil {
t.Fatalf("compose: %v", err)
}
if len(out) == 0 {
t.Fatal("expected composed cover bytes")
}
if _, err := composeLibraryCover([][]byte{[]byte("nope")}, 240, 135); err == nil {
t.Fatal("expected error when no source is decodable")
}
}
func TestLibraryCoverCanvasKeepsSixteenByNine(t *testing.T) {
cases := []struct {
inW, inH int
wantW int
wantH int
checkRatio bool
}{
{0, 0, 480, 270, true},
{640, 480, 640, 360, true}, // 4:3 盒子 → 装进 16:9
{2000, 0, 1920, 1080, true}, // 超上限 → 夹紧
{40, 0, 96, 54, true}, // 低于下限 → 抬到最小宽度
{0, 540, 960, 540, false}, // 只给高度 → 按 16:9 推宽度
}
for _, tc := range cases {
w, h := libraryCoverCanvas(tc.inW, tc.inH)
if w != tc.wantW || h != tc.wantH {
t.Fatalf("libraryCoverCanvas(%d,%d) = %dx%d, want %dx%d", tc.inW, tc.inH, w, h, tc.wantW, tc.wantH)
}
if tc.checkRatio && w != 0 && h != 0 {
ratio := float64(w) / float64(h)
if ratio < 1.7 || ratio > 1.85 {
t.Fatalf("ratio for %dx%d = %.3f, want ~1.78", tc.inW, tc.inH, ratio)
}
}
}
}
// TestLibraryCoverSourcesDedupesSeries 保证封面不会两张都是同一部剧的不同集,
// 并且优先取海报、没有海报时退回剧照、完全没有图的成员被跳过。
func TestLibraryCoverSourcesDedupesSeries(t *testing.T) {
svc := newTestEmbyService(t)
lib := model.Library{Name: "剧集", Path: "/media/tv", Type: "tv", Enabled: true}
if err := svc.repo.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
rows := []model.Media{
{Base: model.Base{ID: "a1"}, LibraryID: lib.ID, Title: "A", Path: "/media/tv/A/S01E01.mkv", SeriesID: "series-a", PosterURL: "https://img/a1.jpg"},
{Base: model.Base{ID: "a2"}, LibraryID: lib.ID, Title: "A", Path: "/media/tv/A/S01E02.mkv", SeriesID: "series-a", PosterURL: "https://img/a2.jpg"},
{Base: model.Base{ID: "b1"}, LibraryID: lib.ID, Title: "B", Path: "/media/tv/B/S01E01.mkv", SeriesID: "series-b", PosterURL: "https://img/b1.jpg"},
{Base: model.Base{ID: "c1"}, LibraryID: lib.ID, Title: "C", Path: "/media/tv/C/S01E01.mkv", SeriesID: "series-c", BackdropURL: "https://img/c1-backdrop.jpg"},
{Base: model.Base{ID: "d1"}, LibraryID: lib.ID, Title: "D", Path: "/media/tv/D/S01E01.mkv", SeriesID: "series-d"},
}
for i := range rows {
if err := svc.repo.DB.Create(&rows[i]).Error; err != nil {
t.Fatalf("create media: %v", err)
}
}
// 竖切三栏:一部剧只出一张,无图成员被跳过 → 正好 3 张。
sources := LibraryCoverSources(t.Context(), svc.repo, lib.ID, libraryCoverSourceLimit)
if len(sources) != 3 {
t.Fatalf("sources = %v, want 3 (一部剧一张、跳过无图成员)", sources)
}
seen := map[string]bool{}
for _, src := range sources {
if seen[src] {
t.Fatalf("duplicate source in %v", sources)
}
seen[src] = true
}
// 同一部剧只会出现一次:具体保留哪一集由查询顺序决定,但绝不能两张。
if seen["https://img/a1.jpg"] && seen["https://img/a2.jpg"] {
t.Fatalf("same series contributed two posters: %v", sources)
}
if !seen["https://img/a1.jpg"] && !seen["https://img/a2.jpg"] {
t.Fatalf("series A is missing from sources: %v", sources)
}
if !seen["https://img/c1-backdrop.jpg"] {
t.Fatalf("member without poster should fall back to backdrop: %v", sources)
}
}
// TestLibraryCoverImageCachesComposedBytes 验证生成结果会落盘复用:同一份
// 内容第二次请求不再回源。
func TestLibraryCoverImageCachesComposedBytes(t *testing.T) {
var hits int32
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.Header().Set("Content-Type", "image/jpeg")
_, _ = w.Write(solidJPEG(t, color.RGBA{R: 180, G: 140, B: 90, A: 255}, 240, 360))
}))
defer upstream.Close()
parsed, err := url.Parse(upstream.URL)
if err != nil {
t.Fatalf("parse upstream: %v", err)
}
cacheCfg := &config.Config{Cache: config.CacheConfig{CacheDir: filepath.Join(t.TempDir(), "cache")}}
proxy := NewImageProxy(cacheCfg, zap.NewNop())
proxy.SetAllowedRemoteHostsProvider(func() []string { return []string{parsed.Host} })
sources := []string{upstream.URL + "/1.jpg", upstream.URL + "/2.jpg", upstream.URL + "/3.jpg", upstream.URL + "/4.jpg"}
first, err := proxy.LibraryCoverImage(t.Context(), "lib-1", sources, 480, 270)
if err != nil {
t.Fatalf("first cover: %v", err)
}
if img := decodeJPEG(t, first); img.Bounds().Dx() != 480 || img.Bounds().Dy() != 270 {
t.Fatalf("size = %v, want 480x270", img.Bounds())
}
hitsAfterFirst := atomic.LoadInt32(&hits)
if hitsAfterFirst != 4 {
t.Fatalf("upstream hits = %d, want one per source", hitsAfterFirst)
}
second, err := proxy.LibraryCoverImage(t.Context(), "lib-1", sources, 480, 270)
if err != nil {
t.Fatalf("second cover: %v", err)
}
if !bytes.Equal(first, second) {
t.Fatal("cached cover bytes differ from the first build")
}
if got := atomic.LoadInt32(&hits); got != hitsAfterFirst {
t.Fatalf("cache miss: upstream hits %d -> %d", hitsAfterFirst, got)
}
// 内容变了要重新生成(成员海报换了新文件)。已缓存的三张海报不会重下,
// 只有新增的那张会回源。
changed := append([]string{}, sources[:3]...)
changed = append(changed, upstream.URL+"/5.jpg")
if _, err := proxy.LibraryCoverImage(t.Context(), "lib-1", changed, 480, 270); err != nil {
t.Fatalf("cover after content change: %v", err)
}
if got := atomic.LoadInt32(&hits); got != hitsAfterFirst+1 {
t.Fatalf("content change should rebuild with one new fetch, hits = %d", got)
}
// 同一批成员的另一个尺寸档位要能共存:Web/App 与 Emby 请求的尺寸不同,
// 不能互相把对方的成品当「旧内容」删掉。
if _, err := proxy.LibraryCoverImage(t.Context(), "lib-1", changed, 400, 225); err != nil {
t.Fatalf("other size cover: %v", err)
}
entries, err := os.ReadDir(LibraryCoverCacheDir(cacheCfg.Cache.CacheDir))
if err != nil {
t.Fatalf("read cover cache dir: %v", err)
}
files := make([]string, 0, len(entries))
for _, entry := range entries {
if !entry.IsDir() {
files = append(files, entry.Name())
}
}
if len(files) != 2 {
t.Fatalf("expected 480x270 and 400x225 variants to coexist, got %v", files)
}
}
// TestLibraryCoverSourceBytesReadsLocalSidecarPoster 覆盖刮削把海报存成
// /media/... 本地绝对路径的情况:这类来源没有 http(s) host,不能走 Fetch,
// 必须按库根目录白名单直接读盘,否则整张拼图会因为「没有可解码的海报」而
// 生成失败(生产环境所有本地库都是这种来源)。
func TestLibraryCoverSourceBytesReadsLocalSidecarPoster(t *testing.T) {
libDir := t.TempDir()
posterPath := filepath.Join(libDir, "poster.jpg")
if err := os.WriteFile(posterPath, solidJPEG(t, color.RGBA{R: 180, G: 90, A: 255}, 200, 300), 0o644); err != nil {
t.Fatalf("write poster: %v", err)
}
proxy := NewImageProxy(&config.Config{Cache: config.CacheConfig{CacheDir: filepath.Join(t.TempDir(), "cache")}}, zap.NewNop())
// 未登记为库根目录时必须拒绝读取。
if _, err := proxy.coverSourceBytes(t.Context(), posterPath); err == nil {
t.Fatal("expected local path outside the library roots to be rejected")
}
proxy.SetLibraryRootsProvider(func() []string { return []string{libDir} })
data, err := proxy.coverSourceBytes(t.Context(), posterPath)
if err != nil {
t.Fatalf("read local poster: %v", err)
}
if len(data) == 0 {
t.Fatal("expected poster bytes")
}
if _, ok := decodeCoverImage(data); !ok {
t.Fatal("local poster should decode")
}
}
// TestLibraryCoverSourcesForLibrarySkipsCustomCover 保证管理员填了自定义
// 封面时不会再生成拼图(沿用自定义图)。
func TestLibraryCoverSourcesForLibrarySkipsCustomCover(t *testing.T) {
svc := newTestEmbyService(t)
lib := model.Library{Name: "电影", Path: "/media/movies", Type: "movie", CoverURL: "https://example.com/custom.jpg"}
if err := svc.repo.Library.Create(t.Context(), &lib); err != nil {
t.Fatalf("create library: %v", err)
}
if err := svc.repo.DB.Create(&model.Media{
Base: model.Base{ID: "m1"},
LibraryID: lib.ID,
Title: "M",
Path: "/media/movies/m1.mkv",
PosterURL: "https://img/m1.jpg",
}).Error; err != nil {
t.Fatalf("create media: %v", err)
}
sources, found := LibraryCoverSourcesForLibrary(t.Context(), svc.repo, lib.ID)
if found == nil || found.CoverURL != lib.CoverURL {
t.Fatalf("library = %#v, want the existing library", found)
}
if len(sources) != 0 {
t.Fatalf("expected no generated sources with a custom cover, got %v", sources)
}
if _, blank := LibraryCoverSourcesForLibrary(t.Context(), svc.repo, "missing"); blank != nil {
t.Fatalf("unknown library should return nil, got %#v", blank)
}
}
+7 -8
View File
@@ -12,14 +12,13 @@ import (
// MediaService offers high-level CRUD over libraries and media items.
type MediaService struct {
cfg *config.Config
log *zap.Logger
repo *repository.Container
cache *RuntimeCacheService
groupedMediaFlight singleflight.Group
mediaSortFlight singleflight.Group
libraryRowsFlight singleflight.Group
recentSeriesFlight singleflight.Group
cfg *config.Config
log *zap.Logger
repo *repository.Container
cache *RuntimeCacheService
groupedMediaFlight singleflight.Group
libraryRowsFlight singleflight.Group
recentSeriesFlight singleflight.Group
}
type MediaVisibility struct {
+16 -31
View File
@@ -5,7 +5,7 @@ import (
"strings"
"time"
"github.com/google/uuid"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
@@ -129,26 +129,6 @@ func (s *MediaService) GroupedMediaVisibleFiltered(
visibility MediaVisibility,
filters MediaListFilters,
) ([]MediaItem, error) {
value, err := s.groupedMediaVisibleFiltered(ctx, libraryID, visibility, filters)
if err != nil {
return nil, err
}
return value.Items, nil
}
// Each immutable grouping has its own revision. Lightweight sort indices can
// reference that revision without retaining another copy of the full rows.
type groupedMediaCacheValue struct {
Items []MediaItem
Revision string
}
func (s *MediaService) groupedMediaVisibleFiltered(
ctx context.Context,
libraryID string,
visibility MediaVisibility,
filters MediaListFilters,
) (*groupedMediaCacheValue, error) {
visibility = ExpandMediaVisibilityForMergedCloudLibraries(ctx, s.repo, visibility)
libraryIDs, err := MergedLibraryIDsForLibrary(ctx, s.repo, libraryID)
if err != nil {
@@ -163,7 +143,7 @@ func (s *MediaService) groupedMediaVisibleFiltered(
value, err, _ := s.groupedMediaFlight.Do(itemsCacheKey, func() (any, error) {
if s.cache != nil {
if cachedObj, ok := s.cache.GetObject(itemsCacheKey); ok {
if cached, ok := cachedObj.(*groupedMediaCacheValue); ok {
if cached, ok := cachedObj.([]MediaItem); ok {
return cached, nil
}
}
@@ -174,8 +154,8 @@ func (s *MediaService) groupedMediaVisibleFiltered(
if err != nil {
return nil, err
}
grouped := &groupedMediaCacheValue{Items: groupMediaVersions(items), Revision: uuid.NewString()}
if s.cache != nil && len(grouped.Items) > 0 {
grouped := groupMediaVersions(items)
if s.cache != nil && len(grouped) > 0 {
s.cache.SetObject(itemsCacheKey, grouped, s.mediaObjectTTL())
}
return grouped, nil
@@ -183,10 +163,10 @@ func (s *MediaService) groupedMediaVisibleFiltered(
if err != nil {
return nil, err
}
if grouped, ok := value.(*groupedMediaCacheValue); ok {
if grouped, ok := value.([]MediaItem); ok {
return grouped, nil
}
return &groupedMediaCacheValue{}, nil
return nil, nil
}
func (s *MediaService) listMediaVisibleForGrouping(ctx context.Context, libraryID string, visibility MediaVisibility, filters MediaListFilters) ([]model.Media, error) {
@@ -200,19 +180,24 @@ func (s *MediaService) listMediaVisibleForGrouping(ctx context.Context, libraryI
AllowedLibraryIDs: visibility.AllowedLibraryIDs,
HiddenLibraryIDs: visibility.HiddenLibraryIDs,
})
// Group before paginating so versions stay together. The search safety limit
// must not truncate a library listing or its reported grouped total.
items, err := s.repo.Media.ListAllByLibrariesFilteredNoCount(ctx, libraryIDs, filter)
// 版本分组的 URL 分页发生在 Go 进程内,响应里的 total 是分组后的数量,
// 不需要数据库再为原始行做一次 COUNT(*)。全量 COUNT 在超大媒体库上
// 会重复扫描整个 library_id 范围,而这里只关心是否存在截断风险。
items, err := s.repo.Media.ListByLibrariesFilteredNoCount(ctx, libraryIDs, 0, maxMediaSearchLimit, filter)
if err != nil {
return nil, err
}
if len(items) >= maxMediaSearchLimit && s.log != nil {
s.log.Warn("media version grouping may be truncated by safety limit",
zap.String("library_id", libraryID),
zap.Int("limit", maxMediaSearchLimit))
}
s.attachLibraryMetadata(ctx, items)
return items, nil
}
// GetMedia returns a single media row.
func (s *MediaService) GetMedia(ctx context.Context, id string) (*model.Media, error) {
media, err := s.repo.Media.FindByID(ctx, id)
func (s *MediaService) GetMedia(ctx context.Context, id string) (*model.Media, error) { media, err := s.repo.Media.FindByID(ctx, id)
if err != nil || media == nil {
return media, err
}
-66
View File
@@ -2,71 +2,17 @@ package service
import (
"context"
"fmt"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
)
type MediaQualityItem struct {
Issue string `json:"issue"`
Media model.Media `json:"media"`
}
func (s *MediaService) QualityIssues(ctx context.Context, issue string, page, pageSize int) ([]MediaQualityItem, int64, error) {
if page < 1 {
page = 1
}
if pageSize < 1 {
pageSize = 50
}
if pageSize > 100 {
pageSize = 100
}
q := s.repo.DB.WithContext(ctx).Model(&model.Media{})
switch issue {
case "missing":
q = q.Where("availability_state = ?", "missing")
case "unmatched":
q = q.Where("scrape_status IN ? OR scrape_status = ''", []string{"pending", "no_match"})
case "poster":
q = q.Where("poster_url IS NULL OR TRIM(poster_url) = ''")
default:
return nil, 0, fmt.Errorf("unknown quality issue: %s", issue)
}
var total int64
if err := q.Count(&total).Error; err != nil {
return nil, 0, err
}
var rows []model.Media
if err := q.Order("updated_at DESC, id ASC").Offset((page - 1) * pageSize).Limit(pageSize).Find(&rows).Error; err != nil {
return nil, 0, err
}
s.attachLibraryMetadata(ctx, rows)
items := make([]MediaQualityItem, 0, len(rows))
for _, media := range rows {
items = append(items, MediaQualityItem{Issue: issue, Media: media})
}
return items, total, nil
}
type MediaSearchFilter struct {
LibraryID string
YearMin int
YearMax int
Genres []string
}
// SearchMedia performs a simple LIKE search across titles.
func (s *MediaService) SearchMedia(ctx context.Context, query string, limit int) ([]model.Media, error) {
return s.SearchMediaVisible(ctx, query, limit, MediaVisibility{IncludeNSFW: true})
}
func (s *MediaService) SearchMediaVisible(ctx context.Context, query string, limit int, visibility MediaVisibility) ([]model.Media, error) {
return s.SearchMediaVisibleFiltered(ctx, query, limit, visibility, MediaSearchFilter{})
}
func (s *MediaService) SearchMediaVisibleFiltered(ctx context.Context, query string, limit int, visibility MediaVisibility, filter MediaSearchFilter) ([]model.Media, error) {
if limit <= 0 {
limit = 50
} else if limit > maxMediaSearchLimit {
@@ -77,10 +23,6 @@ func (s *MediaService) SearchMediaVisibleFiltered(ctx context.Context, query str
IncludeNSFW: visibility.IncludeNSFW,
AllowedLibraryIDs: visibility.AllowedLibraryIDs,
HiddenLibraryIDs: visibility.HiddenLibraryIDs,
LibraryID: filter.LibraryID,
YearMin: filter.YearMin,
YearMax: filter.YearMax,
Genres: filter.Genres,
})
if err != nil {
return nil, err
@@ -103,10 +45,6 @@ func (s *MediaService) SearchMediaVisibleGrouped(ctx context.Context, query stri
}
func (s *MediaService) SearchMediaVisiblePage(ctx context.Context, query string, page, pageSize int, visibility MediaVisibility) ([]model.Media, int64, error) {
return s.SearchMediaVisiblePageFiltered(ctx, query, page, pageSize, visibility, MediaSearchFilter{})
}
func (s *MediaService) SearchMediaVisiblePageFiltered(ctx context.Context, query string, page, pageSize int, visibility MediaVisibility, filter MediaSearchFilter) ([]model.Media, int64, error) {
if pageSize <= 0 {
pageSize = 50
}
@@ -121,10 +59,6 @@ func (s *MediaService) SearchMediaVisiblePageFiltered(ctx context.Context, query
IncludeNSFW: visibility.IncludeNSFW,
AllowedLibraryIDs: visibility.AllowedLibraryIDs,
HiddenLibraryIDs: visibility.HiddenLibraryIDs,
LibraryID: filter.LibraryID,
YearMin: filter.YearMin,
YearMax: filter.YearMax,
Genres: filter.Genres,
})
if err != nil {
return nil, 0, err
-77
View File
@@ -1,77 +0,0 @@
package service
import (
"context"
"fmt"
"sort"
"time"
)
// ListMediaVisibleSortedPage reuses grouped rows and a small position index.
// Only the requested page is copied; pagination does not sort the full library
// again or retain a second full-size MediaItem slice for every sort order.
func (s *MediaService) ListMediaVisibleSortedPage(
ctx context.Context,
libraryID string,
page, pageSize int,
visibility MediaVisibility,
filters MediaListFilters,
spec MediaSortSpec,
history map[string]time.Time,
) ([]MediaItem, int64, error) {
grouped, err := s.groupedMediaVisibleFiltered(ctx, libraryID, visibility, filters)
if err != nil {
return nil, 0, err
}
spec = NormalizeMediaSort(spec.Field, spec.Order)
indices := s.sortedMediaIndices(grouped, spec, history)
page, pageSize = normalizeGroupedMediaPage(page, pageSize)
total := len(grouped.Items)
// Division avoids overflowing (page-1)*pageSize on an excessive page value.
if total == 0 || page-1 > (total-1)/pageSize {
return []MediaItem{}, int64(total), nil
}
start := (page - 1) * pageSize
end := min(start+pageSize, total)
items := make([]MediaItem, 0, end-start)
for _, index := range indices[start:end] {
items = append(items, grouped.Items[index])
}
return items, int64(total), nil
}
func (s *MediaService) sortedMediaIndices(grouped *groupedMediaCacheValue, spec MediaSortSpec, history map[string]time.Time) []int {
// Playback history changes independently of the media cache and is personal.
// Keep this explicit sort live rather than caching another user's history.
if s.cache == nil || spec.Field == "last_played" || len(grouped.Items) == 0 {
return sortMediaIndices(grouped.Items, spec, history)
}
key := fmt.Sprintf("media:obj:sort:%s:%s:%s", grouped.Revision, spec.Field, spec.Order)
value, _, _ := s.mediaSortFlight.Do(key, func() (any, error) {
if cached, ok := s.cache.GetObject(key); ok {
if indices, ok := cached.([]int); ok {
return indices, nil
}
}
indices := sortMediaIndices(grouped.Items, spec, history)
s.cache.SetObject(key, indices, s.mediaObjectTTL())
return indices, nil
})
return value.([]int)
}
func sortMediaIndices(items []MediaItem, spec MediaSortSpec, history map[string]time.Time) []int {
indices := make([]int, len(items))
for i := range indices {
indices[i] = i
}
sort.SliceStable(indices, func(i, j int) bool {
a, b := items[indices[i]], items[indices[j]]
cmp := compareMediaByField(a, b, spec, history)
if cmp == 0 {
cmp = compareID(a.ID, b.ID)
}
return cmp < 0
})
return indices
}
-141
View File
@@ -1,141 +0,0 @@
package service
import (
"fmt"
"math"
"testing"
"time"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
)
func TestSortedMediaPageIncludesRowsBeyondSearchLimit(t *testing.T) {
db := newServiceTestDB(t)
repos := repository.New(db)
lib := model.Library{Name: "Large", Path: "/media/large", Type: "movie", Enabled: true}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatal(err)
}
// This must cross the historical 50,000-row boundary: a smaller fixture
// cannot detect truncation before grouping, counting, and sorting.
now := time.Now()
for start := 0; start <= maxMediaSearchLimit; start += 500 {
var rows []model.Media
for i := start; i < min(start+500, maxMediaSearchLimit+1); i++ {
rows = append(rows, model.Media{
Base: model.Base{ID: fmt.Sprintf("large-%05d", i), CreatedAt: now, UpdatedAt: now},
LibraryID: lib.ID, Path: fmt.Sprintf("/media/large/%05d.mkv", i),
Title: fmt.Sprintf("Movie %05d", i), TMDbID: i + 1,
})
}
if err := db.CreateInBatches(&rows, 100).Error; err != nil {
t.Fatal(err)
}
}
svc := NewMediaService(&config.Config{}, zap.NewNop(), repos)
items, total, err := svc.ListMediaVisibleSortedPage(t.Context(), lib.ID, maxMediaSearchLimit/100+1, 100,
MediaVisibility{IncludeNSFW: true}, MediaListFilters{}, MediaSortSpec{Field: "title", Order: "asc"}, nil)
if err != nil {
t.Fatal(err)
}
if total != int64(maxMediaSearchLimit+1) || len(items) != 1 || items[0].ID != "large-50000" {
t.Fatalf("total=%d, page length=%d, want 50001 rows and final item large-50000", total, len(items))
}
}
func TestSortedMediaPagesReuseIndicesAndRefreshAfterInvalidation(t *testing.T) {
db := newServiceTestDB(t)
repos := repository.New(db)
lib := model.Library{Name: "Movies", Path: "/media/movies", Type: "movie", Enabled: true}
if err := repos.Library.Create(t.Context(), &lib); err != nil {
t.Fatal(err)
}
rows := []model.Media{
{Base: model.Base{ID: "alpha"}, LibraryID: lib.ID, Path: "/media/movies/a.mkv", Title: "Alpha", TMDbID: 1, SizeBytes: 100},
{Base: model.Base{ID: "alpha-alt"}, LibraryID: lib.ID, Path: "/media/movies/a-alt.mkv", Title: "Alpha", TMDbID: 1, SizeBytes: 50},
{Base: model.Base{ID: "beta"}, LibraryID: lib.ID, Path: "/media/movies/b.mkv", Title: "Beta", TMDbID: 2},
{Base: model.Base{ID: "hidden"}, LibraryID: lib.ID, Path: "/media/movies/hidden.mkv", Title: "Hidden", TMDbID: 3, NSFW: true},
}
if err := db.Create(&rows).Error; err != nil {
t.Fatal(err)
}
svc := NewMediaService(&config.Config{}, zap.NewNop(), repos).
SetRuntimeCache(NewRuntimeCacheService(&config.Config{}, zap.NewNop()))
visibility := MediaVisibility{IncludeNSFW: false}
spec := MediaSortSpec{Field: "title", Order: "asc"}
first, total, err := svc.ListMediaVisibleSortedPage(t.Context(), lib.ID, 1, 1, visibility, MediaListFilters{}, spec, nil)
if err != nil || total != 2 || len(first) != 1 || first[0].ID != "alpha" || len(first[0].Versions) != 2 {
t.Fatalf("first=%#v, total=%d, error=%v", first, total, err)
}
grouped, err := svc.groupedMediaVisibleFiltered(t.Context(), lib.ID, visibility, MediaListFilters{})
if err != nil {
t.Fatal(err)
}
index := svc.sortedMediaIndices(grouped, spec, nil)
second, _, err := svc.ListMediaVisibleSortedPage(t.Context(), lib.ID, 2, 1, visibility, MediaListFilters{}, spec, nil)
if err != nil || len(second) != 1 || second[0].ID != "beta" {
t.Fatalf("second=%#v, error=%v", second, err)
}
cachedIndex := svc.sortedMediaIndices(grouped, spec, nil)
if &index[0] != &cachedIndex[0] {
t.Fatal("pagination rebuilt the sorted position index")
}
// The returned page is independently writable without corrupting cached rows.
first[0].Title = "Modified by caller"
if err := db.Model(&model.Media{}).Where("id = ?", "beta").Update("title", "Aardvark").Error; err != nil {
t.Fatal(err)
}
svc.invalidateMediaCache(t.Context())
fresh, _, err := svc.ListMediaVisibleSortedPage(t.Context(), lib.ID, 1, 1, visibility, MediaListFilters{}, spec, nil)
if err != nil || len(fresh) != 1 || fresh[0].ID != "beta" {
t.Fatalf("updated page=%#v, error=%v", fresh, err)
}
empty, _, err := svc.ListMediaVisibleSortedPage(t.Context(), lib.ID, math.MaxInt, 100, visibility, MediaListFilters{}, spec, nil)
if err != nil || empty == nil || len(empty) != 0 {
t.Fatalf("out-of-range page=%#v, error=%v", empty, err)
}
}
func TestMediaSortIndexDoesNotCachePersonalPlaybackHistory(t *testing.T) {
svc := NewMediaService(&config.Config{}, zap.NewNop(), nil).
SetRuntimeCache(NewRuntimeCacheService(&config.Config{}, zap.NewNop()))
grouped := &groupedMediaCacheValue{Revision: "history-test", Items: []MediaItem{
{Media: model.Media{Base: model.Base{ID: "a"}}},
{Media: model.Media{Base: model.Base{ID: "b"}}},
}}
spec := MediaSortSpec{Field: "last_played", Order: "desc"}
first := svc.sortedMediaIndices(grouped, spec, map[string]time.Time{"a": time.Now()})
second := svc.sortedMediaIndices(grouped, spec, map[string]time.Time{"b": time.Now()})
if first[0] != 0 || second[0] != 1 {
t.Fatalf("history sorts = %v, %v", first, second)
}
}
func BenchmarkMediaSortIndex(b *testing.B) {
items := make([]MediaItem, 10000)
for i := range items {
items[i].ID = fmt.Sprint(i)
items[i].Title = fmt.Sprintf("Movie %05d", len(items)-i)
}
spec := MediaSortSpec{Field: "title", Order: "asc"}
b.Run("sort-every-page", func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
SortMediaItems(items, spec.Field, spec.Order, nil)
}
})
b.Run("cached-position-index", func(b *testing.B) {
svc := NewMediaService(&config.Config{}, zap.NewNop(), nil).
SetRuntimeCache(NewRuntimeCacheService(&config.Config{}, zap.NewNop()))
grouped := &groupedMediaCacheValue{Items: items, Revision: "benchmark"}
svc.sortedMediaIndices(grouped, spec, nil)
b.ReportAllocs()
for b.Loop() {
svc.sortedMediaIndices(grouped, spec, nil)
}
})
}
+7
View File
@@ -126,6 +126,13 @@ func ApplyRuntimeSetting(cfg *config.Config, key, value string) {
}
cfg.Cache.ImagesOriginalsMaxSizeMB = n
}
case "cache.images_originals_ttl_hours":
if n, err := strconv.Atoi(value); err == nil {
if n < 0 {
n = 0
}
cfg.Cache.ImagesOriginalsTTLHours = n
}
case "cache.memory_max_size_mb":
n := config.DefaultCacheMemoryMaxSizeMB
if value != "" {
+26 -29
View File
@@ -14,7 +14,6 @@ import (
"errors"
"strings"
"sync"
"time"
"go.uber.org/zap"
@@ -145,34 +144,32 @@ type localMediaProbeTask struct {
}
type existingLocalMedia struct {
AvailabilityState string
MissingSince *time.Time
LibraryRootID string
RelativePath string
Title string
OriginalName string
EpisodeTitle string
SizeBytes int64
DurationSec int
Width int
Height int
VideoCodec string
AudioCodec string
Container string
STRMURL string
FileID string
PosterURL string
BackdropURL string
Overview string
Year int
ReleaseDate string
Rating float32
TMDbID int
BangumiID int
DoubanID string
TheTVDBID string
SeasonNum int
EpisodeNum int
LibraryRootID string
RelativePath string
Title string
OriginalName string
EpisodeTitle string
SizeBytes int64
DurationSec int
Width int
Height int
VideoCodec string
AudioCodec string
Container string
STRMURL string
FileID string
PosterURL string
BackdropURL string
Overview string
Year int
ReleaseDate string
Rating float32
TMDbID int
BangumiID int
DoubanID string
TheTVDBID string
SeasonNum int
EpisodeNum int
// EpisodeFraction 是集号的小数部分(S01E11.5 → 0.5);变化同样要触发刷新,
// 否则已有行上的「半集」标记不会被写回。
EpisodeFraction float64
+1 -3
View File
@@ -33,7 +33,7 @@ func (s *ScannerService) existingLocalMediaSnapshotForRoot(ctx context.Context,
query = query.Where("(library_root_id = ? OR library_root_id = '' OR library_root_id IS NULL)", rootID)
}
rows, err := query.
Select("path", "library_root_id", "relative_path", "title", "original_name", "episode_title", "size_bytes", "duration_sec", "width", "height", "video_codec", "audio_codec", "container", "strm_url", "file_id", "poster_url", "backdrop_url", "overview", "year", "release_date", "rating", "tm_db_id", "bangumi_id", "douban_id", "thetvdb_id", "season_num", "episode_num", "episode_fraction", "genres", "countries", "languages", "nsfw", "scrape_status", "availability_state", "missing_since").
Select("path", "library_root_id", "relative_path", "title", "original_name", "episode_title", "size_bytes", "duration_sec", "width", "height", "video_codec", "audio_codec", "container", "strm_url", "file_id", "poster_url", "backdrop_url", "overview", "year", "release_date", "rating", "tm_db_id", "bangumi_id", "douban_id", "thetvdb_id", "season_num", "episode_num", "episode_fraction", "genres", "countries", "languages", "nsfw", "scrape_status").
Rows()
if err != nil {
return nil, err
@@ -80,8 +80,6 @@ func (s *ScannerService) existingLocalMediaSnapshotForRoot(ctx context.Context,
&row.Languages,
&row.NSFW,
&row.ScrapeStatus,
&row.AvailabilityState,
&row.MissingSince,
); err != nil {
return nil, err
}
+3 -10
View File
@@ -548,7 +548,7 @@ func TestScanLibraryMapsPersistedHostLibraryPath(t *testing.T) {
}
}
func TestRemovePathRetainsVanishedMedia(t *testing.T) {
func TestRemovePathDeletesVanishedMedia(t *testing.T) {
sc, repos := newScannerTestEnv(t)
cache := NewRuntimeCacheService(&config.Config{}, zap.NewNop())
sc.SetRuntimeCache(cache)
@@ -589,15 +589,8 @@ func TestRemovePathRetainsVanishedMedia(t *testing.T) {
if removed != 1 {
t.Fatalf("removed = %d, want 1", removed)
}
if countMedia(t, repos) != 1 {
t.Fatal("missing media metadata must be retained")
}
var missing model.Media
if err := repos.DB.Where("path = ?", file).First(&missing).Error; err != nil {
t.Fatal(err)
}
if missing.AvailabilityState != "missing" || missing.MissingSince == nil {
t.Fatalf("missing state not recorded: %#v", missing)
if countMedia(t, repos) != 0 {
t.Fatal("expected 0 media after removal")
}
if cache.GetJSON(t.Context(), "media:list:test", &cached) {
t.Fatal("vanished media removal should invalidate media cache")
+2 -10
View File
@@ -67,16 +67,8 @@ func TestScanLibraryScansMultipleRootsAndPrunesPerRoot(t *testing.T) {
if err := db.Find(&rows).Error; err != nil {
t.Fatal(err)
}
if len(rows) != 2 {
t.Fatalf("remaining rows = %#v, want both media records preserved", rows)
}
for _, row := range rows {
if row.Path == fileA && row.AvailabilityState != "missing" {
t.Fatalf("vanished file was not marked missing: %#v", row)
}
if row.Path == fileB && row.AvailabilityState != "available" {
t.Fatalf("offline root status was changed: %#v", row)
}
if len(rows) != 1 || rows[0].Path != fileB {
t.Fatalf("remaining rows = %#v, want offline root media preserved", rows)
}
}
+1 -1
View File
@@ -122,7 +122,7 @@ func (s *ScannerService) localMediaScanState(in localMediaScanStateInput) (bool,
}
existing, exists := in.existingMedia[in.cleanPath]
isNewMedia := !exists
if exists && existing.AvailabilityState != "missing" && existing.MissingSince == nil && in.ext != ".strm" && existing.SizeBytes == in.size && !localMetadataNeedsRefresh(existing, in.localMeta) && !localDerivedMetadataNeedsRefresh(existing, in.incoming) {
if exists && in.ext != ".strm" && existing.SizeBytes == in.size && !localMetadataNeedsRefresh(existing, in.localMeta) && !localDerivedMetadataNeedsRefresh(existing, in.incoming) {
// Unchanged rows are skipped, but a row that never received ffprobe data
// still needs a backfill probe so installing ffmpeg later repairs the library.
s.queueProbeBackfillIfNeeded(in.path, in.ext, existing)
@@ -372,14 +372,7 @@ func TestScanLibraryPrunesMissingMedia(t *testing.T) {
if err := db.Model(&model.Media{}).Where("library_id = ?", lib.ID).Count(&count).Error; err != nil {
t.Fatal(err)
}
if count != 2 {
t.Fatalf("media count = %d, want current and missing records preserved", count)
}
var missing model.Media
if err := db.First(&missing, "id = ?", stale.ID).Error; err != nil {
t.Fatal(err)
}
if missing.AvailabilityState != "missing" {
t.Fatalf("stale media state = %q, want missing", missing.AvailabilityState)
if count != 1 {
t.Fatalf("media count = %d, want 1", count)
}
}
-55
View File
@@ -1,55 +0,0 @@
package service
import (
"os"
"path/filepath"
"testing"
"github.com/truewhile/MeBox/internal/model"
)
func TestScanRestoresUnchangedMissingFileAndMetadata(t *testing.T) {
scanner, repos := newScannerTestEnv(t)
root := t.TempDir()
library := model.Library{Name: "Movies", Path: root, Type: "movie", Enabled: true}
if err := repos.Library.Create(t.Context(), &library); err != nil {
t.Fatal(err)
}
path := filepath.Join(root, "Example (2024).mkv")
writeTestFile(t, path, "unchanged content")
if _, err := scanner.ScanLibrary(t.Context(), library.ID); err != nil {
t.Fatal(err)
}
var original model.Media
if err := repos.DB.Where("path = ?", path).First(&original).Error; err != nil {
t.Fatal(err)
}
if err := repos.DB.Model(&original).Updates(map[string]any{
"title": "Manually matched title", "scrape_status": "matched", "tm_db_id": 42,
}).Error; err != nil {
t.Fatal(err)
}
if err := os.Remove(path); err != nil {
t.Fatal(err)
}
if changed, err := scanner.RemovePath(t.Context(), path); err != nil || changed != 1 {
t.Fatalf("mark missing: changed=%d err=%v", changed, err)
}
if changed, err := scanner.RemovePath(t.Context(), path); err != nil || changed != 0 {
t.Fatalf("repeated missing event: changed=%d err=%v", changed, err)
}
writeTestFile(t, path, "unchanged content")
if _, err := scanner.ScanLibrary(t.Context(), library.ID); err != nil {
t.Fatal(err)
}
var restored model.Media
if err := repos.DB.Where("path = ?", path).First(&restored).Error; err != nil {
t.Fatal(err)
}
if restored.ID != original.ID || restored.TMDbID != 42 || restored.Title != "Manually matched title" {
t.Fatalf("restored file lost identity or metadata: %#v", restored)
}
if restored.AvailabilityState != "available" || restored.MissingSince != nil || restored.LastSeenAt == nil {
t.Fatalf("unchanged restored file stayed missing: %#v", restored)
}
}
+15 -44
View File
@@ -2,7 +2,6 @@ package service
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
@@ -13,15 +12,11 @@ import (
"github.com/truewhile/MeBox/internal/model"
)
// RemovePath 将确认不存在的媒体标记为失联;只有可确认的 STRM 扩展名改名
// 才保留软删除墓碑,供后续 ingest 继承元数据。
// RemovePath 移除磁盘上已不存在的媒体记录。普通的删除仍做物理删除;
// 可确认是 STRM 扩展名改名的路径则先保留软删除墓碑,供后续 ingest 继承元数据。
func (s *ScannerService) RemovePath(ctx context.Context, path string) (int64, error) {
if _, err := os.Stat(path); err == nil {
return 0, nil // still exists; nothing to remove
} else if !errors.Is(err, os.ErrNotExist) {
// Permission, I/O, and mount errors are not evidence that a file was
// removed. Preserve records until the path can be inspected reliably.
return 0, nil
}
// 目录整体消失(删除/改名离开):连同其子树下的媒体行一并移除。
// 此前只删 path 精确匹配的行——目录本身通常没有 media 行,导致目录
@@ -38,7 +33,7 @@ func (s *ScannerService) RemovePath(ctx context.Context, path string) (int64, er
return 0, err
}
softIDs := make([]string, 0, len(rows))
missingIDs := make([]string, 0, len(rows))
hardIDs := make([]string, 0, len(rows))
for _, row := range rows {
// LIKE 里的 % _ 是通配符(候选集只会偏大),用 Go 前缀精确过滤,
// 避免对含 % / _ 的路径误删。
@@ -48,12 +43,12 @@ func (s *ScannerService) RemovePath(ctx context.Context, path string) (int64, er
// A vanished STRM with a live sibling (foo.strm <-> foo.mkv.strm)
// is a rename. Keep a soft-deleted tombstone so the later ingest can
// adopt its ID and scraped metadata even if fsnotify delivers the
// remove event before the create event.
// remove event before the create event. A true deletion stays hard.
if row.Path == path && liveSTRMPathAlias(path) {
softIDs = append(softIDs, row.ID)
continue
}
missingIDs = append(missingIDs, row.ID)
hardIDs = append(hardIDs, row.ID)
}
var removed int64
if len(softIDs) > 0 {
@@ -65,12 +60,14 @@ func (s *ScannerService) RemovePath(ctx context.Context, path string) (int64, er
}
removed += res.RowsAffected
}
if len(missingIDs) > 0 {
count, err := s.markMediaMissingByIDs(ctx, missingIDs)
if err != nil {
return removed, err
if len(hardIDs) > 0 {
res := s.repo.DB.WithContext(ctx).Unscoped().
Where("id IN ?", hardIDs).
Delete(&model.Media{})
if res.Error != nil {
return removed, res.Error
}
removed += count
removed += res.RowsAffected
}
if removed > 0 {
s.invalidateMediaCache(ctx)
@@ -85,7 +82,7 @@ const strmRenameTombstoneRetention = 7 * 24 * time.Hour
// purgeExpiredSTRMTombstones keeps the soft-delete grace period bounded.
// It is deliberately restricted to deleted media rows under a STRM alias
// workflow; ordinary missing files retain their metadata for recovery.
// workflow; normal deletions are still hard-deleted immediately.
func (s *ScannerService) purgeExpiredSTRMTombstones(ctx context.Context) {
cutoff := time.Now().Add(-strmRenameTombstoneRetention)
if err := s.repo.DB.WithContext(ctx).Unscoped().
@@ -124,7 +121,7 @@ func (s *ScannerService) pruneMissingMedia(ctx context.Context, libraryID string
}
stale = append(stale, row.ID)
}
return s.markMediaMissingByIDs(ctx, stale)
return s.deleteMediaByIDs(ctx, stale, true)
}
func (s *ScannerService) pruneMissingMediaForRoot(ctx context.Context, libraryID, rootID, rootPath string, seen map[string]struct{}) (int64, error) {
@@ -161,33 +158,7 @@ func (s *ScannerService) pruneMissingMediaForRoot(ctx context.Context, libraryID
}
stale = append(stale, row.ID)
}
return s.markMediaMissingByIDs(ctx, stale)
}
func (s *ScannerService) markMediaMissingByIDs(ctx context.Context, ids []string) (int64, error) {
if len(ids) == 0 {
return 0, nil
}
const batch = 500
var changed int64
now := time.Now()
for i := 0; i < len(ids); i += batch {
end := i + batch
if end > len(ids) {
end = len(ids)
}
result := s.repo.DB.WithContext(ctx).Model(&model.Media{}).
Where("id IN ? AND (availability_state IS NULL OR availability_state <> ?)", ids[i:end], "missing").
Updates(map[string]any{"availability_state": "missing", "missing_since": now})
if result.Error != nil {
return changed, result.Error
}
changed += result.RowsAffected
}
if changed > 0 {
s.invalidateMediaCache(ctx)
}
return changed, nil
return s.deleteMediaByIDs(ctx, stale, true)
}
func pathBelongsToRoot(pathValue, rootPath string) bool {
+1 -1
View File
@@ -202,7 +202,7 @@ func (s *ScannerService) scanLocalLibraryFiles(ctx context.Context, lib *model.L
func existingLocalMediaFileIDs(existingMedia map[string]existingLocalMedia) map[string]string {
seenInodes := make(map[string]string)
for path, existing := range existingMedia {
if existing.FileID != "" && existing.AvailabilityState != "missing" {
if existing.FileID != "" {
seenInodes[existing.FileID] = path
}
}
+3 -10
View File
@@ -175,7 +175,7 @@ func TestIngestPathAdoptsSoftDeletedSTRMRenameTombstone(t *testing.T) {
}
}
func TestRemovePathRetainsTrueDeletionWithoutAlias(t *testing.T) {
func TestRemovePathStillHardDeletesTrueDeletionWithoutAlias(t *testing.T) {
sc, repos := newScannerTestEnv(t)
root := t.TempDir()
lib := model.Library{Name: "Anime", Path: root, Type: "anime", Enabled: true}
@@ -197,15 +197,8 @@ func TestRemovePathRetainsTrueDeletionWithoutAlias(t *testing.T) {
if err := repos.DB.Unscoped().Model(&model.Media{}).Where("path = ?", path).Count(&count).Error; err != nil {
t.Fatal(err)
}
if count != 1 {
t.Fatalf("missing media count = %d, want metadata retained", count)
}
var media model.Media
if err := repos.DB.Where("path = ?", path).First(&media).Error; err != nil {
t.Fatal(err)
}
if media.AvailabilityState != "missing" || media.MissingSince == nil {
t.Fatalf("true deletion did not record missing state: %#v", media)
if count != 0 {
t.Fatalf("true deletion left %d tombstone row(s)", count)
}
}
+1 -13
View File
@@ -50,7 +50,6 @@ type SchedulerService struct {
readerFilesTTLHours int
segments *MediaSegmentService
backup *BackupService
mu sync.Mutex
stopCh chan struct{}
@@ -66,10 +65,6 @@ func (s *SchedulerService) SetTaskTracker(tasks *TaskTrackerService) {
s.tasks = tasks
}
func (s *SchedulerService) SetBackupService(backup *BackupService) {
s.backup = backup
}
func (s *SchedulerService) SetOrganizePipeline(pipeline *OrganizePipelineService) {
s.organizePipeline = pipeline
}
@@ -80,10 +75,10 @@ func (s *SchedulerService) SetExpiryWatcher(watcher *TelegramExpiryWatcher) {
}
// ImageCachePolicy 是一次图片缓存清理要用的策略(全部为 0 表示不做任何清理)。
// 只有容量配额,没有保留时长:淘汰只看占用,不按文件年龄删除。
type ImageCachePolicy struct {
TotalBytes int64
OriginalsBytes int64
OriginalsAge time.Duration
}
func (s *SchedulerService) SetImageCachePolicyProvider(fn func() ImageCachePolicy) {
@@ -201,13 +196,6 @@ func (s *SchedulerService) Start(ctx context.Context) {
run: s.jobTelegramExpiryWarning,
})
}
if s.backup != nil {
s.jobs = append(s.jobs, &scheduledJob{
name: "database_backup",
interval: 5 * time.Minute,
run: s.jobDatabaseBackup,
})
}
for _, j := range s.jobs {
initialDelay := 15 * time.Second
if j.name == "library_scan" || j.name == "organize_source" || j.name == "segment_prewarm" {
+4 -51
View File
@@ -11,53 +11,6 @@ import (
"go.uber.org/zap"
)
const lastAutomaticBackupDateKey = "backup.last_automatic_date"
func (s *SchedulerService) jobDatabaseBackup(ctx context.Context) error {
if s == nil || s.backup == nil || s.repo == nil || s.repo.Setting == nil {
return nil
}
enabled, _ := s.repo.Setting.Get(ctx, "backup.auto_enabled")
if !parseBoolSetting(enabled, false) {
return nil
}
clock, _ := s.repo.Setting.Get(ctx, "backup.auto_time")
if clock == "" {
clock = "03:00"
}
parsed, err := time.Parse("15:04", clock)
if err != nil {
return nil
}
now := s.currentTime().In(time.Local)
dueAt := time.Date(now.Year(), now.Month(), now.Day(), parsed.Hour(), parsed.Minute(), 0, 0, now.Location())
if now.Before(dueAt) {
return nil
}
date := now.Format("2006-01-02")
last, _ := s.repo.Setting.Get(ctx, lastAutomaticBackupDateKey)
if last == date {
return nil
}
if _, err := s.backup.CreateAutomatic(ctx); err != nil {
return err
}
// Record the successful backup before pruning. If retention cleanup fails,
// the next scheduler tick must not create another copy for the same day.
if err := s.repo.Setting.Set(ctx, lastAutomaticBackupDateKey, date); err != nil {
return err
}
retentionValue, _ := s.repo.Setting.Get(ctx, "backup.retention")
retention, err := strconv.Atoi(strings.TrimSpace(retentionValue))
if err != nil || retention < 1 {
retention = 7
}
if err := s.backup.Prune(retention); err != nil {
return err
}
return nil
}
// jobScanLibraries re-walks every enabled library.
//
// 默认关闭:文件变更由 WatcherService 增量入库,无需周期性全量重扫。
@@ -299,18 +252,18 @@ func (s *SchedulerService) jobCleanTranscodeCache(ctx context.Context) error {
return walkAndPrune(s.cacheDir+"/hls", cutoff)
}
// jobCleanImageCache prunes the image proxy cache: 原图按独立配额优先淘汰,
// 总量超限时再淘汰派生成品。全程只看占用,不按文件年龄删除。
// jobCleanImageCache prunes the image proxy cache: 原图按保留时长与独立配额
// 优先淘汰,总量超限时再淘汰派生成品。
func (s *SchedulerService) jobCleanImageCache(ctx context.Context) error {
if s.cacheDir == "" {
return nil
}
policy := s.imageCachePolicy()
if policy.TotalBytes <= 0 && policy.OriginalsBytes <= 0 {
if policy.TotalBytes <= 0 && policy.OriginalsBytes <= 0 && policy.OriginalsAge <= 0 {
return nil
}
imagesDir := filepath.Join(s.cacheDir, "images")
pools := ImageCachePools(imagesDir, policy.OriginalsBytes)
pools := ImageCachePools(imagesDir, policy.OriginalsBytes, policy.OriginalsAge)
res, err := PruneImageCachePools(pools, policy.TotalBytes)
if err != nil {
if s.log != nil {
+4 -4
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"strings"
"time"
"go.uber.org/zap"
@@ -62,9 +63,6 @@ func (b *serviceContainerBuilder) startRealtimeServices() {
b.c.WSHub = NewHub(b.log)
helper.Go(b.log, "ws.hub", b.c.WSHub.Run)
b.c.Tasks = NewTaskTrackerService(b.log, b.c.WSHub)
if b.repos != nil {
b.c.Tasks.SetPersistence(b.repos.DB)
}
b.c.SystemUpdate = NewSystemUpdateService(b.cfg, b.log, b.repos, b.c.Tasks, b.version)
b.c.SSEHub = NewSSEHub(b.log)
@@ -208,7 +206,6 @@ func (b *serviceContainerBuilder) initAccessAndStorageServices() {
b.c.Organizer, b.c.WSHub, b.cfg.Cache.CacheDir,
)
b.c.Scheduler.SetTaskTracker(b.c.Tasks)
b.c.Scheduler.SetBackupService(b.c.Backup)
b.c.Scheduler.SetOrganizePipeline(b.c.OrganizePipeline)
b.c.Scheduler.SetSegments(b.c.Segments)
b.c.Scheduler.SetImageCachePolicyProvider(func() ImageCachePolicy {
@@ -221,6 +218,9 @@ func (b *serviceContainerBuilder) initAccessAndStorageServices() {
if b.cfg.Cache.ImagesOriginalsMaxSizeMB > 0 {
policy.OriginalsBytes = int64(b.cfg.Cache.ImagesOriginalsMaxSizeMB) * 1024 * 1024
}
if b.cfg.Cache.ImagesOriginalsTTLHours > 0 {
policy.OriginalsAge = time.Duration(b.cfg.Cache.ImagesOriginalsTTLHours) * time.Hour
}
return policy
})
// 阅读正文缓存清理:由阅读模块自己按 TTL + 容量 LRU 淘汰(见 reader_content_cache.go)。
+11 -29
View File
@@ -17,8 +17,7 @@ import (
// Optional query `start` (seconds) restarts ffmpeg from that source offset so
// the web player can scrub the full timeline without waiting for a full
// head-to-tail transcode.
func (s *StreamService) ServeHLSPlaylist(w http.ResponseWriter, r *http.Request, mediaID string, scopes ...HLSPlaybackScope) error {
jobKey := hlsPlaybackJobKey(mediaID, scopes)
func (s *StreamService) ServeHLSPlaylist(w http.ResponseWriter, r *http.Request, mediaID string) error {
// 「客户端直连解码」模式下宿主机不提供转码,HLS 一律拒绝,
// 迫使播放器走 direct play 本地解码。
if s.directPlayOnly(r.Context()) {
@@ -28,23 +27,19 @@ func (s *StreamService) ServeHLSPlaylist(w http.ResponseWriter, r *http.Request,
seekGen := parseHLSSeekGen(r)
subtitleStream := parseHLSSubtitleStream(r)
quality := parseHLSQuality(r)
if _, err := s.transcoder.EnsureJobFromSubtitleQuality(r.Context(), mediaID, startSec, seekGen, subtitleStream, quality, scopes...); err != nil {
if _, err := s.transcoder.EnsureJobFromSubtitleQuality(r.Context(), mediaID, startSec, seekGen, subtitleStream, quality); err != nil {
return err
}
s.transcoder.TouchJob(jobKey)
s.transcoder.TouchJob(mediaID)
readyTimeout := 45 * time.Second
if startSec > 0.05 {
// Mid-file HTTP seeks (esp. WMV) need longer before the first segment appears.
readyTimeout = 120 * time.Second
}
if !s.transcoder.WaitReady(r.Context(), jobKey, readyTimeout) {
if !s.transcoder.WaitReady(r.Context(), mediaID, readyTimeout) {
return errors.New("hls playlist not ready")
}
outputDir, err := s.transcoder.hlsOutputDir(jobKey, seekGen)
if err != nil {
return err
}
playlist := filepath.Join(outputDir, "index.m3u8")
playlist := s.transcoder.PlaylistPath(mediaID)
f, err := os.Open(playlist) // #nosec G304 -- playlist path is generated under the transcoder cache directory for this media ID.
if err != nil {
return err
@@ -54,18 +49,12 @@ func (s *StreamService) ServeHLSPlaylist(w http.ResponseWriter, r *http.Request,
w.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Content-Disposition", "inline")
rawQuery := r.URL.RawQuery
if len(scopes) > 0 && scopes[0].SessionID != "" {
query := r.URL.Query()
query.Set("session_id", scopes[0].SessionID)
rawQuery = query.Encode()
}
if rawQuery != "" {
if r.URL.RawQuery != "" {
data, err := io.ReadAll(f)
if err != nil {
return err
}
playlist := appendQueryToHLSSegments(string(data), rawQuery)
playlist := appendQueryToHLSSegments(string(data), r.URL.RawQuery)
_, err = io.WriteString(w, playlist)
return err
}
@@ -182,26 +171,19 @@ func filterHLSSegmentQuery(rawQuery string) string {
}
// ServeHLSSegment writes a single .ts segment from the on-disk cache.
func (s *StreamService) ServeHLSSegment(w http.ResponseWriter, r *http.Request, mediaID, segment string, scopes ...HLSPlaybackScope) error {
jobKey := hlsPlaybackJobKey(mediaID, scopes)
outputDir, err := s.transcoder.hlsOutputDir(jobKey, parseHLSSeekGen(r))
if err != nil {
if len(scopes) > 0 || parseHLSSeekGen(r) > 0 {
return err
}
outputDir = s.transcoder.HLSDir(jobKey) // Legacy internal static-file callers.
}
func (s *StreamService) ServeHLSSegment(w http.ResponseWriter, r *http.Request, mediaID, segment string) error {
s.transcoder.TouchJob(mediaID)
// Only allow segments that look like seg_NNNNN.ts so we cannot be tricked
// into reading arbitrary files via path traversal.
if !strings.HasPrefix(segment, "seg_") || !strings.HasSuffix(segment, ".ts") {
return errors.New("bad segment")
}
full := filepath.Join(outputDir, segment)
full := filepath.Join(s.transcoder.HLSDir(mediaID), segment)
abs, err := filepath.Abs(full)
if err != nil {
return err
}
dir, _ := filepath.Abs(outputDir)
dir, _ := filepath.Abs(s.transcoder.HLSDir(mediaID))
if !pathWithin(abs, dir) {
return errors.New("path escape")
}
+4 -82
View File
@@ -2,7 +2,6 @@ package service
import (
"context"
"encoding/json"
"html"
"strings"
"sync"
@@ -10,16 +9,12 @@ import (
"github.com/google/uuid"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/model"
)
const (
TaskStatusRunning = "running"
TaskStatusCompleted = "completed"
TaskStatusFailed = "failed"
TaskStatusInterrupted = "interrupted"
TaskStatusRunning = "running"
TaskStatusCompleted = "completed"
TaskStatusFailed = "failed"
TaskKindOrganize = "organize"
TaskKindScan = "scan"
@@ -75,26 +70,6 @@ type TaskTrackerService struct {
recent []BackgroundTask
maxRecent int
now func() time.Time
db *gorm.DB
}
// SetPersistence enables durable task history and marks tasks left running by
// a previous process as interrupted. Actual task execution is never resumed
// automatically; this record is an operator-visible recovery point.
func (t *TaskTrackerService) SetPersistence(db *gorm.DB) {
if t == nil || db == nil {
return
}
t.mu.Lock()
t.db = db
now := t.currentTime()
t.mu.Unlock()
err := db.Model(&model.BackgroundTaskRecord{}).
Where("status = ?", TaskStatusRunning).
Updates(map[string]any{"status": TaskStatusInterrupted, "error": "服务在任务完成前重启;请检查文件状态后重新执行", "updated_at": now, "finished_at": now}).Error
if err != nil && t.log != nil {
t.log.Warn("recover interrupted background tasks failed", zap.Error(err))
}
}
type TaskHandle struct {
@@ -130,11 +105,9 @@ func (t *TaskTrackerService) Start(kind, name string, update TaskUpdate) *TaskHa
StartedAt: now,
UpdatedAt: now,
}
applyTaskUpdate(task, update)
t.mu.Lock()
t.active[task.ID] = task
snapshot := cloneBackgroundTask(*task)
t.persist(snapshot)
t.mu.Unlock()
t.publish(snapshot)
return &TaskHandle{tracker: t, id: task.ID}
@@ -159,47 +132,18 @@ func (t *TaskTrackerService) Snapshot() TaskSnapshot {
return TaskSnapshot{}
}
t.mu.Lock()
defer t.mu.Unlock()
active := make([]BackgroundTask, 0, len(t.active))
for _, task := range t.active {
active = append(active, cloneBackgroundTask(*task))
}
recent := make([]BackgroundTask, 0, len(t.recent))
db := t.db
limit := t.maxRecent
if limit < 1 {
limit = 30
}
for _, task := range t.recent {
recent = append(recent, cloneBackgroundTask(task))
}
t.mu.Unlock()
if db != nil {
var rows []model.BackgroundTaskRecord
if err := db.Where("status <> ?", TaskStatusRunning).Order("started_at DESC").Limit(limit).Find(&rows).Error; err == nil {
recent = make([]BackgroundTask, 0, len(rows))
for _, row := range rows {
recent = append(recent, backgroundTaskFromRecord(row))
}
} else if t.log != nil {
t.log.Warn("load background task history failed; using memory history", zap.Error(err))
}
}
return TaskSnapshot{Active: active, Recent: recent}
}
func backgroundTaskFromRecord(row model.BackgroundTaskRecord) BackgroundTask {
task := BackgroundTask{
ID: row.ID, Kind: row.Kind, Name: row.Name, Status: row.Status,
Stage: row.Stage, SourcePath: row.SourcePath, DestPath: row.DestPath,
Message: row.Message, Error: row.Error, StartedAt: row.StartedAt,
UpdatedAt: row.UpdatedAt, FinishedAt: row.FinishedAt,
}
_ = json.Unmarshal([]byte(row.Details), &task.Details)
_ = json.Unmarshal([]byte(row.Metrics), &task.Metrics)
_ = json.Unmarshal([]byte(row.Items), &task.Items)
return task
}
func (t *TaskTrackerService) update(id string, update TaskUpdate) {
now := t.currentTime()
t.mu.Lock()
@@ -211,7 +155,6 @@ func (t *TaskTrackerService) update(id string, update TaskUpdate) {
applyTaskUpdate(task, update)
task.UpdatedAt = now
snapshot := cloneBackgroundTask(*task)
t.persist(snapshot)
t.mu.Unlock()
t.publish(snapshot)
}
@@ -243,7 +186,6 @@ func (t *TaskTrackerService) finish(id string, err error, update TaskUpdate) {
t.recent = t.recent[:t.maxRecent]
}
notifier := t.failureNotifier
t.persist(snapshot)
t.mu.Unlock()
t.publish(snapshot)
@@ -253,26 +195,6 @@ func (t *TaskTrackerService) finish(id string, err error, update TaskUpdate) {
}
}
func (t *TaskTrackerService) persist(task BackgroundTask) {
if t == nil || t.db == nil {
return
}
encode := func(value any) string {
data, _ := json.Marshal(value)
return string(data)
}
row := model.BackgroundTaskRecord{
ID: task.ID, Kind: task.Kind, Name: task.Name, Status: task.Status,
Stage: task.Stage, SourcePath: task.SourcePath, DestPath: task.DestPath,
Message: task.Message, Error: task.Error, Details: encode(task.Details),
Metrics: encode(task.Metrics), Items: encode(task.Items), StartedAt: task.StartedAt,
UpdatedAt: task.UpdatedAt, FinishedAt: task.FinishedAt,
}
if err := t.db.Save(&row).Error; err != nil && t.log != nil {
t.log.Warn("persist background task failed", zap.String("task_id", task.ID), zap.Error(err))
}
}
// SetFailureNotifier 注入任务失败时的管理员通知回调。
func (t *TaskTrackerService) SetFailureNotifier(fn func(ctx context.Context, text string)) {
if t == nil {
@@ -1,78 +0,0 @@
package service
import (
"fmt"
"sync"
"testing"
"go.uber.org/zap"
"github.com/truewhile/MeBox/internal/model"
)
func TestTaskTrackerPersistsHistoryAcrossRestart(t *testing.T) {
db := newServiceTestDB(t)
tracker := NewTaskTrackerService(zap.NewNop(), nil)
tracker.SetPersistence(db)
task := tracker.Start(TaskKindScan, "scan", TaskUpdate{Details: []string{"initial detail"}})
task.Update(TaskUpdate{Metrics: map[string]int64{"visited": 42}})
task.Finish(nil, TaskUpdate{Message: "done"})
restarted := NewTaskTrackerService(zap.NewNop(), nil)
restarted.SetPersistence(db)
snapshot := restarted.Snapshot()
if len(snapshot.Active) != 0 || len(snapshot.Recent) != 1 {
t.Fatalf("unexpected history: %#v", snapshot)
}
recent := snapshot.Recent[0]
if recent.Status != TaskStatusCompleted || recent.Metrics["visited"] != 42 || len(recent.Details) != 1 || recent.FinishedAt == nil {
t.Fatalf("task history lost fields: %#v", recent)
}
}
func TestTaskTrackerConcurrentUpdatesDoNotRegressFinishedTasks(t *testing.T) {
db := newServiceTestDB(t)
sqlDB, err := db.DB()
if err != nil {
t.Fatal(err)
}
sqlDB.SetMaxOpenConns(1)
tracker := NewTaskTrackerService(zap.NewNop(), nil)
tracker.SetPersistence(db)
var workers sync.WaitGroup
for i := 0; i < 12; i++ {
workers.Add(1)
go func(index int) {
defer workers.Done()
task := tracker.Start(TaskKindScan, fmt.Sprintf("scan-%d", index), TaskUpdate{})
task.Update(TaskUpdate{Metrics: map[string]int64{"visited": 1}})
task.Finish(nil, TaskUpdate{})
task.Update(TaskUpdate{Message: "late update"})
_ = tracker.Snapshot()
}(i)
}
workers.Wait()
var completed int64
if err := db.Model(&model.BackgroundTaskRecord{}).Where("status = ?", TaskStatusCompleted).Count(&completed).Error; err != nil || completed != 12 {
t.Fatalf("finished task status regressed: completed=%d err=%v", completed, err)
}
if snapshot := tracker.Snapshot(); len(snapshot.Active) != 0 || len(snapshot.Recent) != 12 {
t.Fatalf("inconsistent task snapshot: %#v", snapshot)
}
}
func TestTaskTrackerMarksAbandonedTasksInterrupted(t *testing.T) {
db := newServiceTestDB(t)
tracker := NewTaskTrackerService(zap.NewNop(), nil)
tracker.SetPersistence(db)
tracker.Start(TaskKindOrganize, "unfinished", TaskUpdate{})
restarted := NewTaskTrackerService(zap.NewNop(), nil)
restarted.SetPersistence(db)
snapshot := restarted.Snapshot()
if len(snapshot.Recent) != 1 || snapshot.Recent[0].Status != TaskStatusInterrupted || snapshot.Recent[0].FinishedAt == nil {
t.Fatalf("abandoned task not marked interrupted: %#v", snapshot)
}
var running int64
if err := db.Model(&model.BackgroundTaskRecord{}).Where("status = ?", TaskStatusRunning).Count(&running).Error; err != nil || running != 0 {
t.Fatalf("stale running tasks: %d err=%v", running, err)
}
}
+32 -89
View File
@@ -1,8 +1,7 @@
// Package service — HLS on-demand transcoder.
//
// TranscoderService spawns ffmpeg processes that segment a source media file
// into HLS (.m3u8 + .ts). Each playback scope owns a cache directory and
// every ffmpeg run writes to its own run-* subdirectory.
// into HLS (.m3u8 + .ts). The output lives under cache.cache_dir/hls/<id>.
// The HTTP layer serves these files directly with a normal http.FileServer.
//
// Encoder selection (read once at startup from the config):
@@ -16,11 +15,9 @@
// plus the kernel module loaded)
//
// Concurrency model:
// - Each authenticated user/player scope has at most one active ffmpeg job.
// - jobs[jobKey] tracks the running goroutine + cancel func; mediaID remains
// the actual media identity for database access and activity reporting.
// - Legacy internal callers keep media-keyed behaviour.
// - Equivalent requests within one scope reuse the current output.
// - Each Media has at most one active ffmpeg job.
// - jobs[mediaID] tracks the running goroutine + cancel func.
// - Calling Start while a job already exists is a no-op.
// - When the playlist file appears on disk we consider the job "ready"
// and unblock the HTTP handler that was waiting on it.
package service
@@ -52,10 +49,9 @@ type TranscoderService struct {
mu sync.Mutex
jobs map[string]*hlsJob
// Start gates are reference-counted so random playback session keys do not
// accumulate after their last starter/stopper has left.
startGateMu sync.Mutex
startGates map[string]*hlsStartGate
// startGates serializes EnsureJobFrom / StopJob per media so concurrent
// playlist hits cannot spawn multiple ffmpeg writers into one HLS dir.
startGates sync.Map // mediaID -> *sync.Mutex
strmResolve func(ctx context.Context, raw string) (*StrmPlayResult, error)
probe *FFprobeService
@@ -70,7 +66,6 @@ type TranscoderService struct {
// hlsJob holds the live state of one ffmpeg run.
type hlsJob struct {
jobKey string
mediaID string
outputDir string
cancel context.CancelFunc
@@ -161,19 +156,14 @@ func (t *TranscoderService) transcodeConfig() *config.Config {
}
}
// HLSDir is the cache root for one internal playback job key.
func (t *TranscoderService) HLSDir(jobKey string) string {
return filepath.Join(t.cfg.Cache.CacheDir, "hls", jobKey)
// HLSDir is the per-media directory that holds index.m3u8 + segment files.
func (t *TranscoderService) HLSDir(mediaID string) string {
return filepath.Join(t.cfg.Cache.CacheDir, "hls", mediaID)
}
// PlaylistPath returns the absolute path of the m3u8 playlist for a media.
func (t *TranscoderService) PlaylistPath(jobKey string) string {
t.mu.Lock()
defer t.mu.Unlock()
if job := t.jobs[jobKey]; job != nil {
return t.jobPlaylistPath(job)
}
return filepath.Join(t.HLSDir(jobKey), "index.m3u8")
func (t *TranscoderService) PlaylistPath(mediaID string) string {
return filepath.Join(t.HLSDir(mediaID), "index.m3u8")
}
// EnsureJob makes sure a transcode is running for mediaID from the start of
@@ -204,8 +194,7 @@ func (t *TranscoderService) EnsureJobFromSubtitle(ctx context.Context, mediaID s
// EnsureJobFromSubtitleQuality is EnsureJobFromSubtitle with an explicit local
// HLS quality profile. Empty qualityID keeps the global transcoder config.
func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, mediaID string, startSec float64, seekGen int64, subtitleStream int, qualityID string, scopes ...HLSPlaybackScope) (string, error) {
jobKey := hlsPlaybackJobKey(mediaID, scopes)
func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, mediaID string, startSec float64, seekGen int64, subtitleStream int, qualityID string) (string, error) {
cfg := t.transcodeConfig()
if !cfg.Transcoder.Enabled {
return "", ErrTranscodeDisabled
@@ -230,31 +219,28 @@ func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, me
return "", ErrMediaNotFound
}
unlock := t.lockJobStart(jobKey)
defer unlock()
gate := t.mediaStartGate(mediaID)
gate.Lock()
defer gate.Unlock()
t.mu.Lock()
if existing, ok := t.jobs[jobKey]; ok {
if existing, ok := t.jobs[mediaID]; ok {
reusable := true
playlist := t.jobPlaylistPath(existing)
if existing.completed {
info, err := os.Stat(playlist)
info, err := os.Stat(t.PlaylistPath(mediaID))
reusable = err == nil && info.Size() > 0
}
if reusable && sameHLSConfiguration(existing, startSec, subtitleStream, qualityID) {
if seekGen > existing.seekGen {
existing.seekGen = seekGen
}
t.touchJobLocked(jobKey)
t.touchJobLocked(mediaID)
t.mu.Unlock()
return playlist, nil
return t.PlaylistPath(mediaID), nil
}
if reusable && !shouldReplaceHLSJobConfigurationQuality(existing, startSec, seekGen, subtitleStream, qualityID) {
t.touchJobLocked(jobKey)
t.touchJobLocked(mediaID)
t.mu.Unlock()
return playlist, nil
return t.PlaylistPath(mediaID), nil
}
prev := t.detachJobLocked(jobKey)
prev := t.detachJobLocked(mediaID)
t.mu.Unlock()
waitJobExit(prev, 12*time.Second)
} else {
@@ -272,9 +258,6 @@ func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, me
input.Quality = quality
t.maybeFillDuration(ctx, m, input)
settings := ffmpegArgSettingsFromConfig(cfg, quality)
// 「原画」档位遇到浏览器可解的 H.264 源时只换封装,避免慢机器上重编码
// 追不上播放(两核软件编码实测 0.1~0.2 倍速)。其它档位要改分辨率,仍重编码。
input.CopyVideo = t.remuxEligible(ctx, m, quality, subtitleStream)
jobEncoder := settings.encoder
if settings.noScale || subtitleStream >= 0 {
jobEncoder = ""
@@ -284,23 +267,12 @@ func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, me
return "", err
}
rootDir := t.HLSDir(jobKey)
// Runs use distinct directories even when Windows keeps an old segment
// open. A delayed request can never read a newer run's seg_00000.ts.
if err := resetHLSDir(rootDir); err != nil {
outDir := t.HLSDir(mediaID)
// Wipe prior segments so a mid-file restart cannot serve stale early chunks.
// Only safe after the previous ffmpeg has exited (waited above / via gate).
if err := resetHLSDir(outDir); err != nil {
return "", err
}
outDir, err := os.MkdirTemp(rootDir, "run-")
if err != nil {
return "", err
}
registered := false
defer func() {
if !registered {
_ = os.RemoveAll(outDir)
_ = os.Remove(rootDir)
}
}()
t.mu.Lock()
if max := t.maxConcurrent(); max > 0 && t.runningJobCountLocked() >= max {
@@ -315,11 +287,7 @@ func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, me
if subtitleStream >= 0 {
jobEncoder = "software (subtitle burn-in)"
}
if input.CopyVideo {
jobEncoder = remuxEncoderLabel
}
job := &hlsJob{
jobKey: jobKey,
mediaID: mediaID,
outputDir: outDir,
cancel: cancel,
@@ -334,8 +302,7 @@ func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, me
ffmpegPath: bin,
config: cfg,
}
t.jobs[jobKey] = job
registered = true
t.jobs[mediaID] = job
t.mu.Unlock()
helper.Go(t.log, "transcoder.monitorIdle", func() { t.monitorIdle(jobCtx, job) })
@@ -343,7 +310,7 @@ func (t *TranscoderService) EnsureJobFromSubtitleQuality(ctx context.Context, me
defer close(job.done)
t.runFFmpeg(jobCtx, job, input)
})
return filepath.Join(outDir, "index.m3u8"), nil
return t.PlaylistPath(mediaID), nil
}
// shouldReplaceHLSJob reports whether an incoming playlist request may cancel
@@ -390,33 +357,9 @@ func sameHLSConfiguration(existing *hlsJob, startSec float64, subtitleStream int
sameHLSStart(existing.startSec, startSec)
}
type hlsStartGate struct {
mu sync.Mutex
users int
}
func (t *TranscoderService) lockJobStart(key string) func() {
t.startGateMu.Lock()
if t.startGates == nil {
t.startGates = make(map[string]*hlsStartGate)
}
gate := t.startGates[key]
if gate == nil {
gate = &hlsStartGate{}
t.startGates[key] = gate
}
gate.users++
t.startGateMu.Unlock()
gate.mu.Lock()
return func() {
gate.mu.Unlock()
t.startGateMu.Lock()
gate.users--
if gate.users == 0 {
delete(t.startGates, key)
}
t.startGateMu.Unlock()
}
func (t *TranscoderService) mediaStartGate(mediaID string) *sync.Mutex {
v, _ := t.startGates.LoadOrStore(mediaID, &sync.Mutex{})
return v.(*sync.Mutex)
}
// detachJobLocked cancels and removes a job from the map without waiting.
+4 -33
View File
@@ -16,9 +16,6 @@ type transcodeInput struct {
StartSec float64
SubtitleStream *int
Quality *LocalHLSQuality
// CopyVideo 为 true 时视频流只换封装(-c:v copy),不重编码。
// 仅当源视频是浏览器可解的老三样(H.264/8bit/4:2:0)且不需要缩放时启用。
CopyVideo bool
}
type ffmpegArgSettings struct {
@@ -55,7 +52,6 @@ func buildFFmpegArgsForInput(cfg *config.Config, input transcodeInput, playlist,
// 原画档位不做硬件缩放的兼容处理,直接走软件编码,保持源分辨率。
settings.encoder = ""
}
copyVideo := input.CopyVideo && input.SubtitleStream == nil
video := ffmpegVideoPlanForSettings(settings)
if input.SubtitleStream != nil {
// Bitmap subtitles must be composited in software. Keeping CUDA/QSV/
@@ -64,17 +60,13 @@ func buildFFmpegArgsForInput(cfg *config.Config, input transcodeInput, playlist,
settings.encoder = ""
video = ffmpegVideoPlanForSettings(settings)
}
if copyVideo {
// 直通:不做缩放、不挂滤镜、不选编码器,整条视频流原样塞进 TS。
video = ffmpegVideoPlan{codec: "copy"}
}
// Mid-file restarts must not use -re: output -ss would otherwise crawl to the
// seek point at 1x wall-clock before emitting the first HLS segment.
realtime := settings.realtime && input.StartSec <= 0.05
args := baseFFmpegArgs(video.preInput, realtime)
args = appendInputAndVideoArgs(args, input, settings, video, copyVideo)
args = appendOutputHLSArgs(args, settings, segments, playlist, copyVideo)
args = appendInputAndVideoArgs(args, input, settings, video)
args = appendOutputHLSArgs(args, settings, segments, playlist)
return args
}
@@ -169,7 +161,7 @@ func baseFFmpegArgs(preInput string, realtime bool) []string {
return args
}
func appendInputAndVideoArgs(args []string, input transcodeInput, settings ffmpegArgSettings, video ffmpegVideoPlan, copyVideo bool) []string {
func appendInputAndVideoArgs(args []string, input transcodeInput, settings ffmpegArgSettings, video ffmpegVideoPlan) []string {
args = append(args, ffmpegHTTPInputArgs(input)...)
ss := ""
if input.StartSec > 0.05 {
@@ -180,11 +172,6 @@ func appendInputAndVideoArgs(args []string, input transcodeInput, settings ffmpe
// scrubbing; CDNs with Range support jump via demuxer seek instead.
if ss != "" {
args = append(args, "-ss", ss)
if copyVideo {
// 复制视频流时 ffmpeg 不能解码,精确 seek 会把关键帧之前的分组丢掉,
// 输出从 P 帧开始(花屏)。退到前一个关键帧,最多提前一两秒。
args = append(args, "-noaccurate_seek")
}
}
args = append(args, "-i", input.Source)
if input.SubtitleStream != nil {
@@ -210,23 +197,7 @@ func appendInputAndVideoArgs(args []string, input transcodeInput, settings ffmpe
return args
}
func appendOutputHLSArgs(args []string, settings ffmpegArgSettings, segments, playlist string, copyVideo bool) []string {
if copyVideo {
// 视频是复制进来的,编码参数一律不适用;只保留音频转 AAC 与切片参数。
// -avoid_negative_ts 让带 B 帧的源在 -ss 之后不会出现负时间戳。
return append(args,
"-avoid_negative_ts", "make_zero",
"-c:a", "aac",
"-ar", "48000",
"-b:a", "128k",
"-ac", "2",
"-f", "hls",
"-hls_time", fmt.Sprintf("%d", settings.segmentSeconds),
"-hls_list_size", "0",
"-hls_segment_filename", segments,
playlist,
)
}
func appendOutputHLSArgs(args []string, settings ffmpegArgSettings, segments, playlist string) []string {
return append(args,
"-pix_fmt", "yuv420p",
"-b:v", settings.bitrate,
+2 -4
View File
@@ -49,18 +49,16 @@ func (t *TranscoderService) runFFmpeg(ctx context.Context, job *hlsJob, input tr
t.mu.Lock()
// Only update the map entry if we are still the registered generation.
if cur, ok := t.jobs[job.key()]; ok && cur == job {
if cur, ok := t.jobs[job.mediaID]; ok && cur == job {
if err == nil && ctx.Err() == nil {
job.completed = true
} else {
delete(t.jobs, job.key())
delete(t.jobs, job.mediaID)
}
}
t.mu.Unlock()
if err != nil || ctx.Err() != nil {
job.cancel()
_ = os.RemoveAll(job.outputDir)
_ = os.Remove(filepath.Dir(job.outputDir)) // Remove the scope root only if empty.
}
t.hub.Publish("transcode", map[string]any{
+10 -13
View File
@@ -26,18 +26,16 @@ func (t *TranscoderService) WaitReady(ctx context.Context, mediaID string, timeo
job, ok := t.jobs[mediaID]
var started time.Time
var completed bool
var playlist string
if ok {
started = job.startedAt
completed = job.completed
playlist = t.jobPlaylistPath(job)
}
t.mu.Unlock()
if !ok {
return false // A failed/exited job cannot become ready later.
}
if ok {
if info, err := os.Stat(playlist); err == nil {
if info, err := os.Stat(t.PlaylistPath(mediaID)); err == nil {
// Allow a small clock skew; reject anything older than this job.
if !info.ModTime().Before(started.Add(-2 * time.Second)) {
t.mu.Lock()
@@ -68,8 +66,9 @@ func (t *TranscoderService) WaitReady(ctx context.Context, mediaID string, timeo
// briefly for it to exit so a subsequent EnsureJobFrom cannot race-write the
// same HLS directory.
func (t *TranscoderService) StopJob(mediaID string) {
unlock := t.lockJobStart(mediaID)
defer unlock()
gate := t.mediaStartGate(mediaID)
gate.Lock()
defer gate.Unlock()
t.mu.Lock()
prev := t.detachJobLocked(mediaID)
@@ -171,7 +170,7 @@ func (t *TranscoderService) monitorIdle(ctx context.Context, job *hlsJob) {
return
case <-ticker.C:
t.mu.Lock()
current, ok := t.jobs[job.key()]
current, ok := t.jobs[job.mediaID]
if !ok || current != job {
t.mu.Unlock()
return
@@ -186,10 +185,11 @@ func (t *TranscoderService) monitorIdle(ctx context.Context, job *hlsJob) {
}
func (t *TranscoderService) stopIdleJob(job *hlsJob, timeout time.Duration) bool {
unlock := t.lockJobStart(job.key())
defer unlock()
gate := t.mediaStartGate(job.mediaID)
gate.Lock()
defer gate.Unlock()
t.mu.Lock()
current := t.jobs[job.key()]
current := t.jobs[job.mediaID]
if current != job {
t.mu.Unlock()
return true
@@ -199,12 +199,9 @@ func (t *TranscoderService) stopIdleJob(job *hlsJob, timeout time.Duration) bool
t.mu.Unlock()
return false
}
t.detachJobLocked(job.key())
t.detachJobLocked(job.mediaID)
t.mu.Unlock()
waitJobExit(job, 12*time.Second)
if job.jobKey != "" {
_ = os.RemoveAll(t.HLSDir(job.key()))
}
t.log.Info("transcode idle timeout", zap.String("media_id", job.mediaID),
zap.Duration("idle_for", idleFor), zap.Duration("timeout", timeout))
return true
-102
View File
@@ -1,102 +0,0 @@
package service
import (
"context"
"encoding/json"
"strings"
"github.com/truewhile/MeBox/internal/model"
)
// 蓝光原盘(m2ts,40 Mbps 级别)在两核小机器上重编码只有 0.1~0.2 倍速,HLS
// 永远追不上播放,播放器等不到切片就报「转码失败」。当源视频本身就是浏览器
// 能解的 H.264 / 8bit / 4:2:0,且用户选的是「原画」档位(不改分辨率)时,
// 这里改走只换封装(-c:v copy)的直通路径:视频不重编码,音频转 AAC,
// CPU 占用接近于零,起播时间从几十秒降到几秒。
//
// 直通只对「原画」生效。1080P/720P/480P 档位意味着要改分辨率,必须重新编码;
// 那类请求仍然按原来的 libx264 路径处理(慢,但码率可控)。
// remuxEncoderLabel 是直通任务在日志与任务面板里的标签。
const remuxEncoderLabel = "remux"
// remuxEligible 判断这次请求能否复制视频流而不是重编码。
func (t *TranscoderService) remuxEligible(ctx context.Context, m *model.Media, quality *LocalHLSQuality, subtitleStream int) bool {
if t == nil || t.repo == nil || m == nil {
return false
}
// 烧录图形字幕要把字幕叠加进画面,必须解码+重编码。
if subtitleStream >= 0 {
return false
}
if !remuxAppliesToQuality(quality) {
return false
}
probe, err := t.repo.MediaProbe.Get(ctx, m.ID)
if err != nil || probe == nil {
// 没有探测缓存就无法确认档次/像素格式,保守走重编码。
return false
}
return remuxCopyableHeight(probe) > 0
}
// remuxAppliesToQuality 只放行「原画」档位(Height 为 0,即不缩放)。
// quality 为 nil 表示调用方没有指定档位(走全局 max_height),同样不直通,
// 否则会绕过配置的分辨率上限给出一个远超预期的码流。
func remuxAppliesToQuality(quality *LocalHLSQuality) bool {
return quality != nil && quality.Height <= 0
}
// remuxCopyableHeight 在源视频可直接复制进 HLS 时返回它的高度,否则返回 0。
func remuxCopyableHeight(probe *model.MediaProbe) int {
if probe == nil || probe.Height <= 0 {
return 0
}
if !strings.EqualFold(strings.TrimSpace(probe.VideoCodec), "h264") {
return 0
}
if !browserSafeH264Payload(probe.Payload) {
return 0
}
return probe.Height
}
// probePayloadStreams 只解析落库 payload 里与直通判断有关的字段。
type probePayloadStreams struct {
Streams []struct {
Type string `json:"type"`
Profile string `json:"profile"`
PixFmt string `json:"pix_fmt"`
} `json:"streams"`
}
// browserSafeH264Payload 只放行 8bit 4:2:0 的 Baseline/Main/High。
// Hi10P(High 10)和 4:2:2 母带浏览器解不出来,复制过去只会黑屏或无声。
func browserSafeH264Payload(payload string) bool {
payload = strings.TrimSpace(payload)
if payload == "" {
// 老数据没有 payload:已经有 codec + 分辨率,按 8bit 常规情况处理。
return true
}
var parsed probePayloadStreams
if err := json.Unmarshal([]byte(payload), &parsed); err != nil {
return false
}
for _, stream := range parsed.Streams {
if !strings.EqualFold(strings.TrimSpace(stream.Type), "video") {
continue
}
switch strings.ToLower(strings.TrimSpace(stream.Profile)) {
case "", "constrained baseline", "baseline", "main", "high":
default:
return false
}
switch strings.ToLower(strings.TrimSpace(stream.PixFmt)) {
case "", "yuv420p", "yuvj420p":
default:
return false
}
return true
}
return true
}
-170
View File
@@ -1,170 +0,0 @@
package service
import (
"strings"
"testing"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/model"
)
// 直通档位必须真的把视频流复制过去:一旦又带上 libx264 / -vf / 码率参数,
// 慢机器上的「原画」就还是几十秒起播。
func TestBuildFFmpegArgsRemuxCopiesVideoStream(t *testing.T) {
cfg := &config.Config{}
cfg.Transcoder.SegmentSeconds = 4
cfg.Transcoder.Realtime = true
cfg.Transcoder.Threads = 2
cfg.Transcoder.MaxHeight = 720
quality, ok := LocalHLSQualityByID("source")
if !ok {
t.Fatal("source quality missing")
}
args := buildFFmpegArgsForInput(cfg, transcodeInput{
Source: "https://cdn.example/x.m2ts",
Headers: map[string]string{"User-Agent": "MeBox-115-GoClient/1.0"},
StartSec: 52.4,
Quality: &quality,
CopyVideo: true,
}, "/cache/hls/k/run-1/index.m3u8", "/cache/hls/k/run-1/seg_%05d.ts")
joined := strings.Join(args, " ")
for _, want := range []string{
"-c:v copy",
"-c:a aac",
"-ar 48000",
"-b:a 128k",
"-ac 2",
"-avoid_negative_ts make_zero",
"-ss 52.400",
"-noaccurate_seek",
"-hls_time 4",
"-hls_segment_filename /cache/hls/k/run-1/seg_%05d.ts",
} {
if !strings.Contains(joined, want) {
t.Errorf("remux args missing %q: %s", want, joined)
}
}
for _, unwanted := range []string{
"libx264",
"-vf ",
"-b:v",
"-maxrate",
"-bufsize",
"-force_key_frames",
"-preset",
"-threads",
"-pix_fmt",
} {
if strings.Contains(joined, unwanted) {
t.Errorf("remux args must not contain %q: %s", unwanted, joined)
}
}
// 中途起播不能加 -re,否则会先按 1 倍速爬到目标位置才出第一个切片。
if strings.Contains(joined, " -re ") {
t.Errorf("mid-file remux must not use -re: %s", joined)
}
}
// 烧录图形字幕要把字幕叠进画面,必须解码重编码,不能被直通标志绕过。
func TestBuildFFmpegArgsRemuxIgnoredForSubtitleBurnIn(t *testing.T) {
cfg := &config.Config{}
cfg.Transcoder.MaxHeight = 720
cfg.Transcoder.SegmentSeconds = 4
quality, _ := LocalHLSQualityByID("source")
subtitle := 4
args := buildFFmpegArgsForInput(cfg, transcodeInput{
Source: "/media/ep.mkv",
Quality: &quality,
CopyVideo: true,
SubtitleStream: &subtitle,
}, "/o/x.m3u8", "/o/seg_%05d.ts")
joined := strings.Join(args, " ")
if strings.Contains(joined, "-c:v copy") {
t.Fatalf("subtitle burn-in must not copy the video stream: %s", joined)
}
if !strings.Contains(joined, "libx264") || !strings.Contains(joined, "-filter_complex") {
t.Fatalf("subtitle burn-in should re-encode: %s", joined)
}
if strings.Contains(joined, "-noaccurate_seek") {
t.Fatalf("encoded path should keep accurate seek: %s", joined)
}
}
// 只有「原画」档位(不缩放)可以直通;其它档位要改分辨率。
func TestRemuxAppliesToQuality(t *testing.T) {
if remuxAppliesToQuality(nil) {
t.Fatal("nil quality must not remux: it falls back to the configured max height")
}
source, _ := LocalHLSQualityByID("source")
if !remuxAppliesToQuality(&source) {
t.Fatal("source quality should remux")
}
for _, id := range []string{"1080", "720", "480"} {
quality, ok := LocalHLSQualityByID(id)
if !ok {
t.Fatalf("quality %s missing", id)
}
if remuxAppliesToQuality(&quality) {
t.Fatalf("quality %s must re-encode", id)
}
}
}
// 只有 8bit 4:2:0 的 H.264 能安全复制;Hi10P / 4:2:2 / HEVC 复制过去浏览器黑屏。
func TestRemuxCopyableHeight(t *testing.T) {
const blurayPayload = `{"container":"mpegts","duration_sec":1451,"bit_rate":42609934,"streams":[{"index":0,"type":"video","codec":"h264","profile":"High","width":1920,"height":1080,"pix_fmt":"yuv420p","frame_rate":"23.976"},{"index":1,"type":"audio","codec":"pcm_bluray","channels":2}]}`
cases := []struct {
name string
probe *model.MediaProbe
want int
}{
{
name: "bluray h264 8bit 420",
probe: &model.MediaProbe{VideoCodec: "h264", Height: 1080, Payload: blurayPayload},
want: 1080,
},
{
name: "no payload falls back to codec and height",
probe: &model.MediaProbe{VideoCodec: "h264", Height: 720},
want: 720,
},
{
name: "hevc cannot be copied into browser hls",
probe: &model.MediaProbe{VideoCodec: "hevc", Height: 1080, Payload: `{"streams":[{"type":"video","profile":"Main 10","pix_fmt":"yuv420p10le"}]}`},
want: 0,
},
{
name: "hi10p h264 is unplayable in browsers",
probe: &model.MediaProbe{VideoCodec: "h264", Height: 1080, Payload: `{"streams":[{"type":"video","profile":"High 10","pix_fmt":"yuv420p10le"}]}`},
want: 0,
},
{
name: "4:2:2 h264 is unplayable in browsers",
probe: &model.MediaProbe{VideoCodec: "h264", Height: 1080, Payload: `{"streams":[{"type":"video","profile":"High 4:2:2","pix_fmt":"yuv422p"}]}`},
want: 0,
},
{
name: "missing probe",
probe: nil,
want: 0,
},
{
name: "unknown height",
probe: &model.MediaProbe{VideoCodec: "h264", Height: 0, Payload: blurayPayload},
want: 0,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := remuxCopyableHeight(tc.probe); got != tc.want {
t.Fatalf("remuxCopyableHeight() = %d, want %d", got, tc.want)
}
})
}
}
-28
View File
@@ -1,28 +0,0 @@
import { api, LONG_REQUEST_TIMEOUT } from './client'
export interface BackupInfo {
filename: string
size: number
created_at: string
automatic: boolean
}
export const backupAPI = {
list: () => api.get<{ items: BackupInfo[] }>('/admin/backups').then((r) => r.data.items ?? []),
create: () => api.post<BackupInfo>('/admin/backups', {}, { timeout: LONG_REQUEST_TIMEOUT }).then((r) => r.data),
remove: (filename: string) => api.delete('/admin/backups', { params: { filename } }),
restore: (filename: string) => api.post<{ message: string; restart_required: boolean }>('/admin/backups/restore', null, { params: { filename }, timeout: LONG_REQUEST_TIMEOUT }).then((r) => r.data),
download: async (filename: string) => {
const response = await api.get<Blob>('/admin/backups/download', {
params: { filename },
responseType: 'blob',
timeout: LONG_REQUEST_TIMEOUT,
})
const url = URL.createObjectURL(response.data)
const anchor = document.createElement('a')
anchor.href = url
anchor.download = filename
anchor.click()
URL.revokeObjectURL(url)
},
}
+4 -6
View File
@@ -175,7 +175,7 @@ export function streamURL(mediaId: string, options: { proxy?: boolean } = {}): s
// hlsURL returns the m3u8 playlist URL fed into hls.js.
// startSec > 0 asks the server to (re)start ffmpeg from that source offset.
export function hlsURL(mediaId: string, startSec = 0, subtitleStream?: number, quality?: string, sessionID?: string): string {
export function hlsURL(mediaId: string, startSec = 0, subtitleStream?: number, quality?: string): string {
const safeStart = Math.max(0, Math.round(startSec * 1000) / 1000)
// Always send start= (including 0) so the server can tell an intentional
// restart-from-head apart from a missing query on a stale refresh.
@@ -188,8 +188,7 @@ export function hlsURL(mediaId: string, startSec = 0, subtitleStream?: number, q
? `&subtitle=${encodeURIComponent(String(subtitleStream))}`
: ''
const qualityQuery = quality ? `&quality=${encodeURIComponent(quality)}` : ''
const session = sessionID ? `&session_id=${encodeURIComponent(sessionID)}` : ''
return `/api/hls/${encodeURIComponent(mediaId)}/index.m3u8?${tokenQuery()}${profileQuery()}${start}${subtitle}${qualityQuery}${bust}${session}`
return `/api/hls/${encodeURIComponent(mediaId)}/index.m3u8?${tokenQuery()}${profileQuery()}${start}${subtitle}${qualityQuery}${bust}`
}
// cloudHlsURL returns the MeBox-proxied 115 cloud HLS master playlist URL.
@@ -203,9 +202,8 @@ export function cloudHlsURL(mediaId: string, definition: string): string {
// Stop an on-demand HLS job. keepalive makes the request survive page
// navigation/tab close, where an axios promise can be discarded by browsers.
export function stopHLSJob(mediaId: string, sessionID?: string): void {
const session = sessionID ? `&session_id=${encodeURIComponent(sessionID)}` : ''
const url = `/api/hls/${encodeURIComponent(mediaId)}?${tokenQuery()}${profileQuery()}${session}`
export function stopHLSJob(mediaId: string): void {
const url = `/api/hls/${encodeURIComponent(mediaId)}?${tokenQuery()}${profileQuery()}`
void fetch(url, {
method: 'DELETE',
credentials: 'same-origin',
+1 -14
View File
@@ -262,15 +262,7 @@ export const mediaAPI = {
})
.then((r) => r.data),
searchPage: (q: string, page = 1, pageSize = 50, options?: {
groupVersions?: boolean
groupSeries?: boolean
libraryId?: string
yearMin?: number
yearMax?: number
genres?: string[]
sort?: 'title_asc' | 'year_desc' | 'rating_desc'
}) =>
searchPage: (q: string, page = 1, pageSize = 50, options?: { groupVersions?: boolean; groupSeries?: boolean }) =>
api
.get<MediaSearchPage>('/media', {
params: {
@@ -279,11 +271,6 @@ export const mediaAPI = {
page_size: pageSize,
group_versions: options?.groupVersions === false ? 0 : undefined,
...(options?.groupSeries ? { group_series: 1 } : {}),
...(options?.libraryId ? { library_id: options.libraryId } : {}),
...(options?.yearMin ? { year_min: options.yearMin } : {}),
...(options?.yearMax ? { year_max: options.yearMax } : {}),
...(options?.genres?.length ? { genre: options.genres } : {}),
...(options?.sort ? { sort: options.sort } : {}),
},
timeout: LONG_REQUEST_TIMEOUT,
})

Some files were not shown because too many files have changed in this diff Show More