Commit Graph

21 Commits

Author SHA1 Message Date
ShukeBta d67dab1f4c refactor scheduler service helpers 2026-06-26 14:54:26 +08:00
ShukeBta 65adca04df fix media workflows and download organization 2026-06-26 08:42:16 +08:00
ShukeBta 192f35d9fa refactor: split modules and harden scraping workflows 2026-06-24 11:59:18 +08:00
ShukeBta 7b881d031c fix organizer diagnostics and site integrations 2026-06-21 09:44:29 +08:00
ShukeBta b492d9f867 Fix license sync and cloud scan scheduling 2026-06-18 01:39:39 +08:00
ShukeBta 0291ab8194 Fix media scraping rename and compose pulls 2026-06-17 23:59:13 +08:00
ShukeBta 850d3db811 Fix scraping workflows subscriptions bot and recycle bin 2026-06-17 19:48:17 +08:00
ShukeBta 479a06f008 Unify organize ingest pipeline 2026-06-14 23:17:52 +08:00
ShukeBta c76c72966a Fix organize targets and cloud transfer controls 2026-06-14 00:27:39 +08:00
ShukeBta 5f42836bbf fix organize trigger task flow 2026-06-13 17:08:56 +08:00
ShukeBta 4e0929bbca fix organize path mapping and task flow 2026-06-13 16:32:38 +08:00
ShukeBta d270e9034f fix: 防止启动整理扫描阻塞登录 2026-06-12 19:06:47 +08:00
ShukeBta 1568ae127a perf: 重构 FTS 索引为 rowid 寻址+触发器维护,根治重启 CPU 占满与扫描卡死
- FTS5 普通列(含 UNINDEXED)不支持索引查找,旧版按 media_id 做
  NOT EXISTS/DELETE 全是整表扫描:启动回填 O(N^2) 烧 CPU 数小时,
  扫描时每个 upsert 一次全表扫描,均隔着全局写锁拖死登录(125s 超时)
- v2 布局:FTS rowid 与 media.rowid 对齐,由 INSERT/UPDATE/DELETE
  触发器实时维护;回填只剩 rowid 点查兜底,且去掉 ORDER BY
- 顺带修复刮削直写 Updates() 后新标题搜不到的问题(触发器覆盖)
- 写门改为 context 感知:长写语句不再让登录请求无限期排队
- warmMediaSearchIndex 延迟 30s 错峰启动
- library_scan 周期任务跳过云盘库(由 cloud_sync 夜间窗口负责),
  首轮延迟从 15s 改为等满一个周期,重启不再立即全量扫描风暴
- prune 改为只取 id/path 并按 500 条批量删除,缩短写锁占用
2026-06-12 06:41:14 +00:00
ShukeBta 23d1fbfddb fix cloud library scale and playback compatibility
Root cause: display-only cloud library filtering was reused by scan jobs, merged cloud mounts could be skipped, and OpenList listing relied on WebDAV/first-page behavior that could cap huge directories around 100 items. Cloud scans also let new items consume probe budget before existing rows with missing track/artwork metadata.\n\nChanges:\n- split display filtering from scannable cloud filtering so merged cloud mounts still scan\n- add OpenList API pagination with WebDAV fallback\n- prioritize existing cloud media missing metadata before new imports\n- restrict automatic cloud sync to one successful 19:00-21:00 daily window and keep manual scan immediate\n- disable startup cloud full-scan by default, with an explicit opt-in setting\n- improve Emby/cloud playback compatibility and cache/search/test coverage from the continued work
2026-06-11 18:31:01 +08:00
ShukeBta 542e85a067 fix: reduce emby load and stabilize strm scan 2026-06-11 00:18:51 +08:00
ShukeBta 0bf983cec8 fix: improve cloud mount scanning feedback 2026-06-10 18:46:30 +08:00
ShukeBta 7d7f3cc758 feat: add cloud transfer and cache optimizations 2026-06-09 19:03:28 +08:00
ShukeBta 76fe0dbde9 feat: scrape metadata after organize 2026-06-09 17:00:24 +08:00
ShukeBta fe0e7a09ce fix: improve media file organization workflow 2026-06-09 13:57:34 +08:00
soldosluka857 b9b7b7052a feat(organize/scan): transfer modes, seeding-safe relocation, inode dedup, incremental scanning
- Organizer: add move/copy/hardlink/symlink transfer modes (default move),
  per-request target_path/transfer_mode overrides, and honor organize.target_dir
  / organize.transfer_mode settings.
- keep_seeding (default on): escalate move->hardlink (cross-device->copy) so the
  qBittorrent source stays in place and continues seeding after organize.
- qBittorrent SetLocation + POST /downloads/relocate to migrate whole torrents
  while keeping them seeding.
- Scanner: FileID (device:inode) hardlink dedup to avoid duplicate recognition
  and double-counted storage; extract single-file ingest.
- Watcher: recursive watch + incremental per-file ingest/remove instead of full
  re-scan; periodic full library scan now gated behind scan.periodic_enabled
  (default off) to reduce disk wear.
- Telegram bot: handle callback_query in polling, declare allowed_updates in
  webhook, answer callbacks.
- Frontend: settings for transfer mode / keep_seeding / periodic scan; organize
  panel target dir + transfer mode overrides.
- Tests for transfer modes, organizer resolution, SetLocation, inode dedup,
  incremental ingest/remove.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-05-30 12:01:46 +08:00
Kiro 4b747c74ca feat: port missing MediaStation features (DLNA, STRM, files, dup, sched, api-configs, emby, storage)
Audit-driven port from the original Python MediaStation. Eight major
subsystems that were absent from the Go rewrite are now in place,
each with its own service, handler, frontend page and smoke-test
assertions.

Backend services
  - service/crypto.go: AES-256-GCM encrypt/decrypt for at-rest secrets
    keyed off the JWT secret. Legacy plaintext rows pass through
    unchanged for smooth upgrades. Unit-tested.
  - service/api_config.go: third-party provider config (TMDb, Bangumi,
    TheTVDB, Fanart, Douban, OpenAI). Seeds defaults on first run.
    Encrypts api_key on write, returns masked 'abc1****wxyz' projection.
  - service/duplicate.go: sparse-sample MD5 (head + middle + tail, 1MiB
    each, plus file-size suffix) duplicate finder. Picks 'best' primary
    (matched > size > id) and marks others is_duplicate=true.
  - service/filemanager.go: server-side allow-listed file browser used
    by the library-path picker. Strict path-traversal protection.
  - service/dlna.go: real SSDP M-SEARCH discovery + AVTransport
    SetAVTransportURI/Play SOAP cast. 30 s discovery cache.
  - service/scheduler.go: 3 recurring background jobs (library_scan
    60min, transcode_cleanup 24h, recycle_purge 24h with 30-day
    cutoff). Status + run-now endpoints.
  - service/cache_cleanup.go: walkAndPrune helper used by scheduler.
  - service/storage.go: DB-only disk-usage breakdown by library and by
    container format.
  - service/emby_compat.go: read-only Emby/Jellyfin shim
    (System/Info, Users, Users/x/Views, Items, PlaybackInfo) so Infuse
    / VidHub / Kodi can browse MediaStationGo libraries.

Model updates
  - Media: new strm_url (302 redirect target), file_hash, is_duplicate,
    duplicate_of fields.
  - APIConfig: new table for encrypted provider secrets.
  - AutoMigrate registers APIConfig.

Stream layer
  - StreamService.ServeFile now redirects 302 to strm_url when set so
    WebDAV / Alist / S3 / HTTP direct links work transparently.

Handlers + routes
  - Authed: GET /files, GET /storage, GET /dlna/devices, POST /dlna/cast,
    PUT/DELETE /media/:id/strm, POST /strm/import,
    POST /duplicates/{scan,unmark}.
  - Admin: GET/PUT/DELETE /admin/api-configs/:provider,
    GET /admin/scheduler, POST /admin/scheduler/:name/run.
  - New /emby/* group: System/Info, Users, Users/:userId/Views,
    Users/:userId/Items, Items/:id/PlaybackInfo (auth-required).

Frontend pages (lazy-loaded, 7 new chunks)
  - DlnaPage: device list + media picker + cast button.
  - FileManagerPage: root selector + breadcrumb + sortable listing.
  - APIConfigsPage: per-provider card with masked-key editor.
  - StoragePage: usage tiles + per-library bars + per-container grid.
  - DuplicatesPage: scan form + grouped report with primary highlight.
  - SchedulerPage: live job table with run-now button (5s refresh).
  - Sidebar reorganised: 自动化 group adds DLNA, 管理 group adds
    存储 / 文件浏览 / 重复文件 / 定时任务 / API 配置.

Smoke test additions (all admin-only)
  - api-configs seeded with 6 providers
  - api-config encrypted in db (sqlite3 enc:v1: prefix check)
  - storage breakdown
  - file browser lists library root + rejects /etc (path traversal)
  - dlna devices endpoint
  - scheduler exposes 3 jobs + run library_scan
  - emby /System/Info + /Users/{x}/Views
  - strm set + stream 302 + strm clear
  - duplicate scan

Verified: go build, go vet, go test (incl. new TestCrypto* suite + the
existing TestParseEpisode/TestCleanQuery/TestSrtToVTT/TestStripASSTags/
TestBuildFFmpegArgs); tsc -b && vite build emits 28 route chunks plus
the deferred hls chunk; main bundle 253 KB / 85 KB gzipped; smoke test
PASS=42 / FAIL=0.
2026-05-15 10:58:32 +00:00