Commit Graph

144 Commits

Author SHA1 Message Date
ShukeBta eec2ee7fe3 Merge remote-tracking branch 'origin/main' into fix/organize-pipeline-cleanup 2026-06-15 18:19:24 +08:00
ShukeBta a5bf4bfdd4 Improve large library storage backends and scans 2026-06-15 18:16:34 +08:00
ShukeBta c9e6adf041 Support Chinese retention rule edits 2026-06-15 02:21:25 +08:00
ShukeBta ea44f21fcf Unify Telegram bot retention rules 2026-06-15 02:11:46 +08:00
ShukeBta 03be4df6e2 Hide duplicate Telegram bot command aliases 2026-06-15 01:43:24 +08:00
ShukeBta 9b9f8eae02 Adapt Sakura-style bot management commands 2026-06-15 01:21:45 +08:00
ShukeBta 406b311a8e Fix login stalls and bot redemption routing 2026-06-15 01:03:07 +08:00
ShukeBta 904701f29f Fix login stalls and bot redemption routing 2026-06-15 01:00:05 +08:00
ShukeBta 479a06f008 Unify organize ingest pipeline 2026-06-14 23:17:52 +08:00
ShukeBta a623702276 Deduplicate organize by multi-source metadata 2026-06-14 01:25:44 +08:00
ShukeBta 4a3ed77395 Fix organize metadata matching and Telegram group privacy 2026-06-14 01:12:55 +08:00
ShukeBta c76c72966a Fix organize targets and cloud transfer controls 2026-06-14 00:27:39 +08:00
ShukeBta 52b772ea45 Fix cloud playback and organize ingest pipeline 2026-06-13 23:21:56 +08:00
ShukeBta 4b2839d753 prefer openlist api credentials 2026-06-13 20:28:23 +08:00
ShukeBta b2b43e58da enforce pure cloud 302 playback 2026-06-13 17:45:10 +08:00
ShukeBta 5f42836bbf fix organize trigger task flow 2026-06-13 17:08:56 +08:00
ShukeBta 4e0929bbca fix organize path mapping and task flow 2026-06-13 16:32:38 +08:00
ShukeBta ce1b234b17 docs: simplify deployment docs and restrict telegram group commands 2026-06-13 14:24:50 +08:00
ShukeBta a177dc61dd fix emby clients and cloud playback controls 2026-06-13 12:08:57 +08:00
ShukeBta ba43117a57 fix: honor STRM playback mode for cloud clients 2026-06-13 00:17:39 +08:00
ShukeBta 5d942f2c42 fix: stabilize STRM and cloud playback 2026-06-12 22:57:50 +08:00
ShukeBta d270e9034f fix: 防止启动整理扫描阻塞登录 2026-06-12 19:06:47 +08:00
ShukeBta 1568ae127a perf: 重构 FTS 索引为 rowid 寻址+触发器维护,根治重启 CPU 占满与扫描卡死
- FTS5 普通列(含 UNINDEXED)不支持索引查找,旧版按 media_id 做
  NOT EXISTS/DELETE 全是整表扫描:启动回填 O(N^2) 烧 CPU 数小时,
  扫描时每个 upsert 一次全表扫描,均隔着全局写锁拖死登录(125s 超时)
- v2 布局:FTS rowid 与 media.rowid 对齐,由 INSERT/UPDATE/DELETE
  触发器实时维护;回填只剩 rowid 点查兜底,且去掉 ORDER BY
- 顺带修复刮削直写 Updates() 后新标题搜不到的问题(触发器覆盖)
- 写门改为 context 感知:长写语句不再让登录请求无限期排队
- warmMediaSearchIndex 延迟 30s 错峰启动
- library_scan 周期任务跳过云盘库(由 cloud_sync 夜间窗口负责),
  首轮延迟从 15s 改为等满一个周期,重启不再立即全量扫描风暴
- prune 改为只取 id/path 并按 500 条批量删除,缩短写锁占用
2026-06-12 06:41:14 +00:00
ShukeBta 91df7521ce fix: 资源占用/登录稳定性/QB整理入库/第三方播放404 综合修复
资源占用(Docker 部署 CPU/内存长期居高):
- 云盘探测预算改为按尝试扣减,杜绝队列满时对每个文件反复入队
  并刷出数万条 WARN(实测日志 41165 条)
- 探测队列满时给文件挂 30 分钟退避 + 告警限速为每分钟一条
- 扫描时每个文件的海报/背景图由同步下载(单张最长 20s)改为
  后台预取队列,云盘大库扫描不再串行拉图数小时
- PlaybackInfo 的云盘 ffprobe 探测改异步(原同步最长 8s,
  既拖慢起播又放大云盘流量),带单飞去重
- 访问日志跳过 /api/health 与静态资源;logging.level/format
  配置真正生效(此前是死配置)

登录稳定性(经常登录报错):
- refresh token 未及时落库期间,刷新请求可识别「待落库令牌」,
  不再把用户踢回登录页;轮换/登出后取消后台补写,防止旧令牌复活

QB 下载整理入库:
- 新增 download.path_mappings 设置:自定义下载器→本程序路径映射
  (每行 客户端路径=本地路径),并复用 compose 环境变量映射规则
- 应用重启后补整理最近 24h 内完成的种子(此前重启即永久漏掉)
- 下载客户端初始化失败仍注册并惰性重连(容器启动顺序免疫)
- 硬链接跨文件系统(EXDEV)自动降级为复制,保种语义不变

第三方播放器 404:
- 播放处理器不再把所有错误吞成 404:媒体不存在→404,
  云盘解析失败/STRM 关闭→502+原因
- 存库的云盘播放 URL 规范化为相对路径,免疫扫描时固化的旧 host
- 云盘媒体 SupportsDirectPlay=false,强制走带鉴权的 DirectStream
2026-06-12 04:31:34 +00:00
ShukeBta a13aba1d9f fix: validate downloader RPC endpoints 2026-06-11 22:58:06 +08:00
ShukeBta 7d0afd0fcb fix: reduce sqlite write pressure during scans 2026-06-11 21:45:53 +08:00
ShukeBta 52d75171b2 fix: allow login during sqlite write pressure 2026-06-11 20:06:55 +08:00
ShukeBta 7274ea19b7 fix: cap CPU and avoid heavy startup work 2026-06-11 19:48:00 +08:00
ShukeBta d9f87c61b6 fix: keep login responsive during background organize 2026-06-11 19:28:49 +08:00
ShukeBta 2ad9606ad5 fix full library display and cloud search visibility 2026-06-11 18:50:05 +08:00
ShukeBta 23d1fbfddb fix cloud library scale and playback compatibility
Root cause: display-only cloud library filtering was reused by scan jobs, merged cloud mounts could be skipped, and OpenList listing relied on WebDAV/first-page behavior that could cap huge directories around 100 items. Cloud scans also let new items consume probe budget before existing rows with missing track/artwork metadata.\n\nChanges:\n- split display filtering from scannable cloud filtering so merged cloud mounts still scan\n- add OpenList API pagination with WebDAV fallback\n- prioritize existing cloud media missing metadata before new imports\n- restrict automatic cloud sync to one successful 19:00-21:00 daily window and keep manual scan immediate\n- disable startup cloud full-scan by default, with an explicit opt-in setting\n- improve Emby/cloud playback compatibility and cache/search/test coverage from the continued work
2026-06-11 18:31:01 +08:00
ShukeBta c22d8271ae fix: use Get instead of FindByID to lookup storage config by type
The scanner was incorrectly using FindByID(mount.Provider) where mount.Provider
is a type string like 'openlist', not a database ID. This caused all cloud
library scans to fail with 'storage config not found'.

Changed to use Get(ctx, mount.Provider) which correctly looks up by type field.

Fixes cloud library scanning for all providers (openlist/quark/115/clouddrive2)
2026-06-11 13:21:37 +08:00
ShukeBta 7ef4af6287 feat: add cloud storage health check on startup
- Validate all enabled cloud storage configs at boot
- Ping each cloud provider to ensure connectivity
- Log health status for better diagnostics
- Help identify cloud storage configuration issues early
2026-06-11 12:54:26 +08:00
ShukeBta 95794f7bbc fix: add storage deletion with library cleanup and scan validation
- Delete storage config now properly removes related libraries and media
- Scanner validates storage config exists before starting cloud scan
- Prevents orphaned scans when storage is removed
- Added repository methods: DeleteByType, FindByID for storage config
- Remove unused storageHandler route to fix compilation
2026-06-11 12:48:32 +08:00
ShukeBta e99d70b017 feat: improve cloud media playback and Emby compatibility
Based on nowen-video and MoviePilot implementations:
- Mark STRM media as IsRemote=true in Emby MediaSource
- Improve cloud media identification using STRMURL field
- Clean up unused cloud play service scaffolding
- Ensure third-party players (Infuse/Emby) recognize cloud media correctly
2026-06-11 12:17:14 +08:00
ShukeBta 39144d70ba fix: translate qBittorrent container paths for auto-organize
- Add path translation for cross-container download paths
- Map common qBittorrent container paths to /downloads
- Fixes 'payload path is not accessible' warnings in logs
- Enables auto-organize to work when qBittorrent runs in separate container
2026-06-11 11:38:08 +08:00
ShukeBta 62b204367c fix: auto-scan cloud libraries on boot + allow CORS for media playback
- Add BootCloudLibraries() to auto-scan all cloud libraries on startup
- Delay 3s to avoid conflict with system init, scan without auto-scrape
- Enable CORS for /api/cloud/play/* and /api/img to support 3rd-party players
- Fixes issue where each user triggers separate cloud library scans
- Fixes issue where Infuse/Emby apps cannot play cloud resources
2026-06-11 11:25:43 +08:00
ShukeBta d90b58ba22 fix: cache cloud artwork before library import 2026-06-11 00:54:29 +08:00
ShukeBta 542e85a067 fix: reduce emby load and stabilize strm scan 2026-06-11 00:18:51 +08:00
ShukeBta 859d6e9d24 implement automatic strm generation 2026-06-10 21:36:55 +08:00
ShukeBta fd0428ee7d fix cloud library mounts and artwork caching 2026-06-10 20:43:42 +08:00
ShukeBta 0bf983cec8 fix: improve cloud mount scanning feedback 2026-06-10 18:46:30 +08:00
ShukeBta 6a892d6549 fix: handle OpenList unicode mount paths 2026-06-10 12:47:39 +08:00
ShukeBta 0f82cb3084 feat: improve OpenList cloud storage support 2026-06-10 00:08:35 +08:00
ShukeBta eab8c3c709 feat: add clouddrive2 storage bridge 2026-06-09 20:22:00 +08:00
ShukeBta bfb0243712 fix: limit ffprobe concurrency 2026-06-09 19:18:21 +08:00
ShukeBta 7d7f3cc758 feat: add cloud transfer and cache optimizations 2026-06-09 19:03:28 +08:00
ShukeBta a55fc502ca feat: add configurable scraper throttling 2026-06-09 17:56:25 +08:00
ShukeBta 76fe0dbde9 feat: scrape metadata after organize 2026-06-09 17:00:24 +08:00
ShukeBta fe0e7a09ce fix: improve media file organization workflow 2026-06-09 13:57:34 +08:00