tcjlj/MeBox
1
0
Fork 0
mirror of https://github.com/truewhile/MeBox.git synced 2026-10-01 03:56:38 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(reader): legado 书源规则引擎 Go 移植 + 阅读 P0 后端

Browse Source 
- internal/service/reader/rule/：逐方法移植 legado analyzeRule 包
  （AnalyzeRule/AnalyzeByJSoup/AnalyzeByJSonPath/AnalyzeByXPath/
  AnalyzeByRegex/AnalyzeUrl/RuleAnalyzer），JS 规则留 P2 接入点
- 书源导入（JSON数组/对象/Base64/URL）、多源聚合搜索（legado 四档
  排序合并）、详情/目录/正文（nextContentUrl 翻页合并）
- 数据模型 4 表注册迁移；/api/reader/* 路由组（书源/搜索/书架/进度/调试）
- 单测 + httptest 全链路端到端测试
- docs/reader-ui-spec.md：legado UI 交互仿制规格（供 P1 前端使用）
This commit is contained in:
truewhile
2026-09-30 15:41:58 +08:00
parent c84f2604ed
commit b8e6418326
28 changed files with 5736 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reader-plan.md
+10 -9
View File
@@ -2,6 +2,9 @@
> 分支：`feature/reading`
> 目标：在首页增加「影视 / 阅读」模式切换，阅读模式完整兼容阅读 3.0（legado）书源体系，覆盖 **文本（bookSourceType=0）、音频（=1）、漫画/图片（=2）** 三类源。
> 实现方式（用户明确要求）：**样式与逻辑全部仿造 refgd/legado 本体**，不参考其他重实现项目；相当于用 Go + React 18 + TypeScript 5 重写该项目。
> 界面与交互的唯一规格：`docs/reader-ui-spec.md`（从 legado 源码逐屏调研产出）。
> 规则引擎的唯一语义基准：legado `app/src/main/java/io/legado/app/model/analyzeRule/` 源码，Go 侧逐方法移植对拍（源码克隆在 `C:\MyProject\_ref\legado`，仅作对照，不进入构建）。
## 1. 范围
@@ -114,12 +117,12 @@ web/src/
| 阶段 | 内容 | 交付物 |
|---|---|---|
| P0 引擎地基 | 规则引擎核心（四分析器 + 规则拆分/组合/变量）+ AnalyzeUrl v1（GET/POST/charset/headers/变量）+ 表结构 + 书源导入/管理 API | 冒烟工具可跑，非 JS 规则单测通过 |
| P1 文本源全链路 + 首页切换 | 搜索聚合（WS 进度）/详情/目录/正文（nextContentUrl 合并、缓存）+ 书架/进度 API；前端首页切换、书架、搜索、详情、文本阅读器 v1 | 用纯规则型文本源完成「搜书→加入→阅读」全流程 |
| P2 JS 与兼容率爬坡 | goja 接入 + `java.*` 桥分批实现 + 加解密族 + URL 完整选项 + replaceRegex + 替换规则管理 + 书源调试页 + 冒烟指标报告 | 主流公开文本源通过率显著提升，形成回归基线 |
| P3 音频源 | 播放列表解析、音频代理（带 UA/Referer）、音频播放器页、进度记忆 | 音频源可听 |
| P4 漫画/图片源 | 图片列表解析（含翻页）、图片代理接入磁盘缓存、漫画阅读器双模式、预加载 | 漫画源可看 |
| P5 体验完善 | 换源、追更（定时刷新目录 + 缓存清理）、发现页、阅读器高级设置、书源备份导出；可选：本地 TXT/EPUB | 完整体验 |
| P0 引擎地基 ✅ | 规则引擎核心（四分析器 + 规则拆分/组合/变量）+ AnalyzeUrl v1（GET/POST/charset/headers/变量/页码模式）+ 表结构 + 书源导入/管理 API + 搜索/详情/目录/正文/书架/进度/调试 API | 已完成：`internal/service/reader/rule/`（规则引擎，~2800 行，对齐 AnalyzeRule/AnalyzeByJSoup/AnalyzeByJSonPath/AnalyzeByXPath/AnalyzeByRegex/AnalyzeUrl/RuleAnalyzer）+ 服务层 + `/api/reader/*` 路由 + 单测/端到端测试全绿 |
| P1 文本源全链路 + 首页切换 | 搜索聚合（WS 进度）/详情/目录/正文（nextContentUrl 合并、缓存）+ 前端首页切换、书架、搜索、详情、文本阅读器 v1（阅读器样式仿 legado：9 宫格点击、主题、翻页动画） | 用纯规则型文本源完成「搜书→加入→阅读」全流程 |
| P2 JS 与兼容率爬坡 | goja 接入 + `java.*` 桥分批实现 + 加解密族 + URL 完整选项 + replaceRegex + 替换规则管理 + 书源调试页（仿 legado 逐条日志流式输出）+ 冒烟指标报告 | 主流公开文本源通过率显著提升，形成回归基线 |
| P3 音频源 | 播放列表解析、音频代理（带 UA/Referer）、音频播放器页（仿 ReadAloudDialog 布局：上一章/播放/下一章/定时/倍速）、进度记忆 | 音频源可听 |
| P4 漫画/图片源 | 图片列表解析（含翻页）、图片代理接入磁盘缓存、漫画阅读器双模式（MangaMenu：顶栏+底部胶囊）、预加载 | 漫画源可看 |
| P5 体验完善 | 换源（ChangeBookSourceDialog 四档排序）、追更（定时刷新目录 + 缓存清理）、发现页（exploreUrl 标签条）、阅读器高级设置（页眉页脚提示、点击区域自定义）、书源编辑器六 Tab、备份导出；可选：本地 TXT/EPUB | 完整体验 |
P0–P2 是主体（约全部工作量 60–70%），P3/P4 相对独立可并行。
@@ -133,7 +136,5 @@ P0–P2 是主体（约全部工作量 60–70%），P3/P4 相对独立可并行
## 9. 参考
- 规则语义基准：https://github.com/gedoor/legado （analyzeRule 源码）
- Java 侧完整 Web 实现先例（语义参考与对拍对象）：https://github.com/hectorqin/reader
- **唯一语义与样式基准**：https://github.com/refgd/legado （规则引擎 analyzeRule 源码 + UI 布局/交互，见 docs/reader-ui-spec.md）
- 书源规则教程：https://mgz0227.github.io/The-tutorial-of-Legado/
- 需求来源 fork：https://github.com/refgd/legado
docs/reader-ui-spec.md
+196
View File
@@ -0,0 +1,196 @@
# legado UI/交互仿制规格（React 18 + Tailwind Web 版实现依据）
> 来源：对 `refgd/legado`（克隆于 `C:\MyProject\_ref\legado`）源码的调研。
> 路径缩写：`<src>` = `app/src/main/java/io/legado/app/`，布局在 `app/src/main/res/layout/`。
## 1. 首页/书架
文件：`<src>ui/main/bookshelf/`（BookshelfFragment、BooksAdapterGrid/List）、`fragment_bookshelf1/2.xml`、`item_bookshelf_grid*.xml`、`item_bookshelf_list*.xml`。主界面底部 BottomNavigation（书架/发现/我的）+ 右上悬浮搜索按钮。
**布局结构（从上到下）**
- style1（标签页风格）：标题行（书架标题 + 下拉箭头，点开分组切换菜单）→ 搜索按钮 + 更多按钮 → 一级分组 tab 条（tab 长按可删分组）→ 次级分组扩展标签 → ViewPager（每 tab 一个书籍列表/网格）。
- style2（单 RecyclerView 混排）：title_bar → SwipeRefresh → 书籍列表（根分组时「分组网格项 + 书籍项」混排，进入分组只显示书籍）→ 空态文案。
- 网格项：封面 + 本地角标 + 未读数角标 + 加载中 + 书名 + 长按遮罩。
- 列表项：封面 + 本地图标 + 有更新标记 + 未读数 + 书名 + 作者 + 最后更新时间 + 当前读到（章节名）+ 最新章节（章节名）+ 长按遮罩。
**交互逻辑**
- 内置分组 id：全部(IdRoot=-100)、本地(IdLocal=-2)、未分组(IdUngrouped=-4)，其余用户分组。
- 单击书 → 直接进入阅读；单击分组 → 进入该分组。
- 长按书 → 跳转书籍详情页；长按分组 → 重命名/删除对话框；style1 长按 tab 删除分组。
- 右上菜单：搜索、Wi-Fi 传书、刷新目录、书架布局切换、分组管理、导出/导入书架、下载离线、本地导入、网址添加、日志。
## 2. 阅读界面（重点）
文件：`<src>ui/book/read/`（ReadBookActivity、ReadMenu、SearchMenu、MangaMenu、config/*Dialog）、`ui/book/read/page/`（ReadView、PageView、ContentTextView、ChapterProvider）。
**布局结构（单 FrameLayout 栈）**：read_view 正文 → 文字选择光标 → read_menu 主菜单 → search_menu 全文搜索 → 朗读回原文浮条。
**正文渲染**
- ReadView 内含 3 个 PageView（prev/cur/next 缓存），ContentTextView 纯 Canvas 自绘：按字号/行距/段距/缩进/两端对齐排版，Web 版用 CSS multi-column 或 JS 分页等效实现。
- 翻页动画（0–5）：0 覆盖、1 平移、2 仿真、3 上下滚动、4 无、5 平移覆盖；速度可调（默认 300ms）。
- 滚动模式支持背景跟随、自动翻页（定时滚动/定时翻页）。
- 图片/漫画书切换独立配置集 + MangaMenu：顶栏 + 底部胶囊（上一页/页码进度条/下一页），支持横滚、缩放。
**点击区域（9 宫格，边缘留 pageTouchClick px）**
- 区块：左上/中上/右上/左中/中/右中/左下/中下/右下。
- 默认：左列与上中=上一页；右列与下中=下一页；正中=呼出菜单。
- 全部可配动作：0 呼出菜单、1 下一页、2 上一页、3 下一章、4 上一章、7 书签、10 目录、11 搜索、12 同步进度等。
**主菜单 ReadMenu（自上而下）**
- 半透明遮罩（点击收起）。
- 顶部 title_bar：返回 + 书名（点击进详情）+ 章节名/章节链接（点击打开原页面）+ 书源按钮（弹菜单：编辑书源/禁用书源等）。
- 底部 bottom_menu（圆角面板）：上一章 | 进度条 | 下一章 → 亮度条 + 自动亮度 → 快捷按钮行（搜索、自动翻页、替换规则、夜间/日间切换）→ 动作面板行（目录、朗读、界面、设置）。
- 菜单配色可跟随页面背景或用主题底色，透明度 35–100。
**设置面板**
- 「界面」三 Tab：文本（字体文件、字重、字号 5–50、字距、行距 0–20、段距、缩进、下划线、阴影）；样式（白天/夜间/E-Ink 三态主题：文字色、背景色/图、强调色、菜单底色、菜单透明度、背景透明度，可恢复/管理/分享）；页面（6 种翻页动画、状态栏深浅、滚动背景跟随、四边距、页眉页脚提示、简繁转换）。
- 「设置」：屏幕方向、保持亮屏、隐藏状态栏、两端/底部对齐、竖排、双页、进度条行为（按页/按章）、音量键/滚轮翻页、点击区域配置、动画速度、自动换源等。
- 页眉页脚提示项：无/章节名/时间/电量/电量百分比/页码/总进度/页码+总页/时间+电量/书名；页眉默认 左=时间/右=电量，页脚默认 左=章节名/右=页码/总页。
**内置主题与配色（assets/defaultData/readConfig.json，6 套 + 自定义）**
| 名称 | 背景(日/夜) | 文字(日/夜) | 强调(日/夜) | 字号 |
|---|---|---|---|---|
| 微信读书 | #FFC0EDC6 / #000000 | #FF0B0B0B / #ADADAD | #E53935 / #FE4D55 | 24 |
| 预设1 | #FFFFFF / #000000 | #000000 / #FFFFFF | #E53935 / #FE4D55 | 20 |
| 预设2(羊皮纸) | #DDC090 / #3C3F43 | #3E3422 / #DCDFE1 | #834E00 / #FE4D55 | 20 |
| 预设3(护眼绿) | #C2D8AA / #3C3F43 | #596C44 / #88C16F | #E53935 / #FE4D55 | 20 |
| 预设4(粉紫) | #DBB8E2 / #3C3F43 | #68516C / #F6AEAE | #801314 / #90BFF5 | 20 |
| 预设5(淡蓝) | #ABCEE0 / #3C3F43 | #3D4C54 / #90BFF5 | #E53935 / #FE4D55 | 20 |
默认排版：textSize 20、letterSpacing 0.1、lineSpacingExtra 12、paragraphSpacing 2、缩进「　　」、padding 上下6/左右16、页脚线 true。未选样式时默认：bg #EEEEEE / 夜 #000000 / E-Ink #FFFFFF，文字 #3E3D3B / 夜 #ADADAD，强调 #E53935 / 夜 #FE4D55。
**音频书播放条（ReadAloudDialog 底部弹层）**
- transport 行：上一章 | 上一个/播放暂停/停止/下一个 | 下一章。
- 定时面板：定时关闭 + 进度条；TTS 语速面板（跟随系统 + 减/加 + 语速条）。
- 底部动作行：目录、主菜单、后台播放、设置。
- 音频播放参数存 Book.readConfig：playMode(0 顺序)、playSpeed(1.0)、openCredits/closeCredits(片头片尾章数)。
## 3. 搜索
文件：`<src>ui/book/search/` + `<src>model/webBook/SearchModel.kt`。
- 布局：顶部 SearchView + 转圈进度条 → 结果流 → 书架命中提示卡 → 搜索历史 chips + 清空 → 右下 开始/停止 悬浮按钮。
- 结果项：封面、在书架角标、书源数角标（可换源数）、书名、作者、分类标签、最新章节、简介。
- 多源并发：所有启用书源（按搜索范围过滤），固定线程池并发（全局设置 threadCount），单源 30s 超时。
- 实时结果流：每源返回即刷新列表；结束后 onSearchFinish(isEmpty, hasMore)（任一源有结果即可翻页，searchPage++）。
- 聚合去重 mergeItems 四档：书名或作者等于关键词 > kind 含关键词 > 书名/作者含关键词 > 其他；同名同作者合并为一项并 addOrigin（角标显示可换源数）；档内按书源数降序。
- 顶部菜单：精准搜索开关、搜索范围（分组/全选）、书源管理、日志。点结果 → 换源对话框/直接打开。
## 4. 发现
`<src>ui/main/explore/` + `ui/book/explore/`。布局：标题栏 → SwipeRefresh → 源选择行（当前源名 + 下拉、源内搜索、标签筛选、更多）→ 二级筛选条 → 分类标签条（横滑 chip，可展开）→ 书籍网格（封面+书名）→ 兜底列表。数据来自启用书源的 exploreUrl + ruleExplore；切源刷新标签；点标签拼 URL 加载、分页加载更多。
## 5. 书籍详情
`<src>ui/book/info/BookInfoActivity.kt`。布局：模糊封面背景 → 标题栏（返回/刷新/菜单）→ 下拉刷新 → 滚动区：封面 + 书名 + 作者 + 最新章节（点击刷新目录）+ 阅读时长 + 分类标签 + 分组行 + 书源行（换源）+ 目录入口行 → 双 Tab（简介 / 目录预览，可全屏、点章节跳读）→ 底部动作区：tv_shelf 加入/移出书架（描边按钮）+ tv_read 开始阅读/继续阅读（实心主色按钮）。菜单含置顶、去书源网页、登录、删除（勾选删缓存）、分享。
## 6. 目录页
`<src>ui/book/toc/`。布局：标题栏（返回 + 书名 + SearchView 搜章节）→ Tab（章节/书签）→ 章节 RecyclerView + 底部信息条（当前位置 + 跳顶/跳底）。菜单：倒序开关、使用净化替换、加载字数、长章分卷合并、TXT 目录规则。倒序即列表反转；卷名行可折叠；已读章节变色；点击章节回传 index 跳读。
## 7. 书源管理
- **主列表**：标题栏（SearchView 过滤 + 菜单）→ RecyclerView（item：域名、复选、启停 Switch、编辑、更多菜单、发现、快速调试、响应耗时条）→ 多选操作栏。菜单：排序（手动/自动/名称/URL/更新时间/响应时间/启用）、分组筛选（启用/禁用/需登录/无分组/启用发现/禁用发现）、添加书源（本地/网络/二维码）。
- **编辑**：标题栏（保存/调试入口）→ 类型下拉 + 六个开关（启用/启用发现/CookieJar/段评/事件监听/自定义按钮）→ 六个 Tab：基本/搜索/发现/详情/目录/正文，每 Tab 为「标签 + 输入框 + 帮助弹层」表单。
- **调试**：顶部 SearchView 输入关键词 → 串行执行 搜索→详情→目录→正文，日志逐条流式显示；可查看各阶段原始 HTML、切换探索分类调试。
## 8. 替换规则
- 列表页：标题栏（搜索 + 菜单：分组管理、启停筛选、添加、导入）→ RecyclerView（名称、分组、内容摘要、启停 Switch）→ 多选操作栏。
- 编辑页字段：规则名 → 分组 → 替换规则（+ 正则开关 + 帮助）→ 替换为 → 作用于标题 / 作用于正文 → 作用范围(书名) → 排除范围 → 超时毫秒（order 隐含）。
## 9. 数据实体字段全集
**BookSource.kt**（rule* 为内嵌对象）
| 字段 | 类型 | 默认值 |
|---|---|---|
| bookSourceUrl | String | "" |
| bookSourceName | String | "" |
| bookSourceGroup | String? | null |
| bookSourceType | Int (0文本/1音频/2图片/3文件/4视频) | 0 |
| bookUrlPattern | String? | null |
| customOrder | Int | 0 |
| enabled | Boolean | true |
| jsLib | String? | null |
| enabledExplore | Boolean | true |
| enabledCookieJar | Boolean? | true |
| concurrentRate | String? | null |
| header | String? | null |
| loginUrl / loginUi / loginCheckJs / coverDecodeJs | String? | null |
| bookSourceComment | String? | null |
| variableComment | String? | null |
| lastUpdateTime | Long | 0 |
| respondTime | Long | 180000 |
| weight | Int | 0 |
| exploreUrl / exploreScreen | String? | null |
| ruleExplore / ruleSearch / ruleBookInfo / ruleToc / ruleContent / ruleReview | 对象 | null |
| eventListener / customButton | Boolean | false |
**Book.kt**
| 字段 | 类型 | 默认值 |
|---|---|---|
| bookUrl (PK) / tocUrl | String | "" / "" |
| origin / originName | String | localTag / "" |
| name / author | String | "" / "" |
| kind / customTag / coverUrl / customCoverUrl / intro / customIntro / charset | String? | null |
| type | Int (BookType) | text |
| group | Long | 0 |
| latestChapterTitle | String? | null |
| latestChapterTime / lastCheckTime | Long | now |
| lastCheckCount / totalChapterNum | Int | 0 |
| durChapterTitle | String? | null |
| durChapterIndex / durVolumeIndex / chapterInVolumeIndex / durChapterPos | Int | 0 |
| durChapterTime | Long | now |
| wordCount | String? | null |
| canUpdate | Boolean | true |
| order / originOrder | Int | 0 |
| variable | String? | null |
| readConfig（ReadConfig 内嵌） | reverseToc、pageAnim、reSegment、imageStyle、useReplaceRule、delTag、ttsEngine、splitLongChapter=true、readSimulating、startDate、startChapter、dailyChapters=3、openCredits=0、closeCredits=0、playMode=0、playSpeed=1.0、manga* 系列、mangaPageAnim | |
**BookChapter.kt**
| 字段 | 类型 | 默认值 |
|---|---|---|
| url / title | String | "" / "" |
| isVolume | Boolean | false |
| baseUrl / bookUrl | String | "" / "" |
| index | Int | 0 |
| isVip / isPay | Boolean | false |
| resourceUrl / tag / wordCount | String? | null |
| start / end | Long? | null |
| startFragmentId / endFragmentId / variable / imgUrl | String? | null |
**ReplaceRule.kt**
| 字段 | 类型 | 默认值 |
|---|---|---|
| id | Long | now |
| name | String | "" |
| group | String? | null |
| pattern | String | "" |
| replacement | String | "" |
| scope | String? | null |
| scopeTitle | Boolean | false |
| scopeContent | Boolean | true |
| excludeScope | String? | null |
| isEnabled | Boolean | true |
| isRegex | Boolean | true |
| timeoutMillisecond | Long | 3000 |
| order | Int | Int.MIN_VALUE |
## 10. 主题与视觉
- 全局：主背景 `#F7F7FA`、主文字 `#DE000000`、品牌色 `#FFF6FBF8`（浅绿白调）。
- 界面主题由 ThemeConfig（themes.json）控制 6 套；阅读菜单/底栏用 bottomBackground（跟随所选界面主题），沉浸模式（readBarStyleFollowPage）直接取页面背景/文字色。
- 强调色 accentColor 用于选中态、光标、Tab 选中；E-Ink 模式菜单改描边。
- 视觉风格：大圆角面板、1dp 描边、玻璃拟态半透明条、图标+文字标签的动作面板。
## Web 版实现要点
- 正文分页仿 ReadView 三页缓存 + 9 宫格点击映射。
- 主题模型照抄 ReadBookConfig.Config（含日/夜/E-Ink 三态字段），内置 6 套主题色值直接使用上表。
- 搜索并发用 Promise 池 + 流式合并，合并逻辑照抄 mergeItems 四档排序。
- 书源/替换编辑器按第 7/8 节 Tab/字段一一对应。
go.mod
+13 -5
View File
@@ -1,6 +1,6 @@
module github.com/truewhile/MeBox
go 1.25.0
go 1.26.0
require (
fyne.io/systray v1.12.2
@@ -19,9 +19,9 @@ require (
github.com/stretchr/testify v1.11.1
github.com/ulikunitz/xz v0.5.15
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.49.0
golang.org/x/crypto v0.55.0
golang.org/x/image v0.45.0
golang.org/x/sync v0.22.0
golang.org/x/sync v0.23.0
golang.org/x/sys v0.47.0
golang.org/x/time v0.15.0
gopkg.in/yaml.v3 v3.0.1
@@ -30,6 +30,12 @@ require (
)
require (
github.com/PaesslerAG/gval v1.0.0 // indirect
github.com/PaesslerAG/jsonpath v0.1.1 // indirect
github.com/PuerkitoBio/goquery v1.13.0 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/antchfx/htmlquery v1.3.6 // indirect
github.com/antchfx/xpath v1.3.6 // indirect
github.com/bytedance/gopkg v0.1.4 // indirect
github.com/bytedance/sonic v1.15.0 // indirect
github.com/bytedance/sonic/loader v0.5.0 // indirect
@@ -38,6 +44,7 @@ require (
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/disintegration/imaging v1.6.3-0.20201218193011-d40f48ce0f09 // indirect
github.com/dlclark/regexp2 v1.12.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/esimov/pigo v1.4.7-0.20240801095032-7465ed14de47 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
@@ -50,6 +57,7 @@ require (
github.com/goccy/go-json v0.10.6 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/godbus/dbus/v5 v5.1.0 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
@@ -90,8 +98,8 @@ require (
go.uber.org/multierr v1.10.0 // indirect
golang.org/x/arch v0.25.0 // indirect
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/net v0.52.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/text v0.42.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
modernc.org/libc v1.70.0 // indirect
go.sum
+91
View File
@@ -1,7 +1,20 @@
fyne.io/systray v1.12.2 h1:Y8DZxgLHsVQt6rY9Zrkkg+j67S7vv/1F2viOWKPpVeA=
fyne.io/systray v1.12.2/go.mod h1:RVwqP9nYMo7h5zViCBHri2FgjXF7H2cub7MAq4NSoLs=
github.com/PaesslerAG/gval v1.0.0 h1:GEKnRwkWDdf9dOmKcNrar9EA1bz1z9DqPIO1+iLzhd8=
github.com/PaesslerAG/gval v1.0.0/go.mod h1:y/nm5yEyTeX6av0OfKJNp9rBNj2XrGhAf5+v24IBN1I=
github.com/PaesslerAG/jsonpath v0.1.0/go.mod h1:4BzmtoM/PI8fPO4aQGIusjGxGir2BzcV0grWtFzq1Y8=
github.com/PaesslerAG/jsonpath v0.1.1 h1:c1/AToHQMVsduPAa4Vh6xp2U0evy4t8SWp8imEsylIk=
github.com/PaesslerAG/jsonpath v0.1.1/go.mod h1:lVboNxFGal/VwW6d9JzIy56bUsYAP6tH/x80vjnCseY=
github.com/PuerkitoBio/goquery v1.13.0 h1:mqHbjD7Jmnul4DTR24LKTjo1uUmHUh072kteGV+xpFM=
github.com/PuerkitoBio/goquery v1.13.0/go.mod h1:Hip5mdBL8K2wEGKJdr27sRaNwIdDajmCwB/ExUPwW+g=
github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.5.2 h1:40yUSXwdkWN851BHCq6uiDhleh7A4+0yIBS+IUAqZVY=
github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.5.2/go.mod h1:FTzydeQVmR24FI0D6XWUOMKckjXehM/jgMn1xC+DA9M=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/antchfx/htmlquery v1.3.6 h1:RNHHL7YehO5XdO8IM8CynwLKONwRHWkrghbYhQIk9ag=
github.com/antchfx/htmlquery v1.3.6/go.mod h1:kcVUqancxPygm26X2rceEcagZFFVkLEE7xgLkGSDl/4=
github.com/antchfx/xpath v1.3.6 h1:s0y+ElRRtTQdfHP609qFu0+c6bglDv20pqOViQjjdPI=
github.com/antchfx/xpath v1.3.6/go.mod h1:i54GszH55fYfBmoZXapTHN8T8tkcHfRgLyVwwqzXNcs=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
@@ -24,6 +37,8 @@ github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/r
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/disintegration/imaging v1.6.3-0.20201218193011-d40f48ce0f09 h1:MJFqtdxTq94XqUgg7DcGCaOIXrDTJE/tPHK66Jshguc=
github.com/disintegration/imaging v1.6.3-0.20201218193011-d40f48ce0f09/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
github.com/dlclark/regexp2 v1.12.0 h1:0j4c5qQmnC6XOWNjP3PIXURXN2gWx76rd3KvgdPkCz8=
github.com/dlclark/regexp2 v1.12.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/esimov/pigo v1.4.7-0.20240801095032-7465ed14de47 h1:48iGRx9HamDuG4pCbPG5IXt4bKHhgn33KGynzHUgeIA=
@@ -62,7 +77,11 @@ github.com/godbus/dbus/v5 v5.1.0 h1:4KLkAxT3aOY8Li4FRJe/KvhoNFFxo0m6fNuFUO8QJUk=
github.com/godbus/dbus/v5 v5.1.0/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8/go.mod h1:wcDNUvekVysuuOpQKo3191zZyTpiI6se1N1ULghS0sw=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
@@ -177,6 +196,7 @@ github.com/ugorji/go/codec v1.3.1 h1:waO7eEiFDwidsBN6agj1vJQ4AG7lh2yqXyOXqhgQuyY
github.com/ugorji/go/codec v1.3.1/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4=
github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=
github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.mongodb.org/mongo-driver/v2 v2.5.0 h1:yXUhImUjjAInNcpTcAlPHiT7bIXhshCTL3jVBkF3xaE=
@@ -191,33 +211,104 @@ go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
golang.org/x/arch v0.25.0 h1:qnk6Ksugpi5Bz32947rkUgDt9/s5qvqDPl/gBKdMJLE=
golang.org/x/arch v0.25.0/go.mod h1:0X+GdSIP+kL5wPmpK7sdkEVTt2XoYP0cSjQSbZBwOi8=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.49.0 h1:+Ng2ULVvLHnJ/ZFEq4KdcDd/cfjrrjjNSXNzxg0Y4U4=
golang.org/x/crypto v0.49.0/go.mod h1:ErX4dUh2UM+CFYiXZRTcMpEcN8b/1gxEuv3nODoYtCA=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.45.0 h1:FMb1nTbH5H9vF55SriQHgFw5GnNL9Jg6L25BwXKzhB0=
golang.org/x/image v0.45.0/go.mod h1:n62x/7RqlwXDvGsSU4u6IUTUf6KghUZ9Bt7cG/T9Fx4=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
internal/handler/reader_routes.go
+281
View File
@@ -0,0 +1,281 @@
// Package handler — 阅读（legado 书源兼容）子系统路由。
package handler
import (
"net/http"
"github.com/gin-gonic/gin"
"github.com/truewhile/MeBox/internal/middleware"
"github.com/truewhile/MeBox/internal/service"
"github.com/truewhile/MeBox/internal/service/reader"
)
func registerReaderRoutes(authed *gin.RouterGroup, svc *service.Container) {
g := authed.Group("/reader")
// 书源管理
g.GET("/sources", readerListSourcesHandler(svc))
g.POST("/sources/import", readerImportSourcesHandler(svc))
g.PATCH("/sources/:id", readerUpdateSourceHandler(svc))
g.DELETE("/sources/:id", readerDeleteSourceHandler(svc))
g.POST("/sources/:id/debug", readerDebugSourceHandler(svc))
// 搜索（多源聚合）
g.POST("/search", readerSearchHandler(svc))
// 详情 / 目录 / 正文
g.GET("/book-info", readerBookInfoHandler(svc))
g.GET("/toc", readerTocHandler(svc))
g.GET("/content", readerContentHandler(svc))
// 书架
g.GET("/books", readerListBooksHandler(svc))
g.POST("/books", readerAddBookHandler(svc))
g.DELETE("/books/:id", readerRemoveBookHandler(svc))
g.PUT("/books/:id/progress", readerSaveProgressHandler(svc))
g.GET("/books/:id/chapters", readerListChaptersHandler(svc))
g.POST("/books/:id/chapters", readerReplaceChaptersHandler(svc))
// 替换净化规则
g.GET("/replace-rules", readerListReplaceRulesHandler(svc))
}
func readerListSourcesHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
sources, err := svc.Reader.ListSources(c.Request.Context())
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"sources": sources})
}
}
func readerImportSourcesHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Text string `json:"text" binding:"required"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
imported, err := svc.Reader.ImportSources(c.Request.Context(), body.Text)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"imported": imported})
}
}
func readerUpdateSourceHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Enabled *bool `json:"enabled"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil || body.Enabled == nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "enabled 字段必填"})
return
}
if err := svc.Reader.UpdateSourceEnabled(c.Request.Context(), c.Param("id"), *body.Enabled); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"ok": true})
}
}
func readerDeleteSourceHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
if err := svc.Reader.DeleteSource(c.Request.Context(), c.Param("id")); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"ok": true})
}
}
func readerDebugSourceHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Key string `json:"key" binding:"required"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
logs, err := svc.Reader.Debug(c.Request.Context(), c.Param("id"), body.Key)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"logs": logs})
}
}
func readerSearchHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Key string `json:"key" binding:"required"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
books, skipped, err := svc.Reader.Search(c.Request.Context(), body.Key)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"books": books, "skipped": skipped})
}
}
func readerBookInfoHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
info, err := svc.Reader.GetBookInfo(
c.Request.Context(),
c.Query("source_id"), c.Query("book_url"),
)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, info)
}
}
func readerTocHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
chapters, err := svc.Reader.GetToc(
c.Request.Context(),
c.Query("source_id"), c.Query("book_url"), c.Query("toc_url"),
)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"chapters": chapters})
}
}
func readerContentHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
content, err := svc.Reader.GetContent(
c.Request.Context(),
c.Query("source_id"), c.Query("book_url"), c.Query("chapter_url"),
)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, content)
}
}
func readerListBooksHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
userID := c.GetString(middleware.CtxUserID)
books, err := svc.Reader.ListBooks(c.Request.Context(), userID)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"books": books})
}
}
func readerAddBookHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Origin reader.SearchOrigin `json:"origin" binding:"required"`
Name string `json:"name" binding:"required"`
Author string `json:"author"`
CoverURL string `json:"cover_url"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
userID := c.GetString(middleware.CtxUserID)
book, err := svc.Reader.AddBook(c.Request.Context(), userID, body.Origin, body.Name, body.Author, body.CoverURL)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, book)
}
}
func readerRemoveBookHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
userID := c.GetString(middleware.CtxUserID)
if err := svc.Reader.RemoveBook(c.Request.Context(), userID, c.Param("id")); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"ok": true})
}
}
func readerSaveProgressHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
ChapterIndex int `json:"chapter_index"`
Pos int `json:"pos"`
ChapterTitle string `json:"chapter_title"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
userID := c.GetString(middleware.CtxUserID)
if err := svc.Reader.SaveProgress(c.Request.Context(), userID, c.Param("id"), body.ChapterIndex, body.Pos, body.ChapterTitle); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"ok": true})
}
}
func readerListChaptersHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
chapters, err := svc.Reader.ListChapters(c.Request.Context(), c.Param("id"))
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"chapters": chapters})
}
}
func readerReplaceChaptersHandler(svc *service.Container) gin.HandlerFunc {
var body struct {
Chapters []reader.ChapterInput `json:"chapters"`
}
return func(c *gin.Context) {
if err := c.ShouldBindJSON(&body); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
if err := svc.Reader.SaveChapters(c.Request.Context(), c.Param("id"), body.Chapters); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"ok": true})
}
}
func readerListReplaceRulesHandler(svc *service.Container) gin.HandlerFunc {
return func(c *gin.Context) {
userID := c.GetString(middleware.CtxUserID)
rules, err := svc.Reader.ListReplaceRules(c.Request.Context(), userID)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"rules": rules})
}
}
internal/handler/routes_authenticated.go
+1
View File
@@ -30,4 +30,5 @@ func registerAuthenticatedRoutes(api *gin.RouterGroup, cfg *config.Config, svc *
registerAuthedDLNAControlRoutes(authed, svc)
registerAuthedFavoriteAndMediaActionRoutes(authed, svc)
registerAuthedPlaybackExtraRoutes(authed, svc)
registerReaderRoutes(authed, svc)
}
internal/model/model.go
+4
View File
@@ -61,5 +61,9 @@ func AllModels() []interface{} {
&StrmDirCache{},
&ScrapeTask{},
&EmbyMount{},
&ReaderBookSource{},
&ReaderBook{},
&ReaderChapter{},
&ReaderReplaceRule{},
}
}
internal/model/reader.go
+83
View File
@@ -0,0 +1,83 @@
// Package model — 阅读（legado 书源兼容）子系统数据模型。
// 字段语义对齐 legado 的 BookSource / Book / BookChapter / ReplaceRule 实体；
// 阅读进度沿用 legado 的做法直接挂在书籍上（durChapter* 字段）。
package model
import (
"time"
)
// ReaderBookSource 书源：原始 JSON 全量存储 + 常用字段冗余列出便于筛选排序。
type ReaderBookSource struct {
Base
Name string `gorm:"type:varchar(255);index" json:"name"`
GroupName string `gorm:"type:varchar(255);index" json:"group"`
Type int `gorm:"default:0" json:"type"` // 0文本 1音频 2图片 3文件 4视频
SourceURL string `gorm:"type:varchar(512);index" json:"source_url"`
RawJSON string `gorm:"type:text" json:"-"`
Enabled bool `gorm:"default:true" json:"enabled"`
EnabledExplore bool `gorm:"default:true" json:"enabled_explore"`
CustomOrder int `json:"custom_order"`
Weight int `json:"weight"`
ConcurrentRate string `gorm:"type:varchar(64)" json:"concurrent_rate"`
Header string `gorm:"type:text" json:"header"` // 书源级请求头 JSON
Comment string `gorm:"type:text" json:"comment"`
Variables string `gorm:"type:text" json:"variables"` // source 变量 JSON
LastUpdateTime int64 `json:"last_update_time"`
LastCheckAt *time.Time `json:"last_check_at"`
RespondTime int64 `json:"respond_time"` // 最近一次调试响应耗时（ms）
}
// ReaderBook 书架条目（含阅读进度，对应 legado Book）。
type ReaderBook struct {
Base
UserID string `gorm:"type:varchar(36);index" json:"user_id"`
Origin string `gorm:"type:varchar(512)" json:"origin"` // 书源 URL
OriginName string `gorm:"type:varchar(255)" json:"origin_name"`
BookURL string `gorm:"type:varchar(512);index" json:"book_url"`
TocURL string `gorm:"type:varchar(512)" json:"toc_url"`
Name string `gorm:"type:varchar(255)" json:"name"`
Author string `gorm:"type:varchar(255)" json:"author"`
Kind string `gorm:"type:varchar(255)" json:"kind"`
CoverURL string `gorm:"type:varchar(512)" json:"cover_url"`
Intro string `gorm:"type:text" json:"intro"`
Charset string `gorm:"type:varchar(32)" json:"charset"`
Type int `gorm:"default:0" json:"type"` // 0文本 1音频 2图片
LatestChapterTitle string `gorm:"type:varchar(512)" json:"latest_chapter_title"`
TotalChapterNum int `json:"total_chapter_num"`
DurChapterIndex int `json:"dur_chapter_index"`
DurChapterPos int `json:"dur_chapter_pos"`
DurChapterTitle string `gorm:"type:varchar(512)" json:"dur_chapter_title"`
DurChapterTime int64 `json:"dur_chapter_time"`
Order int `json:"order"`
Variable string `gorm:"type:text" json:"variable"`
}
// ReaderChapter 章节缓存（对应 legado BookChapter）。
type ReaderChapter struct {
Base
BookID string `gorm:"type:varchar(36);uniqueIndex:idx_reader_book_chapter" json:"book_id"`
Index int `gorm:"uniqueIndex:idx_reader_book_chapter" json:"index"`
URL string `gorm:"type:varchar(512)" json:"url"`
Title string `gorm:"type:varchar(512)" json:"title"`
IsVolume bool `json:"is_volume"`
Tag string `gorm:"type:varchar(255)" json:"tag"`
}
// ReaderReplaceRule 替换净化规则（对应 legado ReplaceRule）。
type ReaderReplaceRule struct {
Base
UserID string `gorm:"type:varchar(36);index" json:"user_id"`
Name string `gorm:"type:varchar(255)" json:"name"`
GroupName string `gorm:"type:varchar(255);index" json:"group"`
Pattern string `gorm:"type:text" json:"pattern"`
Replacement string `gorm:"type:text" json:"replacement"`
Scope string `gorm:"type:varchar(255)" json:"scope"`
ScopeTitle bool `json:"scope_title"`
ScopeContent bool `gorm:"default:true" json:"scope_content"`
ExcludeScope string `gorm:"type:varchar(255)" json:"exclude_scope"`
IsEnabled bool `gorm:"default:true" json:"is_enabled"`
IsRegex bool `gorm:"default:true" json:"is_regex"`
TimeoutMillisecond int64 `gorm:"default:3000" json:"timeout_millisecond"`
Order int `json:"order"`
}
internal/repository/reader_repository.go
+149
View File
@@ -0,0 +1,149 @@
// Package repository — 阅读子系统仓储（书源 / 书架 / 章节 / 替换规则）。
package repository
import (
"context"
"gorm.io/gorm"
"github.com/truewhile/MeBox/internal/model"
)
// ReaderRepository 阅读子系统 GORM 封装。
type ReaderRepository struct {
db *gorm.DB
}
// ListSources 书源列表（按 customOrder 排序）。
func (r *ReaderRepository) ListSources(ctx context.Context) ([]model.ReaderBookSource, error) {
var out []model.ReaderBookSource
err := r.db.WithContext(ctx).Order("custom_order ASC, updated_at DESC").Find(&out).Error
return out, err
}
// GetSource 按 ID 取书源。
func (r *ReaderRepository) GetSource(ctx context.Context, id string) (*model.ReaderBookSource, error) {
var out model.ReaderBookSource
if err := r.db.WithContext(ctx).First(&out, "id = ?", id).Error; err != nil {
return nil, err
}
return &out, nil
}
// GetSourceByURL 按书源 URL 取书源（导入去重用）。
func (r *ReaderRepository) GetSourceByURL(ctx context.Context, sourceURL string) (*model.ReaderBookSource, error) {
var out model.ReaderBookSource
err := r.db.WithContext(ctx).First(&out, "source_url = ?", sourceURL).Error
if err != nil {
return nil, err
}
return &out, nil
}
// CreateSource 新增书源。
func (r *ReaderRepository) CreateSource(ctx context.Context, src *model.ReaderBookSource) error {
return r.db.WithContext(ctx).Create(src).Error
}
// UpdateSource 更新书源字段。
func (r *ReaderRepository) UpdateSource(ctx context.Context, src *model.ReaderBookSource) error {
return r.db.WithContext(ctx).Save(src).Error
}
// DeleteSource 删除书源。
func (r *ReaderRepository) DeleteSource(ctx context.Context, id string) error {
return r.db.WithContext(ctx).Delete(&model.ReaderBookSource{}, "id = ?", id).Error
}
// ListBooks 用户书架（按 order 排序）。
func (r *ReaderRepository) ListBooks(ctx context.Context, userID string) ([]model.ReaderBook, error) {
var out []model.ReaderBook
err := r.db.WithContext(ctx).Where("user_id = ?", userID).Order("`order` ASC, updated_at DESC").Find(&out).Error
return out, err
}
// GetBook 按 ID 取书。
func (r *ReaderRepository) GetBook(ctx context.Context, id string) (*model.ReaderBook, error) {
var out model.ReaderBook
if err := r.db.WithContext(ctx).First(&out, "id = ?", id).Error; err != nil {
return nil, err
}
return &out, nil
}
// FindBookByURL 按用户 + 书源 + 书 URL 查书（加书架去重）。
func (r *ReaderRepository) FindBookByURL(ctx context.Context, userID, origin, bookURL string) (*model.ReaderBook, error) {
var out model.ReaderBook
err := r.db.WithContext(ctx).First(&out, "user_id = ? AND origin = ? AND book_url = ?", userID, origin, bookURL).Error
if err != nil {
return nil, err
}
return &out, nil
}
// CreateBook / UpdateBook / DeleteBook。
func (r *ReaderRepository) CreateBook(ctx context.Context, b *model.ReaderBook) error {
return r.db.WithContext(ctx).Create(b).Error
}
func (r *ReaderRepository) UpdateBook(ctx context.Context, b *model.ReaderBook) error {
return r.db.WithContext(ctx).Save(b).Error
}
func (r *ReaderRepository) DeleteBook(ctx context.Context, userID, id string) error {
return r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
if err := tx.Delete(&model.ReaderBook{}, "id = ? AND user_id = ?", id, userID).Error; err != nil {
return err
}
return tx.Delete(&model.ReaderChapter{}, "book_id = ?", id).Error
})
}
// ReplaceChapters 覆盖式刷新章节列表。
func (r *ReaderRepository) ReplaceChapters(ctx context.Context, bookID string, chapters []model.ReaderChapter) error {
return r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
if err := tx.Delete(&model.ReaderChapter{}, "book_id = ?", bookID).Error; err != nil {
return err
}
if len(chapters) == 0 {
return nil
}
return tx.CreateInBatches(chapters, 500).Error
})
}
// ListChapters 按序取章节。
func (r *ReaderRepository) ListChapters(ctx context.Context, bookID string) ([]model.ReaderChapter, error) {
var out []model.ReaderChapter
err := r.db.WithContext(ctx).Where("book_id = ?", bookID).Order("`index` ASC").Find(&out).Error
return out, err
}
// GetChapter 取指定章节。
func (r *ReaderRepository) GetChapter(ctx context.Context, bookID string, index int) (*model.ReaderChapter, error) {
var out model.ReaderChapter
if err := r.db.WithContext(ctx).First(&out, "book_id = ? AND `index` = ?", bookID, index).Error; err != nil {
return nil, err
}
return &out, nil
}
// ListReplaceRules 用户替换规则（按 order 排序）。
func (r *ReaderRepository) ListReplaceRules(ctx context.Context, userID string) ([]model.ReaderReplaceRule, error) {
var out []model.ReaderReplaceRule
err := r.db.WithContext(ctx).Where("user_id = ?", userID).Order("`order` DESC, created_at ASC").Find(&out).Error
return out, err
}
// CreateReplaceRule / UpdateReplaceRule / DeleteReplaceRule。
func (r *ReaderRepository) CreateReplaceRule(ctx context.Context, rule *model.ReaderReplaceRule) error {
return r.db.WithContext(ctx).Create(rule).Error
}
func (r *ReaderRepository) UpdateReplaceRule(ctx context.Context, rule *model.ReaderReplaceRule) error {
return r.db.WithContext(ctx).Save(rule).Error
}
func (r *ReaderRepository) DeleteReplaceRule(ctx context.Context, userID, id string) error {
return r.db.WithContext(ctx).Delete(&model.ReaderReplaceRule{}, "id = ? AND user_id = ?", id, userID).Error
}
internal/repository/repository.go
+2
View File
@@ -36,6 +36,7 @@ type Container struct {
StrmDirCache *StrmDirCacheRepository
ScrapeTask *ScrapeTaskRepository
EmbyMount *EmbyMountRepository
Reader *ReaderRepository
}
// New 将每个 repository 连接到单个 *gorm.DB。
@@ -68,5 +69,6 @@ func New(db *gorm.DB) *Container {
StrmDirCache: &StrmDirCacheRepository{db: db},
ScrapeTask: &ScrapeTaskRepository{db: db},
EmbyMount: &EmbyMountRepository{db: db},
Reader: &ReaderRepository{db: db},
}
}
internal/service/reader/encoding_test.go
+7
View File
@@ -0,0 +1,7 @@
package reader
import "encoding/base64"
func base64EncodeStr(s string) string {
return base64.StdEncoding.EncodeToString([]byte(s))
}
internal/service/reader/reader.go
+955
View File
@@ -0,0 +1,955 @@
// Package reader — 阅读子系统服务：书源导入管理、搜索、详情、目录、正文。
// 业务语义对齐 legado 的 WebBook / SearchModel / BookSourceDebugModel。
package reader
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"sort"
"strings"
"sync"
"time"
"go.uber.org/zap"
"golang.org/x/sync/errgroup"
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/helper"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service/reader/rule"
)
const (
perSourceTimeout = 30 * time.Second
searchConcurrency = 8
maxContentNextPage = 50 // 正文 nextContentUrl 翻页上限，防死循环
maxBodyBytes = 8 << 20
)
// ReaderService 阅读服务。
type ReaderService struct {
cfg *config.Config
log *zap.Logger
repo *repository.ReaderRepository
http *http.Client
}
// NewReaderService 创建服务。
func NewReaderService(cfg *config.Config, log *zap.Logger, repos *repository.Container) *ReaderService {
return &ReaderService{
cfg: cfg,
log: log,
repo: repos.Reader,
http: helper.NewSiteHTTPClient(30, true),
}
}
// ─── 书源导入与管理 ─────────────────────────────────────────────────────────
// ImportSources 导入书源：支持 JSON 数组 / 单对象 / Base64 / 网络URL。
// 返回导入数量。
func (s *ReaderService) ImportSources(ctx context.Context, text string) (int, error) {
text = strings.TrimSpace(text)
if text == "" {
return 0, fmt.Errorf("导入内容为空")
}
if strings.HasPrefix(text, "http://") || strings.HasPrefix(text, "https://") {
body, _, err := s.execute(ctx, &rule.Request{Method: "GET", URL: text, URLNoQuery: text, Headers: map[string]string{}})
if err != nil {
return 0, fmt.Errorf("拉取书源失败: %w", err)
}
text = body
}
sources := parseSourcePayload(text)
if len(sources) == 0 {
return 0, fmt.Errorf("未识别到有效书源（支持 JSON 数组/对象或 Base64）")
}
imported := 0
for _, raw := range sources {
bs, err := ParseBookSource(raw)
if err != nil || bs.BookSourceURL == "" {
continue
}
// 已存在则更新，否则新建（按书源 URL 去重）
existing, err := s.repo.GetSourceByURL(ctx, bs.BookSourceURL)
now := time.Now()
record := &model.ReaderBookSource{
Name: bs.BookSourceName,
GroupName: strings.TrimSpace(SPtr(bs.BookSourceGroup)),
Type: bs.Type(),
SourceURL: bs.BookSourceURL,
RawJSON: raw,
Enabled: bs.Enabled == nil || *bs.Enabled,
EnabledExplore: bs.EnabledExplore == nil || *bs.EnabledExplore,
CustomOrder: IPtr(bs.CustomOrder),
Weight: IPtr(bs.Weight),
ConcurrentRate: SPtr(bs.ConcurrentRate),
Header: SPtr(bs.Header),
Comment: SPtr(bs.BookSourceComment),
Variables: SPtr(bs.RawVariables),
LastUpdateTime: int64Now(bs.LastUpdateTime),
}
if existing != nil {
existing.Name = record.Name
existing.GroupName = record.GroupName
existing.Type = record.Type
existing.RawJSON = record.RawJSON
existing.Enabled = record.Enabled
existing.EnabledExplore = record.EnabledExplore
existing.CustomOrder = record.CustomOrder
existing.Weight = record.Weight
existing.ConcurrentRate = record.ConcurrentRate
existing.Header = record.Header
existing.Comment = record.Comment
existing.Variables = record.Variables
existing.LastUpdateTime = record.LastUpdateTime
existing.LastCheckAt = &now
if err := s.repo.UpdateSource(ctx, existing); err == nil {
imported++
}
continue
}
record.LastCheckAt = &now
if err := s.repo.CreateSource(ctx, record); err == nil {
imported++
}
}
return imported, nil
}
func int64Now(p *int64) int64 {
if p == nil {
return 0
}
return *p
}
// parseSourcePayload 识别 JSON 数组 / 单对象 / Base64 / 每行一个对象。
func parseSourcePayload(text string) []string {
text = strings.TrimSpace(text)
tryDecode := func(s string) []string {
var arr []json.RawMessage
if err := json.Unmarshal([]byte(s), &arr); err == nil {
var out []string
for _, item := range arr {
out = append(out, string(item))
}
return out
}
var single json.RawMessage
if err := json.Unmarshal([]byte(s), &single); err == nil {
if len(single) > 0 && single[0] == '[' {
return nil
}
return []string{string(single)}
}
return nil
}
if out := tryDecode(text); out != nil {
return out
}
// Base64（含无 padding / URL-safe 变体）
if cleaned := strings.Map(func(r rune) rune {
if r == '\n' || r == '\r' || r == ' ' {
return -1
}
return r
}, text); !strings.HasPrefix(cleaned, "{") {
if decoded, err := base64.StdEncoding.DecodeString(cleaned); err == nil && json.Valid(decoded) {
if out := tryDecode(string(decoded)); out != nil {
return out
}
}
if decoded, err := base64.RawStdEncoding.DecodeString(cleaned); err == nil && json.Valid(decoded) {
if out := tryDecode(string(decoded)); out != nil {
return out
}
}
}
// 每行一个对象
var out []string
for _, line := range strings.Split(text, "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
if items := tryDecode(line); items != nil {
out = append(out, items...)
}
}
return out
}
// ListSources 书源列表。
func (s *ReaderService) ListSources(ctx context.Context) ([]model.ReaderBookSource, error) {
return s.repo.ListSources(ctx)
}
// UpdateSourceEnabled 启停书源。
func (s *ReaderService) UpdateSourceEnabled(ctx context.Context, id string, enabled bool) error {
src, err := s.repo.GetSource(ctx, id)
if err != nil {
return err
}
src.Enabled = enabled
return s.repo.UpdateSource(ctx, src)
}
// DeleteSource 删除书源。
func (s *ReaderService) DeleteSource(ctx context.Context, id string) error {
return s.repo.DeleteSource(ctx, id)
}
// ─── HTTP 执行 ──────────────────────────────────────────────────────────────
// execute 执行 rule.Request，返回（解码后 body, 最终 URL）。
func (s *ReaderService) execute(ctx context.Context, req *rule.Request) (string, string, error) {
var bodyReader io.Reader
if req.Body != "" {
bodyReader = strings.NewReader(req.Body)
}
target := req.URLNoQuery
if target == "" {
target = req.URL
}
httpReq, err := http.NewRequestWithContext(ctx, req.Method, target, bodyReader)
if err != nil {
return "", "", err
}
for k, v := range helper.HTTPHeaderPresets() {
httpReq.Header.Set(k, v)
}
for k, v := range req.Headers {
httpReq.Header.Set(k, v)
}
if req.Method == "POST" {
switch {
case req.IsForm:
httpReq.Header.Set("Content-Type", "application/x-www-form-urlencoded")
case req.IsJSON:
httpReq.Header.Set("Content-Type", "application/json")
}
}
resp, err := s.http.Do(httpReq)
if err != nil {
return "", "", err
}
defer resp.Body.Close()
data, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes))
if err != nil {
return "", "", err
}
charset := req.Charset
if charset == "" {
charset = charsetFromContentType(resp.Header.Get("Content-Type"))
}
body, err := rule.DecodeBytes(data, charset)
if err != nil {
body = string(data)
}
finalURL := resp.Request.URL.String()
if strings.EqualFold(charsetFromContentType(resp.Header.Get("Content-Type")), "xml") &&
!strings.HasPrefix(strings.TrimSpace(body), "<?xml") {
body = "<?xml version=\"1.0\"?>" + body
}
return body, finalURL, nil
}
func charsetFromContentType(ct string) string {
if ct == "" {
return ""
}
idx := strings.Index(strings.ToLower(ct), "charset=")
if idx == -1 {
return ""
}
cs := strings.TrimSpace(ct[idx+len("charset="):])
if i := strings.Index(cs, ";"); i >= 0 {
cs = cs[:i]
}
return strings.Trim(cs, "\"")
}
// ─── 规则执行辅助 ───────────────────────────────────────────────────────────
// newRuleAnalyzer 为指定书源构建规则解析器（注入书源变量）。
func (s *ReaderService) newRuleAnalyzer(bs *BookSource, body, finalURL string) *rule.AnalyzeRule {
ar := rule.NewAnalyzeRule()
ar.SetContent(body, finalURL)
applySourceVariables(ar, bs)
return ar
}
// fetchViaRule 解析 URL 规则并抓取，返回 (body, 最终URL)。
func (s *ReaderService) fetchViaRule(ctx context.Context, urlRule, key string, page int, baseUrl string) (*rule.AnalyzeRule, error) {
req, err := rule.ParseAnalyzeUrl(urlRule, key, page, baseUrl)
if err != nil {
return nil, err
}
if req.Unsupported != nil {
return nil, req.Unsupported
}
body, finalURL, err := s.execute(ctx, req)
if err != nil {
return nil, err
}
ar := rule.NewAnalyzeRule()
ar.SetContent(body, finalURL)
return ar, nil
}
// ─── 搜索 ──────────────────────────────────────────────────────────────────
// SearchBook 搜索结果项（对应 legado SearchBook）。
type SearchBook struct {
Name string `json:"name"`
Author string `json:"author"`
Kind string `json:"kind"`
WordCount string `json:"word_count"`
LatestChapter string `json:"latest_chapter"`
Intro string `json:"intro"`
CoverURL string `json:"cover_url"`
BookURL string `json:"book_url"`
Origins []SearchOrigin `json:"origins"`
}
// SearchOrigin 命中该书目的书源（换源用）。
type SearchOrigin struct {
SourceID string `json:"source_id"`
Origin string `json:"origin"`
OriginName string `json:"origin_name"`
OriginType int `json:"origin_type"`
BookURL string `json:"book_url"`
LatestChapter string `json:"latest_chapter"`
}
type searchHit struct {
book SearchBook
tier int
}
// Search 多源聚合搜索（同步返回，P1 改为 WS 流式推送）。
func (s *ReaderService) Search(ctx context.Context, key string) ([]SearchBook, []SearchSkipped, error) {
sources, err := s.repo.ListSources(ctx)
if err != nil {
return nil, nil, err
}
var enabled []model.ReaderBookSource
for _, src := range sources {
if src.Enabled {
enabled = append(enabled, src)
}
}
if len(enabled) == 0 {
return nil, nil, fmt.Errorf("没有已启用的书源")
}
var (
mu sync.Mutex
hits []searchHit
skipped []SearchSkipped
)
g, gctx := errgroup.WithContext(ctx)
g.SetLimit(searchConcurrency)
for _, src := range enabled {
src := src
g.Go(func() error {
gctxSrc, cancel := context.WithTimeout(gctx, perSourceTimeout)
defer cancel()
books, err := s.searchInSource(gctxSrc, &src, key, 1)
mu.Lock()
defer mu.Unlock()
if err != nil {
skipped = append(skipped, SearchSkipped{SourceID: src.ID, OriginName: src.Name, Reason: err.Error()})
return nil // 单源失败不影响整体
}
for i := range books {
hits = append(hits, searchHit{book: books[i]})
}
return nil
})
}
_ = g.Wait()
return mergeSearchResults(hits, key), skipped, nil
}
// SearchSkipped 搜索失败的书源与原因（书源管理调试用）。
type SearchSkipped struct {
SourceID string `json:"source_id"`
OriginName string `json:"origin_name"`
Reason string `json:"reason"`
}
// tierFor 对应 legado SearchModel 的结果分档：
// 书名或作者等于关键词 > kind 含关键词 > 书名/作者含关键词 > 其他。
func tierFor(b *SearchBook, key string) int {
lk := strings.ToLower(key)
switch {
case strings.EqualFold(b.Name, key) || strings.EqualFold(b.Author, key):
return 0
case strings.Contains(strings.ToLower(b.Kind), lk):
return 1
case strings.Contains(strings.ToLower(b.Name), lk) || strings.Contains(strings.ToLower(b.Author), lk):
return 2
default:
return 3
}
}
// mergeSearchResults 对应 mergeItems：同名同作者合并（多源记录），
// 档间按 tier 顺序，档内按书源数降序。
func mergeSearchResults(hits []searchHit, key string) []SearchBook {
merged := map[string]*SearchBook{}
for _, hit := range hits {
b := hit.book
if b.Name == "" {
continue
}
k := b.Name + "|" + b.Author
if existing, ok := merged[k]; ok {
existing.Origins = append(existing.Origins, b.Origins...)
continue
}
merged[k] = &b
}
out := make([]SearchBook, 0, len(merged))
for _, b := range merged {
sort.SliceStable(b.Origins, func(i, j int) bool {
return b.Origins[i].OriginName < b.Origins[j].OriginName
})
out = append(out, *b)
}
sort.SliceStable(out, func(i, j int) bool {
ti, tj := tierFor(&out[i], key), tierFor(&out[j], key)
if ti != tj {
return ti < tj
}
return len(out[i].Origins) > len(out[j].Origins)
})
return out
}
// searchInSource 单源搜索（对应 WebBook.searchBook）。
func (s *ReaderService) searchInSource(ctx context.Context, src *model.ReaderBookSource, key string, page int) ([]SearchBook, error) {
bs, err := ParseBookSource(src.RawJSON)
if err != nil {
return nil, fmt.Errorf("书源 JSON 解析失败")
}
searchURL := SPtr(bs.SearchURL)
if searchURL == "" {
return nil, fmt.Errorf("书源未配置搜索地址")
}
sr := bs.RuleSearch
if sr == nil || SPtr(sr.BookList) == "" {
return nil, fmt.Errorf("书源未配置搜索列表规则")
}
req, err := rule.ParseAnalyzeUrl(searchURL, key, page, src.SourceURL)
if err != nil {
return nil, err
}
if req.Unsupported != nil {
return nil, req.Unsupported
}
// 书源级请求头
if src.Header != "" {
var headers map[string]any
if json.Unmarshal([]byte(src.Header), &headers) == nil {
for k, v := range headers {
if _, ok := req.Headers[k]; !ok {
req.Headers[k] = fmt.Sprintf("%v", v)
}
}
}
}
body, finalURL, err := s.execute(ctx, req)
if err != nil {
return nil, err
}
ar := rule.NewAnalyzeRule()
ar.SetContent(body, finalURL)
applySourceVariables(ar, bs)
elements, err := ar.GetElements(SPtr(sr.BookList))
if err != nil {
return nil, err
}
var books []SearchBook
for _, el := range elements {
name, err := ar.GetString(SPtr(sr.Name), el, false)
if err != nil {
continue
}
bookURL, err := ar.GetString(SPtr(sr.BookURL), el, true)
if err != nil || bookURL == "" {
continue
}
author, _ := ar.GetString(SPtr(sr.Author), el, false)
kind, _ := ar.GetString(SPtr(sr.Kind), el, false)
cover, _ := ar.GetString(SPtr(sr.CoverURL), el, true)
intro, _ := ar.GetString(SPtr(sr.Intro), el, false)
lastChapter, _ := ar.GetString(SPtr(sr.LastChapter), el, false)
wordCount, _ := ar.GetString(SPtr(sr.WordCount), el, false)
books = append(books, SearchBook{
Name: name,
Author: author,
Kind: kind,
WordCount: wordCount,
LatestChapter: lastChapter,
Intro: intro,
CoverURL: cover,
BookURL: bookURL,
Origins: []SearchOrigin{{
SourceID: src.ID,
Origin: src.SourceURL,
OriginName: firstNonEmpty(src.Name, bs.BookSourceName),
OriginType: src.Type,
BookURL: bookURL,
LatestChapter: lastChapter,
}},
})
}
return books, nil
}
func applySourceVariables(ar *rule.AnalyzeRule, bs *BookSource) {
if bs.Variables == nil {
return
}
vars := map[string]string{}
for k, v := range bs.Variables {
vars[k] = fmt.Sprintf("%v", v)
}
ar.SetBookContext("", vars)
}
func firstNonEmpty(vals ...string) string {
for _, v := range vals {
if v != "" {
return v
}
}
return ""
}
// ─── 详情 / 目录 / 正文 ─────────────────────────────────────────────────────
// BookInfo 书籍详情（对应 legado Book 信息页）。
type BookInfo struct {
Name string `json:"name"`
Author string `json:"author"`
Kind string `json:"kind"`
WordCount string `json:"word_count"`
LatestChapter string `json:"latest_chapter"`
Intro string `json:"intro"`
CoverURL string `json:"cover_url"`
TocURL string `json:"toc_url"`
BookURL string `json:"book_url"`
}
// GetBookInfo 抓取书籍详情。
func (s *ReaderService) GetBookInfo(ctx context.Context, sourceID, bookURL string) (*BookInfo, error) {
src, bs, err := s.loadSource(ctx, sourceID)
if err != nil {
return nil, err
}
bir := bs.RuleBookInfo
if bir == nil {
return nil, fmt.Errorf("书源未配置详情规则")
}
req, err := rule.ParseAnalyzeUrl(bookURL, "", 0, src.SourceURL)
if err != nil {
return nil, err
}
if req.Unsupported != nil {
return nil, req.Unsupported
}
applySourceHeaders(req, src)
body, finalURL, err := s.execute(ctx, req)
if err != nil {
return nil, err
}
ar := rule.NewAnalyzeRule()
ar.SetContent(body, finalURL)
applySourceVariables(ar, bs)
info := &BookInfo{BookURL: bookURL, TocURL: bookURL}
if initRule := SPtr(bir.Init); initRule != "" {
// init 规则返回 JSON 对象时合并字段（对应 legado ruleBookInfo.init）
if initVal, err := ar.GetString(initRule, nil, false); err == nil && initVal != "" {
if strings.HasPrefix(strings.TrimSpace(initVal), "{") {
var m map[string]string
if json.Unmarshal([]byte(initVal), &m) == nil {
if v, ok := m["name"]; ok {
info.Name = v
}
if v, ok := m["author"]; ok {
info.Author = v
}
if v, ok := m["intro"]; ok {
info.Intro = v
}
if v, ok := m["tocUrl"]; ok {
info.TocURL = v
}
}
}
}
}
if v, err := ar.GetString(SPtr(bir.Name), nil, false); err == nil && v != "" {
info.Name = v
}
if v, err := ar.GetString(SPtr(bir.Author), nil, false); err == nil && v != "" {
info.Author = v
}
if v, err := ar.GetString(SPtr(bir.Kind), nil, false); err == nil && v != "" {
info.Kind = v
}
if v, err := ar.GetString(SPtr(bir.WordCount), nil, false); err == nil && v != "" {
info.WordCount = v
}
if v, err := ar.GetString(SPtr(bir.LastChapter), nil, false); err == nil && v != "" {
info.LatestChapter = v
}
if v, err := ar.GetString(SPtr(bir.Intro), nil, false); err == nil && v != "" {
info.Intro = v
}
if v, err := ar.GetString(SPtr(bir.CoverURL), nil, true); err == nil && v != "" {
info.CoverURL = v
}
if v, err := ar.GetString(SPtr(bir.TocURL), nil, true); err == nil && v != "" {
info.TocURL = v
}
return info, nil
}
// TocChapter 目录章节项。
type TocChapter struct {
Index int `json:"index"`
Title string `json:"title"`
URL string `json:"url"`
IsVolume bool `json:"is_volume"`
UpdateTime string `json:"update_time"`
}
// GetToc 抓取目录。
func (s *ReaderService) GetToc(ctx context.Context, sourceID, bookURL, tocURL string) ([]TocChapter, error) {
src, bs, err := s.loadSource(ctx, sourceID)
if err != nil {
return nil, err
}
tr := bs.RuleToc
if tr == nil || SPtr(tr.ChapterList) == "" {
return nil, fmt.Errorf("书源未配置目录规则")
}
if tocURL == "" {
tocURL = bookURL
}
req, err := rule.ParseAnalyzeUrl(tocURL, "", 0, src.SourceURL)
if err != nil {
return nil, err
}
if req.Unsupported != nil {
return nil, req.Unsupported
}
applySourceHeaders(req, src)
body, finalURL, err := s.execute(ctx, req)
if err != nil {
return nil, err
}
ar := rule.NewAnalyzeRule()
ar.SetContent(body, finalURL)
applySourceVariables(ar, bs)
elements, err := ar.GetElements(SPtr(tr.ChapterList))
if err != nil {
return nil, err
}
var chapters []TocChapter
for i, el := range elements {
title, err := ar.GetString(SPtr(tr.ChapterName), el, false)
if err != nil || title == "" {
continue
}
url, err := ar.GetString(SPtr(tr.ChapterURL), el, true)
if err != nil || url == "" {
continue
}
isVolume := false
if SPtr(tr.IsVolume) != "" {
if v, err := ar.GetString(SPtr(tr.IsVolume), el, false); err == nil {
isVolume = v != "" && v != "false" && v != "0"
}
}
updateTime, _ := ar.GetString(SPtr(tr.UpdateTime), el, false)
chapters = append(chapters, TocChapter{
Index: i, Title: title, URL: url, IsVolume: isVolume, UpdateTime: updateTime,
})
}
return chapters, nil
}
// ChapterContent 章节内容（按类型返回文本/音频/图片）。
type ChapterContent struct {
Type string `json:"type"` // text / audio / image
Content string `json:"content,omitempty"`
Tracks []string `json:"tracks,omitempty"` // 音频播放地址（m3u8/直链）
Images []string `json:"images,omitempty"` // 漫画图片列表
}
// GetContent 抓取正文（含 nextContentUrl 翻页合并与净化替换）。
func (s *ReaderService) GetContent(ctx context.Context, sourceID, bookURL, chapterURL string) (*ChapterContent, error) {
src, bs, err := s.loadSource(ctx, sourceID)
if err != nil {
return nil, err
}
cr := bs.RuleContent
if cr == nil || SPtr(cr.Content) == "" {
return nil, fmt.Errorf("书源未配置正文规则")
}
var parts []string
url := chapterURL
for i := 0; i < maxContentNextPage; i++ {
req, err := rule.ParseAnalyzeUrl(url, "", 0, src.SourceURL)
if err != nil {
return nil, err
}
if req.Unsupported != nil {
return nil, req.Unsupported
}
applySourceHeaders(req, src)
body, finalURL, err := s.execute(ctx, req)
if err != nil {
return nil, err
}
ar := rule.NewAnalyzeRule()
ar.SetContent(body, finalURL)
applySourceVariables(ar, bs)
list, err := ar.GetStringList(SPtr(cr.Content), nil, false)
if err != nil {
return nil, err
}
parts = append(parts, strings.Join(list, "\n"))
if SPtr(cr.NextContentURL) == "" {
break
}
next, err := ar.GetString(SPtr(cr.NextContentURL), nil, true)
if err != nil || next == "" || next == url || next == finalURL {
break
}
url = next
}
content := strings.Join(parts, "\n")
if rr := SPtr(cr.ReplaceRegex); rr != "" {
content = rule.ApplyReplaceRegexString(content, rr)
}
out := &ChapterContent{Content: content}
switch src.Type {
case 1:
out.Type = "audio"
out.Tracks = splitURLLines(content)
case 2:
out.Type = "image"
out.Images = splitURLLines(content)
default:
out.Type = "text"
}
return out, nil
}
func splitURLLines(s string) []string {
var out []string
for _, line := range strings.Split(s, "\n") {
line = strings.TrimSpace(line)
if line != "" {
out = append(out, line)
}
}
return out
}
// ─── 书架 ──────────────────────────────────────────────────────────────────
// AddBook 将搜索结果加入书架。
func (s *ReaderService) AddBook(ctx context.Context, userID string, origin SearchOrigin, name, author, coverURL string) (*model.ReaderBook, error) {
if existing, err := s.repo.FindBookByURL(ctx, userID, origin.Origin, origin.BookURL); err == nil && existing != nil {
return existing, nil
}
book := &model.ReaderBook{
UserID: userID,
Origin: origin.Origin,
OriginName: origin.OriginName,
BookURL: origin.BookURL,
Name: name,
Author: author,
CoverURL: coverURL,
Type: origin.OriginType,
}
if err := s.repo.CreateBook(ctx, book); err != nil {
return nil, err
}
return book, nil
}
// ListBooks 书架列表。
func (s *ReaderService) ListBooks(ctx context.Context, userID string) ([]model.ReaderBook, error) {
return s.repo.ListBooks(ctx, userID)
}
// RemoveBook 移出书架。
func (s *ReaderService) RemoveBook(ctx context.Context, userID, id string) error {
return s.repo.DeleteBook(ctx, userID, id)
}
// SaveProgress 保存阅读进度（对应 legado durChapter*）。
func (s *ReaderService) SaveProgress(ctx context.Context, userID, bookID string, chapterIndex, pos int, chapterTitle string) error {
book, err := s.repo.GetBook(ctx, bookID)
if err != nil {
return err
}
if book.UserID != userID {
return fmt.Errorf("无权操作他人书架")
}
book.DurChapterIndex = chapterIndex
book.DurChapterPos = pos
book.DurChapterTitle = chapterTitle
book.DurChapterTime = time.Now().UnixMilli()
return s.repo.UpdateBook(ctx, book)
}
// ListChapters 目录缓存读取。
func (s *ReaderService) ListChapters(ctx context.Context, bookID string) ([]model.ReaderChapter, error) {
return s.repo.ListChapters(ctx, bookID)
}
// ChapterInput 章节保存输入。
type ChapterInput struct {
Index int `json:"index"`
Title string `json:"title"`
URL string `json:"url"`
IsVolume bool `json:"is_volume"`
}
// SaveChapters 覆盖保存章节缓存。
func (s *ReaderService) SaveChapters(ctx context.Context, bookID string, chapters []ChapterInput) error {
models := make([]model.ReaderChapter, 0, len(chapters))
for _, c := range chapters {
models = append(models, model.ReaderChapter{
BookID: bookID,
Index: c.Index,
URL: c.URL,
Title: c.Title,
IsVolume: c.IsVolume,
})
}
return s.repo.ReplaceChapters(ctx, bookID, models)
}
// ListReplaceRules 用户替换规则列表。
func (s *ReaderService) ListReplaceRules(ctx context.Context, userID string) ([]model.ReaderReplaceRule, error) {
return s.repo.ListReplaceRules(ctx, userID)
}
// ─── 书源调试（对应 BookSourceDebugModel 全链路） ───────────────────────────
// Debug 全链路调试：搜索 → 详情 → 目录 → 正文，返回逐条日志。
func (s *ReaderService) Debug(ctx context.Context, sourceID, key string) ([]string, error) {
src, _, err := s.loadSource(ctx, sourceID)
if err != nil {
return nil, err
}
var logs []string
logf := func(format string, args ...any) {
logs = append(logs, fmt.Sprintf(format, args...))
}
logf("搜索关键词: %s", key)
books, err := s.searchInSource(ctx, src, key, 1)
if err != nil {
logf("搜索失败: %v", err)
return logs, nil
}
if len(books) == 0 {
logf("搜索结果为空")
return logs, nil
}
logf("搜索到 %d 条结果", len(books))
for i, b := range books {
if i >= 3 {
break
}
logf("结果[%d] %s / %s", i, b.Name, b.Author)
}
first := books[0]
logf("访问详情页: %s", first.BookURL)
info, err := s.GetBookInfo(ctx, sourceID, first.BookURL)
if err != nil {
logf("详情失败: %v", err)
return logs, nil
}
logf("书名: %s 作者: %s 最新章节: %s", info.Name, info.Author, info.LatestChapter)
logf("访问目录页: %s", info.TocURL)
chapters, err := s.GetToc(ctx, sourceID, first.BookURL, info.TocURL)
if err != nil {
logf("目录失败: %v", err)
return logs, nil
}
logf("共 %d 章", len(chapters))
for i, c := range chapters {
if i >= 3 {
break
}
logf("章节[%d] %s", c.Index, c.Title)
}
// 找第一个非卷章节读正文
for _, c := range chapters {
if c.IsVolume {
continue
}
logf("访问正文: %s", c.URL)
content, err := s.GetContent(ctx, sourceID, first.BookURL, c.URL)
if err != nil {
logf("正文失败: %v", err)
return logs, nil
}
text := content.Content
if len(text) > 200 {
text = text[:200] + "..."
}
logf("正文预览: %s", text)
break
}
logf("调试完成")
return logs, nil
}
// loadSource 加载书源记录与解析结构。
func (s *ReaderService) loadSource(ctx context.Context, sourceID string) (*model.ReaderBookSource, *BookSource, error) {
src, err := s.repo.GetSource(ctx, sourceID)
if err != nil {
return nil, nil, err
}
bs, err := ParseBookSource(src.RawJSON)
if err != nil {
return nil, nil, fmt.Errorf("书源 JSON 解析失败: %w", err)
}
return src, bs, nil
}
func applySourceHeaders(req *rule.Request, src *model.ReaderBookSource) {
if src.Header == "" {
return
}
var headers map[string]any
if json.Unmarshal([]byte(src.Header), &headers) == nil {
for k, v := range headers {
if _, ok := req.Headers[k]; !ok {
req.Headers[k] = fmt.Sprintf("%v", v)
}
}
}
}
internal/service/reader/reader_test.go
+261
View File
@@ -0,0 +1,261 @@
package reader
import (
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/truewhile/MeBox/internal/service/reader/rule"
)
const e2eBookHTML = `<!DOCTYPE html>
<html><body>
<div class="box" id="main">
<div class="item"><h3><a href="/book/1">斗破苍穹</a></h3><span class="author">天蚕土豆</span></div>
<div class="item"><h3><a href="/book/2">凡人修仙传</a></h3><span class="author">忘语</span></div>
</div>
</body></html>`
const e2eBookInfoHTML = `<html><body>
<div class="info"><h1>斗破苍穹</h1><span class="author">天蚕土豆</span>
<p class="intro">三十年河东三十年河西</p>
<a class="toc" href="/book/1/toc.html">查看目录</a></div>
</body></html>`
const e2eTocHTML = `<html><body>
<ul class="chapters">
<li class="vol">第一卷</li>
<li><a href="/book/1/c1.html">第一章 陨落的天才</a></li>
<li><a href="/book/1/c2.html">第二章 斗气大陆</a></li>
</ul>
</body></html>`
const e2eContentHTML = `<html><body><div id="content"> 魂殿来犯，<br>萧炎浴火重生。 </div>
<a class="next" href="/book/1/c1_2.html">下一页</a></body></html>`
const e2eContentPage2HTML = `<html><body><div id="content">少女微凉的手掌传来。</div></body></html>`
// e2eServer 模拟一个完整的书源站点：搜索/详情/目录/正文（含 nextContentUrl 翻页）。
func e2eServer() *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
switch {
case strings.HasPrefix(r.URL.Path, "/search/"):
_, _ = w.Write([]byte(e2eBookHTML))
case strings.HasPrefix(r.URL.Path, "/book/1/toc"):
_, _ = w.Write([]byte(e2eTocHTML))
case strings.HasPrefix(r.URL.Path, "/book/1/c1_2"):
_, _ = w.Write([]byte(e2eContentPage2HTML))
case strings.HasPrefix(r.URL.Path, "/book/1/c"):
_, _ = w.Write([]byte(e2eContentHTML))
case strings.HasPrefix(r.URL.Path, "/book/"):
_, _ = w.Write([]byte(e2eBookInfoHTML))
default:
http.NotFound(w, r)
}
}))
}
// e2eSource 对齐 legado 书源 JSON 结构，覆盖 jsoup 全部四段规则。
func e2eSourceJSON(server string) string {
return fmt.Sprintf(`{
"bookSourceUrl": %q,
"bookSourceName": "测试源",
"bookSourceType": 0,
"searchUrl": "%s/search/{{key}}/1.html",
"ruleSearch": {
"bookList": "class.item",
"name": "tag.h3@tag.a@text",
"bookUrl": "tag.h3@tag.a@href",
"author": "class.author@text"
},
"ruleBookInfo": {
"name": "class.info@tag.h1@text",
"author": "class.info@class.author@text",
"intro": "class.info@class.intro@text",
"tocUrl": "class.info@class.toc@href"
},
"ruleToc": {
"chapterList": "class.chapters@tag.li",
"chapterName": "tag.a@text",
"chapterUrl": "tag.a@href",
"isVolume": "tag.a@text"
},
"ruleContent": {
"content": "id.content@textNodes",
"nextContentUrl": "class.next@href"
}
}`, server, server)
}
// newE2EEngine 执行与 ReaderService 相同的链路（不含 DB）：
// ParseAnalyzeUrl → execute → AnalyzeRule。
type e2eEngine struct {
server string
client *http.Client
}
func (e *e2eEngine) fetch(t *testing.T, urlRule, key string, page int) (*rule.AnalyzeRule, string) {
t.Helper()
req, err := rule.ParseAnalyzeUrl(urlRule, key, page, e.server)
if err != nil {
t.Fatal(err)
}
if req.Unsupported != nil {
t.Fatalf("unsupported: %v", req.Unsupported)
}
httpReq, _ := http.NewRequest("GET", req.URL, nil)
for k, v := range req.Headers {
httpReq.Header.Set(k, v)
}
resp, err := e.client.Do(httpReq)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
ar := rule.NewAnalyzeRule()
ar.SetContent(string(body), resp.Request.URL.String())
return ar, resp.Request.URL.String()
}
func TestEndToEndSourceChain(t *testing.T) {
srv := e2eServer()
defer srv.Close()
bs, err := ParseBookSource(e2eSourceJSON(srv.URL))
if err != nil {
t.Fatal(err)
}
engine := &e2eEngine{server: srv.URL, client: srv.Client()}
// ── 搜索 ──
ar, _ := engine.fetch(t, SPtr(bs.SearchURL), "斗罗", 1)
sr := bs.RuleSearch
els, err := ar.GetElements(SPtr(sr.BookList))
if err != nil {
t.Fatal(err)
}
if len(els) != 2 {
t.Fatalf("search elements = %d", len(els))
}
name, _ := ar.GetString(SPtr(sr.Name), els[0], false)
if name != "斗破苍穹" {
t.Fatalf("search name = %q", name)
}
bookURL, _ := ar.GetString(SPtr(sr.BookURL), els[0], true)
if !strings.Contains(bookURL, "/book/1") {
t.Fatalf("bookURL = %q", bookURL)
}
// ── 详情 ──
ar, _ = engine.fetch(t, bookURL, "", 0)
bir := bs.RuleBookInfo
if got, _ := ar.GetString(SPtr(bir.Name), nil, false); got != "斗破苍穹" {
t.Fatalf("info name = %q", got)
}
if got, _ := ar.GetString(SPtr(bir.Intro), nil, false); got != "三十年河东三十年河西" {
t.Fatalf("info intro = %q", got)
}
tocURL, _ := ar.GetString(SPtr(bir.TocURL), nil, true)
if !strings.Contains(tocURL, "/book/1/toc") {
t.Fatalf("tocURL = %q", tocURL)
}
// ── 目录 ──
ar, _ = engine.fetch(t, tocURL, "", 0)
tr := bs.RuleToc
chEls, err := ar.GetElements(SPtr(tr.ChapterList))
if err != nil {
t.Fatal(err)
}
if len(chEls) != 3 {
t.Fatalf("chapters = %d", len(chEls))
}
var chapters []TocChapter
for i, el := range chEls {
title, _ := ar.GetString(SPtr(tr.ChapterName), el, false)
if title == "" {
continue // 卷名行没有 <a>，取不到标题（服务层同样跳过）
}
url, _ := ar.GetString(SPtr(tr.ChapterURL), el, true)
chapters = append(chapters, TocChapter{Index: i, Title: title, URL: url})
}
if len(chapters) != 2 {
t.Fatalf("chapters = %d", len(chapters))
}
if chapters[0].Title != "第一章 陨落的天才" {
t.Fatalf("chapter0 = %+v", chapters[0])
}
// ── 正文（含 nextContentUrl 翻页合并） ──
var parts []string
url := chapters[0].URL
for i := 0; i < 5; i++ {
ar, finalURL := engine.fetch(t, url, "", 0)
list, err := ar.GetStringList(SPtr(bs.RuleContent.Content), nil, false)
if err != nil {
t.Fatal(err)
}
parts = append(parts, strings.Join(list, "\n"))
next, _ := ar.GetString(SPtr(bs.RuleContent.NextContentURL), nil, true)
if next == "" || next == url || next == finalURL {
break
}
url = next
}
content := strings.Join(parts, "\n")
if !strings.Contains(content, "萧炎浴火重生") || !strings.Contains(content, "少女微凉的手掌") {
t.Fatalf("content = %q", content)
}
}
// ─── 纯函数测试：导入识别 / 搜索合并 ────────────────────────────────────────
func TestParseSourcePayload(t *testing.T) {
// 数组
arr := `[{"bookSourceUrl":"http://a.com","bookSourceName":"A"},{"bookSourceUrl":"http://b.com","bookSourceName":"B"}]`
if got := parseSourcePayload(arr); len(got) != 2 {
t.Fatalf("array payload = %d", len(got))
}
// 单对象
single := `{"bookSourceUrl":"http://a.com","bookSourceName":"A"}`
if got := parseSourcePayload(single); len(got) != 1 {
t.Fatalf("single payload = %d", len(got))
}
// Base64
b64 := base64StdEncode(single)
if got := parseSourcePayload(b64); len(got) != 1 {
t.Fatalf("base64 payload = %d", len(got))
}
}
func TestMergeSearchResults(t *testing.T) {
hit := func(name, author, origin string) searchHit {
return searchHit{book: SearchBook{
Name: name, Author: author,
Origins: []SearchOrigin{{OriginName: origin, BookURL: "u"}},
}}
}
hits := []searchHit{
hit("遮天", "辰东", "源C"),
hit("斗破苍穹", "天蚕土豆", "源B"),
hit("斗破苍穹", "天蚕土豆", "源A"),
hit("斗罗大陆", "唐家三少", "源D"),
}
merged := mergeSearchResults(hits, "斗")
if len(merged) != 3 {
t.Fatalf("merged = %d", len(merged))
}
// 精确命中「斗破苍穹」应排第一（tier 0），且合并两源
if merged[0].Name != "斗破苍穹" || len(merged[0].Origins) != 2 {
t.Fatalf("first = %+v", merged[0])
}
}
func base64StdEncode(s string) string {
return base64EncodeStr(s)
}
internal/service/reader/rule/analyzer.go
+520
View File
@@ -0,0 +1,520 @@
package rule
import (
"net/url"
"strings"
"github.com/PaesslerAG/jsonpath"
"golang.org/x/net/html"
)
// 本文件对应 AnalyzeRule.kt 主类。
// AnalyzeRule 解析规则获取结果（对应 AnalyzeRule）。
type AnalyzeRule struct {
content any
baseUrl string
redirectURL *url.URL
isJSON bool
isRegex bool
// jsRunner 由 P2 阶段的 goja 引擎注入；nil 时 JS 规则报 ErrJsUnsupported。
jsRunner func(js string, result any) (any, error)
// 变量层级（对应 chapter → book → ruleData → source）
chapterVars map[string]string
bookVars map[string]string
vars map[string]string
chapterTitle string
bookName string
sourceGetter func(key string) string
sourcePutter func(key, value string)
ruleCache map[string][]*SourceRule
}
// NewAnalyzeRule 创建解析器。
func NewAnalyzeRule() *AnalyzeRule {
return &AnalyzeRule{ruleCache: map[string][]*SourceRule{}}
}
// SetJSRunner 注入 JS 执行器（P2）。
func (a *AnalyzeRule) SetJSRunner(runner func(js string, result any) (any, error)) {
a.jsRunner = runner
}
// SetContent 对应 setContent(content, baseUrl)。
func (a *AnalyzeRule) SetContent(content any, baseUrl string) *AnalyzeRule {
a.content = content
switch content.(type) {
case *html.Node:
a.isJSON = false
default:
a.isJSON = LooksLikeJSON(anyToString(content))
}
if baseUrl != "" {
a.baseUrl = baseUrl
}
return a
}
// SetBaseUrl 对应 setBaseUrl。
func (a *AnalyzeRule) SetBaseUrl(baseUrl string) *AnalyzeRule {
if baseUrl != "" {
a.baseUrl = baseUrl
}
return a
}
// SetRedirectUrl 对应 setRedirectUrl。
func (a *AnalyzeRule) SetRedirectUrl(u string) *AnalyzeRule {
if strings.HasPrefix(u, "data:") {
return a
}
if parsed, err := url.Parse(u); err == nil {
a.redirectURL = parsed
}
return a
}
// SetChapterContext 设置章节上下文（title 与章节级变量存储）。
func (a *AnalyzeRule) SetChapterContext(title string, vars map[string]string) {
a.chapterTitle = title
if vars != nil {
a.chapterVars = vars
}
}
// SetBookContext 设置书籍上下文（name 与书籍级变量存储）。
func (a *AnalyzeRule) SetBookContext(name string, vars map[string]string) {
a.bookName = name
if vars != nil {
a.bookVars = vars
}
}
// SetSourceVariables 注入书源级变量读写（source.variableMap）。
func (a *AnalyzeRule) SetSourceVariables(getter func(key string) string, putter func(key, value string)) {
a.sourceGetter = getter
a.sourcePutter = putter
}
// jsonpathGet 包装 PaesslerAG/jsonpath.Get。
func jsonpathGet(path string, root any) (v any, err error) {
defer func() {
if r := recover(); r != nil {
v, err = nil, ErrJsUnsupported
}
}()
return jsonpath.Get(path, root)
}
// ─── 变量存取（对应 put/get） ───────────────────────────────────────────────
// Put 对应 put(key, value)：chapter → book → 局部 → source。
func (a *AnalyzeRule) Put(key, value string) string {
switch {
case a.chapterVars != nil:
a.chapterVars[key] = value
case a.bookVars != nil:
a.bookVars[key] = value
default:
if a.vars == nil {
a.vars = map[string]string{}
}
a.vars[key] = value
}
return value
}
// Get 对应 get(key)：特殊键 bookName/title 优先取上下文。
func (a *AnalyzeRule) Get(key string) string {
switch key {
case "bookName":
if a.bookName != "" {
return a.bookName
}
case "title":
if a.chapterTitle != "" {
return a.chapterTitle
}
}
for _, store := range []map[string]string{a.chapterVars, a.bookVars, a.vars} {
if store != nil {
if v, ok := store[key]; ok && v != "" {
return v
}
}
}
if a.sourceGetter != nil {
if v := a.sourceGetter(key); v != "" {
return v
}
}
return ""
}
// ─── JS ────────────────────────────────────────────────────────────────────
func (a *AnalyzeRule) evalJS(js string, result any) (any, error) {
if a.jsRunner == nil {
return nil, ErrJsUnsupported
}
return a.jsRunner(js, result)
}
// ─── 规则拆分缓存 ──────────────────────────────────────────────────────────
func (a *AnalyzeRule) splitSourceRuleCached(ruleStr string) []*SourceRule {
if ruleStr == "" {
return nil
}
if cached, ok := a.ruleCache[ruleStr]; ok {
return cached
}
rules := SplitSourceRule(ruleStr, false, a.isJSON, &a.isRegex)
a.ruleCache[ruleStr] = rules
return rules
}
func (a *AnalyzeRule) putRule(putMap map[string]string) error {
for k, v := range putMap {
s, err := a.GetString(v, nil, false)
if err != nil {
return err
}
a.Put(k, s)
}
return nil
}
func (a *AnalyzeRule) makeDeps() *RuleDeps {
return &RuleDeps{
JS: a.evalJS,
Rule: func(rule string) (string, error) { return a.GetString(rule, nil, false) },
Get: a.Get,
}
}
// ─── 主流程 ────────────────────────────────────────────────────────────────
// GetStringList 对应 getStringList(rule, mContent, isUrl)。
func (a *AnalyzeRule) GetStringList(ruleStr string, mContent any, isUrl bool) ([]string, error) {
if ruleStr == "" {
return nil, nil
}
ruleList := a.splitSourceRuleCached(ruleStr)
return a.getStringListRules(ruleList, mContent, isUrl)
}
func (a *AnalyzeRule) getStringListRules(ruleList []*SourceRule, mContent any, isUrl bool) ([]string, error) {
var result any
content := mContent
if content == nil {
content = a.content
}
if content != nil && len(ruleList) > 0 {
result = content
for _, sourceRule := range ruleList {
if err := a.putRule(sourceRule.putMap); err != nil {
return nil, err
}
resolved, err := sourceRule.MakeUpRule(result, a.makeDeps())
if err != nil {
return nil, err
}
if result == nil {
continue
}
rule := resolved.Rule
if rule != "" || resolved.ReplaceRegex == "" {
switch sourceRule.Mode {
case ModeWebJs:
return nil, ErrWebJSUnsupported
case ModeJs:
result, err = a.evalJS(rule, result)
if err != nil {
return nil, err
}
case ModeJson:
result = newJSONAnalyzer(result).getStringList(rule)
case ModeXPath:
result = newXPathAnalyzer(result).getStringList(rule)
case ModeDefault:
result = newJsoupAnalyzer(result).getStringList(rule)
default:
result = rule
}
}
if resolved.ReplaceRegex != "" {
if lst, ok := result.([]string); ok {
out := make([]string, 0, len(lst))
for _, item := range lst {
out = append(out, applyReplaceRegex(item, resolved))
}
result = out
} else {
result = applyReplaceRegex(resultString(result), resolved)
}
}
}
}
if result == nil {
return nil, nil
}
if s, ok := result.(string); ok {
result = strings.Split(s, "\n")
}
if isUrl {
var urlList []string
if lst, ok := result.([]string); ok {
for _, u := range lst {
abs := a.absolutize(u)
if abs != "" && !containsStr(urlList, abs) {
urlList = append(urlList, abs)
}
}
}
return urlList, nil
}
switch t := result.(type) {
case []string:
return t, nil
case []any:
out := make([]string, len(t))
for i, v := range t {
out[i] = anyToString(v)
}
return out, nil
case []*html.Node:
out := make([]string, len(t))
for i, v := range t {
out[i] = outerHTML(v)
}
return out, nil
default:
return []string{anyToString(result)}, nil
}
}
// GetString 对应 getString(rule, mContent, isUrl)。
func (a *AnalyzeRule) GetString(ruleStr string, mContent any, isUrl bool) (string, error) {
if ruleStr == "" {
return "", nil
}
ruleList := a.splitSourceRuleCached(ruleStr)
return a.getStringRules(ruleList, mContent, isUrl)
}
func (a *AnalyzeRule) getStringRules(ruleList []*SourceRule, mContent any, isUrl bool) (string, error) {
var result any
content := mContent
if content == nil {
content = a.content
}
if content != nil && len(ruleList) > 0 {
result = content
for _, sourceRule := range ruleList {
if err := a.putRule(sourceRule.putMap); err != nil {
return "", err
}
resolved, err := sourceRule.MakeUpRule(result, a.makeDeps())
if err != nil {
return "", err
}
if result == nil {
continue
}
rule := resolved.Rule
if rule != "" || resolved.ReplaceRegex == "" {
switch sourceRule.Mode {
case ModeWebJs:
return "", ErrWebJSUnsupported
case ModeJs:
result, err = a.evalJS(rule, result)
if err != nil {
return "", err
}
case ModeJson:
result = newJSONAnalyzer(result).getString(rule)
case ModeXPath:
result = newXPathAnalyzer(result).getString(rule)
case ModeDefault:
if isUrl {
result = newJsoupAnalyzer(result).getString0(rule)
} else {
result = newJsoupAnalyzer(result).getString(rule)
}
default:
result = rule
}
}
if result != nil && resolved.ReplaceRegex != "" {
result = applyReplaceRegex(resultString(result), resolved)
}
}
}
if result == nil {
result = ""
}
str := resultString(result)
if strings.Contains(str, "&") {
str = html.UnescapeString(str)
}
if isUrl {
if strings.TrimSpace(str) == "" {
return a.baseUrl, nil
}
return a.absolutize(str), nil
}
return str, nil
}
// GetElement 对应 getElement(ruleStr)。
func (a *AnalyzeRule) GetElement(ruleStr string) (any, error) {
if ruleStr == "" {
return nil, nil
}
var result any
content := a.content
ruleList := SplitSourceRule(ruleStr, true, a.isJSON, &a.isRegex)
if content != nil && len(ruleList) > 0 {
result = content
for _, sourceRule := range ruleList {
if err := a.putRule(sourceRule.putMap); err != nil {
return nil, err
}
resolved, err := sourceRule.MakeUpRule(result, a.makeDeps())
if err != nil {
return nil, err
}
if result == nil {
continue
}
rule := resolved.Rule
switch sourceRule.Mode {
case ModeRegex:
result = regexGetElement(resultString(result), splitNotBlankAndTrim(rule, "&&"), 0)
case ModeWebJs:
return nil, ErrWebJSUnsupported
case ModeJs:
result, err = a.evalJS(rule, result)
if err != nil {
return nil, err
}
case ModeJson:
result = newJSONAnalyzer(result).getObject(rule)
case ModeXPath:
result = newXPathAnalyzer(result).getElements(rule)
default:
result = newJsoupAnalyzer(result).getElements(rule)
}
if resolved.ReplaceRegex != "" {
result = applyReplaceRegex(resultString(result), resolved)
}
}
}
return result, nil
}
// GetElements 对应 getElements(ruleStr)：列表获取。
// 注意：与 Kotlin 一致，这里不做 makeUpRule（无 ## 正则段处理）。
func (a *AnalyzeRule) GetElements(ruleStr string) ([]any, error) {
var result any
content := a.content
ruleList := SplitSourceRule(ruleStr, true, a.isJSON, &a.isRegex)
if content != nil && len(ruleList) > 0 {
result = content
for _, sourceRule := range ruleList {
if err := a.putRule(sourceRule.putMap); err != nil {
return nil, err
}
if result == nil {
continue
}
rule := sourceRule.Rule
var err error
switch sourceRule.Mode {
case ModeRegex:
result = regexGetElements(resultString(result), splitNotBlankAndTrim(rule, "&&"), 0)
case ModeWebJs:
return nil, ErrWebJSUnsupported
case ModeJs:
result, err = a.evalJS(rule, result)
if err != nil {
return nil, err
}
case ModeJson:
result = newJSONAnalyzer(result).getList(rule)
case ModeXPath:
result = newXPathAnalyzer(result).getElements(rule)
default:
result = newJsoupAnalyzer(result).getElements(rule)
}
}
}
if result != nil {
switch t := result.(type) {
case []any:
return t, nil
case []string:
out := make([]any, len(t))
for i, v := range t {
out[i] = v
}
return out, nil
case []*html.Node:
out := make([]any, len(t))
for i, v := range t {
out[i] = v
}
return out, nil
}
}
return []any{}, nil
}
// ─── 工具 ──────────────────────────────────────────────────────────────────
func (a *AnalyzeRule) absolutize(u string) string {
if strings.TrimSpace(u) == "" {
return ""
}
if a.redirectURL != nil {
return GetAbsoluteURLParsed(a.redirectURL, u)
}
return GetAbsoluteURL(a.baseUrl, u)
}
// applyReplaceRegex 对应 replaceRegex(result, resolved)。
func applyReplaceRegex(result string, r ResolvedSourceRule) string {
if r.ReplaceRegex == "" {
return result
}
if r.ReplaceFirst {
return regexReplaceFirstOnFirstMatch(r.ReplaceRegex, result, r.Replacement)
}
return regexReplaceAll(r.ReplaceRegex, result, r.Replacement)
}
// resultString 对应 Kotlin 的 result.toString()。
func resultString(result any) string {
switch t := result.(type) {
case nil:
return ""
case string:
return t
case *html.Node:
return outerHTML(t)
case []string:
return strings.Join(t, "\n")
case []*html.Node:
var sb strings.Builder
for _, n := range t {
sb.WriteString(outerHTML(n))
}
return sb.String()
default:
return anyToString(result)
}
}
internal/service/reader/rule/errors.go
+17
View File
@@ -0,0 +1,17 @@
// Package rule 移植自 legado（refgd/legado，GPL-3.0）的规则引擎：
// io.legado.app.model.analyzeRule 包下的 AnalyzeRule / AnalyzeByJSoup /
// AnalyzeByJSonPath / AnalyzeByXPath / AnalyzeByRegex / AnalyzeUrl / RuleAnalyzer。
// 语义以 legado 源码为基准逐项对齐，注释中标注了对应的 Kotlin 方法名。
package rule
import "errors"
var (
// ErrJsUnsupported 书源规则中包含 JS（<js>/@js:/{{}}/js 选项）。
// JS 引擎（goja + java.* 桥）在 P2 阶段接入，届时移除本错误路径。
ErrJsUnsupported = errors.New("书源使用了 JS 规则，当前阶段暂不支持")
// ErrWebJSUnsupported 书源依赖 webView/webJs 抓取，服务端无头浏览器不在支持范围。
ErrWebJSUnsupported = errors.New("书源依赖 webView 抓取，暂不支持")
// ErrTypeUnsupported 书源 URL 声明了 type（zip/file 等），暂不支持。
ErrTypeUnsupported = errors.New("书源 URL 声明了不支持的 type")
)
internal/service/reader/rule/htmlel.go
+263
View File
@@ -0,0 +1,263 @@
package rule
import (
"bytes"
"strings"
"github.com/andybalholm/cascadia"
"golang.org/x/net/html"
)
// 本文件在 golang.org/x/net/html 的 *html.Node 上实现 jsoup 的元素语义，
// 供 jsoup 风格分析器使用。语义对齐 org.jsoup.nodes.Element。
func isElement(n *html.Node) bool {
return n != nil && n.Type == html.ElementNode
}
// childrenElements 对应 Element.children()：直接子元素。
func childrenElements(n *html.Node) []*html.Node {
var out []*html.Node
if n == nil {
return out
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
if isElement(c) {
out = append(out, c)
}
}
return out
}
// collectElements 对应 jsoup Collector.collect(evaluator, root)：
// 前序遍历，包含 root 自身。
func collectElements(root *html.Node, pred func(*html.Node) bool) []*html.Node {
var out []*html.Node
if root == nil {
return out
}
var walk func(n *html.Node)
walk = func(n *html.Node) {
if n.Type == html.ElementNode && pred(n) {
out = append(out, n)
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
// root 自身参与匹配（jsoup select/getElementsByXxx 均包含自身）
if root.Type == html.ElementNode && pred(root) {
out = append(out, root)
}
for c := root.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
return out
}
func hasClassToken(n *html.Node, class string) bool {
for _, a := range n.Attr {
if a.Key == "class" {
for _, tok := range strings.Fields(a.Val) {
if tok == class {
return true
}
}
}
}
return false
}
// getElementsByClass 对应 Element.getElementsByClass（含自身）。
func getElementsByClass(root *html.Node, class string) []*html.Node {
return collectElements(root, func(n *html.Node) bool { return hasClassToken(n, class) })
}
// getElementsByTag 对应 Element.getElementsByTag（含自身）。
func getElementsByTag(root *html.Node, tag string) []*html.Node {
return collectElements(root, func(n *html.Node) bool { return n.Data == tag })
}
// getElementsById 对应 Collector.collect(Evaluator.Id(id), root)（含自身）。
func getElementsById(root *html.Node, id string) []*html.Node {
return collectElements(root, func(n *html.Node) bool {
for _, a := range n.Attr {
if a.Key == "id" && a.Val == id {
return true
}
}
return false
})
}
// getElementsContainingOwnText 对应 Evaluator.ContentsOwnText 语义：
// ownText 包含目标串的元素。
func getElementsContainingOwnText(root *html.Node, text string) []*html.Node {
return collectElements(root, func(n *html.Node) bool {
return strings.Contains(nodeOwnText(n), text)
})
}
// selectCSS 对应 Element.select(css)：以 root 为起点（含自身）执行 CSS 选择。
func selectCSS(root *html.Node, sel string) []*html.Node {
compiled, err := cascadia.Compile(sel)
if err != nil {
return nil
}
return selectWithCompiled(root, compiled)
}
func selectWithCompiled(root *html.Node, sel cascadia.Selector) []*html.Node {
var out []*html.Node
if root.Type == html.ElementNode && sel(root) {
out = append(out, root)
}
var walk func(n *html.Node)
walk = func(n *html.Node) {
for c := n.FirstChild; c != nil; c = c.NextSibling {
if isElement(c) && sel(c) {
out = append(out, c)
}
walk(c)
}
}
walk(root)
return out
}
// nodeOwnText 对应 Element.ownText()：直接子文本节点，规整空白后空格连接。
func nodeOwnText(n *html.Node) string {
var parts []string
for c := n.FirstChild; c != nil; c = c.NextSibling {
if c.Type == html.TextNode {
t := normalizeWhitespace(c.Data)
if t != "" {
parts = append(parts, t)
}
}
}
return strings.Join(parts, " ")
}
// nodeText 对应 Element.text()：全部后代文本规整空白（<br> 记空格，
// 跳过 script/style），多段空白折叠为单个空格。
func nodeText(n *html.Node) string {
var sb bytes.Buffer
var walk func(n *html.Node)
walk = func(n *html.Node) {
if n.Type == html.TextNode {
sb.WriteString(n.Data)
return
}
if n.Type == html.ElementNode && (n.Data == "script" || n.Data == "style") {
return
}
if n.Type == html.ElementNode && n.Data == "br" {
sb.WriteString(" ")
return
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(n)
return normalizeWhitespace(sb.String())
}
// normalizeWhitespace 对应 jsoup TextUtil 的空白规整。
func normalizeWhitespace(s string) string {
return strings.Join(strings.Fields(s), " ")
}
// nodeTextNodes 对应 Element.textNodes()：直接子文本节点（trim 非空）。
func nodeTextNodes(n *html.Node) []string {
var out []string
for c := n.FirstChild; c != nil; c = c.NextSibling {
if c.Type == html.TextNode {
t := strings.TrimSpace(c.Data)
if t != "" {
out = append(out, t)
}
}
}
return out
}
// nodeData 对应 Element.data()：script/style 的原始内容。
func nodeData(n *html.Node) string {
if n.Type != html.ElementNode || (n.Data != "script" && n.Data != "style") {
return ""
}
var sb bytes.Buffer
for c := n.FirstChild; c != nil; c = c.NextSibling {
if c.Type == html.TextNode {
sb.WriteString(c.Data)
}
}
return sb.String()
}
// outerHTML 对应 Element.outerHtml()。
func outerHTML(n *html.Node) string {
var buf bytes.Buffer
if err := html.Render(&buf, n); err != nil {
return ""
}
return buf.String()
}
// outerHTMLNoScript 对应 getResultLast "html" 分支：移除 script/style 后的 outerHtml。
func outerHTMLNoScript(n *html.Node) string {
clone := cloneNodeShallowTree(n)
removeTags(clone, "script")
removeTags(clone, "style")
return outerHTML(clone)
}
func removeTags(n *html.Node, tag string) {
var toRemove []*html.Node
for c := n.FirstChild; c != nil; c = c.NextSibling {
if isElement(c) && c.Data == tag {
toRemove = append(toRemove, c)
}
removeTags(c, tag)
}
for _, r := range toRemove {
n.RemoveChild(r)
}
}
// cloneNodeShallowTree 深拷贝节点树（html.Render 需要）。
func cloneNodeShallowTree(n *html.Node) *html.Node {
c := &html.Node{
Type: n.Type,
DataAtom: n.DataAtom,
Data: n.Data,
Attr: append([]html.Attribute(nil), n.Attr...),
}
for ch := n.FirstChild; ch != nil; ch = ch.NextSibling {
cc := cloneNodeShallowTree(ch)
c.AppendChild(cc)
}
return c
}
func attrValue(n *html.Node, key string) string {
for _, a := range n.Attr {
if a.Key == key {
return a.Val
}
}
return ""
}
// parseHTML 对应 AnalyzeByJSoup.parse：字符串转节点树。
// x/net/html 会补全 <html><body> 结构，选择器从 document 根开始匹配，
// 与 jsoup 以 Document 为根的选择行为一致。
func parseHTML(s string) *html.Node {
nodes, err := html.Parse(strings.NewReader(s))
if err != nil {
return nil
}
return nodes
}
internal/service/reader/rule/json.go
+193
View File
@@ -0,0 +1,193 @@
package rule
import (
"encoding/json"
"strings"
)
// 本文件对应 AnalyzeByJSonPath.kt（Jayway JsonPath 语义，Go 侧用
// PaesslerAG/jsonpath 实现）。
type jsonAnalyzer struct {
root any
}
func newJSONAnalyzer(doc any) *jsonAnalyzer {
switch t := doc.(type) {
case string:
var v any
if err := json.Unmarshal([]byte(t), &v); err != nil {
return &jsonAnalyzer{root: nil}
}
return &jsonAnalyzer{root: v}
default:
return &jsonAnalyzer{root: doc}
}
}
// jsonRead 对应 ctx.read(rule)：路径求值，失败返回 nil。
func jsonRead(root any, path string) any {
if root == nil || path == "" {
return nil
}
v, err := jsonpathGet(path, root)
if err != nil {
return nil
}
return v
}
// jsonValueToString 对应 Kotlin 的 ob.toString() / joinToString("\n")。
func jsonValueToString(ob any) string {
switch t := ob.(type) {
case nil:
return ""
case string:
return t
case []any:
parts := make([]string, len(t))
for i, e := range t {
parts[i] = jsonValueToString(e)
}
return strings.Join(parts, "\n")
default:
return anyToString(t)
}
}
// getString 对应 AnalyzeByJSonPath.getString。
func (a *jsonAnalyzer) getString(rule string) string {
if rule == "" {
return ""
}
ra := NewRuleAnalyzer(rule, true)
rules := ra.SplitRule("&&", "||")
if len(rules) == 1 {
ra.ReSetPos()
result := ra.InnerRule("{$.", 1, 1, func(inner string) string {
return a.getString(inner)
})
if result == "" {
result = jsonValueToString(jsonRead(a.root, rule))
}
return result
}
var textList []string
for _, rl := range rules {
temp := a.getString(rl)
if temp != "" {
textList = append(textList, temp)
if ra.ElementsType() == "||" {
break
}
}
}
return strings.Join(textList, "\n")
}
// getStringList 对应 AnalyzeByJSonPath.getStringList。
func (a *jsonAnalyzer) getStringList(rule string) []string {
var result []string
if rule == "" {
return result
}
ra := NewRuleAnalyzer(rule, true)
rules := ra.SplitRule("&&", "||", "%%")
if len(rules) == 1 {
ra.ReSetPos()
st := ra.InnerRule("{$.", 1, 1, func(inner string) string {
return a.getString(inner)
})
if st == "" {
ob := jsonRead(a.root, rule)
if ob == nil {
return result
}
if lst, ok := ob.([]any); ok {
for _, o := range lst {
result = append(result, jsonValueToString(o))
}
} else {
result = append(result, jsonValueToString(ob))
}
} else {
result = append(result, st)
}
return result
}
var results [][]string
for _, rl := range rules {
temp := a.getStringList(rl)
if len(temp) > 0 {
results = append(results, temp)
if ra.ElementsType() == "||" {
break
}
}
}
if len(results) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(results[0]); i++ {
for _, temp := range results {
if i < len(temp) {
result = append(result, temp[i])
}
}
}
} else {
for _, temp := range results {
result = append(result, temp...)
}
}
}
return result
}
// getObject 对应 getObject：直接返回求值结果。
func (a *jsonAnalyzer) getObject(rule string) any {
return jsonRead(a.root, rule)
}
// getList 对应 getList：要求路径结果为数组（对应 jayway read<ArrayList>，
// 非数组时 legado 侧捕获异常返回空列表）。
func (a *jsonAnalyzer) getList(rule string) []any {
var result []any
if rule == "" {
return result
}
ra := NewRuleAnalyzer(rule, true)
rules := ra.SplitRule("&&", "||", "%%")
if len(rules) == 1 {
ob := jsonRead(a.root, rules[0])
if lst, ok := ob.([]any); ok {
return lst
}
return result
}
var results [][]any
for _, rl := range rules {
temp := a.getList(rl)
if len(temp) > 0 {
results = append(results, temp)
if ra.ElementsType() == "||" {
break
}
}
}
if len(results) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(results[0]); i++ {
for _, temp := range results {
if i < len(temp) {
result = append(result, temp[i])
}
}
}
} else {
for _, temp := range results {
result = append(result, temp...)
}
}
}
return result
}
internal/service/reader/rule/jsoup.go
+626
View File
@@ -0,0 +1,626 @@
package rule
import (
"strings"
"golang.org/x/net/html"
)
// 本文件对应 AnalyzeByJSoup.kt（含 ElementsSingle 索引语法）。
type jsoupAnalyzer struct {
root *html.Node
}
func newJsoupAnalyzer(doc any) *jsoupAnalyzer {
return &jsoupAnalyzer{root: toHTMLNode(doc)}
}
// toHTMLNode 对应 AnalyzeByJSoup.parse(doc)：节点直通，字符串解析为 HTML。
func toHTMLNode(doc any) *html.Node {
switch t := doc.(type) {
case nil:
return nil
case *html.Node:
return t
default:
return parseHTML(anyToString(t))
}
}
// getStringList 对应 AnalyzeByJSoup.getStringList。
func (a *jsoupAnalyzer) getStringList(ruleStr string) []string {
var textS []string
if ruleStr == "" {
return textS
}
isCss := false
elementsRule := ruleStr
if hasPrefixFold(ruleStr, "@CSS:") {
isCss = true
elementsRule = strings.TrimSpace(ruleStr[5:])
}
if elementsRule == "" {
if d := nodeData(a.root); d != "" {
textS = append(textS, d)
} else {
textS = append(textS, "")
}
return textS
}
ra := NewRuleAnalyzer(elementsRule, false)
ruleStrS := ra.SplitRule("&&", "||", "%%")
var results [][]string
for _, ruleStrX := range ruleStrS {
var temp []string
if isCss {
// 对应 isCss 分支：lastIndexOf('@') 分离选择器与提取规则
lastIndex := strings.LastIndex(ruleStrX, "@")
var elements []*html.Node
var lastRule string
if lastIndex == -1 {
elements = selectCSS(a.root, ruleStrX)
lastRule = "text"
} else {
selector := ruleStrX[:lastIndex]
if strings.TrimSpace(selector) == "" {
elements = []*html.Node{a.root}
} else {
elements = selectCSS(a.root, selector)
}
lastRule = ruleStrX[lastIndex+1:]
}
temp = getResultLast(elements, lastRule)
} else {
temp = a.getResultList(ruleStrX)
}
if len(temp) > 0 {
results = append(results, temp)
if ra.ElementsType() == "||" {
break
}
}
}
if len(results) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(results[0]); i++ {
for _, temp := range results {
if i < len(temp) {
textS = append(textS, temp[i])
}
}
}
} else {
for _, temp := range results {
textS = append(textS, temp...)
}
}
}
return textS
}
// getString 对应 getString：多结果以 \n 连接。
func (a *jsoupAnalyzer) getString(ruleStr string) string {
list := a.getStringList(ruleStr)
if len(list) == 0 {
return ""
}
if len(list) == 1 {
return list[0]
}
return strings.Join(list, "\n")
}
// getString0 对应 getString0：只取第一个。
func (a *jsoupAnalyzer) getString0(ruleStr string) string {
list := a.getStringList(ruleStr)
if len(list) == 0 {
return ""
}
return list[0]
}
// getElements 对应 getElements。
func (a *jsoupAnalyzer) getElements(rule string) []*html.Node {
if rule == "" || a.root == nil {
return nil
}
isCss := false
elementsRule := rule
if hasPrefixFold(rule, "@CSS:") {
isCss = true
elementsRule = strings.TrimSpace(rule[5:])
}
ra := NewRuleAnalyzer(elementsRule, false)
ruleStrS := ra.SplitRule("&&", "||", "%%")
var elementsList [][]*html.Node
if isCss {
for _, ruleStr := range ruleStrS {
tempS := selectCSS(a.root, ruleStr)
elementsList = append(elementsList, tempS)
if len(tempS) > 0 && ra.ElementsType() == "||" {
break
}
}
} else {
for _, ruleStr := range ruleStrS {
rsRule := NewRuleAnalyzer(ruleStr, false)
rsRule.Trim()
rs := rsRule.SplitRule("@")
var el []*html.Node
if len(rs) > 1 {
el = []*html.Node{a.root}
for _, rl := range rs {
var es []*html.Node
for _, et := range el {
es = append(es, a.getElementsOf(et, rl)...)
}
el = es
}
} else {
es := newElementsSingle().getElementsSingle(a.root, ruleStr)
el = es
}
elementsList = append(elementsList, el)
if len(el) > 0 && ra.ElementsType() == "||" {
break
}
}
}
var elements []*html.Node
if len(elementsList) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(elementsList[0]); i++ {
for _, es := range elementsList {
if i < len(es) {
elements = append(elements, es[i])
}
}
}
} else {
for _, es := range elementsList {
elements = append(elements, es...)
}
}
}
return elements
}
// getElementsOf 对应私有 getElements(temp, rule) 递归。
func (a *jsoupAnalyzer) getElementsOf(temp *html.Node, rule string) []*html.Node {
if temp == nil || rule == "" {
return nil
}
isCss := false
elementsRule := rule
if hasPrefixFold(rule, "@CSS:") {
isCss = true
elementsRule = strings.TrimSpace(rule[5:])
}
ra := NewRuleAnalyzer(elementsRule, false)
ruleStrS := ra.SplitRule("&&", "||", "%%")
var elementsList [][]*html.Node
if isCss {
for _, ruleStr := range ruleStrS {
tempS := selectCSS(temp, ruleStr)
elementsList = append(elementsList, tempS)
if len(tempS) > 0 && ra.ElementsType() == "||" {
break
}
}
} else {
for _, ruleStr := range ruleStrS {
rsRule := NewRuleAnalyzer(ruleStr, false)
rsRule.Trim()
rs := rsRule.SplitRule("@")
var el []*html.Node
if len(rs) > 1 {
el = []*html.Node{temp}
for _, rl := range rs {
var es []*html.Node
for _, et := range el {
es = append(es, a.getElementsOf(et, rl)...)
}
el = es
}
} else {
el = newElementsSingle().getElementsSingle(temp, ruleStr)
}
elementsList = append(elementsList, el)
if len(el) > 0 && ra.ElementsType() == "||" {
break
}
}
}
var elements []*html.Node
if len(elementsList) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(elementsList[0]); i++ {
for _, es := range elementsList {
if i < len(es) {
elements = append(elements, es[i])
}
}
}
} else {
for _, es := range elementsList {
elements = append(elements, es...)
}
}
}
return elements
}
// getResultList 对应 getResultList：按 "@" 步进，最后一段作为提取规则。
func (a *jsoupAnalyzer) getResultList(ruleStr string) []string {
if ruleStr == "" {
return nil
}
elements := []*html.Node{a.root}
rule := NewRuleAnalyzer(ruleStr, false)
rule.Trim()
rules := rule.SplitRule("@")
last := len(rules) - 1
for i := 0; i < last; i++ {
var es []*html.Node
for _, elt := range elements {
es = append(es, newElementsSingle().getElementsSingle(elt, rules[i])...)
}
elements = es
}
if len(elements) == 0 {
return nil
}
return getResultLast(elements, rules[last])
}
// getResultLast 对应 getResultLast：按最后一个规则提取内容。
func getResultLast(elements []*html.Node, lastRule string) []string {
var textS []string
switch lastRule {
case "text":
for _, element := range elements {
if text := nodeText(element); text != "" {
textS = append(textS, text)
}
}
case "textNodes":
for _, element := range elements {
tn := nodeTextNodes(element)
if len(tn) > 0 {
textS = append(textS, strings.Join(tn, "\n"))
}
}
case "ownText":
for _, element := range elements {
if text := nodeOwnText(element); text != "" {
textS = append(textS, text)
}
}
case "html":
for _, element := range elements {
if h := outerHTMLNoScript(element); h != "" {
textS = append(textS, h)
}
}
case "all":
var sb strings.Builder
for _, element := range elements {
sb.WriteString(outerHTML(element))
}
textS = append(textS, sb.String())
default:
for _, element := range elements {
url := attrValue(element, lastRule)
if url == "" || containsStr(textS, url) {
continue
}
textS = append(textS, url)
}
}
return textS
}
func containsStr(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
}
// ─── ElementsSingle：索引语法（对应 data class ElementsSingle） ─────────────
type indexRange struct {
start *int
end *int
step int
}
type elementsSingle struct {
split byte // '.' 选择 / '!' 排除 / ' ' 无索引
beforeRule string
indexDefault []int
indexes []any // int 或 indexRange
}
func newElementsSingle() *elementsSingle {
return &elementsSingle{split: '.'}
}
func (e *elementsSingle) getElementsSingle(temp *html.Node, rule string) []*html.Node {
e.findIndexSet(rule)
var elements []*html.Node
if e.beforeRule == "" {
elements = childrenElements(temp)
} else {
rules := strings.Split(e.beforeRule, ".")
arg := ""
if len(rules) > 1 {
arg = rules[1]
}
switch rules[0] {
case "children":
elements = childrenElements(temp)
case "class":
if arg != "" {
elements = getElementsByClass(temp, arg)
}
case "tag":
if arg != "" {
elements = getElementsByTag(temp, arg)
}
case "id":
if arg != "" {
elements = getElementsById(temp, arg)
}
case "text":
if arg != "" {
elements = getElementsContainingOwnText(temp, arg)
}
default:
elements = selectCSS(temp, e.beforeRule)
}
}
// 索引集合：slice+set 模拟 Kotlin LinkedHashSet 的插入顺序去重
indexSet := make([]int, 0, len(elements))
seen := make(map[int]bool)
addIndex := func(ix int) {
if !seen[ix] {
seen[ix] = true
indexSet = append(indexSet, ix)
}
}
lenn := len(elements)
lastIndexes := -1
if len(e.indexDefault) > 0 {
lastIndexes = len(e.indexDefault) - 1
} else if len(e.indexes) > 0 {
lastIndexes = len(e.indexes) - 1
}
if len(e.indexes) == 0 {
// 旧式索引：逆向遍历还原顺序
for ix := lastIndexes; ix >= 0; ix-- {
it := e.indexDefault[ix]
if it >= 0 && it < lenn {
addIndex(it)
} else if it < 0 && lenn >= -it {
addIndex(it + lenn)
}
}
} else {
for ix := lastIndexes; ix >= 0; ix-- {
if rg, ok := e.indexes[ix].(indexRange); ok {
start := 0
if rg.start != nil {
start = *rg.start
}
if start < 0 {
start += lenn
}
end := lenn - 1
if rg.end != nil {
end = *rg.end
}
if end < 0 {
end += lenn
}
if (start < 0 && end < 0) || (start >= lenn && end >= lenn) {
continue
}
if start >= lenn {
start = lenn - 1
} else if start < 0 {
start = 0
}
if end >= lenn {
end = lenn - 1
} else if end < 0 {
end = 0
}
if start == end || rg.step >= lenn {
addIndex(start)
continue
}
step := rg.step
if step > 0 {
// 正向步长原样使用
} else if -step < lenn {
step = step + lenn
} else {
step = 1
}
if end > start {
for i := start; i <= end; i += step {
addIndex(i)
}
} else {
for i := start; i >= end; i -= step {
addIndex(i)
}
}
} else {
it := e.indexes[ix].(int)
if it >= 0 && it < lenn {
addIndex(it)
} else if it < 0 && lenn >= -it {
addIndex(it + lenn)
}
}
}
}
if e.split == '!' {
exclude := make(map[int]bool)
for _, ix := range indexSet {
exclude[ix] = true
}
var es []*html.Node
for i, el := range elements {
if !exclude[i] {
es = append(es, el)
}
}
elements = es
} else if e.split == '.' {
var es []*html.Node
for _, ix := range indexSet {
if ix >= 0 && ix < lenn {
es = append(es, elements[ix])
}
}
elements = es
}
return elements
}
// findIndexSet 对应 ElementsSingle.findIndexSet：从右向左解析索引。
// 支持旧式 tag.div.-1:10:2 / tag.div!0:3 与新式 tag.div[!-1, 3:-2:-10, 2]。
func (e *elementsSingle) findIndexSet(rule string) {
rus := []rune(strings.TrimSpace(rule))
n := len(rus)
if n == 0 {
e.split = ' '
e.beforeRule = ""
return
}
var curList []*int // 区间临时列表（逆向压入：右端、左端、间隔）
l := "" // 暂存数字字符串
curMinus := false
head := rus[n-1] == ']'
length := n
if head {
length-- // 跳过尾部 ']'
}
findLoop:
for length >= 0 {
length--
if length < 0 {
break
}
rl := rus[length]
if rl == ' ' {
continue
}
if rl >= '0' && rl <= '9' {
l = string(rl) + l
continue
}
if rl == '-' {
curMinus = true
continue
}
var curInt *int
if l != "" {
v := parseIntSafe(l)
if curMinus {
v = -v
}
curInt = &v
}
if head {
switch rl {
case ':':
curList = append(curList, curInt)
default:
if len(curList) == 0 {
if curInt == nil {
break findLoop // 是 jsoup 选择器而非索引列表
}
e.indexes = append(e.indexes, *curInt)
} else {
rg := indexRange{start: curInt, end: curList[len(curList)-1], step: 1}
if len(curList) == 2 && curList[0] != nil {
rg.step = *curList[0]
}
e.indexes = append(e.indexes, rg)
curList = curList[:0]
}
if rl == '!' {
e.split = '!'
for {
length--
if length < 0 {
break
}
rl = rus[length]
if rl != ' ' {
break
}
}
}
if rl == '[' {
if length < 0 {
length = 0
}
e.beforeRule = string(rus[:length])
return
}
if rl != ',' {
break findLoop
}
}
} else {
if rl == '!' || rl == '.' || rl == ':' {
v := 0
if curInt != nil {
v = *curInt
}
e.indexDefault = append(e.indexDefault, v)
if rl != ':' {
e.split = byte(rl)
e.beforeRule = string(rus[:length])
return
}
} else {
break findLoop
}
}
l = ""
curMinus = false
}
e.split = ' '
e.beforeRule = string(rus)
}
func parseIntSafe(s string) int {
n := 0
for _, c := range s {
if c < '0' || c > '9' {
return 0
}
n = n*10 + int(c-'0')
}
return n
}
internal/service/reader/rule/netutil.go
+180
View File
@@ -0,0 +1,180 @@
package rule
import (
"fmt"
"net/url"
"strings"
"golang.org/x/text/encoding/htmlindex"
"golang.org/x/text/encoding/unicode"
)
// 本文件对应 legado 的 NetworkUtils.kt / EncoderUtils.kt 中与规则引擎相关的函数。
// GetAbsoluteURL 对应 NetworkUtils.getAbsoluteURL(baseURL: String?, relativePath)。
// baseURL 会先截掉 ",{...}" 选项段（substringBefore(",")）。
func GetAbsoluteURL(baseURL, relativePath string) string {
rel := strings.TrimSpace(relativePath)
if isAbsURL(rel) || isDataURL(rel) || strings.HasPrefix(rel, "javascript") {
if strings.HasPrefix(rel, "javascript") {
return ""
}
return rel
}
if baseURL == "" || isDataURL(baseURL) {
return rel
}
base := baseURL
if i := strings.Index(base, ","); i >= 0 {
base = base[:i]
}
baseURLParsed, err := url.Parse(strings.TrimSpace(base))
if err != nil {
return rel
}
return GetAbsoluteURLParsed(baseURLParsed, rel)
}
// GetAbsoluteURLParsed 对应 NetworkUtils.getAbsoluteURL(baseURL: URL?, relativePath)。
func GetAbsoluteURLParsed(base *url.URL, relativePath string) string {
rel := strings.TrimSpace(relativePath)
if base == nil {
return rel
}
if isAbsURL(rel) || isDataURL(rel) {
return rel
}
if strings.HasPrefix(rel, "javascript") {
return ""
}
ref, err := url.Parse(rel)
if err != nil {
return rel
}
return base.ResolveReference(ref).String()
}
// GetBaseUrl 对应 NetworkUtils.getBaseUrl：scheme://host[:port]。
func GetBaseUrl(u string) string {
if len(u) >= 8 && (strings.EqualFold(u[:7], "http://") || (len(u) >= 9 && strings.EqualFold(u[:8], "https://"))) {
if idx := strings.Index(u[8:], "/"); idx >= 0 {
return u[:8+idx]
}
return u
}
return ""
}
// isAbsURL 对应 String.isAbsUrl()。
func isAbsURL(s string) bool {
return strings.HasPrefix(s, "https://") || strings.HasPrefix(s, "http://")
}
func isDataURL(s string) bool {
return strings.HasPrefix(s, "data:")
}
// notNeedEncodingQuery / notNeedEncodingForm 对应 NetworkUtils 的两个 BitSet。
var (
notNeedEncodingQuery = buildEncodingSet("!$&()*+,-./:;=?@[\\]^_`{|}~")
notNeedEncodingForm = buildEncodingSet("*-._")
)
func buildEncodingSet(extra string) map[rune]bool {
set := make(map[rune]bool, 128)
for r := 'a'; r <= 'z'; r++ {
set[r] = true
}
for r := 'A'; r <= 'Z'; r++ {
set[r] = true
}
for r := '0'; r <= '9'; r++ {
set[r] = true
}
for _, r := range extra {
set[r] = true
}
return set
}
func isDigit16(r byte) bool {
return (r >= '0' && r <= '9') || (r >= 'a' && r <= 'f') || (r >= 'A' && r <= 'F')
}
// encodedQuery 对应 NetworkUtils.encodedQuery(str)：判断字符串是否已按
// urlEncode 规范编码（无需再编码返回 true）。
func encodedQuery(s string) bool {
return encodedWith(s, notNeedEncodingQuery)
}
// encodedForm 对应 NetworkUtils.encodedForm(str)。
func encodedForm(s string) bool {
return encodedWith(s, notNeedEncodingForm)
}
func encodedWith(s string, allow map[rune]bool) bool {
rs := []rune(s)
for i := 0; i < len(rs); i++ {
r := rs[i]
if r < 128 && allow[r] {
continue
}
if r == '%' && i+2 < len(rs) && isDigit16(byte(rs[i+1])) && isDigit16(byte(rs[i+2])) {
i += 2
continue
}
return false
}
return len(rs) > 0
}
// JSEscape 对应 EncoderUtils.escape（JS escape() 语义）。
func JSEscape(src string) string {
var sb strings.Builder
for _, r := range src {
code := int(r)
if (code >= 48 && code <= 57) || (code >= 65 && code <= 90) || (code >= 97 && code <= 122) {
sb.WriteRune(r)
continue
}
switch {
case code < 16:
fmt.Fprintf(&sb, "%%0%X", code)
case code < 256:
fmt.Fprintf(&sb, "%%%X", code)
default:
fmt.Fprintf(&sb, "%%u%04X", code)
}
}
return sb.String()
}
// DecodeBytes 按字符集名解码字节流，name 为空时按 UTF-8（带 BOM 处理）。
func DecodeBytes(b []byte, name string) (string, error) {
if len(b) >= 3 && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
return string(b[3:]), nil
}
if name == "" || strings.EqualFold(name, "utf-8") || strings.EqualFold(name, "utf8") {
dec := unicode.UTF8.NewDecoder()
out, err := dec.Bytes(b)
if err != nil {
return string(b), nil
}
return string(out), nil
}
enc, err := htmlindex.Get(name)
if err != nil {
return string(b), nil
}
out, derr := enc.NewDecoder().Bytes(b)
if derr != nil {
return string(b), nil
}
return string(out), nil
}
// LooksLikeJSON 对应 String.isJson()（宽松：trim 后以 { 或 [ 开头）。
func LooksLikeJSON(s string) bool {
s = strings.TrimSpace(s)
return strings.HasPrefix(s, "{") || strings.HasPrefix(s, "[")
}
internal/service/reader/rule/regexan.go
+149
View File
@@ -0,0 +1,149 @@
package rule
import (
"strings"
"github.com/dlclark/regexp2"
)
// 本文件对应 AnalyzeByRegex.kt。Java 正则语义用 regexp2 对齐
// （支持前向后向断言与反向引用），匹配循环对齐 Matcher.find()。
// splitNotBlankAndTrim 对应 String.splitNotBlank("&&")：切分并去空白项。
func splitNotBlankAndTrim(s, sep string) []string {
var out []string
for _, p := range strings.Split(s, sep) {
if t := strings.TrimSpace(p); t != "" {
out = append(out, t)
}
}
return out
}
// regexGetElement 对应 AnalyzeByRegex.getElement：多段正则串联，
// 最终返回第一个匹配的全部分组（含 group 0）。
func regexGetElement(res string, regs []string, index int) []string {
if index >= len(regs) {
return nil
}
re, err := regexp2.Compile(regs[index], regexp2.None)
if err != nil {
return nil
}
m, err := re.FindStringMatchStartingAt(res, 0)
if err != nil || m == nil {
return nil
}
if index+1 == len(regs) {
info := make([]string, 0, len(m.Groups()))
for _, g := range m.Groups() {
if len(g.Captures) > 0 {
info = append(info, g.Captures[0].String())
} else {
info = append(info, "")
}
}
return info
}
var sb strings.Builder
for m != nil {
sb.WriteString(m.String())
m, _ = re.FindNextMatch(m)
}
return regexGetElement(sb.String(), regs, index+1)
}
// regexGetElements 对应 AnalyzeByRegex.getElements：多段正则串联，
// 最终按每个匹配返回一组分组列表。
func regexGetElements(res string, regs []string, index int) [][]string {
if index >= len(regs) {
return nil
}
re, err := regexp2.Compile(regs[index], regexp2.None)
if err != nil {
return nil
}
m, err := re.FindStringMatchStartingAt(res, 0)
if err != nil || m == nil {
return nil
}
if index+1 == len(regs) {
var books [][]string
for m != nil {
info := make([]string, 0, len(m.Groups()))
for _, g := range m.Groups() {
if len(g.Captures) > 0 {
info = append(info, g.Captures[0].String())
} else {
info = append(info, "")
}
}
books = append(books, info)
m, _ = re.FindNextMatch(m)
}
return books
}
var sb strings.Builder
for m != nil {
sb.WriteString(m.String())
m, _ = re.FindNextMatch(m)
}
return regexGetElements(sb.String(), regs, index+1)
}
// regexReplaceAll 对应 Kotlin Regex.replace(result, replacement)
// （Java $N 分组替换语义，regexp2 的 Replace 原生支持）。
func regexReplaceAll(pattern, result, replacement string) string {
re, err := regexp2.Compile(pattern, regexp2.None)
if err != nil {
return strings.ReplaceAll(result, pattern, replacement)
}
out, err := re.Replace(result, replacement, 0, -1)
if err != nil {
return result
}
return out
}
// ApplyReplaceRegexString 应用书源 replaceRegex 字符串（"##pattern##replace[##x]" 格式），
// 对应 ContentRule.replaceRegex 的处理。
func ApplyReplaceRegexString(content, replaceRegex string) string {
if replaceRegex == "" {
return content
}
segs := strings.Split(replaceRegex, "##")
if len(segs) < 2 {
return content
}
pattern := segs[1]
replacement := ""
replaceFirst := false
if len(segs) > 2 {
replacement = segs[2]
}
if len(segs) > 3 {
replaceFirst = true
}
if replaceFirst {
return regexReplaceFirstOnFirstMatch(pattern, content, replacement)
}
return regexReplaceAll(pattern, content, replacement)
}
// regexReplaceFirstOnFirstMatch 对应 replaceRegex 的 replaceFirst 分支：
// 找到第一个匹配（无匹配返回 ""），在匹配文本上做首次替换。
func regexReplaceFirstOnFirstMatch(pattern, result, replacement string) string {
re, err := regexp2.Compile(pattern, regexp2.None)
if err != nil {
return replacement
}
m, err := re.FindStringMatch(result)
if err != nil || m == nil {
return ""
}
out, err := re.Replace(m.String(), replacement, 0, 1)
if err != nil {
return m.String()
}
return out
}
internal/service/reader/rule/rule_test.go
+341
View File
@@ -0,0 +1,341 @@
package rule
import (
"strings"
"testing"
)
// ─── RuleAnalyzer ──────────────────────────────────────────────────────────
func TestRuleAnalyzerSplitAndOr(t *testing.T) {
ra := NewRuleAnalyzer("class.a&&tag.b&&id.c", false)
rules := ra.SplitRule("&&", "||", "%%")
if len(rules) != 3 || rules[0] != "class.a" || rules[1] != "tag.b" || rules[2] != "id.c" {
t.Fatalf("unexpected split: %#v", rules)
}
if ra.ElementsType() != "&&" {
t.Fatalf("elementsType = %q, want &&", ra.ElementsType())
}
// 与 Kotlin 一致：consumeToAny 取最左侧出现的分隔符，
// 之后只按该分隔符切分（混合操作符时右侧保留原样，由上层递归处理）。
ra2 := NewRuleAnalyzer("class.a&&tag.b||id.c", false)
rules2 := ra2.SplitRule("&&", "||", "%%")
if len(rules2) != 2 || rules2[0] != "class.a" || rules2[1] != "tag.b||id.c" {
t.Fatalf("mixed split: %#v", rules2)
}
}
func TestRuleAnalyzerBalancedGroup(t *testing.T) {
// && 在选择器平衡组内不应被切分
ra := NewRuleAnalyzer(`tag.div[class="x&&y"]@text&&class.z`, false)
rules := ra.SplitRule("&&", "||", "%%")
if len(rules) != 2 {
t.Fatalf("unexpected split: %#v", rules)
}
if rules[0] != `tag.div[class="x&&y"]@text` {
t.Fatalf("rule[0] = %q", rules[0])
}
}
func TestRuleAnalyzerSplitByAt(t *testing.T) {
ra := NewRuleAnalyzer("class.bookbox@h4@a@text", false)
ra.Trim()
rules := ra.SplitRule("@")
if len(rules) != 4 {
t.Fatalf("unexpected split: %#v", rules)
}
}
// ─── SourceRule 模式识别 ────────────────────────────────────────────────────
func TestSourceRuleModeDetection(t *testing.T) {
cases := []struct {
rule string
contentIsJSON bool
want Mode
}{
{"class.a@text", false, ModeDefault},
{"$.data.name", false, ModeJson},
{"$[0].name", false, ModeJson},
{"//div[@class='a']/text()", false, ModeXPath},
{"@XPath://div", false, ModeXPath},
{"@Json:$.a", false, ModeJson},
{"@CSS:.a@text", false, ModeDefault},
{"title", true, ModeJson}, // 内容为 JSON 时默认走 Json 模式
{"/html/body", false, ModeXPath},
}
for _, c := range cases {
got := newSourceRule(c.rule, ModeDefault, c.contentIsJSON)
if got.Mode != c.want {
t.Errorf("rule %q mode = %v, want %v", c.rule, got.Mode, c.want)
}
}
}
func TestSourceRuleSplitPut(t *testing.T) {
sr := newSourceRule(`class.a@text@put:{"key1":"class.b@text"}`, ModeDefault, false)
if sr.Rule != "class.a@text" {
t.Fatalf("rule after put split = %q", sr.Rule)
}
if sr.putMap["key1"] != "class.b@text" {
t.Fatalf("putMap = %#v", sr.putMap)
}
}
func TestMakeUpRuleGetVariable(t *testing.T) {
sr := newSourceRule(`@get:{kw}`, ModeDefault, false)
deps := &RuleDeps{Get: func(key string) string { return "搜索词" }}
resolved, err := sr.MakeUpRule(nil, deps)
if err != nil {
t.Fatal(err)
}
if resolved.Rule != "搜索词" {
t.Fatalf("resolved = %q", resolved.Rule)
}
}
func TestSplitSourceRuleJSBlocks(t *testing.T) {
rules := SplitSourceRule(`<js>1+1</js>class.a@text`, false, false, nil)
if len(rules) != 2 || rules[0].Mode != ModeJs || rules[1].Mode != ModeDefault {
t.Fatalf("rules = %#v", rules)
}
if rules[0].Rule != "1+1" {
t.Fatalf("js body = %q", rules[0].Rule)
}
}
// ─── jsoup 分析器 ──────────────────────────────────────────────────────────
const testHTML = `<!DOCTYPE html>
<html><body>
<div class="box" id="main">
<div class="item"><h3><a href="/book/1">斗破苍穹</a></h3><span class="author">天蚕土豆</span><p class="intro">测试<strong>简介</strong></p></div>
<div class="item"><h3><a href="/book/2">凡人修仙传</a></h3><span class="author">忘语</span><p class="intro">凡人流</p></div>
<div class="item"><h3><a href="/book/3">遮天</a></h3><span class="author">辰东</span><p class="intro">九龙拉棺</p></div>
</div>
</body></html>`
func TestJsoupGetElementsAndFields(t *testing.T) {
a := newJsoupAnalyzer(testHTML)
els := a.getElements("class.item")
if len(els) != 3 {
t.Fatalf("elements = %d, want 3", len(els))
}
name := a.getString("class.item.0@tag.h3@tag.a@text")
if name != "斗破苍穹" {
t.Fatalf("name = %q", name)
}
href := a.getString("class.item.0@tag.h3@tag.a@href")
if href != "/book/1" {
t.Fatalf("href = %q", href)
}
author := a.getString("class.item.1@class.author@text")
if author != "忘语" {
t.Fatalf("author = %q", author)
}
// all：拼接所有
names := a.getStringList("class.item@tag.h3@tag.a@text")
if len(names) != 3 || names[2] != "遮天" {
t.Fatalf("names = %#v", names)
}
}
func TestJsoupTextNodesAndOwnText(t *testing.T) {
a := newJsoupAnalyzer(testHTML)
// ownText：不含子元素文本
intro := a.getString("class.item.0@tag.p@ownText")
if intro != "测试" {
t.Fatalf("ownText = %q", intro)
}
// text：含子元素文本（jsoup 不在内联元素间补空格）
full := a.getString("class.item.0@tag.p@text")
if full != "测试简介" {
t.Fatalf("text = %q", full)
}
}
func TestJsoupIndexSyntax(t *testing.T) {
a := newJsoupAnalyzer(testHTML)
// 负索引：最后一个
last := a.getString("class.item.-1@tag.a@text")
if last != "遮天" {
t.Fatalf("last = %q", last)
}
// 新式区间索引
firstTwo := a.getStringList("class.item[0:1]@tag.a@text")
if len(firstTwo) != 2 || firstTwo[0] != "斗破苍穹" || firstTwo[1] != "凡人修仙传" {
t.Fatalf("range = %#v", firstTwo)
}
}
func TestJsoupAndOrPercent(t *testing.T) {
a := newJsoupAnalyzer(testHTML)
// &&：合并两路结果
merged := a.getStringList(`class.item.0@tag.a@text&&class.item.1@tag.a@text`)
if len(merged) != 2 {
t.Fatalf("&& merged = %#v", merged)
}
// ||：第一个非空即停
or := a.getStringList(`id.notexist@text||class.item.0@tag.a@text`)
if len(or) != 1 || or[0] != "斗破苍穹" {
t.Fatalf("|| result = %#v", or)
}
}
func TestJsoupCSSMode(t *testing.T) {
a := newJsoupAnalyzer(testHTML)
// @CSS: 末段为提取规则（与 jsoup 分析器一致的 @ 分离）
name := a.getString("@CSS:#main .item:nth-child(1) a@text")
if name != "斗破苍穹" {
t.Fatalf("@css name = %q", name)
}
href := a.getString("@CSS:.item:nth-child(2) a@href")
if href != "/book/2" {
t.Fatalf("@css href = %q", href)
}
}
// ─── JSONPath 分析器 ───────────────────────────────────────────────────────
const testJSON = `{"data":{"list":[{"title":"第一章","url":"/c/1"},{"title":"第二章","url":"/c/2"}],"name":"测试书","page":2}}`
func TestJSONPathGetString(t *testing.T) {
a := newJSONAnalyzer(testJSON)
if got := a.getString("$.data.name"); got != "测试书" {
t.Fatalf("name = %q", got)
}
if got := a.getString("$.data.list[*].title"); got != "第一章\n第二章" {
t.Fatalf("titles = %q", got)
}
// || 首个非空
if got := a.getString("$.data.missing||$.data.name"); got != "测试书" {
t.Fatalf("|| = %q", got)
}
}
func TestJSONPathInnerRule(t *testing.T) {
a := newJSONAnalyzer(testJSON)
// {$.data.page} 内嵌规则替换
got := a.getString("/api/list/{$.data.page}/next.json")
if got != "/api/list/2/next.json" {
t.Fatalf("inner = %q", got)
}
}
func TestJSONPathListAndElementContext(t *testing.T) {
a := newJSONAnalyzer(testJSON)
list := a.getList("$.data.list[*]")
if len(list) != 2 {
t.Fatalf("list = %#v", list)
}
// 以列表元素为根继续求值（对应 getString(rule, element)）
sub := newJSONAnalyzer(list[0])
if got := sub.getString("$.title"); got != "第一章" {
t.Fatalf("element title = %q", got)
}
}
// ─── XPath 分析器 ──────────────────────────────────────────────────────────
func TestXPathAnalyzer(t *testing.T) {
a := newXPathAnalyzer(testHTML)
if got := a.getString(`//div[@class="item"][1]//a/text()`); got != "斗破苍穹" {
t.Fatalf("xpath = %q", got)
}
hrefs := a.getStringList(`//div[@class="item"]//a/@href`)
if len(hrefs) != 3 {
t.Fatalf("hrefs = %#v", hrefs)
}
els := a.getElements(`//div[@class="item"]`)
if len(els) != 3 {
t.Fatalf("elements = %d", len(els))
}
}
// ─── 正则分析器 ────────────────────────────────────────────────────────────
func TestRegexAnalyzer(t *testing.T) {
content := "第1章 开始 第2章 继续 第3章 结束"
els := regexGetElements(content, []string{`第(\d+)章 ([^ ]+)`}, 0)
if len(els) != 3 || els[0][1] != "1" || els[2][2] != "结束" {
t.Fatalf("regex elements = %#v", els)
}
}
func TestReplaceRegex(t *testing.T) {
a := NewAnalyzeRule()
a.SetContent(testHTML, "http://x.com")
// ## 目标##替换
got, err := a.GetString(`class.item.0@tag.a@text##斗破##破斗`, nil, false)
if err != nil {
t.Fatal(err)
}
if got != "破斗苍穹" {
t.Fatalf("replace = %q", got)
}
}
// ─── AnalyzeUrl ────────────────────────────────────────────────────────────
func TestParseAnalyzeUrlBasic(t *testing.T) {
req, err := ParseAnalyzeUrl("https://example.com/search/{{key}}/1.html", "斗罗", 1, "")
if err != nil {
t.Fatal(err)
}
// 与 legado 一致：路径段不做百分号编码（执行时由 HTTP 客户端转义）
if !strings.Contains(req.URL, "/search/斗罗/1.html") {
t.Fatalf("url = %q", req.URL)
}
}
func TestParseAnalyzeUrlPageList(t *testing.T) {
// <1,20,40>：page=1 → 1；page=2 → 20；page=5 → 40（取最后一档）
for page, want := range map[int]string{1: "1", 2: "20", 5: "40"} {
req, err := ParseAnalyzeUrl("https://e.com/list/<1,20,40>.html", "k", page, "")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(req.URL, want+".html") {
t.Fatalf("page=%d url = %q", page, req.URL)
}
}
}
func TestParseAnalyzeUrlGBKQueryEncoding(t *testing.T) {
req, err := ParseAnalyzeUrl("https://e.com/search.php?keyword={{key}},{\"charset\":\"gbk\"}", "斗罗", 1, "")
if err != nil {
t.Fatal(err)
}
// "斗罗" 的 GBK 编码 = B6 B7 C2 DE
if !strings.Contains(req.URL, "%B6%B7%C2%DE") {
t.Fatalf("gbk url = %q", req.URL)
}
}
func TestParseAnalyzeUrlPostForm(t *testing.T) {
req, err := ParseAnalyzeUrl(`https://e.com/search,{"method":"POST","body":"searchkey={{key}}&submit=go"}`, "斗罗", 1, "")
if err != nil {
t.Fatal(err)
}
if req.Method != "POST" {
t.Fatalf("method = %s", req.Method)
}
if !req.IsForm {
t.Fatalf("body should be form, got %q", req.Body)
}
if !strings.Contains(req.Body, "searchkey=") {
t.Fatalf("body = %q", req.Body)
}
}
func TestParseAnalyzeUrlWebViewUnsupported(t *testing.T) {
req, err := ParseAnalyzeUrl(`https://e.com/x,{"webView":true}`, "k", 1, "")
if err != nil {
t.Fatal(err)
}
if req.Unsupported == nil {
t.Fatal("expected unsupported for webView")
}
}
// ─── httptest 端到端：书源 JSON → 搜索解析全链路（见 reader 包 reader_test.go） ──
internal/service/reader/rule/ruleanalyzer.go
+416
View File
@@ -0,0 +1,416 @@
package rule
import "strings"
// RuleAnalyzer 移植自 legado RuleAnalyzer.kt，逐方法对齐。
// 用于按 "@"/"&&"/"||"/"%%"/"@" 切分规则字符串，切分时跳过引号内与
// 平衡组（[...]、(...)）内的分隔符，避免与选择器或正则内容冲突。
// RuleAnalyzer 是无状态的切分器实例（一次使用）。
type RuleAnalyzer struct {
queue string
pos int
start int
startX int
rule []string
step int
elementsType string
code bool
}
// NewRuleAnalyzer 对应 RuleAnalyzer(data, code)；code=true 时平衡组按
// 代码语义（处理转义、区分 [] 与 ()）处理，用于 jsonPath 规则。
func NewRuleAnalyzer(data string, code bool) *RuleAnalyzer {
return &RuleAnalyzer{queue: data, code: code}
}
// ElementsType 返回上次 splitRule 使用的组合符（"&&"/"||"/"%%"）。
func (a *RuleAnalyzer) ElementsType() string { return a.elementsType }
// Rules 返回切分结果。
func (a *RuleAnalyzer) Rules() []string { return a.rule }
// Trim 对应 trim()：修剪当前规则之前的 "@" 或不可见字符。
func (a *RuleAnalyzer) Trim() {
if a.pos >= len(a.queue) {
return
}
if a.queue[a.pos] == '@' || a.queue[a.pos] < '!' {
a.pos++
for a.pos < len(a.queue) && (a.queue[a.pos] == '@' || a.queue[a.pos] < '!') {
a.pos++
}
a.start = a.pos
a.startX = a.pos
}
}
// ReSetPos 对应 reSetPos()。
func (a *RuleAnalyzer) ReSetPos() {
a.pos = 0
a.startX = 0
}
// consumeTo 对应 consumeTo(seq)。
func (a *RuleAnalyzer) consumeTo(seq string) bool {
a.start = a.pos
offset := strings.Index(a.queue[a.pos:], seq)
if offset == -1 {
return false
}
a.pos += offset
return true
}
// consumeToAny 对应 consumeToAny(seq)。
func (a *RuleAnalyzer) consumeToAny(seqs ...string) bool {
pos := a.pos
for pos != len(a.queue) {
for _, s := range seqs {
if strings.HasPrefix(a.queue[pos:], s) {
a.step = len(s)
a.pos = pos
return true
}
}
pos++
}
return false
}
// findToAny 对应 findToAny(seq: Char)。
func (a *RuleAnalyzer) findToAny(chars ...byte) int {
pos := a.pos
for pos != len(a.queue) {
for _, c := range chars {
if a.queue[pos] == c {
return pos
}
}
pos++
}
return -1
}
// chompCodeBalanced 对应 chompCodeBalanced(open, close)：拉出一个平衡组，
// 存在转义文本，'[' 与参数对 (open/close) 分别计数。
func (a *RuleAnalyzer) chompCodeBalanced(open, close byte) bool {
pos := a.pos
depth := 0
otherDepth := 0
inSingle := false
inDouble := false
for {
if pos == len(a.queue) {
break
}
c := a.queue[pos]
pos++
if c != '\\' {
if c == '\'' && !inDouble {
inSingle = !inSingle
} else if c == '"' && !inSingle {
inDouble = !inDouble
}
if inSingle || inDouble {
continue
}
if c == '[' {
depth++
} else if c == ']' {
depth--
} else if depth == 0 {
if c == open {
otherDepth++
} else if c == close {
otherDepth--
}
}
} else {
pos++
}
if !(depth > 0 || otherDepth > 0) {
break
}
}
if depth > 0 || otherDepth > 0 {
return false
}
a.pos = pos
return true
}
// chompRuleBalanced 对应 chompRuleBalanced(open, close)：引号外才处理转义。
func (a *RuleAnalyzer) chompRuleBalanced(open, close byte) bool {
pos := a.pos
depth := 0
inSingle := false
inDouble := false
for {
if pos == len(a.queue) {
break
}
c := a.queue[pos]
pos++
if c == '\'' && !inDouble {
inSingle = !inSingle
} else if c == '"' && !inSingle {
inDouble = !inDouble
}
if inSingle || inDouble {
continue
}
if c == '\\' {
pos++
continue
}
if c == open {
depth++
} else if c == close {
depth--
}
if !(depth > 0) {
break
}
}
if depth > 0 {
return false
}
a.pos = pos
return true
}
func (a *RuleAnalyzer) chompBalanced(open, close byte) bool {
if a.code {
return a.chompCodeBalanced(open, close)
}
return a.chompRuleBalanced(open, close)
}
// SplitRule 对应 splitRule(vararg split)：把 queue 切分成规则列表。
// 首段按 splits 中最先出现的分隔符确定组合类型，其余按该类型循环切分。
func (a *RuleAnalyzer) SplitRule(splits ...string) []string {
if len(splits) == 1 {
a.elementsType = splits[0]
if !a.consumeTo(a.elementsType) {
a.rule = append(a.rule, a.queue[a.startX:])
return a.rule
}
a.step = len(a.elementsType)
return a.splitRuleNext()
}
if !a.consumeToAny(splits...) {
a.rule = append(a.rule, a.queue[a.startX:])
return a.rule
}
end := a.pos
a.pos = a.start
for {
st := a.findToAny('[', '(')
if st == -1 {
a.rule = []string{a.queue[a.startX:end]}
a.elementsType = a.queue[end : end+a.step]
a.pos = end + a.step
for a.consumeTo(a.elementsType) {
a.rule = append(a.rule, a.queue[a.start:a.pos])
a.pos += a.step
}
a.rule = append(a.rule, a.queue[a.pos:])
return a.rule
}
if st > end {
a.rule = []string{a.queue[a.startX:end]}
a.elementsType = a.queue[end : end+a.step]
a.pos = end + a.step
for a.consumeTo(a.elementsType) && a.pos < st {
a.rule = append(a.rule, a.queue[a.start:a.pos])
a.pos += a.step
}
if a.pos > st {
a.startX = a.start
return a.splitRuleNext()
}
a.rule = append(a.rule, a.queue[a.pos:])
return a.rule
}
a.pos = st
next := byte(')')
if a.queue[a.pos] == '[' {
next = ']'
}
if !a.chompBalanced(a.queue[a.pos], next) {
return []string{a.queue}
}
if end <= a.pos {
break
}
}
a.start = a.pos
return a.splitRuleFirst(splits...)
}
// splitRuleFirst 对应首段匹配的 tailrec 递归（elementsType 尚未确定）。
func (a *RuleAnalyzer) splitRuleFirst(splits ...string) []string {
if len(splits) == 1 {
a.elementsType = splits[0]
if !a.consumeTo(a.elementsType) {
a.rule = append(a.rule, a.queue[a.startX:])
return a.rule
}
a.step = len(a.elementsType)
return a.splitRuleNext()
}
if !a.consumeToAny(splits...) {
a.rule = append(a.rule, a.queue[a.startX:])
return a.rule
}
end := a.pos
a.pos = a.start
for {
st := a.findToAny('[', '(')
if st == -1 {
a.rule = []string{a.queue[a.startX:end]}
a.elementsType = a.queue[end : end+a.step]
a.pos = end + a.step
for a.consumeTo(a.elementsType) {
a.rule = append(a.rule, a.queue[a.start:a.pos])
a.pos += a.step
}
a.rule = append(a.rule, a.queue[a.pos:])
return a.rule
}
if st > end {
a.rule = []string{a.queue[a.startX:end]}
a.elementsType = a.queue[end : end+a.step]
a.pos = end + a.step
for a.consumeTo(a.elementsType) && a.pos < st {
a.rule = append(a.rule, a.queue[a.start:a.pos])
a.pos += a.step
}
if a.pos > st {
a.startX = a.start
return a.splitRuleNext()
}
a.rule = append(a.rule, a.queue[a.pos:])
return a.rule
}
a.pos = st
next := byte(')')
if a.queue[a.pos] == '[' {
next = ']'
}
if !a.chompBalanced(a.queue[a.pos], next) {
return []string{a.queue}
}
if end <= a.pos {
break
}
}
a.start = a.pos
return a.splitRuleFirst(splits...)
}
// splitRuleNext 对应二段匹配 splitRule()（elementsType 已确定）。
func (a *RuleAnalyzer) splitRuleNext() []string {
for {
end := a.pos
a.pos = a.start
for {
st := a.findToAny('[', '(')
if st == -1 {
a.rule = append(a.rule, a.queue[a.startX:end])
a.pos = end + a.step
for a.consumeTo(a.elementsType) {
a.rule = append(a.rule, a.queue[a.start:a.pos])
a.pos += a.step
}
a.rule = append(a.rule, a.queue[a.pos:])
return a.rule
}
if st > end {
a.rule = append(a.rule, a.queue[a.startX:end])
a.pos = end + a.step
for a.consumeTo(a.elementsType) && a.pos < st {
a.rule = append(a.rule, a.queue[a.start:a.pos])
a.pos += a.step
}
if a.pos > st {
a.startX = a.start
return a.splitRuleNext()
}
a.rule = append(a.rule, a.queue[a.pos:])
return a.rule
}
a.pos = st
next := byte(')')
if a.queue[a.pos] == '[' {
next = ']'
}
if !a.chompBalanced(a.queue[a.pos], next) {
return []string{a.queue}
}
if end <= a.pos {
break
}
}
a.start = a.pos
if !a.consumeTo(a.elementsType) {
a.rule = append(a.rule, a.queue[a.startX:])
return a.rule
}
}
}
// InnerRule 对应 innerRule(inner, startStep, endStep, fr) 第一变体：
// 替换所有内嵌规则（如 {$.xxx}），fr 返回空表示该处不是有效内嵌规则。
// 返回替换后的完整字符串；无一替换成功时返回 ""。
func (a *RuleAnalyzer) InnerRule(inner string, startStep, endStep int, fr func(string) string) string {
var sb []byte
for {
if !a.consumeTo(inner) {
break
}
posPre := a.pos
if a.chompCodeBalanced('{', '}') {
frv := fr(a.queue[posPre+startStep : a.pos-endStep])
if frv != "" {
sb = append(sb, a.queue[a.startX:posPre]...)
sb = append(sb, frv...)
a.startX = a.pos
continue
}
}
a.pos += len(inner)
}
if a.startX == 0 {
return ""
}
sb = append(sb, a.queue[a.startX:]...)
return string(sb)
}
// InnerRule2 对应 innerRule(startStr, endStr, fr) 第二变体：无平衡组检查。
// 无一替换成功时返回原串。
func (a *RuleAnalyzer) InnerRule2(startStr, endStr string, fr func(string) string) string {
var sb []byte
for {
if !a.consumeTo(startStr) {
break
}
a.pos += len(startStr)
posPre := a.pos
if a.consumeTo(endStr) {
frv := fr(a.queue[posPre:a.pos])
sb = append(sb, a.queue[a.startX:posPre-len(startStr)]...)
sb = append(sb, frv...)
a.pos += len(endStr)
a.startX = a.pos
}
}
if a.startX == 0 {
return a.queue
}
sb = append(sb, a.queue[a.startX:]...)
return string(sb)
}
internal/service/reader/rule/sourcerule.go
+337
View File
@@ -0,0 +1,337 @@
package rule
import (
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
)
// 本文件对应 AnalyzeRule.kt 中的 SourceRule 内部类与 splitSourceRule。
// Mode 规则模式（对应 AnalyzeRule.Mode）。
type Mode int
const (
ModeXPath Mode = iota
ModeJson
ModeDefault
ModeJs
ModeRegex
ModeWebJs
)
var (
jsPatternRe = regexp.MustCompile(`(?i)<js>([\w\W]*?)</js>|@js:([\w\W]*)`)
webJsPatternRe = regexp.MustCompile(`(?i)@webjs:([\w\W]{5,})`)
putPatternRe = regexp.MustCompile(`(?i)@put:(\{[^}]+?\})`)
evalPatternRe = regexp.MustCompile(`(?i)@get:\{[^}]+?\}|\{\{[\w\W]*?\}\}`)
regexGroupRe = regexp.MustCompile(`\$\d{1,2}`)
)
// makeUpRule 参数类型（对应 SourceRule 的 ruleType 常量）。
const (
paramGet = -2 // @get:{name}
paramJs = -1 // {{js}}
paramText = 0 // 字面文本
paramGroupN = 1 // >0 为 $N 正则分组引用，typ 即分组序号
)
type ruleParam struct {
typ int
val string
}
// SourceRule 是拆分后的单条规则（对应 AnalyzeRule.SourceRule）。
type SourceRule struct {
Rule string
Mode Mode
putMap map[string]string
ruleParams []ruleParam
}
// SplitSourceRule 对应 AnalyzeRule.splitSourceRule(ruleStr, allInOne)。
// contentIsJSON 对应 Kotlin 成员 isJSON（当前内容是否为 JSON）。
// isRegex 对应 AnalyzeRule.isRegex 持久标记，可为 nil。
func SplitSourceRule(ruleStr string, allInOne, contentIsJSON bool, isRegex *bool) []*SourceRule {
if ruleStr == "" {
return nil
}
mode := ModeDefault
start := 0
if allInOne && strings.HasPrefix(ruleStr, ":") {
mode = ModeRegex
if isRegex != nil {
*isRegex = true
}
start = 1
} else if isRegex != nil && *isRegex {
mode = ModeRegex
}
type rulePart struct {
start int
end int
rule string
mode Mode
}
var parts []rulePart
for _, g := range jsPatternRe.FindAllStringSubmatchIndex(ruleStr, -1) {
jsBody := ""
if g[2] >= 0 { // group(1)： <js>...</js>
jsBody = ruleStr[g[2]:g[3]]
} else if g[4] >= 0 { // group(2)： @js:...
jsBody = ruleStr[g[4]:g[5]]
}
parts = append(parts, rulePart{g[0], g[1], jsBody, ModeJs})
}
for _, g := range webJsPatternRe.FindAllStringSubmatchIndex(ruleStr, -1) {
parts = append(parts, rulePart{g[0], g[1], ruleStr[g[2]:g[3]], ModeWebJs})
}
sort.Slice(parts, func(i, j int) bool { return parts[i].start < parts[j].start })
var out []*SourceRule
for _, p := range parts {
if p.start < start {
continue
}
if p.start > start {
if tmp := strings.TrimSpace(ruleStr[start:p.start]); tmp != "" {
out = append(out, newSourceRule(tmp, mode, contentIsJSON))
}
}
out = append(out, newSourceRule(p.rule, p.mode, contentIsJSON))
start = p.end
}
if len(ruleStr) > start {
if tmp := strings.TrimSpace(ruleStr[start:]); tmp != "" {
out = append(out, newSourceRule(tmp, mode, contentIsJSON))
}
}
return out
}
// newSourceRule 对应 SourceRule.init：模式识别 + 分离 put + 拆分 @get/{{}}/$N。
func newSourceRule(ruleStr string, mode Mode, contentIsJSON bool) *SourceRule {
sr := &SourceRule{Mode: mode, putMap: map[string]string{}}
rule := ruleStr
switch {
case mode == ModeJs || mode == ModeRegex:
// 保持原样
case hasPrefixFold(ruleStr, "@CSS:"):
sr.Mode = ModeDefault
case strings.HasPrefix(ruleStr, "@@"):
sr.Mode = ModeDefault
rule = ruleStr[2:]
case hasPrefixFold(ruleStr, "@XPath:"):
sr.Mode = ModeXPath
rule = ruleStr[7:]
case hasPrefixFold(ruleStr, "@Json:"):
sr.Mode = ModeJson
rule = ruleStr[6:]
case contentIsJSON || strings.HasPrefix(ruleStr, "$.") || strings.HasPrefix(ruleStr, "$["):
sr.Mode = ModeJson
case strings.HasPrefix(ruleStr, "/"):
sr.Mode = ModeXPath
}
// 分离 @put:{...}
rule = splitPutRule(rule, sr.putMap)
sr.Rule = rule
sr.splitEvalParams()
return sr
}
func hasPrefixFold(s, prefix string) bool {
return len(s) >= len(prefix) && strings.EqualFold(s[:len(prefix)], prefix)
}
// splitPutRule 对应 splitPutRule：提取并移除 @put:{...} 段。
func splitPutRule(ruleStr string, putMap map[string]string) string {
out := ruleStr
for _, m := range putPatternRe.FindAllStringSubmatch(ruleStr, -1) {
out = strings.Replace(out, m[0], "", 1)
parsed := map[string]string{}
if err := json.Unmarshal([]byte(m[1]), &parsed); err == nil {
for k, v := range parsed {
putMap[k] = v
}
continue
}
// 宽松解析（对应 GSON lenient）：key:val 键值对
pairRe := regexp.MustCompile(`["']?(\w+)["']?\s*:\s*(["']?)([^,{}]*?)\2`)
for _, pm := range pairRe.FindAllStringSubmatch(m[1], -1) {
putMap[pm[1]] = pm[3]
}
}
return out
}
// splitEvalParams 对应 init 中 @get/{{ }} 的拆分循环。
func (sr *SourceRule) splitEvalParams() {
rule := sr.Rule
start := 0
locs := evalPatternRe.FindAllStringIndex(rule, -1)
if len(locs) > 0 {
firstStart := locs[0][0]
prefix := rule[:firstStart]
if sr.Mode != ModeJs && sr.Mode != ModeRegex &&
(firstStart == 0 || !strings.Contains(prefix, "##")) {
sr.Mode = ModeRegex
}
for _, loc := range locs {
if loc[0] > start {
sr.splitRegex(rule[start:loc[0]])
}
tmp := rule[loc[0]:loc[1]]
switch {
case hasPrefixFold(tmp, "@get:"):
sr.ruleParams = append(sr.ruleParams, ruleParam{paramGet, tmp[6 : len(tmp)-1]})
case strings.HasPrefix(tmp, "{{"):
sr.ruleParams = append(sr.ruleParams, ruleParam{paramJs, tmp[2 : len(tmp)-2]})
default:
sr.splitRegex(tmp)
}
start = loc[1]
}
}
if len(rule) > start {
sr.splitRegex(rule[start:])
}
}
// splitRegex 对应 SourceRule.splitRegex：拆分 $N 分组引用。
// $N 匹配只作用于 "##" 前的第一段，"##..." 尾部作为字面参数保留，
// 由 MakeUpRule 重新按 "##" 切分。
func (sr *SourceRule) splitRegex(ruleStr string) {
start := 0
first := strings.Split(ruleStr, "##")[0]
locs := regexGroupRe.FindAllStringIndex(first, -1)
if len(locs) > 0 {
if sr.Mode != ModeJs && sr.Mode != ModeRegex {
sr.Mode = ModeRegex
}
for _, loc := range locs {
if loc[0] > start {
sr.ruleParams = append(sr.ruleParams, ruleParam{paramText, ruleStr[start:loc[0]]})
}
n := 0
fmt.Sscanf(ruleStr[loc[0]+1:loc[1]], "%d", &n)
sr.ruleParams = append(sr.ruleParams, ruleParam{n, ruleStr[loc[0]:loc[1]]})
start = loc[1]
}
}
if len(ruleStr) > start {
sr.ruleParams = append(sr.ruleParams, ruleParam{paramText, ruleStr[start:]})
}
}
// ResolvedSourceRule 对应 AnalyzeRule.ResolvedSourceRule。
type ResolvedSourceRule struct {
Rule string
ReplaceRegex string
Replacement string
ReplaceFirst bool
ParamSize int
}
// RuleDeps 是 MakeUpRule 解析内嵌 {{...}} 时需要的执行环境。
type RuleDeps struct {
// JS 执行 {{js}}（P0 未接入 goja 时返回 ErrJsUnsupported）
JS func(js string, result any) (any, error)
// Rule 执行 {{@rule}} / {{$.rule}} 形式的规则引用
Rule func(rule string) (string, error)
// Get 对应 AnalyzeRule.get(key)
Get func(key string) string
}
// MakeUpRule 对应 SourceRule.makeUpRule(result)：替换 @get/{{}}/$N，
// 再按 "##" 切分出正则段。
func (sr *SourceRule) MakeUpRule(result any, deps *RuleDeps) (ResolvedSourceRule, error) {
resolved := sr.Rule
if len(sr.ruleParams) > 0 {
var infoVal []byte
for i := len(sr.ruleParams) - 1; i >= 0; i-- {
p := sr.ruleParams[i]
switch {
case p.typ >= paramGroupN:
// 对应 Kotlin：(result as? List<String?>) 成功但越界时不插入任何内容
switch lst := result.(type) {
case []string:
if len(lst) > p.typ {
infoVal = prepend(infoVal, lst[p.typ])
}
case []any:
if len(lst) > p.typ {
if s, ok := lst[p.typ].(string); ok {
infoVal = prepend(infoVal, s)
}
}
default:
infoVal = prepend(infoVal, p.val)
}
case p.typ == paramJs:
if isRuleString(p.val) {
s, err := deps.Rule(p.val)
if err != nil {
return ResolvedSourceRule{}, err
}
infoVal = prepend(infoVal, s)
} else {
v, err := deps.JS(p.val, result)
if err != nil {
return ResolvedSourceRule{}, err
}
infoVal = prepend(infoVal, anyToString(v))
}
case p.typ == paramGet:
infoVal = prepend(infoVal, deps.Get(p.val))
default:
infoVal = prepend(infoVal, p.val)
}
}
resolved = string(infoVal)
}
segs := strings.Split(resolved, "##")
r := ResolvedSourceRule{
Rule: strings.TrimSpace(segs[0]),
ReplaceFirst: len(segs) > 3,
ParamSize: len(sr.ruleParams),
}
if len(segs) > 1 {
r.ReplaceRegex = segs[1]
}
if len(segs) > 2 {
r.Replacement = segs[2]
}
return r, nil
}
func prepend(dst []byte, s string) []byte {
return append([]byte(s), dst...)
}
func isRuleString(s string) bool {
return strings.HasPrefix(s, "@") ||
strings.HasPrefix(s, "$.") ||
strings.HasPrefix(s, "$[") ||
strings.HasPrefix(s, "//")
}
// anyToString 对应 Kotlin 值字符串化（整值 Double 去小数，对应 %.0f）。
func anyToString(v any) string {
switch t := v.(type) {
case nil:
return ""
case string:
return t
case float64:
if t == float64(int64(t)) {
return fmt.Sprintf("%.0f", t)
}
return fmt.Sprintf("%v", t)
default:
return fmt.Sprintf("%v", t)
}
}
internal/service/reader/rule/url.go
+348
View File
@@ -0,0 +1,348 @@
package rule
import (
"encoding/json"
"regexp"
"strings"
"golang.org/x/text/encoding/htmlindex"
"golang.org/x/text/encoding/unicode"
)
// 本文件对应 AnalyzeUrl.kt 的 URL 解析部分（不含网络执行与 JS 执行）。
// URLOption 对应 AnalyzeUrl.UrlOption。
type URLOption struct {
Method string `json:"method"`
Charset string `json:"charset"`
Headers map[string]any `json:"headers"`
Body json.RawMessage `json:"body"`
Origin string `json:"origin"`
Retry *int `json:"retry"`
Type string `json:"type"`
WebView json.RawMessage `json:"webView"`
WebJs string `json:"webJs"`
DnsIp string `json:"dnsIp"`
Js string `json:"js"`
BodyJs string `json:"bodyJs"`
ServerID json.RawMessage `json:"serverID"`
WebViewDelayTime json.RawMessage `json:"webViewDelayTime"`
}
// Request 是解析后的可执行请求（供服务层执行）。
type Request struct {
URL string // 最终 URL（GET 时已含重编码后的 query）
URLNoQuery string
Method string
Headers map[string]string
Body string
IsForm bool // Body 为已编码的 form 数据
IsJSON bool // 以 application/json 发送
Charset string
// Unsupported 非 nil 表示该请求依赖当前阶段不支持的能力，
// 值为对应错误（webView/js/type）。
Unsupported error
}
// ParseAnalyzeUrl 对应 AnalyzeUrl.init：URL 规则 → 可执行请求。
// key/page 对应搜索关键词与页码（{{key}}/{{page}}/<1,2,3>）。
func ParseAnalyzeUrl(mUrl, key string, page int, baseUrl string) (*Request, error) {
req := &Request{Method: "GET", Headers: map[string]string{}}
// baseUrl 自身可能带 ",{...}" 选项段，先截断（对应 init 中的 paramPattern）
if st, _, ok := findParamSplit(baseUrl); ok {
baseUrl = baseUrl[:st]
}
ruleUrl := mUrl
// ── analyzeJs：URL 中的 <js>/@js:（P0 不支持） ──
if jsPatternRe.MatchString(ruleUrl) {
req.Unsupported = ErrJsUnsupported
// 移除 JS 块继续解析，便于调试接口展示其余部分
ruleUrl = jsPatternRe.ReplaceAllString(ruleUrl, "")
}
// ── replaceKeyPageJs：{{...}} 与 <页码列表> ──
if strings.Contains(ruleUrl, "{{") && strings.Contains(ruleUrl, "}}") {
var subErr error
ra := NewRuleAnalyzer(ruleUrl, false)
out := ra.InnerRule2("{{", "}}", func(inner string) string {
trimmed := strings.TrimSpace(inner)
switch trimmed {
case "key":
return key
case "page":
p := page
if p < 1 {
p = 1
}
return anyToString(float64(p))
default:
subErr = ErrJsUnsupported
return ""
}
})
if subErr != nil {
req.Unsupported = subErr
} else if out != "" {
ruleUrl = out
}
}
if page >= 1 {
for _, m := range pagePatternRe.FindAllStringSubmatch(ruleUrl, -1) {
pages := strings.Split(m[1], ",")
idx := page
if idx > len(pages) {
idx = len(pages)
}
ruleUrl = strings.ReplaceAll(ruleUrl, m[0], strings.TrimSpace(pages[idx-1]))
}
}
// ── analyzeUrl：分离 URL 与选项 JSON ──
ruleUrl = strings.TrimSpace(ruleUrl)
st, end, hasOpt := findParamSplit(ruleUrl)
urlNoOption := ruleUrl
if hasOpt {
urlNoOption = ruleUrl[:st]
}
urlNoOption = strings.TrimSpace(urlNoOption)
finalURL := GetAbsoluteURL(baseUrl, urlNoOption)
if b := GetBaseUrl(finalURL); b != "" {
baseUrl = b
}
req.URL = finalURL
if hasOpt {
optionStr := strings.TrimSpace(ruleUrl[end:])
var option URLOption
if err := json.Unmarshal([]byte(optionStr), &option); err != nil {
// 宽松重试：去掉可能的前后杂字符
if err2 := json.Unmarshal([]byte(strings.TrimPrefix(optionStr, ",")), &option); err2 != nil {
return req, nil
}
}
switch strings.ToUpper(option.Method) {
case "POST":
req.Method = "POST"
case "HEAD":
req.Method = "HEAD"
}
for k, v := range option.Headers {
req.Headers[k] = anyToString(v)
}
if len(option.Body) > 0 {
var bodyStr string
if err := json.Unmarshal(option.Body, &bodyStr); err == nil {
req.Body = bodyStr
} else {
req.Body = string(option.Body)
}
}
req.Charset = option.Charset
if option.Type != "" && req.Unsupported == nil {
req.Unsupported = ErrTypeUnsupported
}
if option.WebJs != "" && req.Unsupported == nil {
req.Unsupported = ErrWebJSUnsupported
}
if useWebView(option.WebView) && req.Unsupported == nil {
req.Unsupported = ErrWebJSUnsupported
}
if option.Js != "" && req.Unsupported == nil {
req.Unsupported = ErrJsUnsupported
}
if option.BodyJs != "" && req.Unsupported == nil {
req.Unsupported = ErrJsUnsupported
}
}
// ── query / body 编码（对应 analyzeUrl 尾部） ──
if req.Method == "POST" {
req.URLNoQuery = req.URL
body := req.Body
if body != "" && !isJSONStr(body) && !isXMLStr(body) && req.Headers["Content-Type"] == "" {
req.Body = encodeParams(body, req.Charset, false)
req.IsForm = true
} else if isJSONStr(body) && req.Headers["Content-Type"] == "" {
req.IsJSON = true
}
} else {
pos := strings.Index(req.URL, "?")
if pos != -1 {
query := encodeParams(req.URL[pos+1:], req.Charset, true)
req.URLNoQuery = req.URL[:pos]
if query != "" {
req.URL = req.URLNoQuery + "?" + query
} else {
req.URL = req.URLNoQuery
}
} else {
req.URLNoQuery = req.URL
}
}
return req, nil
}
var pagePatternRe = regexp.MustCompile(`<([^>]*)>`)
func useWebView(raw json.RawMessage) bool {
if len(raw) == 0 {
return false
}
var b bool
if err := json.Unmarshal(raw, &b); err == nil {
return b
}
var s string
if err := json.Unmarshal(raw, &s); err == nil {
return s != "" && s != "false"
}
return true
}
func isJSONStr(s string) bool {
t := strings.TrimSpace(s)
if !(strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[")) {
return false
}
return json.Valid([]byte(t))
}
func isXMLStr(s string) bool {
t := strings.TrimSpace(s)
return strings.HasPrefix(t, "<") && strings.HasSuffix(t, ">")
}
// findParamSplit 对应 paramPattern \s*,\s*(?=\{) 的手动实现。
// 返回匹配起点（含逗号前空白）与选项 JSON 起点。
func findParamSplit(s string) (start, end int, ok bool) {
for i := 0; i < len(s); i++ {
if s[i] != ',' {
continue
}
st := i
for st > 0 && isSpaceByte(s[st-1]) {
st--
}
j := i + 1
for j < len(s) && isSpaceByte(s[j]) {
j++
}
if j < len(s) && s[j] == '{' {
return st, j, true
}
}
return 0, 0, false
}
func isSpaceByte(c byte) bool {
return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\v' || c == '\f'
}
// encodeParams 对应 AnalyzeUrl.encodeParams。
func encodeParams(params, charset string, isQuery bool) string {
checkEncoded := charset == ""
cs := charset
if cs == "" {
cs = "utf-8"
}
if isQuery && cs != "escape" {
if encodedQuery(params) {
return params
}
return queryEncode(params, cs)
}
var sb strings.Builder
lenP := len(params)
pos := 0
for pos <= lenP {
if sb.Len() > 0 {
sb.WriteString("&")
}
ampOffset := strings.IndexByte(params[pos:], '&')
if ampOffset == -1 {
ampOffset = lenP
} else {
ampOffset += pos
}
eqOffset := strings.IndexByte(params[pos:ampOffset], '=')
if eqOffset == -1 {
sb.WriteString(appendEncodedStr(params[pos:ampOffset], checkEncoded, cs))
} else {
eqAbs := eqOffset + pos
sb.WriteString(appendEncodedStr(params[pos:eqAbs], checkEncoded, cs))
sb.WriteString("=")
sb.WriteString(appendEncodedStr(params[eqAbs+1:ampOffset], checkEncoded, cs))
}
pos = ampOffset + 1
}
return sb.String()
}
// appendEncodedStr 对应 StringBuilder.appendEncoded。
func appendEncodedStr(value string, checkEncoded bool, charset string) string {
if checkEncoded && encodedForm(value) {
return value
}
if charset == "escape" {
return JSEscape(value)
}
return javaURLEncode(value, charset)
}
// javaURLEncode 对应 java.net.URLEncoder.encode(value, charset)：
// 字母数字与 .-*_ 保留，空格转 +，其余按字符集字节 %XX。
func javaURLEncode(value, charset string) string {
enc := encoderFor(charset)
var sb strings.Builder
for _, r := range value {
switch {
case (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9'),
r == '.', r == '-', r == '*', r == '_':
sb.WriteRune(r)
case r == ' ':
sb.WriteByte('+')
default:
for _, b := range enc(string(r)) {
const hex = "0123456789ABCDEF"
sb.WriteByte('%')
sb.WriteByte(hex[b>>4])
sb.WriteByte(hex[b&0xF])
}
}
}
return sb.String()
}
// queryEncode 对应 hutool RFC3986.UNRESERVED.orNew(...) 的 query 编码：
// 保留 notNeedEncodingQuery 集合内的字符，其余按字符集字节 %XX。
func queryEncode(params, charset string) string {
enc := encoderFor(charset)
var sb strings.Builder
for _, r := range params {
if r < 128 && notNeedEncodingQuery[r] {
sb.WriteRune(r)
continue
}
for _, b := range enc(string(r)) {
const hex = "0123456789ABCDEF"
sb.WriteByte('%')
sb.WriteByte(hex[b>>4])
sb.WriteByte(hex[b&0xF])
}
}
return sb.String()
}
func encoderFor(charset string) func(string) []byte {
if strings.EqualFold(charset, "utf-8") || strings.EqualFold(charset, "utf8") {
utf8Enc := unicode.UTF8
return func(s string) []byte { b, _ := utf8Enc.NewEncoder().Bytes([]byte(s)); return b }
}
enc, err := htmlindex.Get(charset)
if err != nil {
utf8Enc := unicode.UTF8
return func(s string) []byte { b, _ := utf8Enc.NewEncoder().Bytes([]byte(s)); return b }
}
return func(s string) []byte { b, _ := enc.NewEncoder().Bytes([]byte(s)); return b }
}
internal/service/reader/rule/xpath.go
+149
View File
@@ -0,0 +1,149 @@
package rule
import (
"strings"
"github.com/antchfx/htmlquery"
"golang.org/x/net/html"
)
// 本文件对应 AnalyzeByXPath.kt（JsoupXpath 语义，Go 侧用 antchfx/htmlquery）。
type xpathAnalyzer struct {
root *html.Node
}
func newXPathAnalyzer(doc any) *xpathAnalyzer {
return &xpathAnalyzer{root: toHTMLNode(doc)}
}
// xpathResult 对应 getResult(xPath)。
func (a *xpathAnalyzer) result(xPath string) []*html.Node {
if a.root == nil || xPath == "" {
return nil
}
return htmlquery.Find(a.root, xPath)
}
// xpathNodeString 对应 JXNode.asString()。
func xpathNodeString(n *html.Node) string {
if n == nil {
return ""
}
if n.Type == html.TextNode {
return n.Data
}
if n.Type == html.ElementNode {
return htmlquery.InnerText(n)
}
if n.Data != "" {
return n.Data
}
return ""
}
// getElements 对应 AnalyzeByXPath.getElements。
func (a *xpathAnalyzer) getElements(xPath string) []*html.Node {
if xPath == "" {
return nil
}
ra := NewRuleAnalyzer(xPath, false)
rules := ra.SplitRule("&&", "||", "%%")
if len(rules) == 1 {
return a.result(rules[0])
}
var out []*html.Node
var results [][]*html.Node
for _, rl := range rules {
temp := a.getElements(rl)
if len(temp) > 0 {
results = append(results, temp)
if ra.ElementsType() == "||" {
break
}
}
}
if len(results) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(results[0]); i++ {
for _, temp := range results {
if i < len(temp) {
out = append(out, temp[i])
}
}
}
} else {
for _, temp := range results {
out = append(out, temp...)
}
}
}
return out
}
// getStringList 对应 AnalyzeByXPath.getStringList。
func (a *xpathAnalyzer) getStringList(xPath string) []string {
var result []string
ra := NewRuleAnalyzer(xPath, false)
rules := ra.SplitRule("&&", "||", "%%")
if len(rules) == 1 {
for _, n := range a.result(xPath) {
result = append(result, xpathNodeString(n))
}
return result
}
var results [][]string
for _, rl := range rules {
temp := a.getStringList(rl)
if len(temp) > 0 {
results = append(results, temp)
if ra.ElementsType() == "||" {
break
}
}
}
if len(results) > 0 {
if ra.ElementsType() == "%%" {
for i := 0; i < len(results[0]); i++ {
for _, temp := range results {
if i < len(temp) {
result = append(result, temp[i])
}
}
}
} else {
for _, temp := range results {
result = append(result, temp...)
}
}
}
return result
}
// getString 对应 AnalyzeByXPath.getString：多节点以 \n 连接。
func (a *xpathAnalyzer) getString(rule string) string {
ra := NewRuleAnalyzer(rule, false)
rules := ra.SplitRule("&&", "||")
if len(rules) == 1 {
nodes := a.result(rule)
if len(nodes) == 0 {
return ""
}
parts := make([]string, len(nodes))
for i, n := range nodes {
parts[i] = xpathNodeString(n)
}
return strings.Join(parts, "\n")
}
var textList []string
for _, rl := range rules {
temp := a.getString(rl)
if temp != "" {
textList = append(textList, temp)
if ra.ElementsType() == "||" {
break
}
}
}
return strings.Join(textList, "\n")
}
internal/service/reader/source.go
+140
View File
@@ -0,0 +1,140 @@
// Package reader — legado 书源兼容的阅读子系统服务层。
// 本文件对应 legado data/entities/BookSource.kt 的 JSON 结构。
package reader
import (
"encoding/json"
"strings"
)
// BookSource 书源 JSON 结构（字段与 legado 实体一致，未知字段忽略）。
type BookSource struct {
BookSourceURL string `json:"bookSourceUrl"`
BookSourceName string `json:"bookSourceName"`
BookSourceGroup *string `json:"bookSourceGroup"`
BookSourceType *int `json:"bookSourceType"` // 0文本 1音频 2图片 3文件 4视频
BookURLPattern *string `json:"bookUrlPattern"`
CustomOrder *int `json:"customOrder"`
Enabled *bool `json:"enabled"`
EnabledExplore *bool `json:"enabledExplore"`
EnabledCookieJar *bool `json:"enabledCookieJar"`
ConcurrentRate *string `json:"concurrentRate"`
Header *string `json:"header"`
LoginURL *string `json:"loginUrl"`
BookSourceComment *string `json:"bookSourceComment"`
LastUpdateTime *int64 `json:"lastUpdateTime"`
RespondTime *int64 `json:"respondTime"`
Weight *int `json:"weight"`
ExploreURL *string `json:"exploreUrl"`
SearchURL *string `json:"searchUrl"`
RuleExplore *ExploreRule `json:"ruleExplore"`
RuleSearch *SearchRule `json:"ruleSearch"`
RuleBookInfo *BookInfoRule `json:"ruleBookInfo"`
RuleToc *TocRule `json:"ruleToc"`
RuleContent *ContentRule `json:"ruleContent"`
VariableComment *string `json:"variableComment"`
Variables map[string]any `json:"-"`
RawVariables *string `json:"variables"`
}
// SearchRule 搜索规则。
type SearchRule struct {
CheckKeyWord *string `json:"checkKeyWord"`
BookList *string `json:"bookList"`
Name *string `json:"name"`
Author *string `json:"author"`
Kind *string `json:"kind"`
WordCount *string `json:"wordCount"`
LastChapter *string `json:"lastChapter"`
Intro *string `json:"intro"`
CoverURL *string `json:"coverUrl"`
BookURL *string `json:"bookUrl"`
}
// BookInfoRule 详情规则。
type BookInfoRule struct {
Init *string `json:"init"`
Name *string `json:"name"`
Author *string `json:"author"`
Kind *string `json:"kind"`
WordCount *string `json:"wordCount"`
LastChapter *string `json:"lastChapter"`
Intro *string `json:"intro"`
CoverURL *string `json:"coverUrl"`
TocURL *string `json:"tocUrl"`
CanReName *string `json:"canReName"`
DownloadUrls *string `json:"downloadUrls"`
}
// TocRule 目录规则。
type TocRule struct {
PreUpdateJs *string `json:"preUpdateJs"`
ChapterList *string `json:"chapterList"`
ChapterName *string `json:"chapterName"`
ChapterURL *string `json:"chapterUrl"`
IsVolume *string `json:"isVolume"`
UpdateTime *string `json:"updateTime"`
}
// ContentRule 正文规则。
type ContentRule struct {
Content *string `json:"content"`
NextContentURL *string `json:"nextContentUrl"`
WebJs *string `json:"webJs"`
SourceRegex *string `json:"sourceRegex"`
ReplaceRegex *string `json:"replaceRegex"`
ImageStyle *string `json:"imageStyle"`
PayAction *string `json:"payAction"`
}
// ExploreRule 发现规则。
type ExploreRule struct {
BookList *string `json:"bookList"`
Name *string `json:"name"`
Author *string `json:"author"`
Kind *string `json:"kind"`
WordCount *string `json:"wordCount"`
LastChapter *string `json:"lastChapter"`
Intro *string `json:"intro"`
CoverURL *string `json:"coverUrl"`
BookURL *string `json:"bookUrl"`
ExploreURL *string `json:"exploreUrl"`
ExploreKinds *string `json:"exploreKinds"`
CheckKeyWord *string `json:"checkKeyWord"`
}
// ParseBookSource 将书源 JSON 解析为结构体。
func ParseBookSource(raw string) (*BookSource, error) {
var bs BookSource
if err := json.Unmarshal([]byte(raw), &bs); err != nil {
return nil, err
}
if bs.RawVariables != nil && strings.TrimSpace(*bs.RawVariables) != "" {
_ = json.Unmarshal([]byte(*bs.RawVariables), &bs.Variables)
}
return &bs, nil
}
// Type 返回书源类型（默认文本）。
func (b *BookSource) Type() int {
if b.BookSourceType == nil {
return 0
}
return *b.BookSourceType
}
// SPtr 取字符串指针字段值。
func SPtr(s *string) string {
if s == nil {
return ""
}
return *s
}
// IPtr 取整型指针字段值。
func IPtr(i *int) int {
if i == nil {
return 0
}
return *i
}
internal/service/service.go
+2
View File
@@ -12,6 +12,7 @@ import (
"github.com/truewhile/MeBox/internal/config"
"github.com/truewhile/MeBox/internal/helper"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service/reader"
)
// Container 持有在启动时初始化的每个服务。Handler 接收指向它的指针并选择相关字段。
@@ -70,6 +71,7 @@ type Container struct {
Danmaku *DanmakuService
Strm *StrmService
Cloud115 *Cloud115PlaybackService
Reader *reader.ReaderService
Database *DatabaseAdminService
FFTools *FFmpegToolsService
internal/service/service_builder.go
+2
View File
@@ -12,6 +12,7 @@ import (
"github.com/truewhile/MeBox/internal/helper"
"github.com/truewhile/MeBox/internal/model"
"github.com/truewhile/MeBox/internal/repository"
"github.com/truewhile/MeBox/internal/service/reader"
)
type serviceContainerBuilder struct {
@@ -82,6 +83,7 @@ func (b *serviceContainerBuilder) initProviderServices() {
b.c.Fanart = NewFanartProvider(b.cfg, b.log)
b.c.RecognitionWords = NewRecognitionWordsService(b.log, b.repos)
b.c.Danmaku = NewDanmakuService(b.log, b.repos)
b.c.Reader = reader.NewReaderService(b.cfg, b.log, b.repos)
adult := NewAdultProvider(b.log, b.c.APIConfig, b.repos)
b.c.Scraper = NewScraperService(