85 changed files with 8792 additions and 566 deletions
+3
View File
@@ -41,3 +41,6 @@ build/
### Mac OS ### ### Mac OS ###
.DS_Store .DS_Store
/.idea/encodings.xml /.idea/encodings.xml
# Migration script bytecode
scripts/__pycache__/
+150
View File
@@ -0,0 +1,150 @@
# 图片索引缓存(GalleryIndex)— 重写模块说明
## 问题
在线阅读的索引是「gid → 每一页 imgkey」,存在 `cache.db` 的两张表里:
| 表 | 列 | 含义 |
|---|---|---|
| `gidToKey` | `gid`, `key`, `pages` | 画廊头部:gid、画廊 key、总页数 |
| `ImageKeyCache` | `gid`, `imgkey`, `page` | 每页的 imgkey |
旧实现(`GalleryManageService.cacheImagesKey`)依次做四件事:查缓存、判断是否需要重建、
外呼抓 mpv 页、逐页 `INSERT`。由此产生的实际问题:
1. **没有唯一约束**。实测生产缓存库有 4 组重复 gid、97 组重复 `(gid,page)`。
2. **整组替换不是原子的**。先写头部再逐页写,中途失败会留下「头部命中、页 key 缺失」的
半截索引;后续请求直接判定为已缓存,永远取不到图。
3. **用行数判断完整性**。`count(*) == pages` 无法发现「页码 1、3 而 pages=2」这种跳号。
4. **没有并发合并**。同一本画廊被并发打开时,两个构建者会互相删除对方刚写入的行。
5. **逐页写入放大写事务**,抓取与写入之间没有事务边界。
## 交付物与职责
| 文件 | 职责 |
|---|---|
| `Service/GalleryIndexService.java` | 索引的唯一读写入口:同 gid 合并、抓取在事务外、校验后整组短事务替换 |
| `Exception/GalleryIndexException.java` | 分类失败原因:INVALID_LINK / UNAVAILABLE / MALFORMED / TRANSIENT |
| `Configuration/GalleryIndexConfiguration.java` | 缓存库的事务管理器与读写模板(独立文件,未改 SqlConfiguration) |
| `Dao/cache/ImageCacheMapper.java` | 新增页码查询、脏行计数、批量写入(原有方法全部保留) |
| `Maintenance/GalleryIndexMaintenance.java` | 离线体检与修复工具(只删不健康 gid 的整组索引) |
| `scripts/migrate_image_index_cache.{sh,sql}` | 迁移:默认 dry-run、备份、清理、建唯一索引 |
| `Service/GalleryIndexServiceTest.java` | 服务编排的行为约束(无 Spring、无网络) |
| `Maintenance/GalleryIndexMaintenanceTest.java` | 工具在真实临时 SQLite 上的判定与删除范围 |
| `Dao/cache/ImageCacheMapperSqlTest.java` | 注解 SQL 在真实 MyBatis + SQLite 上的验证 |
## 接口
```java
GidToKey ensureIndex(String url) throws GalleryIndexException;
```
返回该画廊的 `gid` / `key` / `pages`。三条不变量:
- **抓取在事务外**:`upstream.fetchPageKeys(url)` 先于任何写事务执行;写事务只在
`publish(...)` 里开启,删除旧索引与批量写入在同一个事务内整体生效。
- **校验先于写入**:行必须满足 gid 与请求一致、页码恰好为 1..N、不重复、imgkey 非空,
否则抛 MALFORMED,此时尚未开启写事务,库中旧索引原样保留。
- **读在快照内**:`lookup` 在同一个读事务里读头部与页码,避免读到整组替换的中间状态。
失败原因:
| Reason | 触发条件 | isPermanent() | 建议处置 |
|---|---|---|---|
| INVALID_LINK | 链接里取不到 gid 或 key | true | 直接回「链接错误」 |
| UNAVAILABLE | 上游返回空页列表(下架/删除) | true | 回「该图片已下架或已被删除」 |
| MALFORMED | 上游页 key 不自洽 | false | 可重试,回「网络波动或其他异常」 |
| TRANSIENT | 上游异常或写库失败 | false | 可重试,回「网络波动或其他异常」 |
## 接入方式(由主模型完成 GalleryManageService 侧改动)
构造函数注入:
```java
final GalleryIndexService galleryIndexService;
```
`cacheImagesKey` 改为:
```java
public String cacheImagesKey(String url) {
Response response = Response.generateResponse();
try {
GidToKey gidToKey = galleryIndexService.ensureIndex(url);
return response.success(objectMapper.valueToTree(gidToKey)).toJSONString();
} catch (GalleryIndexException e) {
response.failure(e.getReason() == GalleryIndexException.Reason.INVALID_LINK
? "链接错误"
: e.isPermanent() ? "该图片已下架或已被删除" : "网络波动或其他异常");
return response.toJSONString();
}
}
```
Spring 装配无需额外配置:`GalleryIndexConfiguration` 已把 `GalleryIndexService` 注册为 bean,
依赖 `ImageCacheMapper`(cache 数据源)与两个 TransactionTemplate。原来的
`isIndexComplete` / `rollbackImageIndex` 私有方法与 `ImageCacheMapper` 的
`countImageKeyCacheByGid` / `delete*ByGid` 仍保留,供旧调用方与非索引路径使用。
读取路径 `getCachedImage` 不需要改动:它按 `(gid,page)` 查 ImageKeyCache,现在会走新加的
`ux_ImageKeyCache_gid_page` 唯一索引。
## 迁移
```bash
# 1. dry-run(只读,不带 --apply 绝不写入)
scripts/migrate_image_index_cache.sh --db /path/to/copy.db
# 2. 先在 SQLite 备份副本上演练
sqlite3 -readonly /home/lionwebsite/cache.db ".backup '/tmp/cache-copy.db'"
# 3. 确认后执行(自动做 .backup 备份并校验)
scripts/migrate_image_index_cache.sh --db /tmp/cache-copy.db --apply
```
迁移只做两类写入:删除不健康 gid 的整组索引(重复头部、半截、跳号、空 imgkey)、
删除孤儿页 key;随后建立 `ux_gidToKey_gid` 与 `ux_ImageKeyCache_gid_page`。
若清理后仍有重复,建唯一索引会失败并整体回滚。真实的 `/home/lionwebsite/cache.db` 未被本次工作修改。
离线修复工具:
```bash
java -cp sqlite-jdbc.jar Maintenance/GalleryIndexMaintenance.java --db /tmp/cache-copy.db # dry-run
java -cp sqlite-jdbc.jar Maintenance/GalleryIndexMaintenance.java --db /tmp/cache-copy.db --apply # 执行
```
对 `/home/lionwebsite` 下的库运行 `--apply` 会被拒绝,除非显式加 `--allow-production`。
## 测试
在 `/tmp` 的独立源码副本中运行(未复制任何生产 runtime config、未改生产库):
```
mvn -o -Djacoco.skip=true \
-Dtest='GalleryIndexServiceTest,GalleryIndexMaintenanceTest,ImageCacheMapperSqlTest' \
-DfailIfNoSpecifiedTests=false test
Tests run: 9, Failures: 0, Errors: 0 -- GalleryIndexMaintenanceTest
Tests run: 21, Failures: 0, Errors: 0 -- GalleryIndexServiceTest
Tests run: 7, Failures: 0, Errors: 0 -- ImageCacheMapperSqlTest
Tests run: 37, Failures: 0, Errors: 0
BUILD SUCCESS
```
覆盖的关键场景:
- 同 gid 24 线程并发只抓取一次、只发布一次;失败结果也被等待者共享;失败后可重试。
- 抓取发生在事务之外;所有删除与写入都在事务内;整组替换只有一个写事务。
- 行数相同但页码跳号、空 imgkey、空 key、pages<=0 均判定为不完整并重建。
- 写入失败整体回滚,旧索引与失败前逐字段一致。
- 上游畸形数据(gid 不符、页码重复/跳号/越界、imgkey 为空)在开启写事务之前被拒绝。
- 大规模索引按 BATCH_SIZE 分片写入;重复发布替换旧行而非累积。
- 注解 SQL 在真实 SQLite 上验证,含唯一约束确实拒绝重复写入。
## 已知边界
- 迁移后 gidToKey 433 条、ImageKeyCache 43704 条(原 443 / 43918)。
被删除的是 4 个冲突或不完整 gid 的整组索引与 214 行关联页 key,由下一次在线阅读重抓。
- 未做性能压测,不承诺加索引后的具体延迟收益,只确认查询计划已由全表扫描变为索引查找。
- lookup 的读事务只是为了拿到一致快照;实测 sqlite-jdbc 不允许在已建立的连接上设置
readOnly,因此读模板只设超时、不设只读标志。
+61
View File
@@ -0,0 +1,61 @@
# 订阅缓存分代化 — 重写模块说明
## 问题
一个订阅账号对应两份内容:`v2ray.txt`(Base64 节点)与 `clash.yaml`(Clash 配置)。
旧实现先 `atomicWrite` 写 v2ray,再 `atomicWrite` 写 clash,两次替换彼此独立:
1. **两份文件可以不同版本**。第二次写入失败(磁盘满、进程被杀)时,用户会拿到
「新 v2ray + 旧 clash」的混合版本,且没有任何状态能表明它已经撕裂。
2. **读取方无法保证配对**。分发路径与备机快照各自 `Files.readAllBytes` 两个路径,
在这两次读之间完成一次刷新,就会读到跨版本组合。
3. **过期刷新可能复活**。在途刷新返回时只比较上游 key 与过滤开关;一次
`invalidateCache` 之后,迟到的响应仍会把内容写回被清空的路径。
4. **失败留半成品**。第一次写成功、第二次失败时,磁盘上就是只有一份文件的账号。
## 交付物与职责
| 文件 | 职责 |
|---|---|
| `Util/SubscriptionCacheStore.java` | 分代存储:两份文件同一分代写入 → 校验 → 指针原子发布;读取始终来自指针指向的同一分代 |
| `Service/SubscriptionRefreshService.java` | 抓取与解析在协调器锁外进行,发布短持锁;`invalidateCache` 递增版本使在途刷新失效 |
| `Service/SubscriptionStandbySnapshotService.java` | 快照构建在读锁内通过同一 store 解析成对内容,保证同版本 |
| `Util/ClashSubscriptionParser.java` | 按 YAML 语义过滤高倍节点并清理 `proxy-groups` 引用(由主模型提供) |
## 目录布局
```
<cacheRoot>/<accountId>/current 指针(Properties 文本,原子替换)
<cacheRoot>/<accountId>/generations/<generation>/v2ray.txt
<cacheRoot>/<accountId>/generations/<generation>/clash.yaml
<cacheRoot>/<accountId>/v2ray.txt + clash.yaml 旧结构,仅在两份都在时按 legacy 读
```
`current` 记录分代名与两份文件的 SHA-256。分代名是定宽 `时间戳-序号`,
因此字典序即时间序;默认只保留最近两个分代(当前与上一个)。
## 不变量
- **发布最后一步才切指针**:先写两份文件,读回校验长度与内容,再原子替换 `current`。
抓取失败、解析失败、任一份写入失败、指针写入失败,都会回滚新分代目录并保留上一个完整版本。
- **读取配对**:`read` / `resolve` / `hasComplete` 走同一份 `load` 契约,每个调用只读一次
指针快照,再按该快照校验双文件哈希。只有两份齐全且匹配才算一个有效版本;
指针缺失才回退 legacy(两份齐全时),指针存在但读不出完整版本(半写、篡改、缺字段)
时按无可用缓存处理,不回退到更旧的半份内容。
- **分发不会发出半份缓存**:`cachedPath` 在无有效版本时返回保证不存在的哨兵路径
(`.unavailable/`),使 `SubService` 判定为 503,而不会被单份 legacy 文件骗过。
`SubService.updateSub` 在读锁内完成 resolve 与 `FileDownload.export`,与 publish/invalidate 互斥。
- **失效不可复活**:`invalidateCache` 在写锁内删除指针/分代/旧结构文件,并移除该账号的
刷新版本标记;在途响应返回时因版本标记对不上而被丢弃,不会重新发布。
- **兼容旧结构**:只有当账号根下 `v2ray.txt` 与 `clash.yaml` 都在时才读为 legacy 版本;
新分代发布成功后旧结构文件被移除,避免两份来源并存。
## 测试
| 测试 | 锁定行为 |
|---|---|
| `Util/SubscriptionCacheStoreTest` | 发布后两份同分代;第二份写入失败 / 指针发布失败均保留旧版本;legacy 双文件才有效;半份 legacy、缺一份分代、内容被篡改、坏指针、指针指向缺失分代均判定不可用;每次查询只读一次指针;哨兵路径保证不存在 |
| `Service/SubscriptionCacheGenerationTest` | 在途旧刷新在失效与新版发布后被丢弃;被丢弃的刷新不留任何缓存 |
| `Service/SubscriptionFilteringTest` | 过滤规则;失败保留上次完整版本;`cachedPath` / `hasCompleteCache` 新契约;legacy 路径定位 |
| `Service/SubscriptionStandbySnapshotServiceTest` | 快照从单一分代读成对内容;不完整 legacy 账号不入快照 |
| `Util/ClashSubscriptionParserTest` | YAML 语义过滤(内联节点与引用)与畸形正文拒绝 |
+30
View File
@@ -0,0 +1,30 @@
# 备用订阅更新事件
主站 `/sub/{client}/{key}` 接受网关的签名事件;`GET` 在客户端类型与缓存路径有效后,同一个主库事务写更新记录和长期收据。签名有效的 `HEAD` 不写记录。普通请求以及无效签名仍走原来的更新记录逻辑。
`POST /internal/subscription-events` 有界读取原始请求(最多 256 KiB),批次最多 100 条。签名算法为 HMAC-SHA256,同步密钥来自 `SUBSCRIPTION_SYNC_SECRET`;签名输入是 `v1\n` + epoch 秒时间戳 + `\n` + 原始请求体 SHA256 十六进制。时间偏差最多 300 秒。请求头为 `X-Subscription-Timestamp` 和 `X-Subscription-Signature`。主站 GET 将单个事件的原始 JSON 使用无 padding 的 base64url 编码放入 `X-Subscription-Event`,同样对原始事件 JSON 签名。
事件字段:`eventId`(规范小写 UUID)、`keyHash`(公开 key 的小写 SHA256)、`client`(v2 或 cat)、`clientIp`(IPv4/IPv6 字面值)、`userAgent`(最多 2048 字符)、`occurredAt`(epoch 毫秒)。允许历史事件,事件时间不可超过未来 5 分钟。签名中的 IP 为权威来源,不能由未经签名的 XFF 覆盖。
响应 `acknowledged` 只在事务已提交后返回,包括已存在的相同事件。冲突事件 ID 返回 400;签名无效返回 403;数据库异常返回 5xx,无确认列表。未知 hash 不确认;历史映射中用户已删除则写终态收据并确认,不猜测用户归属。可见记录按事件时间和 ID 倒序保留每用户最近 10 条,收据独立长期保留。批次任意数据库写入失败会全部回滚。
## 显式迁移
运行前先执行稳定 user_id 关联迁移。新表不会在启动时创建。
1. 先在数据库副本上演练:`python3 scripts/migrate_subscription_events.py /path/to/copy.db`。
2. 停止生产主站服务,确认没有其他写入者。
3. 执行 `python3 scripts/migrate_subscription_events.py /path/to/main.db --apply --service-stopped`。
4. 检查输出 integrity 为 ok,保留输出中的 SQLite 备份,然后启动新程序。
脚本在写入前使用 SQLite backup API 生成权限 0600 的备份,在单个事务中建表并初始化历史 hash,重复执行保留既有收据和历史。不将公开 key 写入历史表或输出;原数据库备份含既有敏感数据,须保留原有保护措施。
快照构建记住所有现有绑定;重置或删除绑定前也保存历史 hash。用户改名时回传事件按 stable user_id 读取当前名字。合法旧订阅标签的 user_id 可以为 NULL,历史映射保留标签并按原标签写回,不创建用户,也不猜测同名新用户归属。可见日志删除不删除历史映射或收据。
Java 编译与测试必须在部署手册指定的 192.168.5.30 执行:
```sh
mvn -Dtest=SubscriptionEventIntegrationTest,SubServiceTest,SubscriptionStandbySnapshotServiceTest,MyBatisNativeConfigurationTest test
```
Python 迁移验证可本机执行:`python3 -m unittest discover -s scripts -p 'test_subscription_events.py' -v`。
+80
View File
@@ -0,0 +1,80 @@
# 任务生命周期与协议 v2
范围:主站下载任务的持久化生命周期、与存储节点的可靠上报协议。对应评估报告第 2、6 节。
## 为什么重写
原实现里创建先`POST`上游、后落库,删除先删库、后等节点,节点上报异步写完后立刻清队列:上下游任一侧失败都会留下「主站与节点说法不一致」的状态,且没有任何东西能把它们对回来。
复现过的具体后果:模拟状态消息发送失败后,完成任务已从节点队列移除、消息却未发出,于是「归档已存在、主站仍显示未完成」。
## 不变量
1. `gid` 只有一个有效生命周期意图。创建先落唯一 `CREATE` 意图(并写一条可见的占位任务行),靠 `gallery(gid)` 主键唯一把并发创建收敛成一个赢家;输家直接得到业务失败。
2. 上游副作用只在意图持久化之后发生。顺序固定为:登记意图 → 只读解析(`prepare`,无 POST)→ 持久化 `SUBMITTING` → 真正 `submit`。
3. `SUBMITTING` 之后绝不自动重发。上游不支持幂等,因此超时/断连/落库失败/重启中断一律落 `UNKNOWN` 等待人工对账处理,而不是再 `POST` 一次。
4. 任务行与周用量在同一短事务里写入;重复落库不重复计量。数据库失败时任务行与意图保持一致(要么都在,要么明确为 `UNKNOWN`),不允许出现「有任务没计量」。
5. 删除失败保留记录。先落 `DELETE_PENDING` 与稳定 `deleteOperationId`,只有节点确认(或文件已不存在)后才在同一事务里清收藏、删任务行并留 `DELETED` 墓碑。
6. 上报按 `revision` 单调推进,按 `operationId` 归属。迟到的旧世代上报被丢弃且不改变当前任务;墓碑(`DELETED`)永久忽略所有上报,已删除任务只能由显式 `CREATE` 重新建立新世代。
7. 回执只在提交之后发送。数据库失败不回 ACK,节点保留快照重投;重复上报也回 ACK 以停止重放,但不会重复通知。
8. 通知与 WebSocket 推送在事务提交之后、由独立执行器发送;慢通知不阻塞状态提交。
9. 只有确认已提交(`SUBMITTED`)或迁移前确实提交过的旧任务才允许下发给节点。`UNKNOWN`、`RESERVED`、`SUBMITTING`、`DELETE_PENDING`、`DELETED` 一律不下发:上游不支持幂等,把未确认/未计量的任务重投会在节点侧变成「下载完成却 0 额度」。
10. 启动时先绑定上报处理器与判定解析器、执行恢复(`SUBMITTING → UNKNOWN`、释放 `RESERVED`),完成之后才允许连接与自动重投;恢复失败时只保持连接,禁止自动派发。
## 状态表
| 层 | 取值 |
| --- | --- |
| `task_operation.state` | `RESERVED` `SUBMITTING` `SUBMITTED` `UNKNOWN` `DELETE_PENDING` `DELETED` |
| 任务展示状态 | `提交中` `已提交` `下载中` `等待压缩` `压缩中` `下载完成` `提交未知` `待删除` `删除失败` |
| 节点上报状态码 | `1` 下载中 `2` 等待压缩 `3` 压缩中 `4` 下载完成(沿用旧码) |
状态码到中文文案只在 `TaskStatus` 一处映射。
## 线协议
帧格式不变:`[messageType(1B)][length(4B, 大端)][JSON]`。旧类型 `0..9` 的字段一个都不动(旧节点 Jackson 会拒绝未知字段)。
| 类型 | 消息 | 关键字段 |
| --- | --- | --- |
| 10 | `ProtocolHelloMessage` | `protocolVersion=2`, `features=["reliable-task-v2"]` |
| 11 | `TaskCommandMessage` | `operationId`, `action=DOWNLOAD\|DELETE`, `gid`, `name` |
| 12 | `TaskReportMessage` | `reportId`, `revision`, `galleryTasks[]`, `operationIds{gid→op}` |
| 13 | `TaskReportAckMessage` | `reportId`, `result` |
流程:主站先发旧 `Identity`,再发类型 10;节点回类型 10 声明 v2 后,才允许使用 `11..13`。未协商成功时自动退化为旧的 `DownloadPostMessage`/`DeleteGalleryMessage`(此时不承诺可靠 ACK,属已知兼容边界)。
## 部署与迁移
1. 用 `scripts/migrate_task_operation_v2.sql` 建表。脚本只有独立语句、不含 `BEGIN/COMMIT`,由外层 wrapper 包事务并在执行前备份。
2. 该脚本还会在 `gallery(gid)` 上建唯一索引。历史库若已有重复 gid,这条语句会失败:先人工确认重复来源再处理,不要直接删行。
3. 主库连接需要 busy 等待(`SqlConfiguration` 已为主数据源设置 `PRAGMA busy_timeout=5000`)。单文件 SQLite 并发开短事务时,没有等待窗口会直接抛出 `SQLITE_BUSY`。生产库建议同时启用 WAL。
4. 旧任务行没有操作记录,首次收到上报时会按显式策略补建一次「世代」(优先采纳上报里的 `operationId`),不会改动其已有状态。
## 验证
`mvn -o test`:549 项,0 失败/错误/跳过。
新增关键用例(`TaskLifecycleIntegrationTest`,真实 SQLite 文件 + 真实事务/触发器/唯一约束,落在 `/tmp` 临时目录):
- 意图与占位行先于上游 `POST`;`SUBMITTING` 先于 `submit`。
- 同 `gid` 6 线程并发创建:上游只被提交一次、任务行不超过一条。
- 上游超时后落 `UNKNOWN`;再次点击创建也不会第二次 `POST`。
- 重启恢复把中断的 `SUBMITTING` 判为 `UNKNOWN`(任务行同步显示「提交未知」),安全释放 `RESERVED` 占位,且不改写 `operationId`、不重发。
- 额度写入失败时落库事务回滚,任务保持可对账的 `提交未知` + `UNKNOWN`。
- 重投/重试复用同一 `operationId`。
- 判定为拒绝的任务(`UNKNOWN` 等)绝不下发,也不退化为无 `operationId` 的旧命令。
- 删除失败保留任务行与意图,重试复用同一删除 ID;成功后清收藏、删行、留墓碑。删除意图 ID 不匹配时不得完成删除。
- 正在提交(`RESERVED`/`SUBMITTING`)的任务拒绝删除;释放预占只清匹配世代的记录与占位行。
- 落库严格匹配本次创建世代(expectedOperationId),被删除/换代后不得复活已删 gid。
- 墓碑永久忽略所有上报(含同一 `operationId`),只允许显式 `CREATE` 用新 `operationId` 重建。
- 旧节点不带 `operationId` 的连续上报保持可接受;新 v2 已知世代缺归属的上报仍被拒。
- 过期 `revision`、缺 `operationId`、待删除任务的上报都被丢弃;未知 gid 忽略但回执。
- 重复上报不重复通知;回执写入失败时 `apply` 返回不可回执结果。
`TaskProtocolV2CodecTest` 锁定类型 10..13 的帧头与字段名,并断言旧 `DownloadPostMessage` 没有混入新字段。
## 已知限制
未做双机真实故障演练。`UNKNOWN` 目前只提供状态与持久化记录:任务行会显示「提交未知」并拒绝自动重投/重试,对账与人工解决的接口由后续批次补齐。
+7
View File
@@ -15,8 +15,15 @@
<description>LionWebsite</description> <description>LionWebsite</description>
<properties> <properties>
<java.version>21</java.version> <java.version>21</java.version>
<tomcat.version>11.0.27</tomcat.version>
<jackson-bom.version>3.1.7</jackson-bom.version>
</properties> </properties>
<dependencies> <dependencies>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>2.6</version>
</dependency>
<dependency> <dependency>
<groupId>org.springframework.boot</groupId> <groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webmvc</artifactId> <artifactId>spring-boot-starter-webmvc</artifactId>
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env bash
# 图片索引(cache.db)迁移:为缓存表补唯一约束与查询索引。
#
# 设计约束:
# * 默认 dry-run,只读检查;必须显式 --apply 才执行写入。
# * 必须显式 --db 指定数据库文件,不读取任何运行时配置,也不推断生产路径。
# * 写库用 sqlite3 -cmd '.bail on',任一语句出错立即以非零码退出,事务不提交。
# * 只以读写模式打开目标库(文件必须已存在,不会新建);dry-run 全用 -readonly。
# * 备份用 SQLite 自身的 .backup(online backup API),而不是 cp:
# cp 会漏掉 WAL 里尚未 checkpoint 的已提交内容。
# * 备份后立即校验备份可读且含两张缓存表,失败则中止并删除无效备份。
# * 不健康(不完整/冲突)的 gid 整组索引会被删除,由下一次在线阅读重新抓取。
# * 唯一约束建在唯一索引上:若清理后仍有重复,建索引会失败并整体回滚。
#
# 用法:
# scripts/migrate_image_index_cache.sh --db /tmp/cache-copy.db # dry-run(只读)
# scripts/migrate_image_index_cache.sh --db /tmp/cache-copy.db --apply # 执行迁移
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SQL_FILE="$SCRIPT_DIR/migrate_image_index_cache.sql"
db_path=""
apply="no"
usage() {
sed -n '2,30p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
}
while [[ $# -gt 0 ]]; do
case "$1" in
--db) db_path="${2:-}"; shift 2 ;;
--apply) apply="yes"; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "未知参数: $1" >&2
exit 2 ;;
esac
done
if [[ -z "$db_path" ]]; then
echo "必须显式指定缓存库: --db <cache.db>;本脚本不会自动推断数据库位置。" >&2
exit 2
fi
if [[ ! -f "$db_path" ]]; then
echo "数据库不存在: $db_path" >&2
exit 2
fi
if [[ ! -f "$SQL_FILE" ]]; then
echo "缺少 SQL: $SQL_FILE" >&2
exit 2
fi
if ! command -v sqlite3 >/dev/null 2>&1; then
echo "缺少 sqlite3 命令行工具" >&2
exit 2
fi
if [[ "$(sqlite3 -readonly "$db_path" \
"select count(*) from sqlite_master where type='table' and name in ('gidToKey','ImageKeyCache');")" != "2" ]]; then
echo "缓存表缺失:$db_path 中找不到 gidToKey 与 ImageKeyCache。" >&2
exit 2
fi
# 体检统计,与 SQL 里实际清理的范围一致:
# bad_gids 将被整组删除的 gid 个数(去重),覆盖:
# 重复头部 / 行数不符 / 页码去重后不足 / 页码区间异常 /
# 空 imgkey / 空 key / 非法 pages
# bad_gid_rows 这些 gid 占用的头部行数(重复头部的 gid 会多于 1 行)
# duplicate_pages 重复的 (gid,page) 组数,仅用于展示,会随所属 gid 一并清理
# orphan_gids 有页 key 但无头部记录的 gid 个数
read_stats() {
sqlite3 -readonly "$db_path" "
select 'header_rows', count(*) from gidToKey;
select 'page_rows', count(*) from ImageKeyCache;
select 'duplicate_gids', count(*) from (select gid from gidToKey group by gid having count(*) > 1);
select 'duplicate_pages', count(*) from (select gid,page from ImageKeyCache group by gid,page having count(*) > 1);
select 'bad_gids', count(distinct g.gid) from gidToKey g
where g.pages is null or g.pages <= 0
or g.key is null or trim(g.key) = ''
or (select count(*) from ImageKeyCache i where i.gid = g.gid) <> g.pages
or (select count(distinct i.page) from ImageKeyCache i where i.gid = g.gid) <> g.pages
or (select min(i.page) from ImageKeyCache i where i.gid = g.gid) <> 1
or (select max(i.page) from ImageKeyCache i where i.gid = g.gid) <> g.pages
or exists (select 1 from ImageKeyCache i
where i.gid = g.gid and (i.imgkey is null or trim(i.imgkey) = ''));
select 'bad_gid_rows', coalesce(sum(c), 0) from (
select count(*) as c from gidToKey g where g.gid in (
select g2.gid from gidToKey g2 where g2.pages is null or g2.pages <= 0
or g2.key is null or trim(g2.key) = ''
or (select count(*) from ImageKeyCache i where i.gid = g2.gid) <> g2.pages
or (select count(distinct i.page) from ImageKeyCache i where i.gid = g2.gid) <> g2.pages
or (select min(i.page) from ImageKeyCache i where i.gid = g2.gid) <> 1
or (select max(i.page) from ImageKeyCache i where i.gid = g2.gid) <> g2.pages
or exists (select 1 from ImageKeyCache i
where i.gid = g2.gid and (i.imgkey is null or trim(i.imgkey) = '')))
group by g.gid);
select 'orphan_gids', count(distinct gid) from ImageKeyCache
where gid not in (select gid from gidToKey);
select 'strict_index', count(*) from sqlite_master
where type='index' and name='ux_ImageKeyCache_gid_page';
"
}
mode_label="dry-run(只读)"
if [[ "$apply" == "yes" ]]; then mode_label="apply(读写)"; fi
echo "=== 缓存库图片索引迁移($mode_label) ==="
echo "数据库: $db_path"
echo "--- 迁移前体检 ---"
read_stats | while IFS='|' read -r name value; do
printf ' %-16s %s\n' "$name" "$value"
done
if [[ "$apply" != "yes" ]]; then
echo
echo "dry-run:未做任何写入。确认无误后加 --apply 执行迁移。"
echo "提示:请先在 SQLite 副本上演练,副本应这样生成(含 WAL 内容):"
echo " sqlite3 -readonly <生产库> \".backup '<副本路径>'\""
exit 0
fi
# ---- 备份:SQLite online backup,包含 WAL 中已提交但未 checkpoint 的内容 ----
backup="$db_path.before-image-index.$(date -u +%Y%m%dT%H%M%SZ)"
if ! sqlite3 "$db_path" ".backup '$backup'"; then
echo "备份失败,未做任何修改: $backup" >&2
exit 3
fi
if [[ ! -s "$backup" ]]; then
echo "备份文件为空,未做任何修改: $backup" >&2
exit 3
fi
if [[ "$(sqlite3 -readonly "$backup" \
"select count(*) from sqlite_master where type='table' and name in ('gidToKey','ImageKeyCache');")" != "2" ]]; then
echo "备份校验失败(缺少缓存表),未做任何修改,已删除无效备份。" >&2
rm -f -- "$backup"
exit 3
fi
echo "已备份(SQLite .backup): $backup"
# ---- 执行迁移:.bail on 保证任一语句失败即退出,事务不提交 ----
sqlite3 -cmd '.bail on' "$db_path" < "$SQL_FILE"
echo "--- 迁移后体检 ---"
read_stats | while IFS='|' read -r name value; do
printf ' %-16s %s\n' "$name" "$value"
done
echo "图片索引迁移完成: $db_path"
+67
View File
@@ -0,0 +1,67 @@
-- 图片索引缓存迁移:为 cache.db 的两张索引表补唯一约束与查询索引。
--
-- 目标(对应本次重写的三条不变量):
-- 1. gid 唯一 —— 同一本画廊只能有一条 gidToKey 记录;
-- 2. (gid,page) 唯一 —— 同一页只能有一个 imgkey,同时天然满足按页查询的索引需求;
-- 3. 只保留「完整」的索引 —— 半截或冲突的 gid 整组删除,让下一次在线阅读重新抓取。
--
-- 为什么用唯一索引而不是重建表:
-- * 唯一索引对 SQLite 与唯一约束等价,都会拒绝重复写入;
-- * 不需要搬表、改列,旧版本二进制仍可读写同一份数据;
-- * 回滚只需 drop index,最坏情况也只需回到脚本自动生成的备份文件。
--
-- 幂等:所有重建索引都带 IF NOT EXISTS;清理语句只按集合删除,重复执行结果相同。
-- 安全:本脚本只做 DELETE 与 CREATE INDEX,不建表、不改列、不碰 gidToKey/ImageKeyCache
-- 之外的任何对象。必须在事务中整体提交,调用方使用 sqlite3 -bail 保证任一语句
-- 失败即终止、事务回滚。
--
-- 执行前请务必先备份数据库文件(迁移脚本 migrate_image_index_cache.sh --apply 会自动备份)。
BEGIN IMMEDIATE;
-- ---------- 1. 冲突 gid:同一 gid 有多条头部记录 ----------
-- 无法判断哪一条正确,整组移除,由下次访问重抓。
CREATE TEMP TABLE __conflict_gid AS
SELECT gid FROM gidToKey GROUP BY gid HAVING COUNT(*) > 1;
DELETE FROM ImageKeyCache WHERE gid IN (SELECT gid FROM __conflict_gid);
DELETE FROM gidToKey WHERE gid IN (SELECT gid FROM __conflict_gid);
-- ---------- 2. 半截或非法 gid:头部与页 key 不一致 ----------
-- 判定同时覆盖:页码数量不符、页码不连续(跳号/重复)、区间不是 1..pages、
-- 头部字段为空、以及存在空的 imgkey 行。
CREATE TEMP TABLE __broken_gid AS
SELECT g.gid
FROM gidToKey g
WHERE g.pages IS NULL OR g.pages <= 0
OR g.key IS NULL OR trim(g.key) = ''
OR (SELECT COUNT(*) FROM ImageKeyCache i WHERE i.gid = g.gid) <> g.pages
OR (SELECT COUNT(DISTINCT i.page) FROM ImageKeyCache i WHERE i.gid = g.gid) <> g.pages
OR (SELECT MIN(i.page) FROM ImageKeyCache i WHERE i.gid = g.gid) <> 1
OR (SELECT MAX(i.page) FROM ImageKeyCache i WHERE i.gid = g.gid) <> g.pages
OR EXISTS (SELECT 1 FROM ImageKeyCache i
WHERE i.gid = g.gid AND (i.imgkey IS NULL OR trim(i.imgkey) = ''));
DELETE FROM ImageKeyCache WHERE gid IN (SELECT gid FROM __broken_gid);
DELETE FROM gidToKey WHERE gid IN (SELECT gid FROM __broken_gid);
-- ---------- 3. 孤儿页 key:没有对应头部记录 ----------
DELETE FROM ImageKeyCache WHERE gid NOT IN (SELECT gid FROM gidToKey);
-- ---------- 4. 唯一约束 ----------
-- 到这里若仍有重复,说明上面的判定有遗漏;CREATE UNIQUE INDEX 会失败,
-- 配合 -bail 让整个事务回滚,不会留下「一半约束」的库。
CREATE UNIQUE INDEX IF NOT EXISTS ux_gidToKey_gid
ON gidToKey (gid);
CREATE UNIQUE INDEX IF NOT EXISTS ux_ImageKeyCache_gid_page
ON ImageKeyCache (gid, page);
-- 按 gid 统计页数会用到 (gid,page) 前缀,按页取 key 也走同一索引,
-- 因此不再单独建 page 索引,避免重复占用写放大。
DROP TABLE __conflict_gid;
DROP TABLE __broken_gid;
COMMIT;
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Create durable subscription event receipts/history, with a SQLite backup.
Stop the production backend before --apply and pass --service-stopped to confirm.
No automatic startup DDL exists. Dry run is read-only; only hashes enter history.
Receipt retention is independent of the most recent ten visible update records.
"""
import argparse
import datetime
import hashlib
import json
from pathlib import Path
import sqlite3
DDL = (
"""CREATE TABLE IF NOT EXISTS subscription_event_receipt (
event_id TEXT PRIMARY KEY NOT NULL, key_hash TEXT NOT NULL,
client TEXT NOT NULL, occurred_at INTEGER NOT NULL, source TEXT NOT NULL,
ingested_at INTEGER NOT NULL, fingerprint TEXT NOT NULL, outcome TEXT NOT NULL
)""",
"""CREATE TABLE IF NOT EXISTS subscription_event_binding (
key_hash TEXT PRIMARY KEY NOT NULL, user_id INTEGER,
user_name TEXT NOT NULL, first_seen_at INTEGER NOT NULL, last_seen_at INTEGER NOT NULL
)""",
"CREATE INDEX IF NOT EXISTS idx_subscription_event_binding_user ON subscription_event_binding(user_id)",
"CREATE INDEX IF NOT EXISTS idx_sub_record_user_time ON sub_update_record(user_id,time DESC,id DESC)",
)
def migrate(database: Path, apply=False, service_stopped=False):
if not database.is_file():
raise ValueError("database must already exist")
if apply and not service_stopped:
raise ValueError("stop backend and supply --service-stopped before applying")
uri = database.resolve().as_uri() + ("?mode=rw" if apply else "?mode=ro")
with sqlite3.connect(uri, uri=True) as db:
for table in ("User", "sub_bind", "sub_update_record"):
columns = {row[1] for row in db.execute(f'PRAGMA table_info("{table}")')}
if not columns or (table != "User" and "user_id" not in columns):
raise ValueError("stable user links migration is required first")
unresolved = db.execute("SELECT count(*) FROM sub_bind sb LEFT JOIN User u ON u.id=sb.user_id WHERE sb.user_id IS NOT NULL AND u.id IS NULL").fetchone()[0]
if unresolved:
raise ValueError("resolve non-null dangling subscription user IDs before migration")
tables = {row[0] for row in db.execute("SELECT name FROM sqlite_master WHERE type='table'")}
result = {"apply": apply, "tables_to_create": sorted({"subscription_event_receipt", "subscription_event_binding"} - tables)}
if not apply:
return result
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
backup = database.with_name(database.name + ".before-subscription-events." + stamp)
# Create mode 600 before any sensitive database bytes are written.
backup.touch(mode=0o600, exist_ok=False)
with sqlite3.connect(backup) as snapshot:
db.backup(snapshot)
try:
db.execute("BEGIN IMMEDIATE")
for statement in DDL:
db.execute(statement)
now = int(datetime.datetime.now(datetime.timezone.utc).timestamp() * 1000)
for key, user_id, name in db.execute("SELECT sb.key,sb.user_id,coalesce(u.username,sb.user) FROM sub_bind sb LEFT JOIN User u ON u.id=sb.user_id"):
key_hash = hashlib.sha256(key.encode("utf-8")).hexdigest()
db.execute("""INSERT INTO subscription_event_binding VALUES(?,?,?,?,?)
ON CONFLICT(key_hash) DO UPDATE SET user_name=excluded.user_name,last_seen_at=excluded.last_seen_at
WHERE subscription_event_binding.user_id IS excluded.user_id""", (key_hash, user_id, name, now, now))
db.commit()
except BaseException:
db.rollback()
raise
result["backup"] = str(backup)
result["history_bindings"] = db.execute("SELECT count(*) FROM subscription_event_binding").fetchone()[0]
result["integrity"] = db.execute("PRAGMA quick_check").fetchone()[0]
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("database", type=Path)
parser.add_argument("--apply", action="store_true")
parser.add_argument("--service-stopped", action="store_true")
args = parser.parse_args()
print(json.dumps(migrate(args.database, args.apply, args.service_stopped), ensure_ascii=False))
+31
View File
@@ -0,0 +1,31 @@
-- 任务操作记录表(协议 v2:先持久化意图,再产生外部副作用)
--
-- 这是显式迁移脚本,不会被应用自动执行,也不对生产库运行。
-- 执行前请备份:sqlite3 LionWebsite.db ".backup LionWebsite.db.before-task-operation"
--
-- 幂等:可重复执行。
create table if not exists task_operation
(
gid integer not null
constraint task_operation_pk
primary key,
operationId varchar(64),
deleteOperationId varchar(64),
state varchar(20) not null,
lastRevision integer not null default 0,
lastReportId varchar(64),
updateTime long not null default 0
);
-- 节点上报消费只按 gid 命中主键,这里额外为「按状态找待处理意图」提供查询入口。
create index if not exists idx_task_operation_state on task_operation (state);
-- gallery.gid 目前只有 table 级主键约束(部分历史库甚至是自增主键的衍生约束),
-- 这里显式建立唯一索引,保证同一 gid 不会出现两条任务行:
-- 「同一 gid 只有一个有效意图」是 v2 生命周期成立的前提。
-- 若该语句因历史重复数据失败,请先人工确认重复行来源再处理,不要直接删除。
create unique index if not exists idx_gallery_gid_unique on gallery (gid);
-- 旧任务行没有操作记录:启动/查询时按 gid 惰性补建为 SUBMITTED(已在上游提交过),
-- 不需要在迁移里批量写入,避免把正在进行的任务判成 UNKNOWN。
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env python3
"""Explicit task-v2 schema migration. Default is read-only; --apply backs up first."""
import argparse
import datetime
import json
from pathlib import Path
import sqlite3
def migrate(database: Path, apply: bool = False) -> dict:
if not database.is_file():
raise ValueError("database must already exist")
with sqlite3.connect(database.resolve().as_uri() + ("?mode=rw" if apply else "?mode=ro"), uri=True) as db:
if not db.execute("pragma table_info(gallery)").fetchall():
raise ValueError("required table missing: gallery")
duplicates = db.execute("select count(*) from (select gid from gallery group by gid having count(*)>1)").fetchone()[0]
if duplicates:
raise ValueError("duplicate gallery gid; reconcile before migration")
present = bool(db.execute("pragma table_info(task_operation)").fetchall())
result = {"apply": apply, "create_operation_table": not present}
if not apply:
return result
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
backup = database.with_name(database.name + ".before-task-v2." + stamp)
with sqlite3.connect(backup) as copy:
db.backup(copy)
backup.chmod(0o600)
sql = Path(__file__).with_name("migrate_task_operation_v2.sql").read_text()
try:
db.executescript("BEGIN IMMEDIATE;\n" + sql + "\nCOMMIT;")
except BaseException:
db.rollback()
raise
result["backup"] = str(backup)
result["operation_rows"] = db.execute("select count(*) from task_operation").fetchone()[0]
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("database", type=Path)
parser.add_argument("--apply", action="store_true")
arguments = parser.parse_args()
print(json.dumps(migrate(arguments.database, arguments.apply), ensure_ascii=False))
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""Add stable user identity to subscription bindings and access records.
The default invocation only reports schema changes. --apply creates a consistent
SQLite backup before the transaction. Run against an isolated database first.
"""
import argparse
import datetime
import json
from pathlib import Path
import sqlite3
def migrate(db_path: Path, apply: bool = False) -> dict:
if not db_path.is_file():
raise ValueError("database must already exist")
uri = db_path.resolve().as_uri() + ("?mode=rw" if apply else "?mode=ro")
with sqlite3.connect(uri, uri=True) as db:
changes = []
for table in ("sub_bind", "sub_update_record"):
columns = {row[1] for row in db.execute(f'PRAGMA table_info("{table}")')}
if not columns:
raise ValueError(f"required table missing: {table}")
if "user_id" not in columns:
changes.append(f"add {table}.user_id")
duplicates = db.execute(
"select count(*) from (select username from User group by username having count(*) > 1)"
).fetchone()[0]
if duplicates:
raise ValueError("ambiguous duplicate usernames; resolve identities before migration")
result = {"apply": apply, "schema_changes": changes}
if not apply:
return result
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
backup = db_path.with_name(db_path.name + ".before-user-links." + stamp)
with sqlite3.connect(backup) as snapshot:
db.backup(snapshot)
backup.chmod(0o600)
db.execute("BEGIN IMMEDIATE")
try:
for change in changes:
table = change.split()[1].split(".")[0]
db.execute(f'ALTER TABLE "{table}" ADD COLUMN user_id INTEGER REFERENCES User(id)')
for table in ("sub_bind", "sub_update_record"):
db.execute(f'UPDATE "{table}" SET user_id=(SELECT id FROM User WHERE username="{table}".user) WHERE user_id IS NULL')
db.execute("CREATE UNIQUE INDEX IF NOT EXISTS User_username_uindex ON User(username)")
db.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_sub_bind_user_id ON sub_bind(user_id) WHERE user_id IS NOT NULL")
db.execute("CREATE INDEX IF NOT EXISTS idx_sub_record_user_id ON sub_update_record(user_id, id)")
db.commit()
except BaseException:
db.rollback()
raise
result["backup"] = str(backup)
result["unresolved_bindings"] = db.execute("select count(*) from sub_bind where user_id is null").fetchone()[0]
result["unresolved_records"] = db.execute("select count(*) from sub_update_record where user_id is null").fetchone()[0]
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("database", type=Path)
parser.add_argument("--apply", action="store_true")
args = parser.parse_args()
print(json.dumps(migrate(args.database, args.apply), ensure_ascii=False))
+82
View File
@@ -0,0 +1,82 @@
import importlib.util
from pathlib import Path
import sqlite3
import tempfile
import unittest
import hashlib
spec = importlib.util.spec_from_file_location("migration", Path(__file__).with_name("migrate_subscription_events.py"))
migration = importlib.util.module_from_spec(spec)
spec.loader.exec_module(migration)
class SubscriptionEventsMigrationTest(unittest.TestCase):
def test_explicit_backup_atomic_and_idempotent(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "main.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY,username TEXT);
CREATE TABLE sub_bind(key TEXT,user TEXT,user_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY,user_id INTEGER,time INTEGER);
INSERT INTO User VALUES(3,'alice');
INSERT INTO sub_bind VALUES('public-test-key','alice',3);
""")
dry = migration.migrate(path)
self.assertEqual(2, len(dry["tables_to_create"]))
with self.assertRaises(ValueError):
migration.migrate(path, True)
first = migration.migrate(path, True, True)
self.assertEqual("ok", first["integrity"])
self.assertEqual(0o600, Path(first["backup"]).stat().st_mode & 0o777)
with sqlite3.connect(first["backup"]) as db:
self.assertEqual(0, db.execute("SELECT count(*) FROM sqlite_master WHERE name='subscription_event_binding'").fetchone()[0])
with sqlite3.connect(path) as db:
row = db.execute("SELECT key_hash,user_id,user_name FROM subscription_event_binding").fetchone()
self.assertEqual((hashlib.sha256(b"public-test-key").hexdigest(), 3, "alice"), row)
db.execute("INSERT INTO subscription_event_receipt VALUES('id','hash','v2',0,'primary',1,'fingerprint','recorded')")
second = migration.migrate(path, True, True)
self.assertEqual([], second["tables_to_create"])
with sqlite3.connect(path) as db:
self.assertEqual(1, db.execute("SELECT count(*) FROM subscription_event_receipt").fetchone()[0])
self.assertEqual(1, db.execute("SELECT count(*) FROM subscription_event_binding").fetchone()[0])
def test_legacy_labels_without_user_ids_remain_null(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "main.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY,username TEXT);
CREATE TABLE sub_bind(key TEXT,user TEXT,user_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY,user_id INTEGER,time INTEGER);
INSERT INTO sub_bind VALUES('legacy-key','legacy-label',NULL);
""")
result = migration.migrate(path, True, True)
self.assertEqual(1, result["history_bindings"])
with sqlite3.connect(path) as db:
row = db.execute("SELECT key_hash,user_id,user_name FROM subscription_event_binding").fetchone()
self.assertEqual((hashlib.sha256(b"legacy-key").hexdigest(), None, "legacy-label"), row)
self.assertIsNone(db.execute("SELECT user_id FROM sub_bind").fetchone()[0])
self.assertEqual(0, db.execute("SELECT count(*) FROM User").fetchone()[0])
migration.migrate(path, True, True)
with sqlite3.connect(path) as db:
self.assertEqual(1, db.execute("SELECT count(*) FROM subscription_event_binding").fetchone()[0])
def test_orphan_binding_aborts_without_ddl(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "main.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY,username TEXT);
CREATE TABLE sub_bind(key TEXT,user TEXT,user_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY,user_id INTEGER,time INTEGER);
INSERT INTO sub_bind VALUES('test','deleted',9);
""")
with self.assertRaises(ValueError):
migration.migrate(path, True, True)
with sqlite3.connect(path) as db:
self.assertEqual(0, db.execute("SELECT count(*) FROM sqlite_master WHERE name LIKE 'subscription_event_%'").fetchone()[0])
if __name__ == "__main__":
unittest.main()
+55
View File
@@ -0,0 +1,55 @@
import importlib.util
from pathlib import Path
import sqlite3
import tempfile
import unittest
spec = importlib.util.spec_from_file_location("migration", Path(__file__).with_name("migrate_user_links.py"))
migration = importlib.util.module_from_spec(spec)
spec.loader.exec_module(migration)
class MigrationTest(unittest.TestCase):
def test_dry_run_idempotent_migration_and_rename(self):
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "test.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY, username TEXT);
CREATE TABLE sub_bind(key TEXT PRIMARY KEY, user TEXT, subscription_account_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY, user TEXT, location TEXT);
INSERT INTO User VALUES(1,'old-name');
INSERT INTO sub_bind VALUES('example-key','old-name',2);
INSERT INTO sub_update_record VALUES(10,'old-name','unknown');
INSERT INTO sub_update_record VALUES(11,'old-name','unknown');
""")
before = path.read_bytes()
self.assertEqual(2, len(migration.migrate(path)["schema_changes"]))
self.assertEqual(before, path.read_bytes())
self.assertEqual(0, migration.migrate(path, True)["unresolved_bindings"])
self.assertEqual([], migration.migrate(path, True)["schema_changes"])
with sqlite3.connect(path) as db:
db.execute("update User set username='new-name' where id=1")
self.assertEqual((1,), db.execute("select user_id from sub_bind").fetchone())
# A slow first lookup must not overwrite the second request's row.
db.execute("update sub_update_record set location='first-location' where id=10")
self.assertEqual(("unknown",), db.execute("select location from sub_update_record where id=11").fetchone())
def test_ambiguous_names_fail_without_modifying_database(self):
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "test.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY, username TEXT);
CREATE TABLE sub_bind(user TEXT);
CREATE TABLE sub_update_record(user TEXT);
INSERT INTO User VALUES(1,'same'),(2,'same');
""")
before = path.read_bytes()
with self.assertRaises(ValueError):
migration.migrate(path, True)
self.assertEqual(before, path.read_bytes())
if __name__ == "__main__":
unittest.main()
@@ -27,6 +27,9 @@ import java.util.Random;
Chat.class, MessageEntity.class, Chat.class, MessageEntity.class,
AbstractMethodError.class, DeleteGalleryMessage.class, DownloadPostMessage.class, DownloadStatusMessage.class, AbstractMethodError.class, DeleteGalleryMessage.class, DownloadPostMessage.class, DownloadStatusMessage.class,
IdentityMessage.class, MaintainMessage.class, ResponseMessage.class, AvailableCheckMessage.class, IdentityMessage.class, MaintainMessage.class, ResponseMessage.class, AvailableCheckMessage.class,
// 协议 v2 的消息与任务操作实体:原生镜像下 Jackson/MyBatis 都要反射构造它们。
ProtocolHelloMessage.class, TaskCommandMessage.class, TaskReportMessage.class, TaskReportAckMessage.class,
TaskOperation.class,
SubscriptionSnapshotMessage.class, SubscriptionSnapshotPayload.class, SubscriptionAccountSnapshot.class, SubscriptionSnapshotMessage.class, SubscriptionSnapshotPayload.class, SubscriptionAccountSnapshot.class,
SubscriptionBindingSnapshot.class, LinkPreviewOptions.class}) SubscriptionBindingSnapshot.class, LinkPreviewOptions.class})
public class CustomBean { public class CustomBean {
@@ -0,0 +1,83 @@
package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Dao.cache.ImageCacheMapper;
import com.lion.lionwebsite.Domain.ImageKeyCache;
import com.lion.lionwebsite.Service.GalleryIndexService;
import com.lion.lionwebsite.Util.GalleryUtil;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import javax.sql.DataSource;
import java.util.List;
/**
* 图片索引服务的事务装配。
*
* <p>单独成文件是为了不触碰既有的 SqlConfiguration:那里的两个 SqlSessionFactory
* 保持原样,这里只额外声明缓存库的事务管理器与模板。
*
* <p>事务管理器必须绑定 datasource-cache —— 索引只写缓存库,
* 若误用主库事务管理器,整组替换就失去原子性(删除与写入落在不同连接上)。
*/
@Configuration
public class GalleryIndexConfiguration {
@Bean
public PlatformTransactionManager cacheTransactionManager(
@Qualifier("datasource-cache") DataSource cacheDataSource) {
return new DataSourceTransactionManager(cacheDataSource);
}
/** 只用于图片索引的短事务发布,超时留给上层请求超时控制,这里显式设为 30 秒。 */
@Bean
public TransactionTemplate cacheTransactionTemplate(
@Qualifier("cacheTransactionManager") PlatformTransactionManager cacheTransactionManager) {
TransactionTemplate template = new TransactionTemplate(cacheTransactionManager);
template.setTimeout(30);
return template;
}
/**
* 只读快照事务,用于一次性读取头部与页码。
*
* <p>读事务不写库,因此只读连接可以与写入并发;它的作用是让「读头部 + 读页码」
* 落在同一个快照上,避免恰好读到整组替换的中间状态。
*/
@Bean
public TransactionTemplate cacheReadTransactionTemplate(
@Qualifier("cacheTransactionManager") PlatformTransactionManager cacheTransactionManager) {
TransactionTemplate template = new TransactionTemplate(cacheTransactionManager);
template.setTimeout(10);
return template;
}
/**
* 这里显式不设置 {@code readOnly}:sqlite-jdbc 不允许在已建立的连接上改只读标志
* (实测抛 SQLException: Cannot change read-only flag after establishing a connection),
* 而事务模板拿到的是连接池里的既有连接,设置只读会让每次读取都直接失败。
* 快照一致性由「同一个事务」保证,不需要额外的只读标志。
*/
@Bean
public GalleryIndexService galleryIndexService(
ImageCacheMapper imageCacheMapper,
@Qualifier("cacheTransactionTemplate") TransactionTemplate cacheTransactionTemplate,
@Qualifier("cacheReadTransactionTemplate") TransactionTemplate cacheReadTransactionTemplate) {
return new GalleryIndexService(imageCacheMapper, cacheTransactionTemplate,
cacheReadTransactionTemplate,
GalleryIndexConfiguration::fetchPageKeys);
}
/**
* 生产环境的上游实现:沿用 GalleryUtil.parseImageKeys(HTTP + JS 解析)。
*
* <p>放在这里而不是服务内部,是为了让服务在测试中可以脱离静态工具类。
*/
static List<ImageKeyCache> fetchPageKeys(String url) throws Exception {
return GalleryUtil.parseImageKeys(url);
}
}
@@ -8,6 +8,11 @@ import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder; import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import com.zaxxer.hikari.HikariDataSource;
import javax.sql.DataSource; import javax.sql.DataSource;
@@ -16,7 +21,9 @@ public class SqlConfiguration {
@Bean(name = "datasource-main") @Bean(name = "datasource-main")
@ConfigurationProperties(prefix = "spring.datasource-main") @ConfigurationProperties(prefix = "spring.datasource-main")
public DataSource dataSourceMain() { public DataSource dataSourceMain() {
return DataSourceBuilder.create().build(); DataSource dataSource = DataSourceBuilder.create().build();
applyBusyTimeout(dataSource);
return dataSource;
} }
@Bean(name = "datasource-cache") @Bean(name = "datasource-cache")
@@ -25,6 +32,18 @@ public class SqlConfiguration {
return DataSourceBuilder.create().build(); return DataSourceBuilder.create().build();
} }
/**
* 给主库连接打开 busy 等待。
*
* <p>主库是单文件 SQLite,任务生命周期会并发开启短事务(创建、状态回执、删除)。
* 默认没有 busy 超时,第二个写事务会立刻拿到 SQLITE_BUSY 并被当作 500;
* 打开等待后,写冲突会按序排队重试而不是失败。
*/
private void applyBusyTimeout(DataSource dataSource) {
if (dataSource instanceof HikariDataSource hikari)
hikari.setConnectionInitSql("PRAGMA busy_timeout=5000");
}
@Bean(name = "sqlSessionFactory-main") @Bean(name = "sqlSessionFactory-main")
public SqlSessionFactory sqlSessionFactoryMain(@Qualifier("datasource-main") DataSource dataSource) throws Exception { public SqlSessionFactory sqlSessionFactoryMain(@Qualifier("datasource-main") DataSource dataSource) throws Exception {
SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean(); SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean();
@@ -48,4 +67,30 @@ public class SqlConfiguration {
public SqlSessionTemplate sqlSessionTemplateCache(@Qualifier("sqlSessionFactory-cache") SqlSessionFactory sqlSessionFactory) { public SqlSessionTemplate sqlSessionTemplateCache(@Qualifier("sqlSessionFactory-cache") SqlSessionFactory sqlSessionFactory) {
return new SqlSessionTemplate(sqlSessionFactory); return new SqlSessionTemplate(sqlSessionFactory);
} }
/**
* 主库事务管理器。
*
* <p>必须显式绑定到 datasource-main:应用里有两个 DataSource,Spring Boot 的自动配置
* 会退避,若没有这个 bean 就没有任何事务管理器,@Transactional 会静默失效。
* @Primary 同时让「不指定名字就注入 TransactionManager」的旧代码继续可用。
*/
@Bean(name = "mainTransactionManager")
@Primary
public DataSourceTransactionManager mainTransactionManager(@Qualifier("datasource-main") DataSource dataSource) {
return new DataSourceTransactionManager(dataSource);
}
/**
* 主库编程式事务模板。
*
* <p>任务生命周期需要把「短事务」放在网络调用之外,并且要求「回调里捕获异常后不再抛出」
* 不能错误提交,因此调用方必须用 {@code execute} 的返回状态或再次抛出决定回滚,
* 而不是在回调里吞掉异常(见 GalleryTaskLifecycleService)。
*/
@Bean(name = "mainTransactionTemplate")
@Primary
public TransactionTemplate mainTransactionTemplate(@Qualifier("mainTransactionManager") DataSourceTransactionManager transactionManager) {
return new TransactionTemplate(transactionManager);
}
} }
@@ -0,0 +1,57 @@
package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Domain.TaskOperation;
import com.lion.lionwebsite.Service.GalleryTaskLifecycleService;
import com.lion.lionwebsite.Service.RemoteService;
import com.lion.lionwebsite.Service.TaskReportHandler;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
/**
* 装配协议 v2 的双向依赖,并做启动恢复。
*
* <p>RemoteService 消费上报时需要生命周期服务,而生命周期服务下发命令时需要 RemoteService。
* 直接互相注入构造参数会形成环,所以这里在两端都构造完成之后用 setter 接上;
* 同时也把「启动时把中断的 SUBMITTING 判为 UNKNOWN、释放 RESERVED 占位」这一步接到应用就绪事件上。
*
* <p>顺序是不可交换的:
* <ol>
* <li>先绑定上报处理器、下发判定解析器与 operationId 解析器;</li>
* <li>再执行启动恢复(SUBMITTING → UNKNOWN、释放 RESERVED);</li>
* <li>最后才允许建立连接与自动重投。连接由本类显式触发,不再由 RemoteService 的
* PostConstruct 在装配前发起,否则会把状态不明的任务盲目重投给节点。</li>
* </ol>
* 恢复失败时仍然允许连接(保持节点在线/心跳),但明确禁止自动派发。
*/
@Component
@Slf4j
@RequiredArgsConstructor
public class TaskLifecycleWiring {
final RemoteService remoteService;
final GalleryTaskLifecycleService lifecycleService;
@EventListener(ApplicationReadyEvent.class)
public void wire() {
TaskReportHandler handler = lifecycleService;
remoteService.setTaskReportHandler(handler);
remoteService.setDownloadOperationIdResolver(lifecycleService::currentOperationId);
remoteService.setDispatchResolver(lifecycleService::resolveDispatch);
boolean recoverySafe = false;
try {
int recovered = lifecycleService.recoverInterruptedSubmissions();
recoverySafe = true;
if (recovered > 0)
log.warn("启动恢复:{} 个中断的创建/预占任务已处理(提交中的标为提交未知)", recovered);
} catch (RuntimeException e) {
// 恢复失败说明库中任务真实状态未知:可以保持连接,但绝不能自动重投/派发。
log.error("启动恢复失败,本次启动将拒绝自动重投未完成任务", e);
}
// 绑定与恢复完成之后才连接:连接成功后的自动重投受 lifecycleReady/recoverySafe 双重约束。
remoteService.startAfterWiring(recoverySafe);
}
}
@@ -62,16 +62,14 @@ public class PublicController {
public String validate(String AuthCode){ public String validate(String AuthCode){
Response response = Response.generateResponse(); Response response = Response.generateResponse();
User user = publicService.getUserId(AuthCode); User user = publicService.getUserId(AuthCode);
// 用 ObjectMapper 组装内层 JSON:手工 String.format 拼用户名时, // 身份字段直接作为结构化 data 返回;共享前端解码器兼容旧版本字符串。
// 名字里带引号或反斜杠会直接产出非法 JSON,前端 JSON.parse 随即失败。
// 这里仍以「JSON 文本」形式放进 data(历史契约,前端按字符串再解析一次)。
ObjectNode identity = CustomUtil.objectMapper.createObjectNode(); ObjectNode identity = CustomUtil.objectMapper.createObjectNode();
identity.put("userId", user.getId()); identity.put("userId", user.getId());
identity.put("username", user.getUsername()); identity.put("username", user.getUsername());
identity.put("isAvailable", !remoteService.isDead()); identity.put("isAvailable", !remoteService.isDead());
// 管理员标记随登录一起下发,前端据此决定是否显示下载人信息与筛选。 // 管理员标记随登录一起下发,前端据此决定是否显示下载人信息与筛选。
identity.put("isAdmin", user.getId() == UserService.ADMIN_USER_ID); identity.put("isAdmin", user.getId() == UserService.ADMIN_USER_ID);
response.success(identity.toString()); response.success(identity);
return response.toJSONString(); return response.toJSONString();
} }
@@ -0,0 +1,33 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.SubscriptionEventService;
import jakarta.servlet.http.HttpServletRequest;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;
import java.io.IOException;
import java.util.Map;
@RestController
public class SubscriptionEventController {
private final SubscriptionEventService events;
public SubscriptionEventController(SubscriptionEventService events) { this.events = events; }
@PostMapping(value = "/internal/subscription-events", produces = "application/json")
public ResponseEntity<Map<String, Object>> ingest(HttpServletRequest request) throws IOException {
// Bound actual bytes even for chunked requests and dishonest Content-Length headers.
byte[] body = request.getInputStream().readNBytes(SubscriptionEventService.MAX_BODY + 1);
if (body.length > SubscriptionEventService.MAX_BODY)
return ResponseEntity.badRequest().body(Map.of("error", "invalid subscription events"));
if (!events.verify(body, request.getHeader("X-Subscription-Timestamp"), request.getHeader("X-Subscription-Signature")))
return ResponseEntity.status(403).body(Map.of("error", "authentication failed"));
try {
return ResponseEntity.ok(events.ingest(events.parseBatch(body)));
} catch (IllegalArgumentException invalid) {
return ResponseEntity.badRequest().body(Map.of("error", "invalid subscription events"));
}
// All infrastructure failures propagate as 5xx; never acknowledge an uncommitted batch.
}
}
@@ -8,6 +8,8 @@ import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Delete; import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Select; import org.apache.ibatis.annotations.Select;
import java.util.List;
@Mapper @Mapper
public interface ImageCacheMapper { public interface ImageCacheMapper {
@Insert("insert into ImageKeyCache values (#{gid}, #{imgkey}, #{page})") @Insert("insert into ImageKeyCache values (#{gid}, #{imgkey}, #{page})")
@@ -31,4 +33,32 @@ public interface ImageCacheMapper {
@Delete("delete from gidToKey where gid=#{gid}") @Delete("delete from gidToKey where gid=#{gid}")
void deleteGidToKey(String gid); void deleteGidToKey(String gid);
/**
* 某个 gid 已落库的页码列表(升序)。
*
* <p>只按行数判断完整性会把「页码 1 和 3、共 2 行」当成完整索引,
* 因此完整性判定改为按页码集合比对——先取回页码,再校验它是否恰好是 1..pages。
*/
@Select("select page from ImageKeyCache where gid=#{gid} order by page")
List<Integer> selectPageNumbersByGid(String gid);
/** 该 gid 下 imgkey 为空的脏行数;任何一行脏行都说明索引不可用。 */
@Select("select count(*) from ImageKeyCache where gid=#{gid} and (imgkey is null or trim(imgkey)='')")
int countInvalidImageKeysByGid(String gid);
/**
* 批量写入页 key(单条多值 INSERT)。
*
* <p>整本画廊的索引可能上千页,逐行 INSERT 会拉长写事务并放大锁竞争;
* 调用方按固定大小分片后,一个写事务里只需少量语句即可写完。
*/
// MyBatis 只在注解值「以 <script> 开头」时才按 XML 动态 SQL 解析,因此这里不能有前导空白。
@Insert("<script>"
+ "insert into ImageKeyCache (gid, imgkey, page) values "
+ "<foreach collection='pages' item='p' separator=','>"
+ "(#{p.gid}, #{p.imgkey}, #{p.page})"
+ "</foreach>"
+ "</script>")
int insertImageKeyCacheBatch(@Param("pages") List<ImageKeyCache> pages);
} }
@@ -20,4 +20,8 @@ public interface CollectMapper {
@Select("select collector from collect where gid=#{gid}") @Select("select collector from collect where gid=#{gid}")
ArrayList<Integer> selectCollectorByGid(int gid); ArrayList<Integer> selectCollectorByGid(int gid);
/** 任务删除完成后清理该 gid 的全部收藏记录(与删任务行同一事务)。 */
@Delete("delete from collect where gid=#{gid}")
void deleteByGid(@Param("gid") int gid);
} }
@@ -36,9 +36,10 @@ public interface GalleryMapper {
@Update(""" @Update("""
update gallery set name=#{name}, link=#{link}, language=#{language}, update gallery set name=#{name}, link=#{link}, language=#{language}, downloader=#{downloader},
pages=#{pages}, status=#{status}, fileSize=#{fileSize}, createTime=#{createTime}, pages=#{pages}, status=#{status}, fileSize=#{fileSize}, createTime=#{createTime},
proceeding=#{proceeding}, resolution=#{resolution}, displayFileSize=#{displayFileSize}, is_download=#{is_download} proceeding=#{proceeding}, resolution=#{resolution}, displayFileSize=#{displayFileSize},
thumb_link=#{thumb_link}, is_download=#{is_download}
where gid=#{gid}""") where gid=#{gid}""")
void updateGallery(Gallery gallery); void updateGallery(Gallery gallery);
@@ -47,4 +48,8 @@ public interface GalleryMapper {
@Delete("delete from gallery where gid=#{gid}") @Delete("delete from gallery where gid=#{gid}")
void deleteGalleryByGid(Integer gid); void deleteGalleryByGid(Integer gid);
/** 只改状态列,用于「待删除」这类不牵动其他字段的意图标记。 */
@Update("update gallery set status=#{status} where gid=#{gid}")
int updateGalleryStatus(@Param("gid") int gid, @Param("status") String status);
} }
@@ -43,49 +43,57 @@ public interface SubMapper {
@Delete("delete from subscription_account where id=#{id}") @Delete("delete from subscription_account where id=#{id}")
void deleteSubscriptionAccount(Integer id); void deleteSubscriptionAccount(Integer id);
@Insert("insert into sub_bind (key, user, subscription_account_id) values (#{key}, #{user}, #{subscriptionAccountId})") @Insert("insert into sub_bind (key, user, subscription_account_id, user_id) values (#{key}, #{user}, #{subscriptionAccountId}, (select id from User where username=#{user}))")
void insertSubBind(SubBind subBind); void insertSubBind(SubBind subBind);
@Select("select sb.key, sb.user, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join subscription_account sa on sa.id=sb.subscription_account_id order by sb.user") @Select("select sb.key, coalesce(u.username, sb.user) as user, sb.user_id as userId, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join User u on u.id=sb.user_id left join subscription_account sa on sa.id=sb.subscription_account_id order by sb.user")
ArrayList<SubBind> selectAllSubBind(); ArrayList<SubBind> selectAllSubBind();
@Select("select sb.key, sb.user, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join subscription_account sa on sa.id=sb.subscription_account_id where sb.key=#{key}") @Select("select sb.key, coalesce(u.username, sb.user) as user, sb.user_id as userId, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join User u on u.id=sb.user_id left join subscription_account sa on sa.id=sb.subscription_account_id where sb.key=#{key}")
SubBind selectSubBind(String key); SubBind selectSubBind(String key);
@Select("select count(key) from sub_bind where key=#{key}") @Select("select count(key) from sub_bind where key=#{key}")
boolean selectSubBindExist(String key); boolean selectSubBindExist(String key);
@Select("select count(*) from sub_bind where user=#{user}") @Select("select count(*) from sub_bind where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
int countSubBindByUser(String user); int countSubBindByUser(String user);
@Update("update sub_bind set subscription_account_id=#{accountId} where user=#{user}") @Update("update sub_bind set subscription_account_id=#{accountId} where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
int updateSubBindAccount(@Param("user") String user, @Param("accountId") Integer accountId); int updateSubBindAccount(@Param("user") String user, @Param("accountId") Integer accountId);
@Update("update sub_bind set key=#{key} where user=#{user}") @Update("update sub_bind set key=#{key} where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
int updateSubBindKey(@Param("user") String user, @Param("key") String key); int updateSubBindKey(@Param("user") String user, @Param("key") String key);
@Delete("delete from sub_bind where user=#{user}") @Delete("delete from sub_bind where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
void deleteSubBind(String user); void deleteSubBind(String user);
@Insert("insert into sub_update_record (user, ip, UA, time, location) values (#{user}, #{ip}, #{UA}, #{time}, #{location})") @Insert("insert into sub_update_record (user, ip, UA, time, location, user_id) values (#{user}, #{ip}, #{UA}, #{time}, #{location}, coalesce(#{userId}, (select id from User where username=#{user})))")
@Options(useGeneratedKeys = true, keyProperty = "id")
void insertSubUpdateRecord(SubUpdateRecord subUpdateRecord); void insertSubUpdateRecord(SubUpdateRecord subUpdateRecord);
@Select("select * from sub_update_record order by time desc") @Select("select id,user,ip,UA,time,location,user_id as userId from sub_update_record order by time desc")
ArrayList<SubUpdateRecord> selectAllSubUpdateRecord(); ArrayList<SubUpdateRecord> selectAllSubUpdateRecord();
@Delete("delete from sub_update_record where user=#{user}") @Delete("delete from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
void deleteSubUpdateRecord(String user); void deleteSubUpdateRecord(String user);
@Select("select count(user) from sub_update_record where user=#{user}") @Select("select count(user) from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
Integer selectUpdateRecordCount(String user); Integer selectUpdateRecordCount(String user);
@Select("select min(id) from sub_update_record where user=#{user}") @Select("select id from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user}) order by time asc,id asc limit 1")
Integer selectMinUpdateRecordId(String user); Integer selectMinUpdateRecordId(String user);
/** 后台补齐归属地:只更新该用户最近一条记录,避免覆盖更早的历史记录。 */ /** 后台补齐归属地:只更新该用户最近一条记录,避免覆盖更早的历史记录。 */
@Update("update sub_update_record set location=#{location} where id=(select max(id) from sub_update_record where user=#{user})") @Update("update sub_update_record set location=#{location} where id=(select max(id) from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user}))")
void updateLatestSubUpdateRecordLocation(@Param("user") String user, @Param("location") String location); void updateLatestSubUpdateRecordLocation(@Param("user") String user, @Param("location") String location);
@Update("update sub_update_record set location=#{location} where id=#{id}")
void updateSubUpdateRecordLocation(@Param("id") int id, @Param("location") String location);
@Update("update sub_bind set user=#{newName} where user_id=#{userId} or (user_id is null and user=#{oldName})")
void renameBoundUser(@Param("userId") int userId, @Param("oldName") String oldName,
@Param("newName") String newName);
@Delete("delete from sub_update_record where id=#{id}") @Delete("delete from sub_update_record where id=#{id}")
void deleteSubUpdateRecordById(int id); void deleteSubUpdateRecordById(int id);
} }
@@ -0,0 +1,35 @@
package com.lion.lionwebsite.Dao.normal;
import com.lion.lionwebsite.Domain.SubscriptionEvent;
import com.lion.lionwebsite.Domain.SubUpdateRecord;
import com.lion.lionwebsite.Domain.SubscriptionEventBinding;
import org.apache.ibatis.annotations.*;
@Mapper
public interface SubscriptionEventMapper {
/** Reserve SQLite's writer before the receipt read; a deferred read-to-write upgrade can fail immediately. */
@Update("update subscription_event_receipt set ingested_at=ingested_at where 0")
void reserveWriter();
@Select("select fingerprint from subscription_event_receipt where event_id=#{eventId}")
String fingerprint(String eventId);
@Insert("insert into subscription_event_receipt(event_id,key_hash,client,occurred_at,source,ingested_at,fingerprint,outcome) values(#{event.eventId},#{event.keyHash},#{event.client},#{event.occurredAt},#{source},#{ingestedAt},#{fingerprint},#{outcome})")
void insertReceipt(@Param("event") SubscriptionEvent event, @Param("source") String source,
@Param("ingestedAt") long ingestedAt, @Param("fingerprint") String fingerprint,
@Param("outcome") String outcome);
@Select("select key_hash as keyHash,user_id as userId,user_name as userName from subscription_event_binding where key_hash=#{hash}")
SubscriptionEventBinding binding(String hash);
@Insert("insert into subscription_event_binding(key_hash,user_id,user_name,first_seen_at,last_seen_at) values(#{hash},#{userId},#{userName},#{now},#{now}) on conflict(key_hash) do update set user_name=excluded.user_name,last_seen_at=excluded.last_seen_at where subscription_event_binding.user_id IS excluded.user_id")
void rememberBinding(@Param("hash") String hash, @Param("userId") Integer userId,
@Param("userName") String userName, @Param("now") long now);
@Insert("insert into sub_update_record(user,ip,UA,time,location,user_id) values(#{user},#{ip},#{UA},#{time},#{location},#{userId})")
@Options(useGeneratedKeys = true, keyProperty = "id")
void insertRecord(SubUpdateRecord record);
@Delete("delete from sub_update_record where (user_id=#{userId} or (user_id is null and user=#{userName})) and id not in (select id from sub_update_record where user_id=#{userId} or (user_id is null and user=#{userName}) order by time desc,id desc limit 10)")
void pruneRecords(@Param("userId") Integer userId, @Param("userName") String userName);
}
@@ -0,0 +1,107 @@
package com.lion.lionwebsite.Dao.normal;
import com.lion.lionwebsite.Domain.TaskOperation;
import org.apache.ibatis.annotations.*;
/**
* 任务操作记录的持久化入口(主库)。
*
* <p>所有写操作都在调用方的事务里执行;这里的注解只描述 SQL,不包含业务判断。
* 主键为 gid,保证「同一 gid 只能有一条有效操作意图」。
*/
@Mapper
public interface TaskOperationRepository {
@Select("select * from task_operation where gid=#{gid}")
TaskOperation selectByGid(int gid);
/** 生成该 gid 的稳定下载 operationId:只有还没有记录时才写入。 */
@Insert("insert or ignore into task_operation (gid, operationId, deleteOperationId, state, lastRevision," +
" lastReportId, updateTime) values (#{gid}, #{operationId}, null, #{state}, 0, null, #{updateTime})")
int insertIfAbsent(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("state") String state, @Param("updateTime") long updateTime);
@Insert("insert into task_operation (gid, operationId, deleteOperationId, state, lastRevision, lastReportId, updateTime)" +
" values (#{gid}, #{operationId}, #{deleteOperationId}, #{state}, #{lastRevision}, #{lastReportId}, #{updateTime})")
void insert(TaskOperation operation);
/** 推进创建意图的状态(RESERVED → SUBMITTING → SUBMITTED/UNKNOWN)。 */
@Update("update task_operation set state=#{state}, updateTime=#{updateTime} where gid=#{gid}")
int updateState(@Param("gid") int gid, @Param("state") String state, @Param("updateTime") long updateTime);
/**
* 带前置条件的创建状态推进:只有当前状态与 operationId 都匹配时才写入。
*
* <p>这是「提交前先确认这一代意图仍然有效」的 CAS:若期间被删除/重建,
* 返回值 0,调用方必须停下,绝不能再产生上游副作用。
*/
@Update("update task_operation set state=#{state}, updateTime=#{updateTime}" +
" where gid=#{gid} and operationId=#{operationId} and state=#{expectedState}")
int compareAndSetCreateState(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("expectedState") String expectedState, @Param("state") String state,
@Param("updateTime") long updateTime);
/**
* 只按当前删除意图 ID 匹配的删除完成:期间若已被删除/重建,则不改动新世代记录。
*/
@Update("update task_operation set deleteOperationId=null, state=#{state}, updateTime=#{updateTime}" +
" where gid=#{gid} and deleteOperationId=#{deleteOperationId}")
int completeDeleteIfSameGeneration(@Param("gid") int gid, @Param("deleteOperationId") String deleteOperationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/**
* 删除「本次创建意图」的预占记录:必须同时匹配 gid 与 operationId,且状态仍是
* RESERVED/SUBMITTING。
*
* <p>三重匹配保证释放流程只能清掉自己那一代:期间被删除/重建(新 operationId 或墓碑)
* 的记录一律不动,避免把新世代抹掉。
*/
@Delete("delete from task_operation where gid=#{gid} and operationId=#{operationId}" +
" and state in ('RESERVED','SUBMITTING')")
int deleteReservationIfSameGeneration(@Param("gid") int gid, @Param("operationId") String operationId);
/** 记录/替换当前下载 operationId 并推进状态。 */
@Update("update task_operation set operationId=#{operationId}, state=#{state}, updateTime=#{updateTime} where gid=#{gid}")
int updateOperationId(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/**
* 给「有记录但缺 operationId」的历史任务补一个稳定 operationId。
* 只在确实缺失时写入,避免覆盖并发创建出来的新世代。
*/
@Update("update task_operation set operationId=#{operationId}, updateTime=#{updateTime}" +
" where gid=#{gid} and operationId is null")
int assignOperationIdIfMissing(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("updateTime") long updateTime);
/**
* 墓碑复活:删除完成后的 gid 允许用全新 operationId 重新创建。
* revision 与回执一并清零,旧操作的迟到上报会因 operationId 不匹配被丢弃。
*/
@Update("update task_operation set operationId=#{operationId}, deleteOperationId=null, state=#{state}," +
" lastRevision=0, lastReportId=null, updateTime=#{updateTime} where gid=#{gid}")
int reactivate(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/** 记录删除意图与其稳定的删除操作 ID(重试必须复用同一个 ID)。 */
@Update("update task_operation set deleteOperationId=#{deleteOperationId}, state=#{state}, updateTime=#{updateTime} where gid=#{gid}")
int updateDeleteIntent(@Param("gid") int gid, @Param("deleteOperationId") String deleteOperationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/**
* 写入状态上报回执(revision + reportId)。
*
* <p>回执与任务行更新在同一个事务里提交,提交成功之后才回 ACK;
* 因此节点重复投递同一份上报时,只能看到已经提交过的回执,不会重复落库或重复通知。
*/
@Update("update task_operation set lastRevision=#{revision}, lastReportId=#{reportId}, updateTime=#{updateTime}" +
" where gid=#{gid}")
int updateReportReceipt(@Param("gid") int gid, @Param("revision") long revision,
@Param("reportId") String reportId, @Param("updateTime") long updateTime);
@Select("select * from task_operation where state=#{state}")
java.util.List<TaskOperation> selectByState(String state);
@Delete("delete from task_operation where gid=#{gid}")
int deleteByGid(int gid);
}
@@ -14,4 +14,10 @@ public class SubBind {
String subscriptionAccountName; String subscriptionAccountName;
boolean subscriptionAccountEnabled; boolean subscriptionAccountEnabled;
boolean filterHighMultiplier; boolean filterHighMultiplier;
Integer userId;
public SubBind(String key, String user, Integer accountId, String accountName,
boolean accountEnabled, boolean filterHighMultiplier) {
this(key, user, accountId, accountName, accountEnabled, filterHighMultiplier, null);
}
} }
@@ -16,4 +16,9 @@ public class SubUpdateRecord {
String UA; String UA;
Date time; Date time;
String location; String location;
Integer userId;
public SubUpdateRecord(int id, String user, String ip, String UA, Date time, String location) {
this(id, user, ip, UA, time, location, null);
}
} }
@@ -0,0 +1,16 @@
package com.lion.lionwebsite.Domain;
import lombok.AllArgsConstructor;
import lombok.Data;
/** Validated, authenticated event. Never contains the public subscription key. */
@Data
@AllArgsConstructor
public class SubscriptionEvent {
private String eventId;
private String keyHash;
private String client;
private String clientIp;
private String userAgent;
private long occurredAt;
}
@@ -0,0 +1,10 @@
package com.lion.lionwebsite.Domain;
import lombok.Data;
@Data
public class SubscriptionEventBinding {
private String keyHash;
private Integer userId;
private String userName;
}
@@ -0,0 +1,53 @@
package com.lion.lionwebsite.Domain;
import lombok.Data;
/**
* 单个 gid 的下载生命周期操作记录(协议 v2 的持久化意图)。
*
* <p>这张表是「先持久化意图、再产生外部副作用」的落点:
* <ul>
* <li>CREATE 意图先于上游 POST 落库,用来保证同一 gid 只有一个未完成意图;</li>
* <li>提交结果不确定时只能停在 {@link #STATE_UNKNOWN},绝不自动重复上游 POST;</li>
* <li>DELETE 意图先于节点删除落库,节点确认后(或文件已不存在)才在同一事务里删任务行并留墓碑。</li>
* </ul>
*
* <p>旧任务行没有对应操作记录,查询时按 gid 惰性补建(见 GalleryTaskLifecycleService)。
*/
@Data
public class TaskOperation {
/** 已登记创建意图,尚未向上游提交。 */
public static final String STATE_RESERVED = "RESERVED";
/** 正在向上游提交;此时不可再发第二次 POST。 */
public static final String STATE_SUBMITTING = "SUBMITTING";
/** 上游确认已接受下载请求。 */
public static final String STATE_SUBMITTED = "SUBMITTED";
/**
* 提交结果未知:超时、断连或在 SUBMITTING 期间重启。
* 上游不支持幂等,因此这里只能等对账(reconcile),不能自动重发。
*/
public static final String STATE_UNKNOWN = "UNKNOWN";
/** 已登记删除意图,等待节点确认。 */
public static final String STATE_DELETE_PENDING = "DELETE_PENDING";
/** 节点已确认删除,任务行也随之删除;本行作为墓碑保留,用于停止迟到状态的重放。 */
public static final String STATE_DELETED = "DELETED";
private int gid;
/** 当前有效的下载操作 ID;同一 gid 的 DOWNLOAD 重试必须复用它。 */
private String operationId;
/** 独立于下载操作 ID 的删除操作 ID,删除重试复用它。 */
private String deleteOperationId;
private String state;
/** 最近一次应用的状态上报 revision;用于丢弃乱序/重放的旧上报。 */
private long lastRevision;
/** 节点已确认收到的最新上报 ID;用于重复 ACK 的幂等判断。 */
private String lastReportId;
private long updateTime;
}
@@ -0,0 +1,44 @@
package com.lion.lionwebsite.Exception;
/**
* 图片索引构建失败。
*
* <p>调用方需要区分三类结果,才能给出正确的用户可见响应:
* 链接本身有问题({@link Reason#INVALID_LINK},属于请求错误)、
* 画廊已不存在({@link Reason#UNAVAILABLE},重试无意义)、
* 上游或数据库暂时不可用({@link Reason#TRANSIENT},可以重试)。
*/
public class GalleryIndexException extends Exception {
public enum Reason {
/** 链接无法解析出 gid 或 key。 */
INVALID_LINK,
/** 上游明确表示画廊不存在(下架/删除/解析不到页 key)。 */
UNAVAILABLE,
/** 上游返回的页 key 无法自洽(gid 不匹配、页码重复/跳号、imgkey 为空)。 */
MALFORMED,
/** 上游波动或索引写入失败,重试可能成功。 */
TRANSIENT
}
private final Reason reason;
public GalleryIndexException(Reason reason, String message) {
super(message);
this.reason = reason;
}
public GalleryIndexException(Reason reason, String message, Throwable cause) {
super(message, cause);
this.reason = reason;
}
public Reason getReason() {
return reason;
}
/** 重试没有意义(链接错误或画廊不可用);上游数据畸形与会话级故障仍可重试。 */
public boolean isPermanent() {
return reason == Reason.INVALID_LINK || reason == Reason.UNAVAILABLE;
}
}
@@ -0,0 +1,319 @@
package com.lion.lionwebsite.Maintenance;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* 缓存库图片索引的离线体检与修复工具。
*
* <p>它是独立可执行类(只用 JDBC,不依赖 Spring / MyBatis),因此可以在服务停止后
* 直接对数据库文件运行,也可以先对生产库的副本演练。</p>
*
* <pre>
* java -cp sqlite-jdbc.jar GalleryIndexMaintenance.java --db /path/cache.db
* java -cp sqlite-jdbc.jar GalleryIndexMaintenance.java --db /path/cache.db --apply
* </pre>
*
* <p>它只做两类写入:删除不完整或有冲突的 gid 整组索引(gidToKey 头部加该 gid 的全部
* 页 key),以及删除没有头部记录的孤儿页 key。被删掉的 gid 由下一次在线阅读重新抓取,
* 因此不需要猜测保留哪一组 key。工具不建表、不改列、不碰其它业务表。</p>
*/
public final class GalleryIndexMaintenance {
/** 生产运行时数据库所在目录;默认拒绝直接写入,必须显式加 --allow-production。 */
private static final String PRODUCTION_DIR = "/home/lionwebsite";
private static final String PLAIN_REPAIR_NOTE =
"这些画廊将在下次被访问时重新抓取。";
public static void main(String[] args) {
Map<String, String> options = parseOptions(args);
if (options.containsKey("help") || !options.containsKey("db")) {
printUsage();
System.exit(options.containsKey("help") ? 0 : 2);
return;
}
Path database = Path.of(options.get("db")).toAbsolutePath().normalize();
boolean apply = options.containsKey("apply");
if (!Files.isRegularFile(database)) {
System.err.println("数据库不存在: " + database);
System.exit(2);
return;
}
if (apply && isProductionPath(database) && !options.containsKey("allow-production")) {
System.err.println("拒绝写入生产库路径 " + database
+ "。请先在副本上演练;确需在本机生产库执行时显式加 --allow-production,"
+ "并确保服务已停止。");
System.exit(3);
return;
}
try (Connection connection = open(database)) {
Report before = inspect(connection);
System.out.println(before.render(apply ? "修复前" : "体检结果"));
if (!apply) {
System.out.println();
System.out.println("dry-run:未做任何写入。确认无误后加 --apply 执行修复。");
System.exit(before.isHealthy() ? 0 : 1);
return;
}
if (before.isHealthy()) {
System.out.println();
System.out.println("索引已健康,无需修复。");
return;
}
connection.setAutoCommit(false);
int removed = repair(connection, before.badGids, before.orphanGids);
connection.commit();
System.out.println();
System.out.println("已删除 " + removed + " 个 gid 的索引整组," + PLAIN_REPAIR_NOTE);
Report after = inspect(connection);
System.out.println();
System.out.println(after.render("修复后"));
if (!after.isHealthy()) {
System.err.println("修复后仍未达到健康状态,请人工检查。");
System.exit(4);
}
} catch (SQLException e) {
System.err.println("执行失败: " + e.getMessage());
e.printStackTrace(System.err);
System.exit(5);
}
}
private static Connection open(Path database) throws SQLException {
Connection connection = DriverManager.getConnection("jdbc:sqlite:" + database);
try (Statement statement = connection.createStatement()) {
statement.execute("pragma busy_timeout=5000");
statement.execute("pragma foreign_keys=off");
}
return connection;
}
/** 是否位于生产运行时目录;工具据此拒绝在未显式确认时写入生产库。 */
static boolean isProductionPath(Path database) {
return database.startsWith(Path.of(PRODUCTION_DIR).toAbsolutePath().normalize());
}
// ---------- 体检 ----------
static Report inspect(Connection connection) throws SQLException {
Report report = new Report();
report.hasHeaderTable = tableExists(connection, "gidToKey");
report.hasPageTable = tableExists(connection, "ImageKeyCache");
if (!report.hasHeaderTable || !report.hasPageTable)
return report;
report.headerRows = count(connection, "select count(*) from gidToKey");
report.pageRows = count(connection, "select count(*) from ImageKeyCache");
report.duplicateHeaderGids = count(connection,
"select count(*) from (select gid from gidToKey group by gid having count(*) > 1)");
report.duplicatePageGroups = count(connection,
"select count(*) from (select gid, page from ImageKeyCache group by gid, page having count(*) > 1)");
try (PreparedStatement statement = connection.prepareStatement(BAD_GID_QUERY);
ResultSet rows = statement.executeQuery()) {
while (rows.next()) {
String gid = rows.getString(1);
report.badGids.add(gid);
report.badReasons.put(gid, describe(rows));
}
}
try (PreparedStatement statement = connection.prepareStatement(
"select distinct gid from ImageKeyCache where gid not in (select gid from gidToKey)");
ResultSet rows = statement.executeQuery()) {
while (rows.next())
report.orphanGids.add(rows.getString(1));
}
return report;
}
/**
* 一个 gid 触发的全部不健康条件集中在一句里检查,避免多次全表扫描。
*
* <p>必须按 gid 分组后再判断:重复 gid(同一 gid 有多条头部记录)本身就是要清除的对象,
* 而重复行里的某一条可能恰好与页 key 数一致,逐行判断会把它当成健康记录漏掉。
*/
private static final String BAD_GID_QUERY = """
select g.gid as gid,
count(*) as header_rows,
max(g.pages) as pages,
min(g.key) as key,
(select count(*) from ImageKeyCache i where i.gid = g.gid) as page_rows,
(select count(distinct i.page) from ImageKeyCache i where i.gid = g.gid) as distinct_pages,
(select coalesce(min(i.page), 0) from ImageKeyCache i where i.gid = g.gid) as min_page,
(select coalesce(max(i.page), 0) from ImageKeyCache i where i.gid = g.gid) as max_page,
(select count(*) from ImageKeyCache i where i.gid = g.gid
and (i.imgkey is null or trim(i.imgkey) = '')) as dirty_keys
from gidToKey g
group by g.gid
having count(*) > 1
or max(g.pages) is null or max(g.pages) <= 0
or min(g.key) is null or trim(min(g.key)) = ''
or (select count(*) from ImageKeyCache i where i.gid = g.gid) <> max(g.pages)
or (select count(distinct i.page) from ImageKeyCache i where i.gid = g.gid) <> max(g.pages)
or (select coalesce(min(i.page), 0) from ImageKeyCache i where i.gid = g.gid) <> 1
or (select coalesce(max(i.page), 0) from ImageKeyCache i where i.gid = g.gid) <> max(g.pages)
or (select count(*) from ImageKeyCache i where i.gid = g.gid
and (i.imgkey is null or trim(i.imgkey) = '')) > 0
order by g.gid
""";
private static String describe(ResultSet row) throws SQLException {
List<String> reasons = new ArrayList<>();
int headerRows = row.getInt("header_rows");
int pages = row.getInt("pages");
String key = row.getString("key");
int pageRows = row.getInt("page_rows");
int distinctPages = row.getInt("distinct_pages");
int minPage = row.getInt("min_page");
int maxPage = row.getInt("max_page");
int dirtyKeys = row.getInt("dirty_keys");
if (headerRows > 1) reasons.add("重复头部记录 " + headerRows + " 条");
if (key == null || key.isBlank()) reasons.add("key 为空");
if (pages <= 0) reasons.add("pages 非法(" + pages + ")");
if (pageRows != pages) reasons.add("页 key 行数 " + pageRows + " 与 pages " + pages + " 不符");
if (distinctPages != pages) reasons.add("页码去重后 " + distinctPages + " 个");
if (minPage != 1 || maxPage != pages) reasons.add("页码区间 " + minPage + "-" + maxPage);
if (dirtyKeys > 0) reasons.add("imgkey 为空 " + dirtyKeys + " 行");
return String.join("、", reasons);
}
// ---------- 修复 ----------
/** 删除不健康 gid 的整组索引与孤儿页 key,返回被移除索引的 gid 个数。 */
static int repair(Connection connection, List<String> badGids, List<String> orphanGids)
throws SQLException {
int touched = 0;
try (PreparedStatement pages = connection.prepareStatement("delete from ImageKeyCache where gid = ?");
PreparedStatement header = connection.prepareStatement("delete from gidToKey where gid = ?")) {
for (String gid : badGids) {
pages.setString(1, gid);
pages.executeUpdate();
header.setString(1, gid);
header.executeUpdate();
touched++;
}
for (String gid : orphanGids) {
pages.setString(1, gid);
if (pages.executeUpdate() > 0)
touched++;
}
}
return touched;
}
// ---------- 基础设施 ----------
private static boolean tableExists(Connection connection, String table) throws SQLException {
try (PreparedStatement statement = connection.prepareStatement(
"select count(*) from sqlite_master where type='table' and name=?")) {
statement.setString(1, table);
try (ResultSet rows = statement.executeQuery()) {
return rows.next() && rows.getInt(1) > 0;
}
}
}
private static int count(Connection connection, String sql) throws SQLException {
try (Statement statement = connection.createStatement();
ResultSet rows = statement.executeQuery(sql)) {
return rows.next() ? rows.getInt(1) : 0;
}
}
private static Map<String, String> parseOptions(String[] args) {
Map<String, String> options = new LinkedHashMap<>();
for (int i = 0; i < args.length; i++) {
String arg = args[i];
if (!arg.startsWith("--"))
continue;
String name = arg.substring(2);
if (name.equals("apply") || name.equals("help") || name.equals("allow-production")) {
options.put(name, "true");
continue;
}
if (i + 1 >= args.length) {
System.err.println("缺少参数值: " + arg);
System.exit(2);
}
options.put(name, args[++i]);
}
return options;
}
private static void printUsage() {
System.out.println("""
用法: java -cp <sqlite-jdbc.jar> GalleryIndexMaintenance.java --db <cache.db> [--apply] [--allow-production]
--db <path> 必填,缓存库文件
--apply 执行修复;不加则只做 dry-run 体检
--allow-production 允许写入 /home/lionwebsite 下的数据库(默认拒绝)
退出码: 0 健康或修复成功 / 1 存在待修复项(dry-run) / 2 参数错误 /
3 拒绝写生产路径 / 4 修复后仍不健康 / 5 执行异常
""");
}
private GalleryIndexMaintenance() { }
/** 一次体检结果。 */
static final class Report {
boolean hasHeaderTable;
boolean hasPageTable;
int headerRows;
int pageRows;
int duplicateHeaderGids;
int duplicatePageGroups;
final List<String> badGids = new ArrayList<>();
final List<String> orphanGids = new ArrayList<>();
final Map<String, String> badReasons = new LinkedHashMap<>();
boolean isHealthy() {
return hasHeaderTable && hasPageTable && badGids.isEmpty() && orphanGids.isEmpty();
}
String render(String title) {
StringBuilder out = new StringBuilder();
out.append("=== 图片索引").append(title).append("(").append(Instant.now()).append(")===\n");
if (!hasHeaderTable || !hasPageTable) {
out.append("缺少缓存表: gidToKey=").append(hasHeaderTable)
.append(" ImageKeyCache=").append(hasPageTable).append('\n');
return out.toString();
}
out.append("gidToKey 行数: ").append(headerRows).append('\n');
out.append("ImageKeyCache 行数: ").append(pageRows).append('\n');
out.append("重复 gid 组: ").append(duplicateHeaderGids).append('\n');
out.append("重复 (gid,page) 组: ").append(duplicatePageGroups).append('\n');
out.append("不完整或冲突 gid: ").append(badGids.size()).append('\n');
int limit = 20;
for (int i = 0; i < badGids.size() && i < limit; i++) {
String gid = badGids.get(i);
out.append(" - gid=").append(gid).append(' ')
.append(badReasons.getOrDefault(gid, "")).append('\n');
}
if (badGids.size() > limit)
out.append(" ... 其余 ").append(badGids.size() - limit).append(" 个略\n");
out.append("孤儿页 key 的 gid: ").append(orphanGids.size()).append('\n');
out.append(isHealthy() ? "结论: 健康\n" : "结论: 需要修复\n");
return out.toString();
}
}
}
@@ -21,6 +21,20 @@ public class AbstractMessage {
public static final byte SUBSCRIPTION_SNAPSHOT_MESSAGE = 9; public static final byte SUBSCRIPTION_SNAPSHOT_MESSAGE = 9;
// ---- 协议 v2(可靠任务)。旧类型 0..9 的字段保持不变,不能新增字段。 ----
/** 能力协商:{@code {protocolVersion: 2, features: ["reliable-task-v2"]}}。 */
public static final byte PROTOCOL_HELLO_MESSAGE = 10;
/** 带稳定 operationId 的任务命令(DOWNLOAD/DELETE)。 */
public static final byte TASK_COMMAND_MESSAGE = 11;
/** 带 reportId + revision 的状态上报,替代旧的无确认上报。 */
public static final byte TASK_REPORT_MESSAGE = 12;
/** 主站提交事务成功后的上报回执,节点据此清理快照。 */
public static final byte TASK_REPORT_ACK_MESSAGE = 13;
public byte messageType; public byte messageType;
public int messageId; public int messageId;
@@ -45,6 +45,10 @@ public class MessageCodec extends ByteToMessageCodec<AbstractMessage> {
case AbstractMessage.MAINTAIN_MESSAGE -> objectMapper.readValue(metadata, MaintainMessage.class); case AbstractMessage.MAINTAIN_MESSAGE -> objectMapper.readValue(metadata, MaintainMessage.class);
case AbstractMessage.AVAILABLE_CHECK_MESSAGE -> objectMapper.readValue(metadata, AvailableCheckMessage.class); case AbstractMessage.AVAILABLE_CHECK_MESSAGE -> objectMapper.readValue(metadata, AvailableCheckMessage.class);
case AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE -> objectMapper.readValue(metadata, SubscriptionSnapshotMessage.class); case AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE -> objectMapper.readValue(metadata, SubscriptionSnapshotMessage.class);
case AbstractMessage.PROTOCOL_HELLO_MESSAGE -> objectMapper.readValue(metadata, ProtocolHelloMessage.class);
case AbstractMessage.TASK_COMMAND_MESSAGE -> objectMapper.readValue(metadata, TaskCommandMessage.class);
case AbstractMessage.TASK_REPORT_MESSAGE -> objectMapper.readValue(metadata, TaskReportMessage.class);
case AbstractMessage.TASK_REPORT_ACK_MESSAGE -> objectMapper.readValue(metadata, TaskReportAckMessage.class);
default -> null; default -> null;
}; };
@@ -0,0 +1,30 @@
package com.lion.lionwebsite.Message;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.ArrayList;
import java.util.List;
/** 协议 v2 能力协商(类型 10)。双方都以对方声明了 version>=2 为前提才启用新消息。 */
@Data
@NoArgsConstructor
public class ProtocolHelloMessage extends AbstractMessage {
{
messageType = PROTOCOL_HELLO_MESSAGE;
}
public static final int PROTOCOL_V2 = 2;
public static final String FEATURE_RELIABLE_TASK_V2 = "reliable-task-v2";
private int protocolVersion = PROTOCOL_V2;
private List<String> features = new ArrayList<>(List.of(FEATURE_RELIABLE_TASK_V2));
/** 对端是否声明了 v2 可靠任务能力。 */
public boolean supportsReliableTasks() {
return protocolVersion >= PROTOCOL_V2
&& features != null && features.contains(FEATURE_RELIABLE_TASK_V2);
}
}
@@ -0,0 +1,29 @@
package com.lion.lionwebsite.Message;
import lombok.Data;
/**
* 任务命令(类型 11)。
*
* <p>只有确认对端支持 v2 之后才发送。同一逻辑操作在重试/重连后必须复用同一个
* {@link #operationId},节点据此返回原结果而不重复执行副作用。
*/
@Data
public class TaskCommandMessage extends AbstractMessage {
{
messageType = TASK_COMMAND_MESSAGE;
}
public static final String ACTION_DOWNLOAD = "DOWNLOAD";
public static final String ACTION_DELETE = "DELETE";
/** 稳定操作 ID:DOWNLOAD 与 DELETE 各自独立。 */
private String operationId;
private String action;
private int gid;
private String name;
}
@@ -0,0 +1,29 @@
package com.lion.lionwebsite.Message;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* 上报回执(类型 13)。
*
* <p>只在任务行与回执已经提交之后才发送;重复上报(同一 reportId)也会得到回执,
* 让节点停止重放。{@link #result} 0 表示已落库。
*/
@Data
@NoArgsConstructor
public class TaskReportAckMessage extends AbstractMessage {
{
messageType = TASK_REPORT_ACK_MESSAGE;
}
public static final byte RESULT_SUCCESS = 0;
private String reportId;
private byte result;
public TaskReportAckMessage(String reportId, byte result) {
this.reportId = reportId;
this.result = result;
}
}
@@ -0,0 +1,32 @@
package com.lion.lionwebsite.Message;
import com.lion.lionwebsite.Domain.GalleryTask;
import lombok.Data;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* 节点状态上报(类型 12)。
*
* <p>相比旧的无确认上报,这里带上单调递增的 {@link #revision} 与唯一 {@link #reportId}:
* 主站据此丢弃乱序/重放的旧快照,并在事务提交后回 {@link TaskReportAckMessage},
* 节点只有在收到匹配回执后才清理自己的快照。
*/
@Data
public class TaskReportMessage extends AbstractMessage {
{
messageType = TASK_REPORT_MESSAGE;
}
/** 本次上报的唯一 ID,重复投递同一份快照时保持不变。 */
private String reportId;
/** 节点侧单调递增版本号;主站只接受比已记录更大的 revision。 */
private long revision;
private GalleryTask[] galleryTasks;
/** gid → 产生该状态的下载 operationId,用于丢弃属于旧操作的迟到上报。 */
private Map<Integer, String> operationIds = new LinkedHashMap<>();
}
@@ -0,0 +1,262 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.cache.ImageCacheMapper;
import com.lion.lionwebsite.Domain.GidToKey;
import com.lion.lionwebsite.Domain.ImageKeyCache;
import com.lion.lionwebsite.Exception.GalleryIndexException;
import com.lion.lionwebsite.Util.GalleryUtil;
import com.lion.lionwebsite.Util.SingleFlight;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.transaction.support.TransactionOperations;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
/**
* 图片索引(gid 到每页 imgkey)的唯一读写入口。
*
* <p>这个类替换掉原先「读缓存 / 写缓存 / 回滚」分散在业务服务里、并且逐页 INSERT
* 的写法。三件事在这里被固定下来:
*
* <ol>
* <li><b>同 gid 合并</b>:并发请求同一本画廊时只有一个线程真的去抓上游,
* 其余线程等待同一个结果(含失败),因此不会出现两个构建者互相删除对方写入的索引。</li>
* <li><b>抓取在事务外</b>:上游 HTTP 可能耗时十几秒,绝不能占着 SQLite 写事务,
* 否则整个缓存库的写入都会排队。</li>
* <li><b>整组短事务发布</b>:一个事务里先删该 gid 的旧索引,再批量写入新索引。
* 失败整体回滚,磁盘上仍保留旧的完整索引;只有写入成功这一个状态是对外可见的。</li>
* </ol>
*
* <p>依赖全部由构造函数注入:mapper 负责落库,operations 负责事务边界,
* upstream 负责把 mpv 页解析成页 key 列表。测试可以直接给假的实现,
* 不需要起 Spring 上下文,也不需要网络。
*/
public class GalleryIndexService {
private static final Logger log = LoggerFactory.getLogger(GalleryIndexService.class);
/** 单条 INSERT 携带的最大行数,避免 SQL 过长。 */
static final int BATCH_SIZE = 200;
private final ImageCacheMapper mapper;
private final TransactionOperations operations;
private final TransactionOperations readOperations;
private final GalleryKeyUpstream upstream;
private final SingleFlight<String, GidToKey> builds = new SingleFlight<>();
public GalleryIndexService(ImageCacheMapper mapper, TransactionOperations operations,
GalleryKeyUpstream upstream) {
this(mapper, operations, operations, upstream);
}
/**
* @param operations 发布索引的写事务(删除加批量写入必须整体生效)
* @param readOperations 读取索引的一致性快照事务;传 null 表示不额外开启读事务
*/
public GalleryIndexService(ImageCacheMapper mapper, TransactionOperations operations,
TransactionOperations readOperations, GalleryKeyUpstream upstream) {
if (mapper == null || operations == null || upstream == null)
throw new IllegalArgumentException("mapper/operations/upstream 不能为空");
this.mapper = mapper;
this.operations = operations;
this.readOperations = readOperations;
this.upstream = upstream;
}
/**
* 保证该画廊的索引可用,返回 gid 到 key 的映射。
*
* @param url 画廊链接,形如 https://exhentai.org/g/1596929/f08534d87d/
* @throws GalleryIndexException 链接错误 / 画廊不可用 / 暂时失败,见 GalleryIndexException.Reason
*/
public GidToKey ensureIndex(String url) throws GalleryIndexException {
return ensureIndex(url, this::buildAndPublish);
}
/**
* 同 ensureIndex,但允许调用方提供构建动作。
*
* <p>离线重建需要「先检查再决定是否重建」的不同策略,因此把构建步骤作为参数暴露出来;
* 生产调用使用默认实现即可。
*/
GidToKey ensureIndex(String url, IndexBuilder builder) throws GalleryIndexException {
GalleryLink link = GalleryLink.parse(url);
GidToKey cached = lookup(link);
if (cached != null)
return cached;
try {
return builds.run(link.gid(), () -> {
// 进入临界区后必须再看一眼:等待期间别的线程可能已经把索引建好了。
GidToKey existing = lookup(link);
return existing != null ? existing : builder.build(url);
});
} catch (GalleryIndexException e) {
throw e;
} catch (Exception e) {
throw new GalleryIndexException(GalleryIndexException.Reason.TRANSIENT,
"构建图片索引失败 gid=" + link.gid(), e);
}
}
/**
* 已缓存且完整则返回索引,否则返回 null。
*
* <p>完整性由页码集合判定,而不是行数:历史重复行与半截写入都会让行数与页数不符。
*/
GidToKey lookup(GalleryLink link) {
if (readOperations != null)
return readOperations.execute(status -> lookupInTransaction(link));
return lookupInTransaction(link);
}
/**
* 在同一个读事务里读头部与页码,得到一个一致的快照。
*
* <p>两个独立的 SELECT 之间可能夹着另一个线程的整组替换,于是头部是旧的、页码是新的
* (或相反),完整性判断会得出错误结论。SQLite 的读事务能看到固定快照,避免这种交错。
*/
private GidToKey lookupInTransaction(GalleryLink link) {
GidToKey header = mapper.selectKeyByGid(link.gid());
if (header == null)
return null;
if (header.getPages() <= 0 || header.getKey() == null || header.getKey().isBlank())
return null;
if (!isComplete(header))
return null;
return header;
}
private boolean isComplete(GidToKey header) {
List<Integer> pages = mapper.selectPageNumbersByGid(header.getGid());
if (pages == null || pages.size() != header.getPages())
return false;
if (mapper.countInvalidImageKeysByGid(header.getGid()) != 0)
return false;
// 页码必须是严格递增的 1..pages,重复或跳号都视为不完整。
for (int i = 0; i < pages.size(); i++) {
Integer page = pages.get(i);
if (page == null || page != i + 1)
return false;
}
return true;
}
/** 抓上游(事务外)后短事务整组替换。 */
private GidToKey buildAndPublish(String url) throws GalleryIndexException {
GalleryLink link = GalleryLink.parse(url);
List<ImageKeyCache> pageKeys;
try {
// 抓取必须发生在事务之外:上游可能耗时十几秒,占着 SQLite 写事务会拖垮整个缓存库。
pageKeys = upstream.fetchPageKeys(url);
} catch (Exception e) {
throw new GalleryIndexException(GalleryIndexException.Reason.TRANSIENT,
"抓取图片索引失败 gid=" + link.gid(), e);
}
if (pageKeys == null || pageKeys.isEmpty())
throw new GalleryIndexException(GalleryIndexException.Reason.UNAVAILABLE,
"该图片已下架或已被删除 gid=" + link.gid());
// 落库前先校验上游数据自洽:整组替换一旦开始就没有回头路,
// 与其把畸形页码写进唯一约束再回滚,不如在此之前就判为失败。
validatePageKeys(link, pageKeys);
GidToKey header = new GidToKey();
header.setGid(link.gid());
header.setKey(link.key());
header.setPages(pageKeys.size());
publish(header, pageKeys);
return header;
}
/**
* 校验上游返回的页 key 是否自洽。
*
* <p>四条要求缺一不可,否则索引会以「看起来完整」的样子写进库:
* gid 属于请求的那本画廊、页码恰好是 1..N、页码不重复、imgkey 非空。
* 任一不满足都抛出 {@link GalleryIndexException.Reason#MALFORMED},此时尚未开启写事务,
* 库中旧索引保持原样。
*/
void validatePageKeys(GalleryLink link, List<ImageKeyCache> pageKeys) throws GalleryIndexException {
int expected = pageKeys.size();
Set<Integer> seen = new HashSet<>(expected * 2);
for (ImageKeyCache row : pageKeys) {
if (row == null || row.getGid() == null || !row.getGid().equals(link.gid()))
throw malformed(link, "上游页 key 的 gid 与请求不一致");
if (row.getImgkey() == null || row.getImgkey().isBlank())
throw malformed(link, "上游页 key 的 imgkey 为空");
int page = row.getPage();
if (page < 1 || page > expected)
throw malformed(link, "上游页码越界: " + page + "(共 " + expected + " 页)");
if (!seen.add(page))
throw malformed(link, "上游页码重复: " + page);
}
if (seen.size() != expected)
throw malformed(link, "上游页码不连续,去重后 " + seen.size() + " 个,期望 " + expected + " 个");
}
private GalleryIndexException malformed(GalleryLink link, String detail) {
log.warn("上游图片索引非法 gid={} detail={}", link.gid(), detail);
return new GalleryIndexException(GalleryIndexException.Reason.MALFORMED,
detail + " gid=" + link.gid());
}
/**
* 在单个事务里整组替换索引。
*
* <p>失败时事务整体回滚:要么看到旧的完整索引,要么看到新的完整索引,
* 不存在「头部已更新、页 key 缺失」的中间态。
*/
void publish(GidToKey header, List<ImageKeyCache> pageKeys) {
operations.executeWithoutResult(status -> {
mapper.deleteImageKeyCacheByGid(header.getGid());
mapper.deleteGidToKey(header.getGid());
mapper.insertGidToKey(header);
writeBatches(pageKeys);
});
log.info("图片索引已整体替换 gid={} pages={}", header.getGid(), header.getPages());
}
private void writeBatches(List<ImageKeyCache> pageKeys) {
for (int start = 0; start < pageKeys.size(); start += BATCH_SIZE) {
int end = Math.min(start + BATCH_SIZE, pageKeys.size());
mapper.insertImageKeyCacheBatch(new ArrayList<>(pageKeys.subList(start, end)));
}
}
/** 便于日志与错误信息使用的解析结果。 */
record GalleryLink(String gid, String key) {
static GalleryLink parse(String url) throws GalleryIndexException {
String[] segments = url == null ? null : url.split("/");
Integer gid = GalleryUtil.parseGid(url);
if (gid == null || segments == null || segments.length <= 5 || segments[5].isBlank())
throw new GalleryIndexException(GalleryIndexException.Reason.INVALID_LINK,
"链接错误");
return new GalleryLink(String.valueOf(gid), segments[5].strip());
}
}
/**
* 上游取页 key 的端口。
*
* <p>生产实现只是把 URL 转交给 GalleryUtil.parseImageKeys;
* 单独抽出来是为了让本服务在测试里完全脱离网络与静态工具类。
*/
@FunctionalInterface
public interface GalleryKeyUpstream {
List<ImageKeyCache> fetchPageKeys(String url) throws Exception;
}
/**
* 构建索引的动作。允许抛出业务异常,因此不用 java.util.function.Function
* (后者不能抛受检异常,会让调用方被迫包装成运行时异常)。
*/
@FunctionalInterface
interface IndexBuilder {
GidToKey build(String url) throws GalleryIndexException;
}
}
@@ -46,6 +46,16 @@ public class GalleryManageService {
final PushService pushService; final PushService pushService;
/**
* 任务生命周期(协议 v2)。创建/删除都先走它做持久化意图,再产生外部副作用。
* 用字段注入是为了兼容现有构造式 mock 测试与旧调用点。
*/
@org.springframework.beans.factory.annotation.Autowired(required = false)
GalleryTaskLifecycleService lifecycleService;
@org.springframework.beans.factory.annotation.Autowired
GalleryIndexService galleryIndexService;
/** /**
* 创建任务 * 创建任务
* @param link 任务链接 * @param link 任务链接
@@ -79,6 +89,20 @@ public class GalleryManageService {
pushService.taskCreateReport(user.getUsername(), taskName, response); pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString(); return response.toJSONString();
} }
// 协议 v2 生命周期:gid 唯一意图先落库(占位行「提交中」),提交前持久化 SUBMITTING,
// 上游结果不确定时报 UNKNOWN 而不是自动重发;差额额度与任务行在同一短事务里写入。
if (lifecycleService != null) {
response = lifecycleService.createTask(gid, taskName, link, targetResolution, user.getId());
Gallery current = galleryMapper.selectGalleryByGid(gid);
taskName = current == null || current.getName() == null ? taskName : current.getName();
// 业务结果由生命周期服务给出,data 用结构化对象下发(前端解码器兼容旧字符串)。
if (response.isSuccess() && current != null)
response.success(objectMapper.valueToTree(current));
pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString();
}
Gallery gallery; Gallery gallery;
//判断数据库中是否有这个任务,有则返回状态 //判断数据库中是否有这个任务,有则返回状态
@@ -124,7 +148,7 @@ public class GalleryManageService {
// Do not overwrite an immediate node status with the original submitted state. // Do not overwrite an immediate node status with the original submitted state.
Gallery current = galleryMapper.selectGalleryByGid(gallery.getGid()); Gallery current = galleryMapper.selectGalleryByGid(gallery.getGid());
response.success((current == null ? gallery : current).toString()); response.success(objectMapper.valueToTree(current == null ? gallery : current));
pushService.taskCreateReport(user.getUsername(), taskName, response); pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString(); return response.toJSONString();
@@ -184,7 +208,7 @@ public class GalleryManageService {
try { try {
gallery = GalleryUtil.parse(link, false, null); gallery = GalleryUtil.parse(link, false, null);
if (gallery != null) if (gallery != null)
response.success(objectMapper.valueToTree(gallery).toString()); response.success(objectMapper.valueToTree(gallery));
else else
response.failure("查询失败"); response.failure("查询失败");
} catch (Exception e) { } catch (Exception e) {
@@ -193,7 +217,7 @@ public class GalleryManageService {
} }
else else
response.success(objectMapper.valueToTree(gallery).toString()); response.success(objectMapper.valueToTree(gallery));
return response.toJSONString(); return response.toJSONString();
} }
@@ -212,7 +236,7 @@ public class GalleryManageService {
response.failure("未找到该图片,请使用链接"); response.failure("未找到该图片,请使用链接");
else else
response.success(gallery.toString()); response.success(objectMapper.valueToTree(gallery));
return response.toJSONString(); return response.toJSONString();
} }
@@ -245,7 +269,7 @@ public class GalleryManageService {
if (userId == UserService.ADMIN_USER_ID) if (userId == UserService.ADMIN_USER_ID)
fillDownloaderNames(galleries); fillDownloaderNames(galleries);
response.success(objectMapper.valueToTree(galleries).toString()); response.success(objectMapper.valueToTree(galleries));
return response.toJSONString(); return response.toJSONString();
} }
@@ -302,7 +326,7 @@ public class GalleryManageService {
Gallery gallery = galleryMapper.selectGalleryByName("%" + name + "%"); Gallery gallery = galleryMapper.selectGalleryByName("%" + name + "%");
if (gallery != null) if (gallery != null)
response.success(objectMapper.valueToTree(gallery).toString()); response.success(objectMapper.valueToTree(gallery));
else else
response.failure("没有找到该名字的图片"); response.failure("没有找到该名字的图片");
@@ -320,7 +344,7 @@ public class GalleryManageService {
Gallery[] galleries = galleryMapper.selectGalleryByDownloader(userMapper.selectUserByAuthCode(AuthCode).getId()); Gallery[] galleries = galleryMapper.selectGalleryByDownloader(userMapper.selectUserByAuthCode(AuthCode).getId());
if (galleries.length > 0) if (galleries.length > 0)
response.success(objectMapper.valueToTree(galleries).toString()); response.success(objectMapper.valueToTree(galleries));
else else
response.failure("您未下载图片"); response.failure("您未下载图片");
@@ -358,8 +382,46 @@ public class GalleryManageService {
return response.toJSONString(); return response.toJSONString();
} }
// 通过授权后才落库并通知节点,避免被拒请求仍删除节点文件。 // 通过授权后才产生外部副作用。协议 v2 的删除顺序:
// 先落「待删除」意图 → 节点确认(或文件已不存在)→ 才在同一事务删任务行并留墓碑。
// 失败时任务行与意图都保留,可沿原记录重试并复用同一个删除 operationId。
log.info("删除图片{}", gallery.getName()); log.info("删除图片{}", gallery.getName());
if (lifecycleService != null) {
// 被拒绝(例如任务仍在提交中)时必须回业务失败:
// 绝不能退避到裸删除,否则会与并发的创建 POST 同时执行并写回已删/重建的 gid。
GalleryTaskLifecycleService.DeleteIntent intent = lifecycleService.beginDelete(gallery.getGid());
if (!intent.accepted()) {
response.failure(intent.message());
log.info("拒绝删除 gid={}:{}", gid, intent.message());
return response.toJSONString();
}
byte result = remoteService.sendDeleteCommand(intent.deleteOperationId(),
gallery.getGid(), gallery.getName());
switch (result) {
case 0 -> {
lifecycleService.completeDelete(gallery.getGid(), intent.deleteOperationId());
response.success();
}
// 文件已不存在等于目标达成:同样完成本地清理,避免留下永远删不掉的任务行。
case ErrorCode.FILE_NOT_FOUND -> {
lifecycleService.completeDelete(gallery.getGid(), intent.deleteOperationId());
response.success();
}
case ErrorCode.IO_ERROR -> {
lifecycleService.failDelete(gallery.getGid());
response.failure("图片:" + gallery.getName() + "删除失败,IO错误");
}
// 节点无响应/超时:保留记录与待删除意图,可重试。
default -> {
lifecycleService.failDelete(gallery.getGid());
response.failure("图片:" + gallery.getName() + "删除失败,节点无响应");
}
}
if (!response.isSuccess())
log.info(response.getData());
return response.toJSONString();
}
galleryMapper.deleteGalleryByGid(gallery.getGid()); galleryMapper.deleteGalleryByGid(gallery.getGid());
switch (remoteService.deleteGallery(gallery)) { switch (remoteService.deleteGallery(gallery)) {
case ErrorCode.IO_ERROR -> response.failure("图片:" + gallery.getName() + "删除失败,IO错误"); case ErrorCode.IO_ERROR -> response.failure("图片:" + gallery.getName() + "删除失败,IO错误");
@@ -399,76 +461,25 @@ public class GalleryManageService {
lastResetAmountTime == null || lastResetAmountTime.getValue() == null lastResetAmountTime == null || lastResetAmountTime.getValue() == null
? "" : lastResetAmountTime.getValue()); ? "" : lastResetAmountTime.getValue());
response.success(objectMapper.valueToTree(data).toString()); response.success(objectMapper.valueToTree(data));
return response.toJSONString(); return response.toJSONString();
} }
public String cacheImagesKey(String url) { public String cacheImagesKey(String url) {
Response response = Response.generateResponse();
// 畸形链接在这里就拦掉:段数不足会让 url.split("/")[5] 抛 ArrayIndexOutOfBoundsException,
// 而 parseGid 也会对无 /g/ 的链接返回 null,两者都不该变成 500。
String[] segments = url == null ? null : url.split("/");
Integer parsedGid = parseGidFromLink(url);
if (parsedGid == null || segments == null || segments.length <= 5) {
response.failure("链接错误");
return response.toJSONString();
}
String gid = String.valueOf(parsedGid);
GidToKey gidToKey = imageCacheMapper.selectKeyByGid(gid);
//已缓存过且页 key 完整,直接返回
if(gidToKey != null && isIndexComplete(gidToKey)) {
return response.success(objectMapper.valueToTree(gidToKey)).toJSONString();
}
// 半截缓存(历史崩溃或旧版本写入中断)先清干净再整体重建,否则残留的页 key 会重复累积。
if (gidToKey != null) {
log.warn("图片索引缓存不完整,重建 gid={} 记录页数={}", gid, gidToKey.getPages());
rollbackImageIndex(gid);
}
boolean inserted = false;
try { try {
gidToKey = new GidToKey(); GidToKey index = galleryIndexService.ensureIndex(url);
gidToKey.setGid(gid); return Response.generateResponse().success(objectMapper.valueToTree(index)).toJSONString();
gidToKey.setKey(segments[5].strip()); } catch (com.lion.lionwebsite.Exception.GalleryIndexException error) {
ArrayList<ImageKeyCache> imageKeyCaches = GalleryUtil.parseImageKeys(url); String message = switch (error.getReason()) {
if(imageKeyCaches == null) case INVALID_LINK -> "链接错误";
return response.failure("该图片已下架或已被删除").toJSONString(); case UNAVAILABLE -> "该图片已下架或已被删除";
gidToKey.setPages(imageKeyCaches.size()); case MALFORMED, TRANSIENT -> "网络波动或其他异常";
imageCacheMapper.insertGidToKey(gidToKey); };
inserted = true; log.warn("缓存图片索引失败 reason={}", error.getReason());
for (ImageKeyCache imageKeyCache : imageKeyCaches) return Response._failure(message);
imageCacheMapper.insertImageKeyCache(imageKeyCache); } catch (RuntimeException error) {
response.success(objectMapper.valueToTree(gidToKey)); log.warn("缓存图片索引失败 errorType={}", error.getClass().getSimpleName());
}catch (Exception e){ return Response._failure("网络波动或其他异常");
// 索引必须整体生效:先写 gidToKey 再逐页写 key,中途失败会留下「gidToKey 命中、
// 页 key 缺失」的半截缓存,后续请求直接返回已缓存而永远取不到图。
// 因此任何异常都回滚已写入的部分再回业务失败。Jackson 3 的解析异常继承
// RuntimeException 而非 IOException,只用 IOException 会漏掉,这里统一兜住。
if (inserted)
rollbackImageIndex(gid);
log.warn("缓存图片索引失败 gid={} errorType={}", gid, e.getClass().getSimpleName());
response.failure("网络波动或其他异常");
}
return response.toJSONString();
}
/** 页 key 行数与 gidToKey 记录不一致时视为半截缓存,需要重建。 */
private boolean isIndexComplete(GidToKey gidToKey) {
Integer pages = gidToKey.getPages();
if (pages == null || pages <= 0)
return false;
return imageCacheMapper.countImageKeyCacheByGid(gidToKey.getGid()) == pages;
}
/** 删除某个 gid 已写入的索引(页 key 与 gidToKey),用于回滚或重建。 */
private void rollbackImageIndex(String gid) {
try {
imageCacheMapper.deleteImageKeyCacheByGid(gid);
imageCacheMapper.deleteGidToKey(gid);
} catch (Exception e) {
log.warn("回滚图片索引缓存失败 gid={}", gid, e);
} }
} }
@@ -517,10 +528,24 @@ public class GalleryManageService {
Gallery[] galleries = galleryMapper.selectUnDoneGalleries(); Gallery[] galleries = galleryMapper.selectUnDoneGalleries();
if(galleries != null && galleries.length != 0) { if(galleries != null && galleries.length != 0) {
log.info("发送未下载完成图片至节点,{}本", galleries.length); // 只重投「确实已提交上游」的任务:提交未知/预占中/待删除的任务重投会变成
for (Gallery gallery : galleries) // 未计量副作用(节点侧显示下载完成但没有任何额度记录)。
int sent = 0;
int skipped = 0;
for (Gallery gallery : galleries) {
if (lifecycleService != null
&& !lifecycleService.resolveDispatch(gallery.getGid()).dispatchable()) {
skipped++;
continue;
}
remoteService.addGalleryToQueue(gallery); remoteService.addGalleryToQueue(gallery);
response.success(String.format("发送未下载完成图片至节点,%s本", galleries.length)); sent++;
}
log.info("发送未下载完成图片至节点,{}本(跳过需要对账的{}本)", sent, skipped);
if (sent == 0)
response.failure("没有可安全重投的任务,需先人工对账提交未知的任务");
else
response.success(String.format("发送未下载完成图片至节点,%s本", sent));
}else{ }else{
response.failure("当前没有未下载完成的图片"); response.failure("当前没有未下载完成的图片");
} }
@@ -536,6 +561,13 @@ public class GalleryManageService {
return Response._success("下载完成"); return Response._success("下载完成");
if(remoteService.isDead()) if(remoteService.isDead())
return Response._failure("节点不在线,无法重试"); return Response._failure("节点不在线,无法重试");
// 提交结果未知(UNKNOWN)或正在删除的任务不能重试:上游不支持幂等,重发会让
// 「可能已受理但未计量」的任务在节点侧变成下载完成却 0 额度。这类任务需人工对账。
if (lifecycleService != null) {
var decision = lifecycleService.resolveDispatch(gid);
if (decision.kind() == GalleryTaskLifecycleService.DispatchDecision.Kind.REFUSE)
return Response._failure("该任务提交结果未知或正在删除,请先人工对账后再重试");
}
RemoteService.RetryResult retryResult = remoteService.retryGallery(gallery); RemoteService.RetryResult retryResult = remoteService.retryGallery(gallery);
if(retryResult.success()) if(retryResult.success())
@@ -0,0 +1,783 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.*;
import com.lion.lionwebsite.Domain.*;
import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import com.lion.lionwebsite.Message.TaskCommandMessage;
import com.lion.lionwebsite.Message.TaskReportMessage;
import com.lion.lionwebsite.Util.PreparedGallery;
import com.lion.lionwebsite.Util.Response;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionTemplate;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
/**
* 下载任务的持久化生命周期(协议 v2)。
*
* <p>核心约束:
* <ol>
* <li><b>先保存意图,再产生外部副作用</b>:同一 gid 的唯一 CREATE 意图先落库(占位任务行 + RESERVED 操作记录),
* 通过 gid 的主键唯一约束把并发创建收敛成「一个赢家」;</li>
* <li><b>SUBMITTING 之后绝不自动重发</b>:上游不支持幂等,提交结果不确定时只能落 UNKNOWN,等显式对账/解决;</li>
* <li><b>提交结果与额度用一个短事务写入</b>:Gallery 与 weekUsedAmount 要么一起成功,要么一起回滚,
* 事务提交之后才把 DOWNLOAD 命令下发给节点;</li>
* <li><b>删除失败保留记录</b>:先落待删除意图,节点确认(或文件已不存在)后才在同一事务里删任务行并留墓碑;
* 失败则保留 Gallery 与 DELETE_PENDING 意图,可沿原记录重试且复用同一删除 operationId。</li>
* </ol>
*
* <p>所有网络调用都发生在事务之外;事务回调内部不做 I/O,也不吞异常(吞掉会错误提交)。
*/
@Service
@Slf4j
public class GalleryTaskLifecycleService implements TaskReportHandler {
final GalleryMapper galleryMapper;
final CollectMapper collectMapper;
final CustomConfigurationMapper configurationMapper;
final UserMapper userMapper;
final TaskOperationRepository taskOperationRepository;
final UpstreamGateway upstreamGateway;
final RemoteService remoteService;
/**
* 短全局创建预占锁。
*
* <p>只有「登记唯一 CREATE 意图」这一小段数据库写入在锁内,网络调用一律在锁外。
* 单进程内它把并发同 gid 创建直接收敛成一个赢家;跨进程的正确性仍由
* {@code task_operation}/{@code gallery} 的主键唯一约束兜底。
*/
private final Object reservationLock = new Object();
/**
* 测试钩子:进入上游只读解析之前执行。
* 用来让并发删除发生在预占之后、SUBMITTING 之前。生产环境是空实现。
*/
volatile Runnable beforePrepare = () -> { };
/**
* 测试钩子:SUBMITTING 已持久化之后、真正 POST 上游之前执行。
* 用来制造「提交瞬间任务被删除」的 late generation 场景。生产环境是空实现。
*/
volatile Runnable afterMarkSubmitting = () -> { };
/** 主库事务模板;测试可注入内存 sqlite 的事务模板。 */
TransactionTemplate transactions;
public GalleryTaskLifecycleService(GalleryMapper galleryMapper,
CollectMapper collectMapper,
CustomConfigurationMapper configurationMapper,
UserMapper userMapper,
TaskOperationRepository taskOperationRepository,
UpstreamGateway upstreamGateway,
RemoteService remoteService,
@Qualifier("mainTransactionTemplate") TransactionTemplate transactions) {
this.galleryMapper = galleryMapper;
this.collectMapper = collectMapper;
this.configurationMapper = configurationMapper;
this.userMapper = userMapper;
this.taskOperationRepository = taskOperationRepository;
this.upstreamGateway = upstreamGateway;
this.remoteService = remoteService;
this.transactions = transactions;
}
/** 测试用:替换事务模板或在没有 Spring 事务时退化为直连执行。 */
void setTransactions(TransactionTemplate transactions) {
this.transactions = transactions;
}
/**
* 一次「登记创建意图」的结果。
*
* <p>只有赢家能继续做外部副作用;输家(已存在有效意图 / 数据库竞争失败)
* 一律转成业务失败,绝不再 POST 上游。
*/
private enum ReserveOutcome {
/** 已成功登记唯一 CREATE 意图,可以做只读解析与提交。 */
RESERVED,
/** 同一 gid 已存在有效意图(或任务行)。 */
DUPLICATE,
/** 登记时的数据库竞争/失败:没有产生任何副作用,必须转业务失败。 */
CONTENDED
}
/** 一次「登记创建意图」的返回:结果 + 赢家持有的操作记录。 */
private record Reservation(ReserveOutcome outcome, TaskOperation operation) {
}
/**
* 单次下发决策。
*
* <p>把「能不能把这个任务交给节点」从「下载是否已提交」里拆出来:
* 未确认上游提交结果(UNKNOWN / 预占 / 待删除 / 删除墓碑)绝不能下发,
* 否则未计量的人工对账任务可能在节点侧变成「下载完成却 0 额度」。
*/
public static final class DispatchDecision {
public enum Kind {
/** 当前 operation 已是 SUBMITTED:可以按稳定 operationId 下发。 */
READY,
/** 迁移前的旧任务行:没有操作记录但历史上确实提交过下载。 */
LEGACY,
/** 不可下发:状态不明或被删除,下发会产生未计量副作用。 */
REFUSE
}
private final Kind kind;
private final String operationId;
private DispatchDecision(Kind kind, String operationId) {
this.kind = kind;
this.operationId = operationId;
}
public Kind kind() {
return kind;
}
public String operationId() {
return operationId;
}
/** 只有 READY/LEGACY 允许真正下发到节点。 */
public boolean dispatchable() {
return kind != Kind.REFUSE && operationId != null;
}
/** 拒绝下发(供调用方与测试构造)。 */
public static DispatchDecision refuse() {
return REFUSE;
}
/** 可直接下发(供测试构造)。 */
public static DispatchDecision ready(String operationId) {
return new DispatchDecision(Kind.READY, operationId);
}
}
private static final DispatchDecision REFUSE = new DispatchDecision(DispatchDecision.Kind.REFUSE, null);
/**
* 一次删除登记的返回。
*
* <p>用显式结果替代「null 即退避到裸删除」的旧语义:
* 被拒绝时调用方必须回业务失败,不能绕过待删除意图直接删节点文件。
*/
public record DeleteIntent(boolean accepted, String deleteOperationId, String message) {
static DeleteIntent refused(String message) {
return new DeleteIntent(false, null, message);
}
static DeleteIntent accepted(String deleteOperationId) {
return new DeleteIntent(true, deleteOperationId, null);
}
}
/** 在事务里执行;没有配置事务模板时候直接执行(仅便于构造 mock 测试)。 */
private <T> T inTransaction(java.util.function.Supplier<T> body) {
TransactionTemplate template = transactions;
if (template == null)
return body.get();
return template.execute(status -> {
try {
return body.get();
} catch (RuntimeException e) {
// 回调里必须把异常继续抛出,否则 Spring 会把只读了一半的事务当成正常提交。
status.setRollbackOnly();
throw e;
}
});
}
/**
* 创建任务的持久化前置:在同一个事务里登记唯一 CREATE 意图。
*
* <p>同时写入可见的占位任务行(状态「提交中」),这样提交期间任务列表就能解析该 gid,
* 不会出现「上游已受理但本地查不到」的空窗。
*
* @return 已登记意图(赢家)返回它;同一 gid 已有有效意图返回 {@code null}(输家)。
*/
private Reservation reserveCreateIntent(int gid, String placeholderName, int downloaderId) {
// 锁内只做数据库写入:把并发同 gid 创建串行化,网络调用留在锁外。
synchronized (reservationLock) {
try {
TaskOperation operation = inTransaction(() -> {
TaskOperation existingOperation = taskOperationRepository.selectByGid(gid);
Gallery existingGallery = galleryMapper.selectGalleryByGid(gid);
if (existingOperation != null) {
// 只有「删除已完成」的墓碑才允许用全新 operationId 重新创建(显式重建)。
if (!TaskOperation.STATE_DELETED.equals(existingOperation.getState()))
return null;
if (existingGallery != null)
return null;
String operationId = TaskProtocol.newDownloadOperationId();
taskOperationRepository.reactivate(gid, operationId, TaskOperation.STATE_RESERVED, now());
insertPlaceholder(gid, placeholderName, downloaderId);
return taskOperationRepository.selectByGid(gid);
}
if (existingGallery != null)
return null;
TaskOperation created = new TaskOperation();
created.setGid(gid);
created.setOperationId(TaskProtocol.newDownloadOperationId());
created.setState(TaskOperation.STATE_RESERVED);
created.setUpdateTime(now());
taskOperationRepository.insert(created);
insertPlaceholder(gid, placeholderName, downloaderId);
return created;
});
return operation == null
? new Reservation(ReserveOutcome.DUPLICATE, null)
: new Reservation(ReserveOutcome.RESERVED, operation);
} catch (RuntimeException e) {
// 数据库竞争/失败发生在任何外部副作用之前,绝不能穿透成 500,也不能继续提交。
log.warn("登记创建意图失败,按业务失败处理 gid={} error={}", gid, e.getClass().getSimpleName());
return new Reservation(ReserveOutcome.CONTENDED, null);
}
}
}
/** 写入占位任务行:让提交期间的任务在列表里可见。 */
private void insertPlaceholder(int gid, String name, int downloaderId) {
Gallery placeholder = new Gallery();
placeholder.setGid(gid);
placeholder.setName(name);
placeholder.setStatus(TaskStatus.PENDING_SUBMIT);
placeholder.setDownloader(downloaderId);
placeholder.setCreateTime(System.currentTimeMillis() / 1000);
placeholder.setProceeding(0);
galleryMapper.insertGallery(placeholder);
}
/**
* 标记为正在提交,并用 CAS 确认这一代意图仍然有效。
*
* <p>只有当前状态与 operationId 都匹配(RESERVED → SUBMITTING)才返回 true;
* 期间被删除或重建返回 false,调用方必须停下,绝不能再 POST 上游。
*
* @return true 表示已持久化 SUBMITTING,可以安全产生上游副作用
*/
boolean markSubmitting(String operationId, int gid) {
return inTransaction(() -> taskOperationRepository.compareAndSetCreateState(gid, operationId,
TaskOperation.STATE_RESERVED, TaskOperation.STATE_SUBMITTING, now()) == 1);
}
/**
* 提交结果未知(超时/断连/中断)。保留意图,等待人工对账,不自动重发。
*
* <p>TaskOperation 与 Gallery 状态在同一个短事务里更新:
* 操作落 UNKNOWN 而任务行仍显示「提交中」会让运营侧误以为还能自动重试,
* 因此两边必须一致地表达「不确定,需要人工对账」。
*/
void markUnknown(String operationId, int gid) {
inTransaction(() -> {
int advanced = taskOperationRepository.compareAndSetCreateState(gid, operationId,
TaskOperation.STATE_SUBMITTING, TaskOperation.STATE_UNKNOWN, now());
// 只有本次这一代确实从 SUBMITTING 推进到 UNKNOWN 时才同步任务行:
// 若期间已被删除/重建(CAS 返回 0),绝不能再改那条记录。
if (advanced != 1)
return null;
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery != null && !TaskStatus.UNKNOWN.equals(gallery.getStatus()))
galleryMapper.updateGalleryStatus(gid, TaskStatus.UNKNOWN);
return null;
});
}
/**
* 提交成功后的「一次性短事务」:写入 Gallery 行、累计周用量、把操作推进为 SUBMITTED。
* 三者同一个事务,任何一个失败都一起回滚,避免出现「任务存在但用量未计」这类不一致。
*/
/**
* 提交成功后的落库(带世代校验)。
*
* @param expectedOperationId 本次提交所归属的 operationId;只有它仍是当前有效意图时才能落库,
* 否则(被删除/重建)必须抛错,避免复活已删 gid 或污染新世代
*/
int persistSubmitted(Gallery gallery, int downloaderId, String expectedOperationId) {
return inTransaction(() -> {
TaskOperation operation = taskOperationRepository.selectByGid(gallery.getGid());
// 严格匹配本次世代:任务被删除、被重建或换了 operationId 时都不能落库。
if (operation == null || TaskOperation.STATE_DELETED.equals(operation.getState())
|| TaskOperation.STATE_DELETE_PENDING.equals(operation.getState()))
throw new IllegalStateException("任务已删除或没有有效创建意图,禁止落库复活 gid=" + gallery.getGid());
if (expectedOperationId == null || !expectedOperationId.equals(operation.getOperationId()))
throw new IllegalStateException("创建意图世代不匹配,禁止落库 gid=" + gallery.getGid());
// 幂等:同一意图重复落库(例如上游已提交后重跑)不得把用量累计两次。
boolean alreadyCounted = TaskOperation.STATE_SUBMITTED.equals(operation.getState());
gallery.setDownloader(downloaderId);
gallery.set_download(true);
if (!TaskStatus.SUBMITTED.equals(gallery.getStatus()))
gallery.setStatus(TaskStatus.SUBMITTED);
Gallery existing = galleryMapper.selectGalleryByGid(gallery.getGid());
if (existing == null)
galleryMapper.insertGallery(gallery);
else
galleryMapper.updateGallery(gallery);
if (!alreadyCounted)
configurationMapper.incrementConfiguration(CustomConfiguration.WEEK_USED_AMOUNT, gallery.getFileSize());
taskOperationRepository.updateState(gallery.getGid(), TaskOperation.STATE_SUBMITTED, now());
return 1;
});
}
/**
* 预占式创建:登记 gid 唯一意图(提交任务行占位)→ 只读解析上游 → 标 SUBMITTING → 真正提交。
*
* <p>上游提交发生在事务之外,且只在 SUBMITTING 已持久化之后执行一次。网络异常一律落 UNKNOWN。
*/
public Response createTask(int gid, String placeholderName, String link, String targetResolution, int downloaderId) {
Response response = Response.generateResponse();
Reservation reservation = reserveCreateIntent(gid, placeholderName, downloaderId);
if (reservation.outcome() == ReserveOutcome.DUPLICATE) {
response.failure("任务队列已有此任务,请点击查找任务");
return response;
}
if (reservation.outcome() == ReserveOutcome.CONTENDED) {
// 预占阶段的数据库竞争不产生任何副作用;转业务失败,绝不下发上游、绝不穿透成 500。
response.failure("任务提交失败,请稍后重试");
return response;
}
// 解析上游是只读步骤;真正的 POST 只在 SUBMITTING 持久化之后发生。
// 生命周期约束:RESERVED 阶段被中断是安全的(可释放重来),SUBMITTING 之后一律按 UNKNOWN 处理。
return submitPreparedIntent(gid, reservation.operation(), link, targetResolution, downloaderId);
}
private Response submitPreparedIntent(int gid, TaskOperation operation, String link, String targetResolution,
int downloaderId) {
Response response = Response.generateResponse();
beforePrepare.run();
PreparedGallery prepared;
try {
prepared = upstreamGateway.prepare(link);
} catch (IOException e) {
log.warn("解析任务链接失败 gid={}", gid, e);
releaseReservation(gid, operation.getOperationId());
response.failure("IO错误,可能是网络波动");
return response;
} catch (RuntimeException e) {
log.warn("解析任务链接异常 gid={}", gid, e);
releaseReservation(gid, operation.getOperationId());
response.failure("提交任务失败,未知原因,请检查链接是否正常");
return response;
}
if (prepared == null || prepared.gallery() == null) {
releaseReservation(gid, operation.getOperationId());
response.failure("提交任务失败,未知原因,请检查链接是否正常");
return response;
}
Gallery gallery = prepared.gallery();
gallery.setGid(gid);
// CAS 确认这一代意图仍然有效(未被并发删除/重建),并把状态持久化为 SUBMITTING。
// 只有成功之后才允许真正 POST;失败说明任务已不在当前生命周期里,直接放弃。
if (!markSubmitting(operation.getOperationId(), gid)) {
log.warn("创建意图已失效(可能被并发删除),放弃提交 gid={} operationId={}",
gid, operation.getOperationId());
response.failure("任务已被删除或状态已变化,请刷新任务列表");
return response;
}
afterMarkSubmitting.run();
Gallery submitted;
try {
submitted = upstreamGateway.submit(prepared, targetResolution);
} catch (ResolutionNotMatchException e) {
releaseReservation(gid, operation.getOperationId());
response.failure("提交失败,分辨率不存在");
return response;
} catch (IOException | RuntimeException e) {
// 超时或断连时上游可能已经受理,无法确认:只能记 UNKNOWN,绝不自动重发。
log.warn("上游提交结果未知 gid={} operationId={}", gid, operation.getOperationId(), e);
markUnknown(operation.getOperationId(), gid);
response.failure("上游提交结果未知,请稍后在任务列表中确认,不要重复提交");
return response;
}
if (submitted == null || !TaskStatus.SUBMITTED.equals(submitted.getStatus())) {
markUnknown(operation.getOperationId(), gid);
response.failure("上游提交结果未知,请稍后在任务列表中确认,不要重复提交");
return response;
}
submitted.setGid(gid);
try {
persistSubmitted(submitted, downloaderId, operation.getOperationId());
} catch (RuntimeException e) {
// 上游已受理但本地落库失败:保留 SUBMITTING 意图并标 UNKNOWN,任务行仍在列表里可对账。
log.error("提交成功后落库失败 gid={} operationId={}", gid, operation.getOperationId(), e);
markUnknown(operation.getOperationId(), gid);
response.failure("任务已提交,但本地保存失败,请在任务列表中确认");
return response;
}
// 只有事务提交之后才向节点下发命令;重复下发由节点按 operationId 去重。
byte dispatchResult = dispatchDownload(gid, submitted.getName());
Gallery current = galleryMapper.selectGalleryByGid(gid);
Gallery payload = current == null ? submitted : current;
if (dispatchResult == 0)
response.success(objectMapper.valueToTree(payload));
else
response.failure("任务已保存,但节点未确认接收;请刷新任务列表后重试");
return response;
}
/**
* 提交成功后的落库 + 下发。落库(含用量)在一个短事务里完成,事务提交后才向节点发命令。
*
* <p>先解析当前世代再落库:世代不存在(未提交过 / 已删除)时不落库也不下发。
*
* @return 已持久化的 Gallery;数据库失败或世代无效时抛出异常,调用方回业务失败且不下发节点。
*/
public Gallery persistAndDispatch(Gallery gallery, int downloaderId) {
DispatchDecision decision = resolveDispatch(gallery.getGid());
if (!decision.dispatchable())
throw new IllegalStateException("任务没有可下发的有效世代 gid=" + gallery.getGid());
persistSubmitted(gallery, downloaderId, decision.operationId());
dispatchDownload(gallery.getGid(), gallery.getName());
return gallery;
}
/** 向节点下发 DOWNLOAD 命令(复用当前有效 operationId),返回节点结果码。 */
public byte dispatchDownload(int gid, String name) {
DispatchDecision decision = resolveDispatch(gid);
if (!decision.dispatchable())
return -1;
return remoteService.sendTaskCommand(TaskCommandMessage.ACTION_DOWNLOAD, decision.operationId(), gid, name);
}
/**
* 判定一个 gid 能否下发给节点,并为迁移前的旧任务补齐稳定 operationId。
*
* <p>规则:
* <ul>
* <li>v2 任务只有 {@code SUBMITTED} 可下发;{@code UNKNOWN} 必须人工对账,
* 否则未计量的未知任务会在节点侧变成「下载完成却 0 额度」;</li>
* <li>预占中的 {@code RESERVED}/{@code SUBMITTING}、待删除与删除墓碑都不下发;</li>
* <li>没有任何操作记录、但任务行确实存在且状态已是「已提交」及之后的历史任务,
* 补建一个稳定 operationId 后再下发(这是唯一的 legacy 兼容路径)。</li>
* </ul>
*/
public DispatchDecision resolveDispatch(int gid) {
TaskOperation operation = taskOperationRepository.selectByGid(gid);
if (operation != null && TaskOperation.STATE_SUBMITTED.equals(operation.getState())
&& operation.getOperationId() != null)
return new DispatchDecision(DispatchDecision.Kind.READY, operation.getOperationId());
// 没有操作记录,或只有「旧世代」记录(缺 operationId):必须先确认任务行确实已提交过,
// 再补一个稳定 operationId;UNKNOWN/预占/待删除一律拒绝。
if (operation != null && !TaskOperation.STATE_SUBMITTED.equals(operation.getState()))
return REFUSE;
if (!isLegacySubmitted(gid))
return REFUSE;
String freshId = ensureStableOperationId(gid);
return freshId == null ? REFUSE : new DispatchDecision(DispatchDecision.Kind.LEGACY, freshId);
}
/** 迁移前的旧任务行:没有操作记录,但状态表明上游确实已受理过下载,可安全重投。 */
private boolean isLegacySubmitted(int gid) {
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null || gallery.getStatus() == null)
return false;
return switch (gallery.getStatus()) {
case TaskStatus.SUBMITTED, TaskStatus.DOWNLOADING, TaskStatus.WAITING_COMPRESS,
TaskStatus.COMPRESSING -> true;
// 提交未知 / 提交中 / 待删除 / 删除失败 一律不算已提交,不能下发。
default -> false;
};
}
/**
* 为一个已经真正提交过的旧任务持久化稳定 operationId。
*
* <p>必须是短事务写入、先落库后返回,之后重投/重连才复用同一个 ID;
* 用 {@code insert or ignore} + 回读,避免与并发创建互相覆盖。
*
* @return 稳定 operationId;无法安全建立时返回 null(调用方不下发)
*/
String ensureStableOperationId(int gid) {
// 与创建预占共用同一把短锁(锁内只有数据库写),并发为同一 gid 补 ID 时只能有一个赢家。
synchronized (reservationLock) {
String candidate = TaskProtocol.newDownloadOperationId();
return inTransaction(() -> {
if (taskOperationRepository.selectByGid(gid) == null)
taskOperationRepository.insertIfAbsent(gid, candidate, TaskOperation.STATE_SUBMITTED, now());
else
taskOperationRepository.assignOperationIdIfMissing(gid, candidate, now());
TaskOperation current = taskOperationRepository.selectByGid(gid);
if (current == null || current.getOperationId() == null)
return null;
// 只有确实处于 SUBMITTED 的世代才可下发;并发写入的其他状态一律拒绝。
return TaskOperation.STATE_SUBMITTED.equals(current.getState()) ? current.getOperationId() : null;
});
}
}
/**
* 取消预占(解析失败、分辨率不存在等:还没有产生任何上游副作用)。
*
* <p>只删除与本次创建 intent 匹配的 RESERVED 记录:若期间该 gid 已被删除并重建
* (新的 operationId / 墓碑),这里绝不能把新世代记录抹掉。占位行同理只删「提交中」。
*/
private void releaseReservation(int gid, String operationId) {
inTransaction(() -> {
// 只删与本代 operationId 匹配且仍停在 RESERVED/SUBMITTING 的记录。
int removed = taskOperationRepository.deleteReservationIfSameGeneration(gid, operationId);
// 只有确实删掉了本代预占记录,才能删对应的占位行:
// 若期间已出现新的 RESERVED 代,删除返回 0,此时绝不能误删新占位。
if (removed == 1) {
Gallery existing = galleryMapper.selectGalleryByGid(gid);
if (existing != null && TaskStatus.PENDING_SUBMIT.equals(existing.getStatus()))
galleryMapper.deleteGalleryByGid(gid);
}
return null;
});
}
/**
* 删除的持久化部分。
*
* <p>正在创建意图(RESERVED/SUBMITTING)的任务不能删除:此刻 DELETE 与创建 POST 会同时发生,
* 随后创建落库还会把已删/重建的 gid 写回。这里直接拒绝,由用户在提交结果明确后再操作。
*
* @return 显式结果:被拒时调用方必须回业务失败,不能退避到裸删除
*/
public DeleteIntent beginDelete(int gid) {
return inTransaction(() -> {
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null)
return DeleteIntent.refused("删除失败,该图片不存在");
TaskOperation operation = taskOperationRepository.selectByGid(gid);
if (operation != null && (TaskOperation.STATE_RESERVED.equals(operation.getState())
|| TaskOperation.STATE_SUBMITTING.equals(operation.getState())))
return DeleteIntent.refused("任务正在提交中,请等待提交结果明确后再删除");
if (operation == null) {
// 历史任务行没有操作记录:直接落一条 DELETE_PENDING 意图,避免中间态缺字段。
TaskOperation fresh = new TaskOperation();
fresh.setGid(gid);
fresh.setOperationId(TaskProtocol.newDownloadOperationId());
fresh.setDeleteOperationId(TaskProtocol.newDeleteOperationId());
fresh.setState(TaskOperation.STATE_DELETE_PENDING);
fresh.setUpdateTime(now());
taskOperationRepository.insert(fresh);
galleryMapper.updateGalleryStatus(gid, TaskStatus.DELETE_PENDING);
return DeleteIntent.accepted(fresh.getDeleteOperationId());
}
if (operation.getDeleteOperationId() == null) {
operation.setDeleteOperationId(TaskProtocol.newDeleteOperationId());
operation.setUpdateTime(now());
}
taskOperationRepository.updateDeleteIntent(gid, operation.getDeleteOperationId(),
TaskOperation.STATE_DELETE_PENDING, now());
galleryMapper.updateGalleryStatus(gid, TaskStatus.DELETE_PENDING);
return DeleteIntent.accepted(operation.getDeleteOperationId());
});
}
/**
* 删除的完成部分:节点已确认(或文件已不存在)后,在一个短事务里删任务行、收藏并留墓碑。
*
* <p>事务提交前若失败,任务行与 DELETE_PENDING 意图都保留,可以继续重试。
* 删除意图 ID 必须匹配(防止并发删除/重建时抹掉别的一代),随后只把 DELETE_PENDING
* 世代推进为 DELETED;墓碑(DELETED 且无删除意图)不允许在这里被重新激活。
*
* @return true 表示本次删除意图已完成并留下墓碑
*/
public boolean completeDelete(int gid, String deleteOperationId) {
return inTransaction(() -> {
TaskOperation operation = taskOperationRepository.selectByGid(gid);
if (operation == null || operation.getDeleteOperationId() == null
|| !operation.getDeleteOperationId().equals(deleteOperationId))
return false;
// 先清收藏再删任务行:收藏一侧在部分历史库上带有指向 gallery 的外键。
collectMapper.deleteByGid(gid);
galleryMapper.deleteGalleryByGid(gid);
int updated = taskOperationRepository.completeDeleteIfSameGeneration(gid, deleteOperationId,
TaskOperation.STATE_DELETED, now());
return updated == 1;
});
}
/** 删除失败:保留任务行并记一条可重试的失败状态。 */
public void failDelete(int gid) {
inTransaction(() -> {
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null)
return null;
gallery.setStatus(TaskStatus.DELETE_FAILED);
galleryMapper.updateGallery(gallery);
return null;
});
}
/**
* 启动恢复:把上次中断在 SUBMITTING 的意图判为 UNKNOWN,并安全释放还停在 RESERVED 的预占。
*
* <p>这段恢复必须发生在任何自动重投之前:
* <ul>
* <li>{@code SUBMITTING}:上游可能已经受理但本地无法确认,只能落 UNKNOWN(操作与任务行同时
* 更新为「提交未知」),交人工对账,绝不自动再 POST;</li>
* <li>{@code RESERVED}:还没有产生任何上游副作用,重启后可以直接释放预占
* (删掉匹配的 RESERVED 记录与「提交中」占位行),让用户重新提交。</li>
* </ul>
*
* @return 本次恢复(判未知 + 释放预占)处理的任务数
*/
public int recoverInterruptedSubmissions() {
return inTransaction(() -> {
int recovered = 0;
for (TaskOperation operation : taskOperationRepository.selectByState(TaskOperation.STATE_SUBMITTING)) {
int advanced = taskOperationRepository.compareAndSetCreateState(operation.getGid(), operation.getOperationId(),
TaskOperation.STATE_SUBMITTING, TaskOperation.STATE_UNKNOWN, now());
if (advanced != 1)
continue;
Gallery gallery = galleryMapper.selectGalleryByGid(operation.getGid());
if (gallery != null && !TaskStatus.UNKNOWN.equals(gallery.getStatus()))
galleryMapper.updateGalleryStatus(operation.getGid(), TaskStatus.UNKNOWN);
recovered++;
}
for (TaskOperation operation : taskOperationRepository.selectByState(TaskOperation.STATE_RESERVED)) {
// 还没 POST:释放匹配的预占记录与占位行,允许重启后重新提交。
// 只有确实删掉了这条本代 RESERVED 记录,才删其占位行;否则不动(可能已换代)。
int removed = taskOperationRepository.deleteReservationIfSameGeneration(operation.getGid(),
operation.getOperationId());
if (removed == 1) {
Gallery gallery = galleryMapper.selectGalleryByGid(operation.getGid());
if (gallery != null && TaskStatus.PENDING_SUBMIT.equals(gallery.getStatus()))
galleryMapper.deleteGalleryByGid(operation.getGid());
}
recovered++;
}
return recovered;
});
}
/** 已完成状态上报的处理:丢弃旧 revision、旧 operation、待删除任务与重复上报。 */
@Override
public CommittedReport apply(TaskReportMessage report) {
if (report == null || report.getReportId() == null || report.getGalleryTasks() == null)
return null;
Applied applied;
try {
applied = inTransaction(() -> applyInTransaction(report));
} catch (RuntimeException e) {
// 数据库失败:不返回可回执结果,节点会保留快照并稍后重投;失败写入绝不清任务。
log.warn("应用节点上报失败,未回执 reportId={}", report.getReportId(), e);
return null;
}
if (applied == null)
return null;
return new CommittedReport(report.getReportId(), applied.newlyCompleted(), applied.acceptedTasks());
}
/** 事务内应用上报的结果:新完成的任务与真正被接受的进度更新。 */
private record Applied(List<Gallery> newlyCompleted, List<GalleryTask> acceptedTasks) {
}
/**
* 在一个事务里应用上报。
*
* @return 非 null 表示已经提交(可以回 ACK,重复上报也会返回结果以停止重放)。
*/
private Applied applyInTransaction(TaskReportMessage report) {
long revision = report.getRevision();
List<Gallery> newlyCompleted = new ArrayList<>();
List<GalleryTask> acceptedTasks = new ArrayList<>();
for (GalleryTask task : report.getGalleryTasks()) {
if (task == null)
continue;
TaskOperation operation = taskOperationRepository.selectByGid(task.getGid());
Gallery gallery = galleryMapper.selectGalleryByGid(task.getGid());
if (gallery == null) {
// 未知 gid(任务已删除):忽略但 ACK,让节点停止重放。
log.debug("忽略节点上报的未知任务 gid={}", task.getGid());
continue;
}
String expectedOperationId = report.getOperationIds() == null
? null : report.getOperationIds().get(task.getGid());
if (operation == null) {
// 旧任务行(迁移前创建)没有操作记录:按显式生成策略补建并采纳上报里的 operationId。
operation = adoptGeneration(task.getGid(), expectedOperationId);
} else if (TaskOperation.STATE_DELETED.equals(operation.getState())) {
// 墓碑必须永久忽略旧上报:删除已完成的任务只能由显式 CREATE 重新建立新世代。
// (gallery 已被删除时上面的 null 分支已经提前忽略,这里防御性兜底。)
log.debug("忽略已删除 gid 的迟到上报 gid={}", task.getGid());
continue;
} else if (TaskOperation.STATE_DELETE_PENDING.equals(operation.getState())) {
log.info("忽略待删除任务的迟到上报 gid={}", task.getGid());
continue;
} else if (operation.getLastRevision() > 0 && revision <= operation.getLastRevision()) {
log.debug("丢弃过期上报 gid={} revision={} lastRevision={}",
task.getGid(), revision, operation.getLastRevision());
continue;
} else if (operation.getOperationId() != null) {
// v2 任务必须有归属:缺 operationId 或与当前世代不符都不能改当前任务,
// 否则「删除后重建」的新世代会被旧上报污染。
if (expectedOperationId == null || !operation.getOperationId().equals(expectedOperationId)) {
log.info("丢弃无法归属/旧世代的迟到上报 gid={} 上报={} 当前={}",
task.getGid(), expectedOperationId, operation.getOperationId());
continue;
}
}
String status = TaskStatus.fromReport(task.getStatus());
if (status == null) {
log.warn("未知上报状态码 gid={} status={}", task.getGid(), task.getStatus());
continue;
}
gallery.setProceeding(task.getProceeding());
if (task.getName() != null)
gallery.setName(task.getName());
boolean justCompleted = TaskStatus.COMPLETE.equals(status)
&& !TaskStatus.COMPLETE.equals(gallery.getStatus());
gallery.setStatus(status);
galleryMapper.updateGallery(gallery);
taskOperationRepository.updateReportReceipt(task.getGid(), revision, report.getReportId(), now());
acceptedTasks.add(task);
if (justCompleted)
newlyCompleted.add(gallery);
}
return new Applied(newlyCompleted, acceptedTasks);
}
/**
* 为迁移前创建、尚无操作记录的历史任务建立一次「世代」:优先采纳上报里的 operationId;
* 旧节点不带 operationId 时保留 null,让后续同世代的进度上报仍能被接受。
*
* <p>不在这里凭空生成随机 ID:那会让第一份进度上报之后的所有旧上报都因「无法归属」被丢弃。
* 真正的稳定 operationId 会在首次自动下发时用 {@code ensureStableOperationId} 落库,
* 之后该世代即可被严格匹配。不会覆盖任何已有任务状态。
*/
private TaskOperation adoptGeneration(int gid, String reportedOperationId) {
TaskOperation operation = new TaskOperation();
operation.setGid(gid);
operation.setOperationId(reportedOperationId);
operation.setState(TaskOperation.STATE_SUBMITTED);
operation.setUpdateTime(now());
taskOperationRepository.insert(operation);
return operation;
}
private static long now() {
return System.currentTimeMillis() / 1000;
}
/** 供装配层解析当前有效 operationId(避免装配代码访问包私有字段)。 */
public String currentOperationId(int gid) {
TaskOperation operation = taskOperationRepository.selectByGid(gid);
return operation == null ? null : operation.getOperationId();
}
}
@@ -54,35 +54,16 @@ public class QueryService {
} }
Document parse = Jsoup.parse(result); Document parse = Jsoup.parse(result);
Elements elements = parse.select("body > div.ido > div:nth-child(2) > table > tbody > tr"); java.util.List<GalleryForQuery> galleries;
ArrayList<GalleryForQuery> galleries = new ArrayList<>();
ListIterator<Element> elementListIterator = elements.listIterator();
try { try {
elementListIterator.next(); //skip firstlink galleries = com.lion.lionwebsite.Util.GallerySearchParser.parse(parse);
}catch (NullPointerException | NoSuchElementException e){ } catch (IOException | RuntimeException error) {
response.failure("没有搜索到结果"); log.warn("上游搜索页面格式不支持: {}", error.getClass().getSimpleName());
return response.toJSONString(); return response.failure("上游搜索页面格式不支持").toJSONString();
} }
if (galleries.isEmpty()) return response.failure("没有搜索到结果").toJSONString();
while (elementListIterator.hasNext()){ response.success(objectMapper.valueToTree(galleries));
Element element = elementListIterator.next();
GalleryForQuery gallery = new GalleryForQuery();
gallery.setType(element.child(0).text()); //type
String src = element.child(1).select("img").attr("src");
if (src.startsWith("data"))
src = element.child(1).select("img").attr("data-src");
src = src.replace("https://s.exhentai.org", "");
gallery.setThumbnailUrl(src); //thumbnailSrc
gallery.setUploadTime(element.child(1).select("div > div [onclick]").get(1).text()); //uploadTime
gallery.setLink(element.child(2).child(0).attr("href")); //link
gallery.setGid(GalleryUtil.parseGid(gallery.getLink()) + ""); //gid
gallery.setName(element.child(2).child(0).child(0).text()); //name
gallery.setPage(Integer.parseInt(element.child(3).child(1).text().split(" ")[0])); //page
galleries.add(gallery);
}
response.success(objectMapper.valueToTree(galleries).toString());
Elements nextLink = parse.select("#unext"); Elements nextLink = parse.select("#unext");
if(nextLink.hasAttr("href")) if(nextLink.hasAttr("href"))
response.set("next", nextLink.attr("href")); response.set("next", nextLink.attr("href"));
@@ -37,6 +37,7 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException; import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicBoolean;
import java.util.List;
@Service @Service
@Data @Data
@@ -76,12 +77,131 @@ public class RemoteService {
final SubscriptionStandbySnapshotService subscriptionStandbySnapshotService; final SubscriptionStandbySnapshotService subscriptionStandbySnapshotService;
/**
* 协议 v2 上报处理器。用 setter 注入而不是构造参数,避免 RemoteService 与
* GalleryTaskLifecycleService 之间的依赖环(后者需要 RemoteService 下发命令)。
*/
volatile TaskReportHandler taskReportHandler;
/**
* gid → 当前有效的下载 operationId。由生命周期服务在装配期注册(保持依赖方向单向)。
* 未注册或返回 null 时,重投/重试退化为旧的无 operationId 命令。
*/
volatile java.util.function.IntFunction<String> downloadOperationIdResolver;
/**
* gid → 下发判定(是否允许下发 + 用哪个稳定 operationId)。
*
* <p>替代「能解析出 operationId 就下发」的旧语义:UNKNOWN / 预占中 / 待删除的任务
* 必须被明确拒绝,否则未计量的人工对账任务可能在节点侧变成「下载完成却 0 额度」。
*/
volatile java.util.function.IntFunction<GalleryTaskLifecycleService.DispatchDecision> dispatchResolver;
/**
* 生命周期装配是否已经完成(handler/resolver 已绑定 + 启动恢复已执行)。
*
* <p>在它为 true 之前,绝不自动重投未完成任务:启动早期还不知道哪些任务是
* SUBMITTED、哪些是 UNKNOWN,盲目下发会制造未计量副作用。
*/
volatile boolean lifecycleReady;
/**
* 启动恢复是否安全完成。恢复失败时拒绝自动重投,避免在状态不明的库上继续派发。
*/
volatile boolean recoverySafe;
/**
* 节点是否声明了 v2 可靠任务能力。只有协商成功之后才发送类型 11..13,
* 否则继续走旧的无确认上报,保持向后兼容。
*/
volatile boolean v2Negotiated;
/** 本次连接的 Hello 协商结果;收到节点 Hello(或超时)后完成。 */
volatile CompletableFuture<Boolean> helloNegotiation;
/** 明确「未下发」的结果码:与 -1(发送失败/超时)区分开。 */
public static final byte REFUSED = -3;
/** 事务提交之后的通知/推送执行器:外部延迟(Telegram、慢浏览器)不得阻塞状态提交。 */
final ExecutorService notificationExecutor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "gallery-notify");
thread.setDaemon(true);
return thread;
});
/**
* 协议握手的专用线程。hello 等待响应是阻塞调用,绝不能放在 Netty 事件循环上,
* 否则响应本身要由同一个事件循环投递,会直接死锁。
*/
final ExecutorService handshakeExecutor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "node-protocol-handshake");
thread.setDaemon(true);
return thread;
});
final ExecutorService subscriptionSyncExecutor = Executors.newSingleThreadExecutor(r -> { final ExecutorService subscriptionSyncExecutor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "subscription-standby-sync"); Thread thread = new Thread(r, "subscription-standby-sync");
thread.setDaemon(true); thread.setDaemon(true);
return thread; return thread;
}); });
/** 测试/装配用:绑定上报处理器。 */
public void setTaskReportHandler(TaskReportHandler taskReportHandler) {
this.taskReportHandler = taskReportHandler;
}
public void setDownloadOperationIdResolver(java.util.function.IntFunction<String> resolver) {
this.downloadOperationIdResolver = resolver;
}
public void setDispatchResolver(java.util.function.IntFunction<GalleryTaskLifecycleService.DispatchDecision> resolver) {
this.dispatchResolver = resolver;
}
/**
* 由装配层在「绑定 handler/resolver + 完成启动恢复」之后调用:标记可以安全派发并建立连接。
*
* <p>初始连接不再放在 {@code @PostConstruct}:那时 handler/resolver 都还没绑定,
* 连接成功会把尚未确认的 UNKNOWN/预占任务当作普通任务重投。
*
* @param recoverySafe 启动恢复是否成功;false 时只允许连接,不允许自动重投
*/
public void startAfterWiring(boolean recoverySafe) {
this.recoverySafe = recoverySafe;
this.lifecycleReady = true;
initChannel();
}
/** 把 gid 解析成下发判定;未注册解析器或解析失败时按「拒绝下发」处理。 */
private GalleryTaskLifecycleService.DispatchDecision resolveDispatchDecision(int gid) {
java.util.function.IntFunction<GalleryTaskLifecycleService.DispatchDecision> resolver = dispatchResolver;
if (resolver == null)
return null;
try {
return resolver.apply(gid);
} catch (Exception e) {
log.warn("解析任务下发判定失败 gid={}", gid, e);
return null;
}
}
private String resolveDownloadOperationId(int gid) {
java.util.function.IntFunction<String> resolver = downloadOperationIdResolver;
if (resolver == null)
return null;
try {
return resolver.apply(gid);
} catch (Exception e) {
log.warn("解析任务 operationId 失败 gid={}", gid, e);
return null;
}
}
/** 节点是否已完成 v2 能力协商(供健康检查/测试观察)。 */
public boolean isReliableProtocolReady() {
return v2Negotiated;
}
/** /**
* 处理节点上报的任务状态。原先这些 JDBC 读写直接跑在 Netty 的 IO 线程上, * 处理节点上报的任务状态。原先这些 JDBC 读写直接跑在 Netty 的 IO 线程上,
* N 条上报会阻塞该 IO 线程,连带拖慢心跳与请求响应;改为单线程顺序执行, * N 条上报会阻塞该 IO 线程,连带拖慢心跳与请求响应;改为单线程顺序执行,
@@ -120,10 +240,14 @@ public class RemoteService {
@Value("${subscription.standby.sync-enabled:false}") @Value("${subscription.standby.sync-enabled:false}")
boolean subscriptionSyncEnabled; boolean subscriptionSyncEnabled;
@PostConstruct /**
void init() { * 这里刻意不再自动连接。
initChannel(); *
} * <p>原实现在 PostConstruct 里立即 initChannel(),而那一刻 TaskLifecycleWiring
* 还没绑定上报处理器、operationId 解析器,也没执行启动恢复:连接成功后会马上把
* 未完成任务(含 UNKNOWN/预占)当作普通任务重投给节点。
* 现在改为由装配层在完成绑定与恢复后调用 startAfterWiring(boolean)。
*/
public boolean initChannel(){ public boolean initChannel(){
if (stopping || !connecting.compareAndSet(false, true)) if (stopping || !connecting.compareAndSet(false, true))
@@ -175,12 +299,32 @@ public class RemoteService {
channel = channelFuture.channel(); channel = channelFuture.channel();
closeMonitorSocket(); closeMonitorSocket();
channel.writeAndFlush(new IdentityMessage("lionwebsite")); channel.writeAndFlush(new IdentityMessage("lionwebsite"));
// 协议 v2:先发旧的 Identity,再发 hello 做能力协商。
//子节点上线时,发送未完成的任务 // 协商完成之前不允许下发任何任务,也不允许自动重投未完成任务。
resetUndone(); CompletableFuture<Boolean> negotiation = queueHelloHandshake();
// 节点刚上线时可能没有任何快照(例如刚重启),必须推一次; // 在协商线程上等待结果,再决定重投;绝不阻塞 Netty 事件循环。
// 这同时会刷新节点「最近收到快照」的时刻,避免内容未变时被判过期。 handshakeExecutor.execute(() -> {
boolean reliable = awaitNegotiation(negotiation);
// 订阅快照与任务重投都要求生命周期装配已完成;装配前只保持连接。
requestSubscriptionSync(); requestSubscriptionSync();
if (!reliable) {
log.info("节点未确认 v2 能力,跳过自动重投未完成任务");
return;
}
if (!lifecycleReady) {
log.info("生命周期装配未就绪,跳过自动重投未完成任务");
return;
}
if (!recoverySafe) {
log.error("启动恢复未安全完成,拒绝自动重投未完成任务");
return;
}
try {
resetUndone();
} catch (RuntimeException e) {
log.warn("重投未完成任务失败", e);
}
});
return true; return true;
}catch (Exception e){ }catch (Exception e){
log.error("connect node failed, wait for node back online", e); log.error("connect node failed, wait for node back online", e);
@@ -297,6 +441,8 @@ public class RemoteService {
failPendingRequests(); failPendingRequests();
if (channel != null) channel.close(); if (channel != null) channel.close();
statusApplyExecutor.shutdownNow(); statusApplyExecutor.shutdownNow();
notificationExecutor.shutdownNow();
handshakeExecutor.shutdownNow();
subscriptionSyncExecutor.shutdownNow(); subscriptionSyncExecutor.shutdownNow();
networkGroup.shutdownGracefully(); networkGroup.shutdownGracefully();
eventLoopGroup.shutdownGracefully(); eventLoopGroup.shutdownGracefully();
@@ -315,7 +461,11 @@ public class RemoteService {
} }
public void resetUndone(){ public void resetUndone(){
if (channelFuture.channel() == null || !channelFuture.channel().isActive()) // 只有「连接在线 + v2 已协商 + 生命周期装配完成 + 启动恢复安全完成」才允许批量重投;
// 任何一项不满足都说明还不知道任务真实状态,盲目下发会制造未计量副作用。
if (channelFuture == null || channelFuture.channel() == null || !channelFuture.channel().isActive())
return;
if (!v2Negotiated || !lifecycleReady || !recoverySafe)
return; return;
Gallery[] galleries = galleryMapper.selectUnDoneGalleries(); Gallery[] galleries = galleryMapper.selectUnDoneGalleries();
@@ -328,15 +478,135 @@ public class RemoteService {
} }
public byte addGalleryToQueue(Gallery gallery){ public byte addGalleryToQueue(Gallery gallery){
GalleryTaskLifecycleService.DispatchDecision decision = resolveDispatchDecision(gallery.getGid());
if (decision != null) {
// 明确拒绝(UNKNOWN/预占/待删除等):绝不能退化为无 operationId 的旧命令。
if (!decision.dispatchable())
return REFUSED;
return sendTaskCommand(TaskCommandMessage.ACTION_DOWNLOAD, decision.operationId(),
gallery.getGid(), gallery.getName());
}
// 兼容:装配层未注册判定解析器时(如单元测试),退回只按 operationId 解析的旧行为。
String operationId = resolveDownloadOperationId(gallery.getGid());
if (operationId != null)
return sendTaskCommand(TaskCommandMessage.ACTION_DOWNLOAD, operationId, gallery.getGid(), gallery.getName());
return legacyDownloadCommand(gallery.getGid(), gallery.getName());
}
/** 旧的无 operationId 下载命令,供未启用 v2 或无法解析 operationId 时退避使用。 */
private byte legacyDownloadCommand(int gid, String name) {
GalleryTask galleryTask = new GalleryTask(); GalleryTask galleryTask = new GalleryTask();
galleryTask.setGid(gallery.getGid()); galleryTask.setGid(gid);
galleryTask.setName(gallery.getName()); galleryTask.setName(name);
DownloadPostMessage message = new DownloadPostMessage(); DownloadPostMessage message = new DownloadPostMessage();
message.setGalleryTask(galleryTask); message.setGalleryTask(galleryTask);
return sendRequest(message, 10, TimeUnit.SECONDS); return sendRequest(message, 10, TimeUnit.SECONDS);
} }
/**
* 在专用线程上执行 hello 协商,并返回一个在「收到节点 Hello」或「等待超时」后完成的 future。
*
* <p>节点的应答是 Hello 而不是 ResponseMessage,因此绝不能走 sendRequest 的响应等待器
* (那会白等 10 秒超时,排在后面的自动重投也只能 10 秒后才开始)。
* 这里直接 writeAndFlush,并让 channelRead 的 Hello 分支完成 future;
* 超时用 completeOnTimeout 处理,写失败立刻完成。
*/
private CompletableFuture<Boolean> queueHelloHandshake() {
// 10 秒内没等到节点 Hello 就按「未协商」完成,避免后续重投被无限期挂起。
CompletableFuture<Boolean> result = new CompletableFuture<Boolean>()
.completeOnTimeout(false, 10, TimeUnit.SECONDS);
helloNegotiation = result;
if (stopping) {
result.complete(false);
return result;
}
try {
handshakeExecutor.execute(() -> {
v2Negotiated = false;
try {
Channel target = channel;
if (target == null || !target.isActive()) {
result.complete(false);
return;
}
// Hello 不写入 promiseHashMap:节点回的是 Hello,没有 ResponseMessage 等待者。
target.writeAndFlush(new ProtocolHelloMessage()).addListener(future -> {
if (!future.isSuccess()) {
if (!stopping)
log.warn("协议协商发送失败: {}", future.cause() == null
? "unknown" : future.cause().getClass().getSimpleName());
result.complete(false);
}
});
} catch (RuntimeException e) {
log.warn("协议协商失败", e);
result.complete(false);
}
});
} catch (java.util.concurrent.RejectedExecutionException e) {
if (!stopping) log.warn("安排协议协商失败", e);
result.complete(false);
}
return result;
}
/** 等待协商结果;超时或中断都按「未协商」处理(宁可不下发,也不盲目重投)。 */
boolean awaitNegotiation(CompletableFuture<Boolean> negotiation) {
if (negotiation == null)
return v2Negotiated;
try {
// future 自身已在 10 秒时按「未协商」完成,这里留少量余量即可。
return negotiation.get(12, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return false;
} catch (Exception e) {
return false;
}
}
/**
* 下发一条带稳定 operationId 的任务命令;未协商 v2 时退化为旧的 DownloadPostMessage,
* 保证新旧节点组合仍可用(旧节点上无法保证可靠 ACK,这是已知的兼容边界)。
*/
public byte sendTaskCommand(String action, String operationId, int gid, String name) {
if (!v2Negotiated)
return TaskCommandMessage.ACTION_DELETE.equals(action)
? legacyDeleteCommand(gid, name)
: legacyDownloadCommand(gid, name);
TaskCommandMessage command = new TaskCommandMessage();
command.setOperationId(operationId);
command.setAction(action);
command.setGid(gid);
command.setName(name);
return sendRequest(command, 10, TimeUnit.SECONDS);
}
/** 删除命令:带独立稳定 operationId,重试复用同一个 ID。 */
public byte sendDeleteCommand(String operationId, int gid, String name) {
if (!v2Negotiated)
return legacyDeleteCommand(gid, name);
TaskCommandMessage command = new TaskCommandMessage();
command.setOperationId(operationId);
command.setAction(TaskCommandMessage.ACTION_DELETE);
command.setGid(gid);
command.setName(name);
return sendRequest(command, 10, TimeUnit.SECONDS);
}
/** 旧的无 operationId 删除命令,供未协商 v2 时退避使用。 */
private byte legacyDeleteCommand(int gid, String name) {
Gallery gallery = new Gallery();
gallery.setGid(gid);
gallery.setName(name);
return deleteGallery(gallery);
}
public RetryResult retryGallery(Gallery gallery){ public RetryResult retryGallery(Gallery gallery){
// 提交结果未知或被删除的任务不能重试:节点可能已经把未计量任务当成新下载。
GalleryTaskLifecycleService.DispatchDecision decision = resolveDispatchDecision(gallery.getGid());
if (decision != null && !decision.dispatchable())
return new RetryResult(false, "该任务提交结果未知或正在删除,需要人工对账后再重试");
CompletableFuture<String> statusFuture = new CompletableFuture<>(); CompletableFuture<String> statusFuture = new CompletableFuture<>();
retryStatusWaiters.computeIfAbsent(gallery.getGid(), ignored -> new CopyOnWriteArrayList<>()) retryStatusWaiters.computeIfAbsent(gallery.getGid(), ignored -> new CopyOnWriteArrayList<>())
.add(statusFuture); .add(statusFuture);
@@ -344,6 +614,13 @@ public class RemoteService {
byte submitResult = addGalleryToQueue(gallery); byte submitResult = addGalleryToQueue(gallery);
if(submitResult != 0 && !statusFuture.isDone()) if(submitResult != 0 && !statusFuture.isDone())
return new RetryResult(false, "节点未接受重试请求"); return new RetryResult(false, "节点未接受重试请求");
// 竞态兜底:本方法读到的 Gallery 是调用方传入的旧快照,注册等待者与节点被通知之间
// 节点可能刚完成清理(例如 ACK 之后已清队列、重复 DOWNLOAD 只回放 receipt 不再产生新上报)。
// 此时按「当前库里的真实状态」收尾:只认重新读到的完成状态,绝不把传入的旧进行中状态
// 当作节点的新确认返回。
String completedStatus = currentCompletedStatus(gallery.getGid());
if (completedStatus != null && !statusFuture.isDone())
statusFuture.complete(completedStatus);
return new RetryResult(true, statusFuture.get(10, TimeUnit.SECONDS)); return new RetryResult(true, statusFuture.get(10, TimeUnit.SECONDS));
}catch (TimeoutException e){ }catch (TimeoutException e){
return new RetryResult(false, "节点已收到重试请求,但未及时返回任务状态"); return new RetryResult(false, "节点已收到重试请求,但未及时返回任务状态");
@@ -368,6 +645,22 @@ public class RemoteService {
waiters.forEach(waiter -> waiter.complete(status)); waiters.forEach(waiter -> waiter.complete(status));
} }
/**
* 重新读库确认任务是否已真正完成;只用于「节点已接受重试」之后的竞态兜底。
*
* <p>读库失败一律返回 null:宁可继续等上报,也不能凭一次失败的读把未知状态说成完成。
*/
private String currentCompletedStatus(int gid) {
try {
Gallery current = galleryMapper.selectGalleryByGid(gid);
if (current != null && TaskStatus.COMPLETE.equals(current.getStatus()))
return current.getStatus();
} catch (RuntimeException e) {
log.warn("重试时读取任务当前状态失败 gid={}", gid, e);
}
return null;
}
public record RetryResult(boolean success, String message) {} public record RetryResult(boolean success, String message) {}
public byte deleteGallery(Gallery gallery){ public byte deleteGallery(Gallery gallery){
@@ -447,6 +740,18 @@ public class RemoteService {
} }
}); });
} }
else if(msg instanceof ProtocolHelloMessage hello) {
// 节点声明了 v2 可靠任务能力;此后才允许使用类型 11..13。
v2Negotiated = hello.supportsReliableTasks();
log.info("节点协议协商完成 v2={} version={}", v2Negotiated, hello.getProtocolVersion());
CompletableFuture<Boolean> negotiation = helloNegotiation;
if (negotiation != null)
negotiation.complete(v2Negotiated);
}
else if(msg instanceof TaskReportMessage report) {
// 事务提交与 ACK 都放在单线程消费队列里顺序执行,避免并发写库。
statusApplyExecutor.execute(() -> consumeV2Report(report));
}
else if(msg instanceof ResponseMessage rsm) { else if(msg instanceof ResponseMessage rsm) {
Promise<AbstractMessage> promise = promiseHashMap.remove(rsm.messageId); Promise<AbstractMessage> promise = promiseHashMap.remove(rsm.messageId);
if(promise != null) if(promise != null)
@@ -456,6 +761,71 @@ public class RemoteService {
} }
} }
/** 消费协议 v2 上报:事务提交 → 唤醒重试等待者 → 回 ACK → 独立执行器做通知/推送。 */
private void consumeV2Report(TaskReportMessage report) {
try {
TaskReportHandler handler = taskReportHandler;
if (handler == null) {
log.warn("收到 v2 上报但未装配处理器,暂不回执 reportId={}", report.getReportId());
return;
}
TaskReportHandler.CommittedReport committed = handler.apply(report);
if (committed == null) {
// 提交失败:不回 ACK,节点保留快照后重投。
log.warn("v2 上报未提交,等待节点重投 reportId={}", report.getReportId());
return;
}
// 先完成等待者:apply 已确认提交,这一步绝不能因随后的 ACK 写入抛异常而丢失。
completeRetryStatusWaitersFor(committed.acceptedTasks());
// 提交成功之后才回执,让节点清理快照。
Channel target = channel;
if (target != null && target.isActive())
target.writeAndFlush(new TaskReportAckMessage(committed.reportId(), TaskReportAckMessage.RESULT_SUCCESS));
dispatchNotifications(report, committed);
} catch (Exception e) {
log.warn("处理 v2 上报失败 reportId={}", report.getReportId(), e);
}
}
/** 事务提交之后、ACK 与通知之前,按「真正被接受的更新」唤醒等待重试结果的调用方。
* 被过滤的过期/旧 operation/待删除任务不在 acceptedTasks 里,不得完成等待者。 */
private void completeRetryStatusWaitersFor(List<GalleryTask> acceptedTasks) {
if (acceptedTasks == null || acceptedTasks.isEmpty())
return;
for (GalleryTask task : acceptedTasks) {
if (task == null)
continue;
String status = TaskStatus.fromReport(task.getStatus());
if (status != null)
completeRetryStatusWaiters(task.getGid(), status);
}
}
/** 通知/推送在独立执行器里执行,外部延迟不会阻塞状态提交。 */
private void dispatchNotifications(TaskReportMessage report, TaskReportHandler.CommittedReport committed) {
// 只推送事务里真正被接受的更新(含下载中/压缩中等中间状态);
// 过期、旧世代、待删除任务已被过滤掉,不会把它们带到前端。
GalleryTask[] accepted = committed.acceptedTasks() == null
? new GalleryTask[0]
: committed.acceptedTasks().toArray(new GalleryTask[0]);
List<Gallery> newlyCompleted = committed.newlyCompleted();
try {
notificationExecutor.execute(() -> {
try {
for (Gallery gallery : newlyCompleted)
pushService.downloadComplete(gallery);
if (accepted.length > 0)
webSocketService.updateTaskProcessing(accepted);
} catch (Exception e) {
log.warn("任务完成通知/推送失败", e);
}
});
} catch (java.util.concurrent.RejectedExecutionException e) {
// 通知队列已关闭只是「推不出去」,既不能影响已提交的 ACK,也不能影响重试等待者。
log.warn("通知任务被拒绝,状态仍已提交 reportId={}", report.getReportId());
}
}
/** 把节点上报的任务状态写回库里并推送给前端,按上报顺序逐个处理。 */ /** 把节点上报的任务状态写回库里并推送给前端,按上报顺序逐个处理。 */
private void applyReportedStatus(GalleryTask[] galleryTasks) { private void applyReportedStatus(GalleryTask[] galleryTasks) {
for (GalleryTask galleryTask : galleryTasks) { for (GalleryTask galleryTask : galleryTasks) {
@@ -5,6 +5,7 @@ import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper; import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.SubBind; import com.lion.lionwebsite.Domain.SubBind;
import com.lion.lionwebsite.Domain.SubUpdateRecord; import com.lion.lionwebsite.Domain.SubUpdateRecord;
import com.lion.lionwebsite.Domain.SubscriptionEvent;
import com.lion.lionwebsite.Domain.SubscriptionAccount; import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.FileDownload; import com.lion.lionwebsite.Util.FileDownload;
@@ -17,6 +18,10 @@ import lombok.extern.slf4j.Slf4j;
import org.jsoup.Jsoup; import org.jsoup.Jsoup;
import org.jsoup.select.Elements; import org.jsoup.select.Elements;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.transaction.support.TransactionTemplate;
import jakarta.annotation.PreDestroy;
import java.io.IOException; import java.io.IOException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
@@ -39,6 +44,13 @@ import java.util.function.Supplier;
final RemoteService remoteService; final RemoteService remoteService;
final SubscriptionStateCoordinator stateCoordinator; final SubscriptionStateCoordinator stateCoordinator;
@Autowired
@Qualifier("mainTransactionTemplate")
TransactionTemplate transactions;
@Autowired(required = false)
SubscriptionEventService subscriptionEvents;
/** /**
* 归属地查询是同步外呼(最长十几秒),且失败只影响记录里的一列展示。 * 归属地查询是同步外呼(最长十几秒),且失败只影响记录里的一列展示。
* 若放在订阅分发路径上,用户取一次订阅就得先等它完成,因此改为后台补齐。 * 若放在订阅分发路径上,用户取一次订阅就得先等它完成,因此改为后台补齐。
@@ -48,11 +60,12 @@ import java.util.function.Supplier;
Executor locationExecutor = newFixedLocationExecutor(); Executor locationExecutor = newFixedLocationExecutor();
private static ExecutorService newFixedLocationExecutor() { private static ExecutorService newFixedLocationExecutor() {
return Executors.newFixedThreadPool(2, r -> { return new java.util.concurrent.ThreadPoolExecutor(2, 2, 0, java.util.concurrent.TimeUnit.MILLISECONDS,
new java.util.concurrent.ArrayBlockingQueue<>(128), r -> {
Thread thread = new Thread(r, "sub-location-resolver"); Thread thread = new Thread(r, "sub-location-resolver");
thread.setDaemon(true); thread.setDaemon(true);
return thread; return thread;
}); }, new java.util.concurrent.ThreadPoolExecutor.AbortPolicy());
} }
public String insertSubscriptionAccount(String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) { public String insertSubscriptionAccount(String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) {
@@ -179,6 +192,7 @@ import java.util.function.Supplier;
String key = RandomUtil.randomString(8); String key = RandomUtil.randomString(8);
while (subMapper.selectSubBindExist(key)) while (subMapper.selectSubBindExist(key))
key = RandomUtil.randomString(8); key = RandomUtil.randomString(8);
rememberEventBindings();
subMapper.updateSubBindKey(user, key); subMapper.updateSubBindKey(user, key);
subMapper.deleteSubUpdateRecord(user); subMapper.deleteSubUpdateRecord(user);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
@@ -211,6 +225,16 @@ import java.util.function.Supplier;
} }
public void updateSub(HttpServletResponse response, HttpServletRequest request, String client, String key) { public void updateSub(HttpServletResponse response, HttpServletRequest request, String client, String key) {
Lock lock = stateCoordinator.readLock();
lock.lock();
try {
distributeSubscription(response, request, client, key);
} finally {
lock.unlock();
}
}
private void distributeSubscription(HttpServletResponse response, HttpServletRequest request, String client, String key) {
if (key == null || client == null) if (key == null || client == null)
return; return;
SubBind subBind = subMapper.selectSubBind(key); SubBind subBind = subMapper.selectSubBind(key);
@@ -220,8 +244,9 @@ import java.util.function.Supplier;
} }
String ip = resolveClientIp(request); String ip = resolveClientIp(request);
String ua = request.getHeader("User-Agent"); String ua = request.getHeader("User-Agent");
if (ua == null) return; SubscriptionEvent event = subscriptionEvents == null ? null : subscriptionEvents.requestEvent(request, key, client);
recordUpdate(subBind.getUser(), ip, ua); if (ua == null && event == null) return;
if (event == null) recordUpdate(subBind, ip, ua);
if (!"v2".equals(client) && !"cat".equals(client)) { if (!"v2".equals(client) && !"cat".equals(client)) {
sendStatus(response, HttpServletResponse.SC_BAD_REQUEST, "client error"); sendStatus(response, HttpServletResponse.SC_BAD_REQUEST, "client error");
return; return;
@@ -231,6 +256,8 @@ import java.util.function.Supplier;
sendStatus(response, HttpServletResponse.SC_SERVICE_UNAVAILABLE, "subscription unavailable"); sendStatus(response, HttpServletResponse.SC_SERVICE_UNAVAILABLE, "subscription unavailable");
return; return;
} }
if (event != null && !"HEAD".equalsIgnoreCase(request.getMethod()))
subscriptionEvents.recordPrimary(event, subBind);
FileDownload.export(request, response, path.toString()); FileDownload.export(request, response, path.toString());
} }
@@ -259,24 +286,34 @@ import java.util.function.Supplier;
|| "0:0:0:0:0:0:0:1".equals(address); || "0:0:0:0:0:0:0:1".equals(address);
} }
private void recordUpdate(String user, String ip, String ua) { private void recordUpdate(SubBind binding, String ip, String ua) {
String user = binding.getUser();
// 先落一条 unknown 记录并立即返回,归属地随后在后台补齐; // 先落一条 unknown 记录并立即返回,归属地随后在后台补齐;
// 这样取订阅的延迟不再取决于第三方定位站点的响应时间。 // 这样取订阅的延迟不再取决于第三方定位站点的响应时间。
subMapper.insertSubUpdateRecord(new SubUpdateRecord(0, user, ip, ua, new Date(), "unknown")); SubUpdateRecord record = new SubUpdateRecord(0, user, ip, ua, new Date(), "unknown");
record.setUserId(binding.getUserId());
subMapper.insertSubUpdateRecord(record);
if (subMapper.selectUpdateRecordCount(user) > 10) if (subMapper.selectUpdateRecordCount(user) > 10)
subMapper.deleteSubUpdateRecordById(subMapper.selectMinUpdateRecordId(user)); subMapper.deleteSubUpdateRecordById(subMapper.selectMinUpdateRecordId(user));
locationExecutor.execute(() -> enrichLocation(user, ip)); // Capture the generated row ID: another request may arrive before the lookup finishes.
if (record.getId() > 0) {
try {
locationExecutor.execute(() -> enrichLocation(record.getId(), ip));
} catch (java.util.concurrent.RejectedExecutionException busy) {
log.debug("位置查询队列已满,保留 unknown recordId={}", record.getId());
}
}
} }
/** 后台补齐最近一条记录的归属地;任何失败都只记日志,不影响已落库的记录。 */ /** 后台按记录 ID 补齐归属地;任何失败都只记日志,不影响已落库的记录。 */
private void enrichLocation(String user, String ip) { private void enrichLocation(int recordId, String ip) {
try { try {
String location = resolveLocation(ip); String location = resolveLocation(ip);
if (location == null || location.isBlank() || "unknown".equals(location)) if (location == null || location.isBlank() || "unknown".equals(location))
return; return;
subMapper.updateLatestSubUpdateRecordLocation(user, location); subMapper.updateSubUpdateRecordLocation(recordId, location);
} catch (Exception e) { } catch (Exception e) {
log.debug("补齐订阅更新记录归属地失败 user={}", user, e); log.debug("补齐订阅更新记录归属地失败 recordId={}", recordId, e);
} }
} }
@@ -300,30 +337,41 @@ import java.util.function.Supplier;
} }
private String deleteSubBindUnlocked(String user) { private String deleteSubBindUnlocked(String user) {
rememberEventBindings();
subMapper.deleteSubBind(user); subMapper.deleteSubBind(user);
subMapper.deleteSubUpdateRecord(user); subMapper.deleteSubUpdateRecord(user);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
return Response._success("删除成功"); return Response._success("删除成功");
} }
private void rememberEventBindings() {
if (subscriptionEvents != null) subscriptionEvents.rememberBindings(subMapper.selectAllSubBind());
}
private String withWriteLock(Supplier<String> action) { private String withWriteLock(Supplier<String> action) {
Lock stateLock = stateCoordinator.writeLock(); Lock stateLock = stateCoordinator.writeLock();
stateLock.lock(); stateLock.lock();
try { try {
return action.get(); return transactions == null ? action.get() : transactions.execute(ignored -> action.get());
} finally { } finally {
stateLock.unlock(); stateLock.unlock();
} }
} }
@PreDestroy
void shutdownLocationResolver() {
if (locationExecutor instanceof ExecutorService service)
service.shutdownNow();
}
private SubscriptionAccount firstEnabledAccount() { private SubscriptionAccount firstEnabledAccount() {
return subMapper.selectAllSubscriptionAccounts().stream().filter(SubscriptionAccount::isEnabled).findFirst().orElse(null); return subMapper.selectAllSubscriptionAccounts().stream().filter(SubscriptionAccount::isEnabled).findFirst().orElse(null);
} }
private String accountJson(SubscriptionAccount account) { private tools.jackson.databind.JsonNode accountJson(SubscriptionAccount account) {
account.setV2Url(refreshService.v2Url(account)); account.setV2Url(refreshService.v2Url(account));
account.setClashUrl(refreshService.clashUrl(account)); account.setClashUrl(refreshService.clashUrl(account));
return CustomUtil.objectMapper.valueToTree(account).toString(); return CustomUtil.objectMapper.valueToTree(account);
} }
private static void sendStatus(HttpServletResponse response, int status, String message) { private static void sendStatus(HttpServletResponse response, int status, String message) {
@@ -0,0 +1,224 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.*;
import com.lion.lionwebsite.Util.CustomUtil;
import jakarta.annotation.PreDestroy;
import jakarta.servlet.http.HttpServletRequest;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionTemplate;
import tools.jackson.databind.JsonNode;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Clock;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.Lock;
@Service
public class SubscriptionEventService {
public static final int MAX_BODY = 256 * 1024;
private final SubscriptionEventMapper events;
private final SubMapper subscriptions;
private final UserMapper users;
private final TransactionTemplate transactions;
private final SubscriptionStateCoordinator coordinator;
@Value("${subscription.standby.sync-secret:}") String syncSecret;
Clock clock = Clock.systemUTC();
private final ExecutorService locations = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS,
new ArrayBlockingQueue<>(128), runnable -> {
Thread thread = new Thread(runnable, "subscription-event-location");
thread.setDaemon(true);
return thread;
}, new ThreadPoolExecutor.AbortPolicy());
public SubscriptionEventService(SubscriptionEventMapper events, SubMapper subscriptions, UserMapper users,
@Qualifier("mainTransactionTemplate") TransactionTemplate transactions,
SubscriptionStateCoordinator coordinator) {
this.events = events;
this.subscriptions = subscriptions;
this.users = users;
this.transactions = transactions;
this.coordinator = coordinator;
}
/** Auth failure deliberately discloses neither signature nor payload. */
public boolean verify(byte[] body, String timestamp, String signature) {
if (body == null || body.length > MAX_BODY || syncSecret == null || syncSecret.isBlank()
|| timestamp == null || !timestamp.matches("[0-9]{1,12}")
|| signature == null || !signature.matches("[0-9a-fA-F]{64}")) return false;
try {
long seconds = Long.parseLong(timestamp);
if (Math.abs(clock.instant().getEpochSecond() - seconds) > 300) return false;
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(syncSecret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
byte[] expected = mac.doFinal(("v1\n" + timestamp + "\n" + sha256(body)).getBytes(StandardCharsets.UTF_8));
return MessageDigest.isEqual(expected, HexFormat.of().parseHex(signature));
} catch (Exception invalid) { return false; }
}
/** Invalid or absent signed headers fall back to the ordinary logging path. */
public SubscriptionEvent requestEvent(HttpServletRequest request, String key, String client) {
String encoded = request.getHeader("X-Subscription-Event");
if (encoded == null || encoded.length() > 8192 || !encoded.matches("[A-Za-z0-9_-]+")) return null;
try {
byte[] body = Base64.getUrlDecoder().decode(encoded);
if (!verify(body, request.getHeader("X-Subscription-Timestamp"), request.getHeader("X-Subscription-Signature"))) return null;
SubscriptionEvent event = parse(CustomUtil.objectMapper.readTree(body));
String ua = Objects.requireNonNullElse(request.getHeader("User-Agent"), "");
if (!event.getKeyHash().equals(sha256(key.getBytes(StandardCharsets.UTF_8)))
|| !event.getClient().equals(client) || !event.getUserAgent().equals(ua)) return null;
return event;
} catch (Exception invalid) { return null; }
}
public List<SubscriptionEvent> parseBatch(byte[] body) {
try {
if (body == null || body.length > MAX_BODY) throw new IllegalArgumentException();
JsonNode root = CustomUtil.objectMapper.readTree(body);
JsonNode list = root.get("events");
if (!root.isObject() || list == null || !list.isArray() || list.size() > 100) throw new IllegalArgumentException();
List<SubscriptionEvent> result = new ArrayList<>();
for (JsonNode item : list) result.add(parse(item));
return result;
} catch (Exception invalid) { throw new IllegalArgumentException("invalid subscription events"); }
}
SubscriptionEvent parse(JsonNode node) {
if (node == null || !node.isObject()) throw new IllegalArgumentException();
String id = text(node, "eventId", 36);
if (!UUID.fromString(id).toString().equals(id)) throw new IllegalArgumentException();
String hash = text(node, "keyHash", 64);
if (!hash.matches("[0-9a-f]{64}")) throw new IllegalArgumentException();
String client = text(node, "client", 3);
if (!client.equals("v2") && !client.equals("cat")) throw new IllegalArgumentException();
String ip = text(node, "clientIp", 64);
// Literal IPv4/IPv6 only: no host names, URL fragments or query injection into geolocation.
if (ip.isEmpty() || !ip.matches("[0-9A-Fa-f:.]+")) throw new IllegalArgumentException();
String ua = text(node, "userAgent", 2048);
if (ua.chars().anyMatch(c -> c < 32 || c == 127)) throw new IllegalArgumentException();
JsonNode time = node.get("occurredAt");
if (time == null || !time.isIntegralNumber() || !time.canConvertToLong()) throw new IllegalArgumentException();
long occurred = time.longValue();
if (occurred < 0 || occurred > clock.millis() + 300_000) throw new IllegalArgumentException();
return new SubscriptionEvent(id, hash, client, ip, ua, occurred);
}
private static String text(JsonNode node, String field, int max) {
JsonNode value = node.get(field);
if (value == null || !value.isString() || value.stringValue().length() > max) throw new IllegalArgumentException();
return value.stringValue();
}
public Map<String, Object> ingest(List<SubscriptionEvent> batch) {
Lock lock = coordinator.readLock();
lock.lock();
try {
// Return acknowledgements only after execute has returned and the entire transaction committed.
List<String> acknowledged = transactions.execute(status -> {
events.reserveWriter();
List<String> ids = new ArrayList<>();
for (SubscriptionEvent event : batch) if (persist(event, "standby", null)) ids.add(event.getEventId());
return ids;
});
return Map.of("acknowledged", Objects.requireNonNull(acknowledged));
} finally { lock.unlock(); }
}
public void recordPrimary(SubscriptionEvent event, SubBind binding) {
transactions.executeWithoutResult(status -> {
events.reserveWriter();
if (!persist(event, "primary", binding)) throw new IllegalStateException("subscription user unavailable");
});
}
private boolean persist(SubscriptionEvent event, String source, SubBind current) {
String fingerprint = sha256((event.getEventId() + "\n" + event.getKeyHash() + "\n" + event.getClient()
+ "\n" + event.getOccurredAt() + "\n" + event.getClientIp() + "\n" + event.getUserAgent()).getBytes(StandardCharsets.UTF_8));
String old = events.fingerprint(event.getEventId());
if (old != null) {
if (!old.equals(fingerprint)) throw new IllegalArgumentException("event ID conflict");
return true;
}
Integer userId = current == null ? null : current.getUserId();
String userName = current == null ? null : current.getUser();
if (current == null) {
boolean found = false;
for (SubBind binding : subscriptions.selectAllSubBind()) {
if (sha256(binding.getKey().getBytes(StandardCharsets.UTF_8)).equals(event.getKeyHash())) {
userId = binding.getUserId();
userName = binding.getUser();
found = true;
break;
}
}
if (!found) {
SubscriptionEventBinding binding = events.binding(event.getKeyHash());
if (binding == null) return false;
userId = binding.getUserId();
userName = binding.getUserName();
}
}
if (userId != null) {
User user = users.selectUserById(userId);
if (user == null) {
events.insertReceipt(event, source, clock.millis(), fingerprint, "user_deleted");
return true;
}
userName = user.getUsername();
}
// NULL user_id is a valid legacy subscription label, not a missing/deleted User.
// Keep it NULL even if someone later creates a User with the same label.
if (userName == null || userName.isBlank()) return false;
events.rememberBinding(event.getKeyHash(), userId, userName, clock.millis());
SubUpdateRecord record = new SubUpdateRecord(0, userName, event.getClientIp(), event.getUserAgent(),
new Date(event.getOccurredAt()), "unknown", userId);
events.insertRecord(record);
events.pruneRecords(userId, userName);
events.insertReceipt(event, source, clock.millis(), fingerprint, "recorded");
// Never perform network calls before commit. A dropped/rolled-back record has no enrichment task.
org.springframework.transaction.support.TransactionSynchronizationManager.registerSynchronization(
new org.springframework.transaction.support.TransactionSynchronization() {
@Override public void afterCommit() { enrich(record.getId(), event.getClientIp()); }
});
return true;
}
public void rememberBindings(Collection<SubBind> bindings) {
for (SubBind binding : bindings) {
Integer userId = binding.getUserId();
String name = binding.getUser();
if (userId != null) {
User user = users.selectUserById(userId);
if (user != null) name = user.getUsername();
}
if (name != null && !name.isBlank()) events.rememberBinding(
sha256(binding.getKey().getBytes(StandardCharsets.UTF_8)), userId, name, clock.millis());
}
}
private void enrich(int recordId, String ip) {
if (recordId <= 0) return;
try { locations.execute(() -> {
String location = SubService.resolveLocation(ip);
if (location != null && !location.isBlank() && !location.equals("unknown")) {
try { subscriptions.updateSubUpdateRecordLocation(recordId, location); }
catch (Exception ignored) { /* Optional display enrichment; no request failure or sensitive log. */ }
}
}); } catch (RejectedExecutionException ignored) { }
}
public static String sha256(byte[] body) {
try { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(body)); }
catch (Exception failure) { throw new IllegalStateException("SHA-256 unavailable"); }
}
@PreDestroy void close() { locations.shutdownNow(); }
}
@@ -2,7 +2,9 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.SubMapper; import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.SubscriptionAccount; import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Util.ClashSubscriptionParser;
import com.lion.lionwebsite.Util.SubscriptionClientProfile; import com.lion.lionwebsite.Util.SubscriptionClientProfile;
import com.lion.lionwebsite.Util.SubscriptionCacheStore;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.hc.client5.http.classic.methods.HttpGet; import org.apache.hc.client5.http.classic.methods.HttpGet;
@@ -57,6 +59,12 @@ public class SubscriptionRefreshService {
/** 时间源,测试可替换以固定成功时刻。 */ /** 时间源,测试可替换以固定成功时刻。 */
Clock clock = Clock.systemDefaultZone(); Clock clock = Clock.systemDefaultZone();
/**
* 分代缓存存储。默认按 {@link #cacheRoot} 懒构造;测试可覆写为受控子类来模拟写入失败。
* 保持为普通字段而不是构造参数,是为了不改动既有两参构造器(Spring 与老测试都在用)。
*/
SubscriptionCacheStore cacheStore;
private long refreshSequence; private long refreshSequence;
// Accessed only while holding the coordinator write lock. // Accessed only while holding the coordinator write lock.
private final Map<Integer, Long> latestRefresh = new HashMap<>(); private final Map<Integer, Long> latestRefresh = new HashMap<>();
@@ -78,18 +86,19 @@ public class SubscriptionRefreshService {
// Network access and parsing never hold the shared subscription lock. // Network access and parsing never hold the shared subscription lock.
try { try {
boolean filter = account.isFilterHighMultiplier();
String v2 = processV2(firstLine(download(v2Url(account), SubscriptionClientProfile.forAccount(accountId, false))), String v2 = processV2(firstLine(download(v2Url(account), SubscriptionClientProfile.forAccount(accountId, false))),
account.isFilterHighMultiplier(), highMultiplierThreshold); filter, highMultiplierThreshold);
List<String> clash = processClash(download(clashUrl(account), SubscriptionClientProfile.forAccount(accountId, true)), // 按 YAML 语义过滤高倍节点并同步清理 proxy-groups 引用,不再靠行缩进猜测。
account.isFilterHighMultiplier(), highMultiplierThreshold); String clash = ClashSubscriptionParser.transform(
download(clashUrl(account), SubscriptionClientProfile.forAccount(accountId, true)),
filter, highMultiplierThreshold);
stateLock.lock(); stateLock.lock();
try { try {
if (!isCurrent(account, version)) if (!isCurrent(account, version))
return false; return false;
Path dir = Paths.get(cacheRoot, String.valueOf(accountId)); // 两份内容写进同一分代,校验通过后指针最后发布:任一步失败都保留上次完整版本。
Files.createDirectories(dir); store().publish(accountId, v2.getBytes(StandardCharsets.UTF_8), clash.getBytes(StandardCharsets.UTF_8));
atomicWrite(dir.resolve("v2ray.txt"), v2.getBytes(StandardCharsets.UTF_8));
atomicWrite(dir.resolve("clash.yaml"), String.join("\n", clash).concat("\n").getBytes(StandardCharsets.UTF_8));
subMapper.markSubscriptionRefreshSuccess(accountId, clock.millis()); subMapper.markSubscriptionRefreshSuccess(accountId, clock.millis());
return true; return true;
} finally { } finally {
@@ -162,56 +171,6 @@ public class SubscriptionRefreshService {
return URLDecoder.decode(node.substring(hash + 1), StandardCharsets.UTF_8); return URLDecoder.decode(node.substring(hash + 1), StandardCharsets.UTF_8);
} }
private List<String> processClash(List<String> source, boolean filter, double threshold) {
Set<String> removed = new HashSet<>();
List<String> result = new ArrayList<>();
boolean inProxies = false;
boolean skipNode = false;
for (String line : source) {
if (line.equals("proxies:")) {
inProxies = true;
skipNode = false;
result.add(line);
continue;
}
if (line.equals("proxy-groups:")) {
inProxies = false;
skipNode = false;
result.add(line);
continue;
}
if (inProxies && line.matches("^\\s{2}-\\s+name:.*")) {
String name = clashName(line);
skipNode = filter && isHigh(name, threshold);
if (skipNode)
removed.add(name);
else
result.add(line);
continue;
}
if (skipNode)
continue;
if (!inProxies && !removed.isEmpty() && line.trim().startsWith("- ")) {
String ref = line.trim().substring(2).trim();
if (removed.contains(unquote(ref)))
continue;
}
result.add(line);
}
return result;
}
private static String clashName(String line) {
int index = line.indexOf("name:");
return unquote(line.substring(index + 5).trim());
}
private static String unquote(String value) {
if (value.length() >= 2 && ((value.startsWith("\"") && value.endsWith("\"")) || (value.startsWith("'") && value.endsWith("'"))))
return value.substring(1, value.length() - 1);
return value;
}
private static boolean isHigh(String name, double threshold) { private static boolean isHigh(String name, double threshold) {
Matcher matcher = MULTIPLIER.matcher(name); Matcher matcher = MULTIPLIER.matcher(name);
return matcher.find() && Double.parseDouble(matcher.group(1)) > threshold; return matcher.find() && Double.parseDouble(matcher.group(1)) > threshold;
@@ -251,22 +210,17 @@ public class SubscriptionRefreshService {
} }
} }
private static void atomicWrite(Path target, byte[] data) throws IOException { /**
Path temp = target.resolveSibling(target.getFileName() + ".tmp"); * 当前应分发的缓存文件。只在一整个版本有效时返回真实路径;否则返回一个保证不存在的
Files.write(temp, data); * 哨兵路径,使分发端按「订阅暂不可用」处理,而不是发出半份旧缓存。
try { */
Files.move(temp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(temp, target, StandardCopyOption.REPLACE_EXISTING);
}
}
public Path cachedPath(Integer accountId, String client) { public Path cachedPath(Integer accountId, String client) {
return Paths.get(cacheRoot, String.valueOf(accountId), client.equals("v2") ? "v2ray.txt" : "clash.yaml"); SubscriptionCacheStore store = store();
return store.resolve(accountId, client).orElseGet(() -> store.unavailablePath(accountId, client));
} }
public boolean hasCompleteCache(Integer accountId) { public boolean hasCompleteCache(Integer accountId) {
return Files.isRegularFile(cachedPath(accountId, "v2")) && Files.isRegularFile(cachedPath(accountId, "cat")); return store().hasComplete(accountId);
} }
public void invalidateCache(Integer accountId) { public void invalidateCache(Integer accountId) {
@@ -274,12 +228,15 @@ public class SubscriptionRefreshService {
stateLock.lock(); stateLock.lock();
try { try {
latestRefresh.remove(accountId); // In-flight responses must not restore invalidated content. latestRefresh.remove(accountId); // In-flight responses must not restore invalidated content.
Files.deleteIfExists(cachedPath(accountId, "v2")); store().invalidate(accountId);
Files.deleteIfExists(cachedPath(accountId, "cat"));
} catch (IOException e) {
log.warn("清理失效订阅缓存失败 accountId={}", accountId, e);
} finally { } finally {
stateLock.unlock(); stateLock.unlock();
} }
} }
private synchronized SubscriptionCacheStore store() {
if (cacheStore == null)
cacheStore = new SubscriptionCacheStore(cacheRoot);
return cacheStore;
}
} }
@@ -6,8 +6,10 @@ import com.lion.lionwebsite.Domain.SubBind;
import com.lion.lionwebsite.Domain.SubscriptionAccount; import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Message.*; import com.lion.lionwebsite.Message.*;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.SubscriptionCacheStore;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct; import jakarta.annotation.PostConstruct;
@@ -16,8 +18,6 @@ import javax.crypto.spec.SecretKeySpec;
import java.io.ByteArrayOutputStream; import java.io.ByteArrayOutputStream;
import java.io.IOException; import java.io.IOException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest; import java.security.MessageDigest;
import java.util.*; import java.util.*;
import java.util.zip.GZIPOutputStream; import java.util.zip.GZIPOutputStream;
@@ -30,6 +30,9 @@ public class SubscriptionStandbySnapshotService {
private final SubscriptionStateCoordinator stateCoordinator; private final SubscriptionStateCoordinator stateCoordinator;
private final ObjectMapper objectMapper = CustomUtil.objectMapper; private final ObjectMapper objectMapper = CustomUtil.objectMapper;
@Autowired(required = false)
SubscriptionEventService subscriptionEvents;
@Value("${subscription.standby.sync-secret:}") @Value("${subscription.standby.sync-secret:}")
String syncSecret; String syncSecret;
@@ -39,6 +42,12 @@ public class SubscriptionStandbySnapshotService {
@Value("${subscription.cache-root:sub/accounts}") @Value("${subscription.cache-root:sub/accounts}")
String cacheRoot; String cacheRoot;
/**
* 与 {@link SubscriptionRefreshService} 共用的分代缓存解析器。
* 测试可直接覆写;未设置时按 {@link #cacheRoot} 懒构造。
*/
SubscriptionCacheStore cacheStore;
@PostConstruct @PostConstruct
void validateConfiguration() { void validateConfiguration() {
if (syncEnabled && (syncSecret == null || syncSecret.isBlank())) if (syncEnabled && (syncSecret == null || syncSecret.isBlank()))
@@ -51,12 +60,14 @@ public class SubscriptionStandbySnapshotService {
try { try {
Map<Integer, SubscriptionAccountSnapshot> accountMap = new HashMap<>(); Map<Integer, SubscriptionAccountSnapshot> accountMap = new HashMap<>();
for (SubscriptionAccount account : subMapper.selectAllSubscriptionAccounts()) { for (SubscriptionAccount account : subMapper.selectAllSubscriptionAccounts()) {
Path v2Path = cachedPath(account.getId(), "v2"); if (!account.isEnabled())
Path clashPath = cachedPath(account.getId(), "cat");
if (!account.isEnabled() || !Files.isRegularFile(v2Path) || !Files.isRegularFile(clashPath))
continue; continue;
byte[] v2 = Files.readAllBytes(v2Path); // 从同一分代读出成对内容,保证快照里两份文件同版本;无完整版本则跳过该账号。
byte[] clash = Files.readAllBytes(clashPath); Optional<SubscriptionCacheStore.Pair> pair = store().read(account.getId());
if (pair.isEmpty())
continue;
byte[] v2 = pair.get().v2();
byte[] clash = pair.get().clash();
SubscriptionAccountSnapshot snapshot = new SubscriptionAccountSnapshot(); SubscriptionAccountSnapshot snapshot = new SubscriptionAccountSnapshot();
snapshot.setAccountId(account.getId()); snapshot.setAccountId(account.getId());
snapshot.setEnabled(true); snapshot.setEnabled(true);
@@ -71,7 +82,9 @@ public class SubscriptionStandbySnapshotService {
List<SubscriptionAccountSnapshot> accounts = new ArrayList<>(accountMap.values()); List<SubscriptionAccountSnapshot> accounts = new ArrayList<>(accountMap.values());
accounts.sort(Comparator.comparing(SubscriptionAccountSnapshot::getAccountId)); accounts.sort(Comparator.comparing(SubscriptionAccountSnapshot::getAccountId));
List<SubscriptionBindingSnapshot> bindings = new ArrayList<>(); List<SubscriptionBindingSnapshot> bindings = new ArrayList<>();
for (SubBind bind : subMapper.selectAllSubBind()) { List<SubBind> currentBindings = subMapper.selectAllSubBind();
if (subscriptionEvents != null) subscriptionEvents.rememberBindings(currentBindings);
for (SubBind bind : currentBindings) {
if (bind.getSubscriptionAccountId() == null || !accountMap.containsKey(bind.getSubscriptionAccountId())) if (bind.getSubscriptionAccountId() == null || !accountMap.containsKey(bind.getSubscriptionAccountId()))
continue; continue;
SubscriptionBindingSnapshot snapshot = new SubscriptionBindingSnapshot(); SubscriptionBindingSnapshot snapshot = new SubscriptionBindingSnapshot();
@@ -132,7 +145,9 @@ public class SubscriptionStandbySnapshotService {
private static String hex(byte[] input) { return HexFormat.of().formatHex(input); } private static String hex(byte[] input) { return HexFormat.of().formatHex(input); }
private Path cachedPath(Integer accountId, String client) { private synchronized SubscriptionCacheStore store() {
return Path.of(cacheRoot, String.valueOf(accountId), "v2".equals(client) ? "v2ray.txt" : "clash.yaml"); if (cacheStore == null)
cacheStore = new SubscriptionCacheStore(cacheRoot);
return cacheStore;
} }
} }
@@ -0,0 +1,23 @@
package com.lion.lionwebsite.Service;
import java.util.UUID;
/** 协议 v2 的操作 ID 生成与解析。稳定 ID 是「节点重发/重试不重复执行副作用」的依据。 */
public final class TaskProtocol {
private TaskProtocol() {
}
/** 生成一个新的稳定操作 ID(下载或删除),形如 {@code op-<uuid>}。 */
public static String newOperationId(String action) {
return action + "-" + UUID.randomUUID();
}
public static String newDownloadOperationId() {
return newOperationId("dl");
}
public static String newDeleteOperationId() {
return newOperationId("del");
}
}
@@ -0,0 +1,37 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.Gallery;
import com.lion.lionwebsite.Domain.GalleryTask;
import com.lion.lionwebsite.Message.TaskReportMessage;
import java.util.List;
/**
* 协议 v2 状态上报的消费入口。
*
* <p>定义成接口是为了断开依赖环:RemoteService 需要把上报交给生命周期服务,
* 而生命周期服务需要有 RemoteService 才能下发命令。RemoteService 通过可选的 setter
* 注入本接口(见 {@code RemoteService#setTaskReportHandler}),因此构造期没有环。
*/
public interface TaskReportHandler {
/**
* 提交一份上报。
*
* @return 已成功提交(可回 ACK)时返回提交结果;数据库失败等「不可回 ACK」的情况返回 null。
* 返回 null 时节点会保留快照并稍后重投,这正是「失败写入不得清任务」的要求。
*/
CommittedReport apply(TaskReportMessage report);
/**
* 已提交的结果。
*
* @param reportId 本次上报 ID(回执用)
* @param newlyCompleted 本次「首次」完成的任务,通知与 WebSocket 由独立执行器在事务之后处理,
* 因此慢通知不会阻塞数据库消费线程;重复上报不会重复出现在这里
* @param acceptedTasks 事务内实际被接受的进度更新(含下载中/压缩中等中间状态,已排除过期、
* 旧世代与待删除任务),WebSocket 只推这一批,避免前端看到被丢弃的迟到状态
*/
record CommittedReport(String reportId, List<Gallery> newlyCompleted, List<GalleryTask> acceptedTasks) {
}
}
@@ -0,0 +1,45 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.GalleryTask;
/**
* 任务业务状态码到展示文案的唯一映射点。
*
* <p>状态以稳定代码表达,中文文案只出现在展示层(WebSocket/列表),
* 避免把「失败/待重试/结果未知」塞进成功失败提示字符串。
*/
public final class TaskStatus {
public static final String SUBMITTED = "已提交";
/** 已登记创建意图、尚未确认上游提交;用于让任务列表在提交期间也能看到占位任务。 */
public static final String PENDING_SUBMIT = "提交中";
public static final String DOWNLOADING = "下载中";
public static final String WAITING_COMPRESS = "等待压缩";
public static final String COMPRESSING = "压缩中";
public static final String COMPLETE = "下载完成";
public static final String UNKNOWN = "提交未知";
public static final String DELETE_PENDING = "待删除";
public static final String DELETE_FAILED = "删除失败";
private TaskStatus() {
}
/**
* 节点上报的进度状态映射为业务文案;未知状态码返回 null,由调用方忽略。
*
* <p>GalleryTask 里的状态码是可变 static byte,不是编译期常量,因此不能用 switch case;
* 这里用整数比较,避免为改常量引入风险。
*/
public static String fromReport(byte reportStatus) {
int status = reportStatus;
if (status == GalleryTask.DOWNLOADING)
return DOWNLOADING;
if (status == GalleryTask.DOWNLOAD_COMPLETE)
return WAITING_COMPRESS;
if (status == GalleryTask.COMPRESSING)
return COMPRESSING;
if (status == GalleryTask.COMPRESS_COMPLETE)
return COMPLETE;
return null;
}
}
@@ -0,0 +1,47 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.Gallery;
import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import com.lion.lionwebsite.Util.GalleryUtil;
import com.lion.lionwebsite.Util.PreparedGallery;
import org.springframework.stereotype.Component;
import java.io.IOException;
/**
* 上游图站的读写边界。
*
* <p>分成两步是协议 v2 的核心约束:
* <ol>
* <li>{@link #prepare} 只做读取(页面、分辨率),绝不产生下载副作用;</li>
* <li>调用方必须在持久化 SUBMITTING 之后才调用 {@link #submit},因为那一步才真正 POST。</li>
* </ol>
*
* <p>拆成接口是为了让生命周期服务可以在测试里注入假实现,也方便后续把抓取/解析/提交
* 进一步分开;默认实现 {@link GalleryUtilAdapter} 直接转发到现有 GalleryUtil。
*/
public interface UpstreamGateway {
/** 只读解析:失败返回 null 表示链接无效或页面不可用,不产生任何副作用。 */
PreparedGallery prepare(String link) throws IOException;
/** 真正提交下载请求。抛异常表示结果未知,由调用方落 UNKNOWN,不做自动重发。 */
Gallery submit(PreparedGallery prepared, String targetResolution)
throws IOException, ResolutionNotMatchException;
/** 默认实现:转发现有 GalleryUtil 的拆分方法。 */
@Component
class GalleryUtilAdapter implements UpstreamGateway {
@Override
public PreparedGallery prepare(String link) throws IOException {
return GalleryUtil.prepare(link);
}
@Override
public Gallery submit(PreparedGallery prepared, String targetResolution)
throws IOException, ResolutionNotMatchException {
return GalleryUtil.submit(prepared, targetResolution);
}
}
}
@@ -3,6 +3,7 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.CollectMapper; import com.lion.lionwebsite.Dao.normal.CollectMapper;
import com.lion.lionwebsite.Dao.normal.GalleryMapper; import com.lion.lionwebsite.Dao.normal.GalleryMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper; import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.User; import com.lion.lionwebsite.Domain.User;
import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor; import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
@@ -10,8 +11,13 @@ import com.lion.lionwebsite.Util.Response;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.transaction.support.TransactionTemplate;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.concurrent.locks.Lock;
import java.util.function.Supplier;
import static com.lion.lionwebsite.Util.CustomUtil.objectMapper; import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
@@ -28,6 +34,11 @@ public class UserService{
final TaskHandlerInterceptor taskHandlerInterceptor; final TaskHandlerInterceptor taskHandlerInterceptor;
@Autowired SubMapper subMapper;
@Autowired SubscriptionStateCoordinator subscriptionState;
@Autowired RemoteService remoteService;
@Autowired @Qualifier("mainTransactionTemplate") TransactionTemplate transactions;
public String addAuthCode(String targetAuthCode, String people) { public String addAuthCode(String targetAuthCode, String people) {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
@@ -66,12 +77,22 @@ public class UserService{
Response response = Response.generateResponse(); Response response = Response.generateResponse();
try{ try{
if(userMapper.isExist(targetAuthCode) != 0){ return coordinated(() -> {
userMapper.updateUsername(targetAuthCode, newUsername); if (userMapper.isExist(targetAuthCode) == 0)
response.success("修改成功"); return response.failure("授权码不存在").toJSONString();
}else { if (newUsername == null || newUsername.isBlank())
response.failure("授权码不存在"); return response.failure("用户名不能为空").toJSONString();
} String name = newUsername.trim();
User current = userMapper.selectUserByAuthCode(targetAuthCode);
User existing = userMapper.selectUserByUsername(name);
if (existing != null && (current == null || existing.getId() != current.getId()))
return response.failure("用户名已存在").toJSONString();
userMapper.updateUsername(targetAuthCode, name);
if (subMapper != null && current != null)
subMapper.renameBoundUser(current.getId(), current.getUsername(), name);
if (remoteService != null) remoteService.requestSubscriptionSync();
return response.success("修改成功").toJSONString();
});
}catch (Exception e){ }catch (Exception e){
log.error("修改用户名失败", e); log.error("修改用户名失败", e);
response.failure("修改失败"); response.failure("修改失败");
@@ -85,17 +106,22 @@ public class UserService{
if(userMapper.isExist(targetAuthCode) != 0) if(userMapper.isExist(targetAuthCode) != 0)
try{ try{
coordinated(() -> {
User user = userMapper.selectUserByAuthCode(targetAuthCode); User user = userMapper.selectUserByAuthCode(targetAuthCode);
//修改下载人以及取消收藏
ArrayList<Integer> galleries = collectMapper.selectGidByCollector(user.getId()); ArrayList<Integer> galleries = collectMapper.selectGidByCollector(user.getId());
for(Integer gallery: galleries) for(Integer gallery: galleries)
collectMapper.disCollect(gallery, user.getId()); collectMapper.disCollect(gallery, user.getId());
galleryMapper.updateGalleryDownloader(3, user.getId()); galleryMapper.updateGalleryDownloader(ADMIN_USER_ID, user.getId());
if (subMapper != null) {
subMapper.deleteSubUpdateRecord(user.getUsername());
subMapper.deleteSubBind(user.getUsername());
}
userMapper.deleteUserByAuthCode(targetAuthCode); userMapper.deleteUserByAuthCode(targetAuthCode);
return "deleted";
});
response.success("删除成功"); response.success("删除成功");
taskHandlerInterceptor.updateAuthCodes(); taskHandlerInterceptor.updateAuthCodes();
if (remoteService != null) remoteService.requestSubscriptionSync();
}catch (Exception e){ }catch (Exception e){
log.error("删除授权码失败", e); log.error("删除授权码失败", e);
response.failure("删除失败"); response.failure("删除失败");
@@ -108,27 +134,29 @@ public class UserService{
} }
public String alterStatus(String AuthCode, boolean isEnable){ public String alterStatus(String AuthCode, boolean isEnable){
Response response = Response.generateResponse(); try {
String result = coordinated(() -> {
if(userMapper.isExist(AuthCode) == 1){ if (userMapper.isExist(AuthCode) != 1)
return Response._failure("该用户不存在");
int id = userMapper.selectUserByAuthCode(AuthCode).getId(); int id = userMapper.selectUserByAuthCode(AuthCode).getId();
userMapper.updateIsEnableById(id, isEnable); userMapper.updateIsEnableById(id, isEnable);
galleryMapper.updateGalleryDownloader(3, id); if (!isEnable) galleryMapper.updateGalleryDownloader(ADMIN_USER_ID, id);
response.success("修改用户状态成功"); return Response._success("修改用户状态成功");
taskHandlerInterceptor.updateAuthCodes(); });
// Refresh the in-memory identity view only after the database commits.
if (result.contains("\"result\":\"success\"")) taskHandlerInterceptor.updateAuthCodes();
return result;
} catch (RuntimeException error) {
log.warn("修改用户状态失败: {}", error.getClass().getSimpleName());
return Response._failure("修改用户状态失败");
} }
else{
response.failure("该用户不存在");
}
return response.toJSONString();
} }
public String getAllUser() { public String getAllUser() {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
User[] users = userMapper.selectAllUser(); User[] users = userMapper.selectAllUser();
response.success(objectMapper.valueToTree(users).toString()); response.success(objectMapper.valueToTree(users));
return response.toJSONString(); return response.toJSONString();
} }
@@ -150,4 +178,14 @@ public class UserService{
User user = userMapper.selectUserByAuthCode(AuthCode); User user = userMapper.selectUserByAuthCode(AuthCode);
return user != null && user.getId() == ADMIN_USER_ID; return user != null && user.getId() == ADMIN_USER_ID;
} }
private String coordinated(Supplier<String> action) {
Lock lock = subscriptionState == null ? null : subscriptionState.writeLock();
if (lock != null) lock.lock();
try {
return transactions == null ? action.get() : transactions.execute(ignored -> action.get());
} finally {
if (lock != null) lock.unlock();
}
}
} }
@@ -0,0 +1,45 @@
package com.lion.lionwebsite.Util;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.DumperOptions;
import java.util.*;
import java.util.regex.Pattern;
/** YAML semantics rather than indentation determine nodes and references. */
public final class ClashSubscriptionParser {
private static final Pattern MULTIPLIER = Pattern.compile("(\\d+(?:\\.\\d+)?)x\\s*$", Pattern.CASE_INSENSITIVE);
private ClashSubscriptionParser() { }
public static String transform(List<String> lines, boolean filter, double threshold) {
Object loaded = new Yaml().load(String.join("\n", lines));
if (!(loaded instanceof Map<?, ?> source)) throw new IllegalArgumentException("Clash 上游必须为 YAML 对象");
if (!(source.get("proxies") instanceof List<?> proxies)) throw new IllegalArgumentException("Clash 上游缺少 proxies 列表");
Map<Object, Object> output = new LinkedHashMap<>(source);
Set<String> removed = new HashSet<>();
List<Object> kept = new ArrayList<>();
for (Object node : proxies) {
if (!(node instanceof Map<?, ?> fields) || !(fields.get("name") instanceof String name))
throw new IllegalArgumentException("Clash 节点缺少名称");
var multiplier = MULTIPLIER.matcher(name);
if (filter && multiplier.find() && Double.parseDouble(multiplier.group(1)) > threshold)
removed.add(name);
else kept.add(node);
}
output.put("proxies", kept);
if (source.get("proxy-groups") instanceof List<?> groups) {
List<Object> result = new ArrayList<>();
for (Object group : groups) {
if (!(group instanceof Map<?, ?> fields)) throw new IllegalArgumentException("Clash 分组必须为对象");
Map<Object, Object> copy = new LinkedHashMap<>(fields);
if (fields.get("proxies") instanceof List<?> references)
copy.put("proxies", references.stream().filter(ref -> !removed.contains(ref)).toList());
result.add(copy);
}
output.put("proxy-groups", result);
}
DumperOptions options = new DumperOptions();
options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK);
options.setPrettyFlow(true);
return new Yaml(options).dump(output);
}
}
@@ -0,0 +1,97 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.Gallery;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;
import java.util.*;
import java.util.regex.Pattern;
/** Pure parsing: upstream requests and download submission live outside this class. */
public final class GalleryPageParser {
private GalleryPageParser() { }
/** 归档页 #db 单元格开头:分辨率标签 + 正数体积 + 受支持单位;后缀费用文本(Free、50GP)忽略。 */
private static final Pattern RESOLUTION_ENTRY =
Pattern.compile("^(Original|(\\d+)x)\\s+(\\d+(?:\\.\\d+)?)\\s*([A-Za-z]+)(?=\\s|$)", Pattern.CASE_INSENSITIVE);
public static Gallery metadata(Document document, String link, int gid) throws IOException {
String title = document.select("#gn").text();
if (title.isBlank()) title = document.select("#gj").text();
if (title.isBlank()) throw new IOException("上游页面缺少画廊名称,可能是错误页");
Map<String, String> fields = new HashMap<>();
for (Element row : document.select("#gdd tr")) {
var cells = row.select("td");
if (cells.size() < 2) continue;
String name = cells.get(0).text().replace(":", "").strip().toLowerCase(Locale.ROOT);
fields.put(name, cells.get(1).text().strip());
}
var pages = Pattern.compile("\\d+").matcher(fields.getOrDefault("length", ""));
if (!pages.find()) throw new IOException("上游页面缺少画廊页数");
Gallery result = new Gallery();
result.setGid(gid); result.setLink(link); result.setName(title + " [" + gid + "]");
result.setCreateTime(System.currentTimeMillis() / 1000); result.setProceeding(0);
result.setLanguage(fields.getOrDefault("language", "").replace("\u00a0", " "));
try {
result.setPages(Integer.parseInt(pages.group()));
result.setFileSize(CustomUtil.stringToFileSize(fields.getOrDefault("file size", "").replace(" ", "")));
} catch (RuntimeException failure) { throw new IOException("上游画廊页数或文件大小格式不支持", failure); }
if (result.getPages() <= 0) throw new IOException("上游画廊页数必须为正数");
result.setDisplayFileSize(CustomUtil.fileSizeToString(result.getFileSize()));
Element thumbnail = document.selectFirst("#gd1 [style*=url]");
var image = Pattern.compile("url\\(\\s*['\"]?(.*?)['\"]?\\s*\\)").matcher(thumbnail == null ? "" : thumbnail.attr("style"));
String imageUrl = image.find() ? image.group(1) : document.select("#gd1 img").attr("src");
result.setThumb_link(imageUrl.replace("https://s.exhentai.org", ""));
return result;
}
public static String archiveLink(Document document) throws IOException {
for (Element anchor : document.select("#gd5 a")) {
var matcher = Pattern.compile("https?://[^'\"\\s]+").matcher(anchor.attr("onclick"));
if (matcher.find()) return matcher.group();
String href = anchor.absUrl("href");
if (!href.isBlank() && (anchor.text().toLowerCase(Locale.ROOT).contains("download") || href.contains("archive")))
return href;
}
throw new IOException("上游页面缺少归档入口");
}
/**
* 解析归档页 #db 的可选分辨率。每个单元格合并后的文本形如 {@code Original 36.31 MiB Free}、
* {@code 800x 5.65 MiB Free}、{@code 2560x N/A N/A},单段布局形如 {@code 1280x 12.5 MiB},
* 都由同一条锚定规则覆盖;只看开头(分辨率 + 正数体积 + 单位),故 N/A 与费用文本不匹配,
* 配对也不会跨 {@code <td>} 借用相邻格。体积须能被 {@link CustomUtil#stringToFileSize(String)}
* 解析为正数,宽度须为正且不溢出。结果按宽度升序、{@code Original} 最后;无可用项抛 IOException。
*/
public static Map<String, String> resolutions(Document document) throws IOException {
TreeMap<Integer, String> sizes = new TreeMap<>();
String original = null;
for (Element cell : document.select("#db td")) {
var matcher = RESOLUTION_ENTRY.matcher(cell.text().replace('\u00a0', ' ').strip());
if (!matcher.find()) continue;
String size = matcher.group(3) + matcher.group(4);
try {
if (CustomUtil.stringToFileSize(size) <= 0) continue;
} catch (RuntimeException unsupported) {
continue;
}
if (matcher.group(1).equalsIgnoreCase("Original")) {
original = size;
continue;
}
int width;
try {
width = Integer.parseInt(matcher.group(2));
} catch (NumberFormatException outOfRange) {
continue;
}
if (width <= 0) continue;
sizes.put(width, size);
}
Map<String, String> result = new LinkedHashMap<>();
sizes.forEach((width, size) -> result.put(width + "x", size));
if (original != null) result.put("Original", original);
if (result.isEmpty()) throw new IOException("上游页面没有可用分辨率");
return result;
}
}
@@ -0,0 +1,74 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.GalleryForQuery;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Pattern;
/** Search entries support table and thumbnail layouts; metadata excludes title/tag text. */
public final class GallerySearchParser {
private GallerySearchParser() { }
public static List<GalleryForQuery> parse(Document document) throws IOException {
List<GalleryForQuery> result = new ArrayList<>();
for (Element row : document.select("tr, .gl1t")) {
if (row.tagName().equals("tr") && !row.select(".gl1t").isEmpty()) continue;
Element link = row.selectFirst("a[href*=/g/]");
if (link == null) continue;
String href = link.attr("href");
Integer gid = GalleryUtil.parseGid(href);
if (gid == null) continue;
GalleryForQuery gallery = new GalleryForQuery();
gallery.setGid(gid.toString()); gallery.setLink(href);
Element title = row.selectFirst(".glink");
String name = title == null ? "" : title.text();
if (name.isBlank()) {
for (Element candidate : row.select("a[href*=/g/]")) {
if (!candidate.text().isBlank()) {
name = candidate.text();
break;
}
}
}
gallery.setName(name);
Element type = row.selectFirst(".cn, .cs");
Element firstCell = row.selectFirst("td");
gallery.setType(type != null ? type.text() : firstCell == null ? "" : firstCell.text());
Element image = row.selectFirst("img");
String thumbnail = image == null ? "" : image.attr("src");
if (thumbnail.isBlank() || thumbnail.startsWith("data:")) thumbnail = image == null ? "" : image.attr("data-src");
gallery.setThumbnailUrl(thumbnail.replace("https://s.exhentai.org", ""));
gallery.setPage(pageCount(row, gid));
Element date = row.selectFirst("[id^=posted_]");
if (date != null) gallery.setUploadTime(date.text());
else {
var dates = Pattern.compile("\\d{4}-\\d{2}-\\d{2}(?:\\s+\\d{2}:\\d{2})?").matcher(row.text());
gallery.setUploadTime(dates.find() ? dates.group() : "");
}
if (gallery.getName().isBlank()) throw new IOException("上游搜索结果缺少名称 gid=" + gid);
result.add(gallery);
}
return result;
}
private static int pageCount(Element row, int gid) throws IOException {
Element metadata = row.clone();
// Extended rows can put title and metadata in the same cell: remove only
// the title links/tags, retaining their containing cells and page counts.
metadata.select("a[href*=/g/], .glink, .gt, .gtl, .gtd, .glthumb").remove();
var expression = Pattern.compile("^(\\d+)\\s+pages?$", Pattern.CASE_INSENSITIVE);
for (Element element : metadata.getAllElements()) {
var pages = expression.matcher(element.text().strip());
if (!pages.matches()) continue;
try {
int count = Integer.parseInt(pages.group(1));
if (count > 0) return count;
} catch (NumberFormatException invalid) {
throw new IOException("上游搜索结果页数格式不支持 gid=" + gid, invalid);
}
}
throw new IOException("上游搜索结果缺少页数 gid=" + gid);
}
}
@@ -10,12 +10,13 @@ import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.classic.methods.HttpPost; import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.config.RequestConfig; import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.client5.http.entity.EntityBuilder; import org.apache.hc.client5.http.entity.EntityBuilder;
import org.apache.hc.client5.http.entity.mime.MultipartEntityBuilder; import org.apache.hc.client5.http.entity.UrlEncodedFormEntity;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient; import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse; import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients; import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType; import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.HttpEntity; import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.message.BasicNameValuePair;
import org.apache.hc.core5.util.Timeout; import org.apache.hc.core5.util.Timeout;
import org.jsoup.Jsoup; import org.jsoup.Jsoup;
import org.jsoup.nodes.Document; import org.jsoup.nodes.Document;
@@ -28,6 +29,8 @@ import java.io.BufferedReader;
import java.io.File; import java.io.File;
import java.io.IOException; import java.io.IOException;
import java.io.InputStreamReader; import java.io.InputStreamReader;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.util.*; import java.util.*;
import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicInteger;
@@ -40,7 +43,7 @@ public class GalleryUtil {
static String POST = "post"; static String POST = "post";
static String GET = "get"; static String GET = "get";
static String FORM_DATA = "formData"; static String FORM_URLENCODED = "formUrlEncoded";
static String JSON = "json"; static String JSON = "json";
@@ -58,6 +61,8 @@ public class GalleryUtil {
/** Reusable HTTP client —不要每次请求新建 */ /** Reusable HTTP client —不要每次请求新建 */
private static final CloseableHttpClient httpClient = HttpClients.custom() private static final CloseableHttpClient httpClient = HttpClients.custom()
// An upstream download POST has no idempotency key; never replay it on 429/503.
.disableAutomaticRetries()
.setDefaultRequestConfig(RequestConfig.custom() .setDefaultRequestConfig(RequestConfig.custom()
.setConnectTimeout(Timeout.ofMilliseconds(5_000)) .setConnectTimeout(Timeout.ofMilliseconds(5_000))
.setConnectionRequestTimeout(Timeout.ofMilliseconds(5_000)) .setConnectionRequestTimeout(Timeout.ofMilliseconds(5_000))
@@ -82,6 +87,12 @@ public class GalleryUtil {
* @throws ResolutionNotMatchException 没有目标分辨率 * @throws ResolutionNotMatchException 没有目标分辨率
*/ */
public static Gallery parse(String url, boolean isDownload, String targetResolution) throws IOException, ResolutionNotMatchException { public static Gallery parse(String url, boolean isDownload, String targetResolution) throws IOException, ResolutionNotMatchException {
PreparedGallery prepared = prepare(url);
if (prepared == null) return null;
return isDownload ? submit(prepared, targetResolution) : prepared.gallery();
}
public static PreparedGallery prepare(String url) throws IOException {
String origin = verifyLink(url); String origin = verifyLink(url);
if(origin == null){ if(origin == null){
@@ -94,66 +105,25 @@ public class GalleryUtil {
log.warn("链接无法解析出 gid,按无效链接处理"); log.warn("链接无法解析出 gid,按无效链接处理");
return null; return null;
} }
Gallery gallery = new Gallery(); Document galleryDoc = Jsoup.parse(requests(url, GET, null, null), url);
gallery.setLink(url); Gallery gallery = GalleryPageParser.metadata(galleryDoc, url, gid);
gallery.setCreateTime(System.currentTimeMillis()/1000); String archiveLink = GalleryPageParser.archiveLink(galleryDoc);
gallery.setGid(gid);
gallery.setProceeding(0);
//访问图片页面
String galleryPage = requests(url, GET, null, null);
Document galleryDoc = Jsoup.parse(galleryPage);
//收集图片基本信息
gallery.setName(galleryDoc.select("#gn").text() + " [" + gallery.getGid() + ']');
gallery.setLanguage(galleryDoc.select("#gdd > table > tbody > tr:nth-child(4) > td.gdt2").text().replace("%nbps", ""));
gallery.setPages(Integer.parseInt(galleryDoc.select("#gdd > table > tbody > tr:nth-child(6) > td.gdt2").text().split(" ")[0]));
gallery.setFileSize(CustomUtil.stringToFileSize(galleryDoc.select("#gdd > table > tbody > tr:nth-child(5) > td.gdt2").text().replace(" ", "")));
gallery.setDisplayFileSize(CustomUtil.fileSizeToString(gallery.getFileSize()));
String thumb_link = galleryDoc.select("#gd1 > div").getFirst().attr("style");
gallery.setThumb_link(thumb_link.substring(thumb_link.indexOf("url(") + 4, thumb_link.lastIndexOf(")")).replace("https://s.exhentai.org", ""));
//查找下载页面链接并初始化参数
String download_link = galleryDoc.select("#gd5 > p:nth-child(2) > a").attr("onclick").split("'")[1];
HashMap<String, String> headers = new HashMap<>(); HashMap<String, String> headers = new HashMap<>();
headers.put("origin", origin); headers.put("referer", url);
headers.put("referer", download_link); Document downloadDoc = Jsoup.parse(requests(archiveLink, GET, headers, null), archiveLink);
String download_link = downloadDoc.select("#hathdl_form").attr("abs:action");
//访问下载页面,获取分辨率 if (download_link.isBlank()) throw new IOException("上游页面缺少下载提交表单");
String downloadPage = requests(download_link, GET, headers, null); gallery.setAvailableResolution(GalleryPageParser.resolutions(downloadDoc));
Document downloadDoc = Jsoup.parse(downloadPage);
Elements resolutions = downloadDoc.select("#db > div > table > tbody > tr > td");
download_link = downloadDoc.select("#hathdl_form").attr("action");
Map<String, String> availableResolution = new LinkedHashMap<>();
Map<String, String> tempResolutions = new LinkedHashMap<>();
ArrayList<Integer> tempResolutionKeys = new ArrayList<>();
for(Element element: resolutions){
String[] parameter = element.select("td > p").text().split(" ");
if(parameter[1].contains("N/A"))
continue;
tempResolutions.put(parameter[0], parameter[1] + parameter[2]);
if(parameter[0].contains("x")){
tempResolutionKeys.add(Integer.valueOf(parameter[0].replace("x", "")));
}
}
tempResolutionKeys.sort(Comparator.comparingInt(o -> o));
for(Integer resolution: tempResolutionKeys){
availableResolution.put(resolution.toString()+"x", tempResolutions.get(resolution+"x"));
}
availableResolution.put("Original", tempResolutions.get("Original"));
gallery.setAvailableResolution(availableResolution);
//如果不是下载,则直接返回
if(!isDownload){
gallery.setStatus("等待确认下载"); gallery.setStatus("等待确认下载");
return gallery; return new PreparedGallery(gallery, download_link, origin);
} }
public static Gallery submit(PreparedGallery prepared, String targetResolution) throws IOException, ResolutionNotMatchException {
Gallery gallery = prepared.gallery();
String download_link = prepared.formUrl();
HashMap<String, String> headers = new HashMap<>();
headers.put("origin", prepared.origin());
headers.put("referer", download_link);
//如果目标分辨率不存在,抛出错误 //如果目标分辨率不存在,抛出错误
if(!gallery.getAvailableResolution().containsKey(targetResolution)){ if(!gallery.getAvailableResolution().containsKey(targetResolution)){
log.warn("目标分辨率不存在,可用分辨率: {}", gallery.getAvailableResolution()); log.warn("目标分辨率不存在,可用分辨率: {}", gallery.getAvailableResolution());
@@ -161,15 +131,16 @@ public class GalleryUtil {
} }
gallery.setResolution(targetResolution); gallery.setResolution(targetResolution);
gallery.setFileSize(CustomUtil.stringToFileSize(availableResolution.get(targetResolution).replace(" ", ""))); gallery.setFileSize(CustomUtil.stringToFileSize(gallery.getAvailableResolution().get(targetResolution).replace(" ", "")));
gallery.setDisplayFileSize(CustomUtil.fileSizeToString(gallery.getFileSize())); gallery.setDisplayFileSize(CustomUtil.fileSizeToString(gallery.getFileSize()));
//提交下载请求 //提交下载请求
HashMap<String, String> body = new HashMap<>(); HashMap<String, String> body = new HashMap<>();
body.put("payload", FORM_DATA); // The upstream form has no enctype: match the browser's default encoding.
body.put("payload", FORM_URLENCODED);
body.put("hathdl_xres", targetResolution.replace("x", "").replace("Original", "org")); body.put("hathdl_xres", targetResolution.replace("x", "").replace("Original", "org"));
downloadPage = requests(download_link, POST, headers, body); String downloadPage = requests(download_link, POST, headers, body);
downloadDoc = Jsoup.parse(downloadPage); Document downloadDoc = Jsoup.parse(downloadPage);
//判断下载请求是否提交成功 //判断下载请求是否提交成功
if(downloadDoc.select("#db > p:nth-child(2) > strong").text().startsWith("#")) if(downloadDoc.select("#db > p:nth-child(2) > strong").text().startsWith("#"))
@@ -192,14 +163,12 @@ public class GalleryUtil {
if(content.trim().isEmpty()) if(content.trim().isEmpty())
return null; return null;
Document document = Jsoup.parse(content); Document document = Jsoup.parse(content);
if(document.select("body > script").size() < 2) String script = document.select("script").stream().map(Element::data).collect(java.util.stream.Collectors.joining("\n"));
return null; if (!script.contains("imagelist") || !script.contains("mpvkey")) return null;
Element script = document.select("body > script").get(1);
String[] scripts = script.html().split("\n");
ArrayList<ImageKeyCache> imageKeyCaches = new ArrayList<>(); ArrayList<ImageKeyCache> imageKeyCaches = new ArrayList<>();
AtomicInteger page = new AtomicInteger(1); AtomicInteger page = new AtomicInteger(1);
gid2MpvKey.put(gid, scripts[1].split("=")[1].replace(";", "").replace("\"", "").replace(" ", "")); gid2MpvKey.put(gid, JavaScriptAssignments.string(script, "mpvkey"));
JsonNode nodes = parseImagelist(scripts[2]); JsonNode nodes = objectMapper.readTree(JavaScriptAssignments.array(script, "imagelist"));
nodes.forEach((n) -> { nodes.forEach((n) -> {
ImageKeyCache imageKeyCache = new ImageKeyCache(); ImageKeyCache imageKeyCache = new ImageKeyCache();
imageKeyCache.setGid(gid); imageKeyCache.setGid(gid);
@@ -247,12 +216,14 @@ public class GalleryUtil {
gid2MpvKey.remove(parseGid(url) + ""); gid2MpvKey.remove(parseGid(url) + "");
return; return;
} }
Document document = Jsoup.parse(content); try {
Element script = document.select("body > script").get(1); String scripts = Jsoup.parse(content).select("script").stream().map(Element::data)
String[] scripts = script.html().split("\n"); .collect(java.util.stream.Collectors.joining("\n"));
String mpvKey = scripts[1].split("=")[1].replace(";", "").replace("\"", "").replace(" ", ""); gid2MpvKey.put(parseGid(url) + "", JavaScriptAssignments.string(scripts, "mpvkey"));
gid2MpvKey.put(parseGid(url) + "", mpvKey); } catch (IOException e) {
log.info("刷新key:{}", mpvKey); gid2MpvKey.remove(parseGid(url) + "");
log.warn("上游mpv脚本无法解析,缓存已失效");
}
} }
public static String getImageUrl(String mpvKey, ImageKeyCache imageKeyCache) { public static String getImageUrl(String mpvKey, ImageKeyCache imageKeyCache) {
@@ -326,9 +297,18 @@ public class GalleryUtil {
if(url.length() < 40) if(url.length() < 40)
return null; return null;
if(!url.contains("/g/") || !url.contains("exhentai")) try {
URI uri = URI.create(url);
String scheme = uri.getScheme();
if ((!"https".equalsIgnoreCase(scheme) && !"http".equalsIgnoreCase(scheme))
|| !"exhentai.org".equalsIgnoreCase(uri.getHost())
|| uri.getRawUserInfo() != null || uri.getPort() != -1
|| uri.getPath() == null || !uri.getPath().matches("/g/[0-9]+/[A-Za-z0-9]+/?"))
return null; return null;
return ""; return scheme.toLowerCase(Locale.ROOT) + "://exhentai.org";
} catch (IllegalArgumentException invalidUrl) {
return null;
}
} }
/** /**
@@ -363,16 +343,18 @@ public class GalleryUtil {
httpPost.addHeader(header.getKey(), header.getValue()); httpPost.addHeader(header.getKey(), header.getValue());
if(body != null) { if(body != null) {
// Keep the caller's form intact; payload is a local encoding selector.
body = new HashMap<>(body);
String payload; String payload;
if((payload = body.remove("payload")).equals(JSON)) { if((payload = body.remove("payload")).equals(JSON)) {
EntityBuilder entityBuilder = EntityBuilder.create(); EntityBuilder entityBuilder = EntityBuilder.create();
entityBuilder.setContentType(ContentType.APPLICATION_JSON); entityBuilder.setContentType(ContentType.APPLICATION_JSON);
entityBuilder.setText(objectMapper.writeValueAsString(body)); entityBuilder.setText(objectMapper.writeValueAsString(body));
httpPost.setEntity(entityBuilder.build()); httpPost.setEntity(entityBuilder.build());
} else if(payload.equals(FORM_DATA)) { } else if(payload.equals(FORM_URLENCODED)) {
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create(); var fields = body.entrySet().stream()
body.forEach(multipartEntityBuilder::addTextBody); .map(entry -> new BasicNameValuePair(entry.getKey(), entry.getValue())).toList();
httpPost.setEntity(multipartEntityBuilder.build()); httpPost.setEntity(new UrlEncodedFormEntity(fields, StandardCharsets.UTF_8));
} }
} }
httpResponse = httpClient.execute(httpPost); httpResponse = httpClient.execute(httpPost);
@@ -382,13 +364,15 @@ public class GalleryUtil {
int statusCode = httpResponse.getCode(); int statusCode = httpResponse.getCode();
StringBuilder stringBuilder = new StringBuilder(); StringBuilder stringBuilder = new StringBuilder();
if (statusCode == 200 && responseEntity != null) { if (statusCode == 200 && responseEntity != null) {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(responseEntity.getContent()))) { try (BufferedReader reader = new BufferedReader(new InputStreamReader(responseEntity.getContent(), StandardCharsets.UTF_8))) {
String str; String str;
while ((str = reader.readLine()) != null) while ((str = reader.readLine()) != null)
stringBuilder.append(str).append("\n"); stringBuilder.append(str).append("\n");
} }
} else { } else {
log.warn("{}:{}", url, statusCode); // Never log gallery tokens, cookies or an arbitrary upstream error body.
throw new IOException("上游 HTTP " + statusCode + " (" + method.toUpperCase(Locale.ROOT)
+ " " + URI.create(url).getHost() + ")");
} }
return stringBuilder.toString(); return stringBuilder.toString();
} }
@@ -0,0 +1,34 @@
package com.lion.lionwebsite.Util;
import java.io.IOException;
import java.util.regex.Pattern;
/** Extracts data assignments without depending on script position or line layout. */
public final class JavaScriptAssignments {
private JavaScriptAssignments() { }
public static String string(String source, String name) throws IOException {
var matcher = Pattern.compile("\\b" + Pattern.quote(name) + "\\s*=\\s*(['\"])(.*?)\\1", Pattern.DOTALL).matcher(source);
if (!matcher.find()) throw new IOException("上游脚本缺少 " + name);
return matcher.group(2);
}
public static String array(String source, String name) throws IOException {
var matcher = Pattern.compile("\\b" + Pattern.quote(name) + "\\s*=\\s*\\[").matcher(source);
if (!matcher.find()) throw new IOException("上游脚本缺少 " + name);
int start = matcher.end() - 1, depth = 0; char quote = 0; boolean escaped = false;
for (int i = start; i < source.length(); i++) {
char c = source.charAt(i);
if (quote != 0) {
if (escaped) escaped = false;
else if (c == '\\') escaped = true;
else if (c == quote) quote = 0;
continue;
}
if (c == '"' || c == '\'') quote = c;
else if (c == '[') depth++;
else if (c == ']' && --depth == 0) return source.substring(start, i + 1);
}
throw new IOException("上游脚本的 " + name + " 不完整");
}
}
@@ -0,0 +1,6 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.Gallery;
/** A read-only upstream preparation; creating this value never submits a download. */
public record PreparedGallery(Gallery gallery, String formUrl, String origin) { }
@@ -0,0 +1,328 @@
package com.lion.lionwebsite.Util;
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.AtomicMoveNotSupportedException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.security.MessageDigest;
import java.util.Comparator;
import java.util.HexFormat;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Stream;
/**
* 订阅缓存的分代存储:v2ray 与 clash 两份内容必须同时生效。
*
* <p>旧实现把两份文件各自原子替换,第二份失败时用户会拿到「新 v2ray + 旧 clash」的
* 混合版本。这里把两份文件写进同一个不可变分代目录,校验通过后用一次原子替换发布
* 指针(manifest),因此抓取、解析、写入或发布任何一步失败,都只会保留上一个完整分代。
*
* <p>目录结构:
* <pre>
* &lt;cacheRoot&gt;/&lt;accountId&gt;/current 指针(原子替换,最后发布)
* &lt;cacheRoot&gt;/&lt;accountId&gt;/generations/&lt;generation&gt;/v2ray.txt
* &lt;cacheRoot&gt;/&lt;accountId&gt;/generations/&lt;generation&gt;/clash.yaml
* &lt;cacheRoot&gt;/&lt;accountId&gt;/v2ray.txt + clash.yaml 旧结构,两份都在才按 legacy 读
* </pre>
*
* <p>读取始终从指针指向的同一个分代里取两份文件,所以拿到的一定是同版本内容。
*/
@Slf4j
public class SubscriptionCacheStore {
private static final String MANIFEST_FILE = "current";
private static final String GENERATION_DIR = "generations";
private static final String V2_FILE = "v2ray.txt";
private static final String CLASH_FILE = "clash.yaml";
/** 保留目录名,结构上永远不存在,用于表达「当前没有可分发版本」。 */
private static final String UNAVAILABLE_DIR = ".unavailable";
private static final String MANIFEST_GENERATION = "generation";
private static final String MANIFEST_V2_SHA = "v2-sha256";
private static final String MANIFEST_CLASH_SHA = "clash-sha256";
/** 旧结构读出来的版本的 generation 标记;真实分代名是定宽时间戳,不会与之碰撞。 */
private static final String LEGACY_GENERATION = "legacy";
/** 保留分代数:当前与上一个,足够回退且不会无限增长。 */
private static final int RETAINED_GENERATIONS = 2;
private static final AtomicLong SEQUENCE = new AtomicLong();
private final Path root;
public SubscriptionCacheStore(String cacheRoot) {
this(Paths.get(cacheRoot));
}
public SubscriptionCacheStore(Path root) {
this.root = root;
}
/** 同一个分代目录里读出的一对内容;两份文件必然同版本。 */
public record Pair(String generation, byte[] v2, byte[] clash) { }
public Path accountDir(int accountId) {
return root.resolve(String.valueOf(accountId));
}
/** 旧结构的账号根文件路径。仅供兼容测试定位 legacy 文件,不作为回退目标。 */
public Path legacyPath(int accountId, String client) {
return accountDir(accountId).resolve(fileName(client));
}
/**
* 没有完整可分发版本时返回的哨兵路径。
*
* <p>它保证不存在,因此调用方用 {@code Files.isRegularFile} 判定就会走「订阅暂不可用」
* 分支,而不会被半份 legacy 文件骗过。
*/
public Path unavailablePath(int accountId, String client) {
return accountDir(accountId).resolve(UNAVAILABLE_DIR).resolve(fileName(client));
}
private static String fileName(String client) {
return "v2".equals(client) ? V2_FILE : CLASH_FILE;
}
/**
* 写入一份新分代并在最后发布指针。
*
* <p>失败时抛出异常,且不改变指针与上一个分代,调用方据此保留上次的完整版本。
*/
public Pair publish(int accountId, byte[] v2, byte[] clash) throws IOException {
Path account = accountDir(accountId);
Files.createDirectories(account);
String generation = nextGeneration();
Path dir = account.resolve(GENERATION_DIR).resolve(generation);
boolean published = false;
try {
Files.createDirectories(dir);
writeFile(dir.resolve(V2_FILE), v2);
writeFile(dir.resolve(CLASH_FILE), clash);
// 发布前校验:两份文件都必须能按写入内容读回,避免把半截内容挂上指针。
verify(dir.resolve(V2_FILE), v2);
verify(dir.resolve(CLASH_FILE), clash);
publishManifest(account, generation, sha256(v2), sha256(clash));
published = true;
} finally {
if (!published)
deleteRecursively(dir);
}
prune(account, generation);
removeLegacy(account);
return new Pair(generation, v2, clash);
}
/** 读取当前发布内容;只有构成一整个有效版本时才返回。 */
public Optional<Pair> read(int accountId) {
return Optional.ofNullable(load(accountDir(accountId)));
}
/**
* 解析当前应分发的单个文件。
*
* <p>与 {@link #read}、{@link #hasComplete} 使用同一份配对契约:只有两份文件
* 齐全且哈希校验通过时才返回路径,因此分发路径不会发出半份或已损坏的缓存。
*/
public Optional<Path> resolve(int accountId, String client) {
Path account = accountDir(accountId);
Pair pair = load(account);
if (pair == null)
return Optional.empty();
Path path = LEGACY_GENERATION.equals(pair.generation())
? account.resolve(fileName(client))
: account.resolve(GENERATION_DIR).resolve(pair.generation()).resolve(fileName(client));
return Files.isRegularFile(path) ? Optional.of(path) : Optional.empty();
}
/** 是否已有可分发的一对内容(分代两份都在且哈希匹配,或旧结构两份都在)。 */
public boolean hasComplete(int accountId) {
return load(accountDir(accountId)) != null;
}
/** 删除该账号的全部缓存(指针、分代、旧结构文件),使后续请求判定为无缓存。 */
public void invalidate(int accountId) {
Path account = accountDir(accountId);
deleteIfExists(account.resolve(MANIFEST_FILE));
deleteRecursively(account.resolve(GENERATION_DIR));
deleteIfExists(account.resolve(V2_FILE));
deleteIfExists(account.resolve(CLASH_FILE));
}
/** 写入接缝:测试覆写以模拟第二份文件写入失败。 */
protected void writeFile(Path target, byte[] data) throws IOException {
atomicWrite(target, data);
}
/** 指针发布接缝:测试覆写以模拟 manifest 发布失败。 */
protected void publishManifest(Path accountDir, String generation, String v2Sha256, String clashSha256)
throws IOException {
atomicWrite(accountDir.resolve(MANIFEST_FILE),
manifestText(generation, v2Sha256, clashSha256).getBytes(StandardCharsets.UTF_8));
}
/**
* 读取「一整个版本」:一次读入指针快照,再按该快照定位分代并校验双文件哈希。
*
* <p>指针只读一次,因此并发 {@link #publish} 切换指针时不会出现「用新分代名配旧哈希」
* 或哈希字段缺失导致的空指针;要么拿到完整的新版本,要么拿到完整的旧版本。
*/
private Pair load(Path account) {
Properties manifest = readManifest(account);
if (manifest == null) {
// 指针文件不存在才是「旧结构」。指针存在但读不出来说明它已损坏,同样按无缓存处理。
return Files.isRegularFile(account.resolve(MANIFEST_FILE)) ? null : loadLegacy(account);
}
Pair generation = loadGeneration(account, manifest);
if (generation != null)
return generation;
// 指针存在却读不出完整版本(半写、被篡改、内容缺失)时不得回退到 legacy:
// 那会发出一个比指针更旧的版本,掩盖损坏。此时按无可用缓存处理。
return null;
}
private Pair loadGeneration(Path account, Properties manifest) {
String generation = manifest.getProperty(MANIFEST_GENERATION);
if (generation == null || generation.isBlank() || generation.contains("/") || generation.contains("..")) {
log.warn("订阅缓存指针内容非法");
return null;
}
String v2Sha256 = manifest.getProperty(MANIFEST_V2_SHA);
String clashSha256 = manifest.getProperty(MANIFEST_CLASH_SHA);
if (v2Sha256 == null || clashSha256 == null) {
log.warn("订阅缓存指针缺少哈希字段 generation={}", generation);
return null;
}
Path dir = account.resolve(GENERATION_DIR).resolve(generation);
byte[] v2 = readFile(dir.resolve(V2_FILE));
byte[] clash = readFile(dir.resolve(CLASH_FILE));
if (v2 == null || clash == null) {
log.warn("订阅缓存分代内容缺失 generation={}", generation);
return null;
}
if (!matches(v2, v2Sha256) || !matches(clash, clashSha256)) {
log.warn("订阅缓存分代内容校验失败 generation={}", generation);
return null;
}
return new Pair(generation, v2, clash);
}
/** 旧结构:只有账号根下两份文件都在才构成一个有效版本。 */
private Pair loadLegacy(Path account) {
byte[] v2 = readFile(account.resolve(V2_FILE));
byte[] clash = readFile(account.resolve(CLASH_FILE));
if (v2 == null || clash == null)
return null;
return new Pair(LEGACY_GENERATION, v2, clash);
}
/** 指针读取接缝:包内可见,便于测试统计「每次查询只读一次指针」。 */
Properties readManifest(Path account) {
Path manifest = account.resolve(MANIFEST_FILE);
if (!Files.isRegularFile(manifest))
return null;
try {
Properties properties = new Properties();
properties.load(new StringReader(Files.readString(manifest, StandardCharsets.UTF_8)));
return properties;
} catch (IOException e) {
log.warn("读取订阅缓存指针失败: {}", e.getMessage());
return null;
}
}
private static byte[] readFile(Path path) {
try {
return Files.isRegularFile(path) ? Files.readAllBytes(path) : null;
} catch (IOException e) {
log.warn("读取订阅缓存文件失败 path={} error={}", path, e.getMessage());
return null;
}
}
private static void verify(Path path, byte[] expected) throws IOException {
byte[] written = Files.readAllBytes(path);
if (written.length != expected.length || !MessageDigest.isEqual(written, expected))
throw new IOException("订阅缓存写入校验失败 " + path.getFileName());
}
private static boolean matches(byte[] content, String expectedSha256) {
return expectedSha256 != null && expectedSha256.equals(sha256(content));
}
private static String manifestText(String generation, String v2Sha256, String clashSha256) {
return MANIFEST_GENERATION + "=" + generation + "\n"
+ MANIFEST_V2_SHA + "=" + v2Sha256 + "\n"
+ MANIFEST_CLASH_SHA + "=" + clashSha256 + "\n";
}
/** 分代名定宽可排序:时间戳在前保证单调,序号消除同毫秒碰撞。 */
private static String nextGeneration() {
return String.format("%013d-%06d", System.currentTimeMillis(), SEQUENCE.incrementAndGet() % 1_000_000);
}
private void prune(Path account, String current) {
Path dir = account.resolve(GENERATION_DIR);
List<Path> generations;
try (Stream<Path> entries = Files.list(dir)) {
generations = entries.filter(Files::isDirectory)
.sorted(Comparator.comparing(path -> path.getFileName().toString(), Comparator.reverseOrder()))
.toList();
} catch (IOException e) {
log.warn("清理订阅缓存分代失败 path={} error={}", dir, e.getMessage());
return;
}
for (int index = RETAINED_GENERATIONS; index < generations.size(); index++) {
Path stale = generations.get(index);
if (!stale.getFileName().toString().equals(current))
deleteRecursively(stale);
}
}
/** 分代发布成功后旧结构文件被取代,避免两份来源同时存在。 */
private void removeLegacy(Path account) {
deleteIfExists(account.resolve(V2_FILE));
deleteIfExists(account.resolve(CLASH_FILE));
}
private static void atomicWrite(Path target, byte[] data) throws IOException {
Path temp = target.resolveSibling(target.getFileName() + ".tmp");
Files.write(temp, data);
try {
Files.move(temp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(temp, target, StandardCopyOption.REPLACE_EXISTING);
}
}
private static void deleteIfExists(Path path) {
try {
Files.deleteIfExists(path);
} catch (IOException e) {
log.warn("删除订阅缓存文件失败 path={} error={}", path, e.getMessage());
}
}
private static void deleteRecursively(Path path) {
if (!Files.exists(path))
return;
try (Stream<Path> entries = Files.walk(path)) {
entries.sorted(Comparator.reverseOrder()).forEach(SubscriptionCacheStore::deleteIfExists);
} catch (IOException e) {
log.warn("删除订阅缓存目录失败 path={} error={}", path, e.getMessage());
}
}
private static String sha256(byte[] input) {
try {
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(input));
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
}
@@ -231,6 +231,22 @@ class MyBatisNativeConfigurationTest {
"同名 XML 映射文件应登记"); "同名 XML 映射文件应登记");
} }
@Test
void subscriptionEventMapperRegistersEventAndBindingReflection() {
var processor = new MyBatisNativeConfiguration.MyBatisBeanFactoryInitializationAotProcessor();
var factory = new org.springframework.beans.factory.support.DefaultListableBeanFactory();
var definition = new RootBeanDefinition(MapperFactoryBean.class);
definition.getPropertyValues().add("mapperInterface", com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper.class);
factory.registerBeanDefinition("subscriptionEventMapper", definition);
var hints = new RuntimeHints();
var contribution = processor.processAheadOfTime(factory);
assertNotNull(contribution);
contribution.applyTo(new StubGenerationContext(hints), new NoOpInitializationCode());
assertTrue(RuntimeHintsPredicates.reflection().onType(com.lion.lionwebsite.Domain.SubscriptionEvent.class).test(hints));
assertTrue(RuntimeHintsPredicates.reflection().onType(com.lion.lionwebsite.Domain.SubscriptionEventBinding.class).test(hints));
assertTrue(RuntimeHintsPredicates.proxies().forInterfaces(com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper.class).test(hints));
}
/** applyTo 只用 runtimeHints,初始化代码侧给一个记录型空实现即可。 */ /** applyTo 只用 runtimeHints,初始化代码侧给一个记录型空实现即可。 */
private static final class NoOpInitializationCode private static final class NoOpInitializationCode
implements org.springframework.beans.factory.aot.BeanFactoryInitializationCode { implements org.springframework.beans.factory.aot.BeanFactoryInitializationCode {
@@ -82,14 +82,7 @@ class PublicControllerTest {
verify(publicService, times(1)).logIpAddress(anyString()); verify(publicService, times(1)).logIpAddress(anyString());
} }
/** /** 身份直接内嵌为 JSON 对象,保留 result/data 字段。 */
* 返回体形如 {"result":"success","data":"{\"userId\": 7, ...}"}——
* data 是「JSON 文本的字符串」(历史契约,前端按字符串解析后再反序列化)。
*
* <p>内层 JSON 改由 ObjectMapper 生成(原先手工 String.format 拼接,用户名带引号
* 会产出非法 JSON),因此不再有冒号后的空格;这里断言的是字段与取值,
* 断言的字符串形式随之调整。
*/
@Test @Test
void validateReturnsIdentityAndNodeAvailability() throws Exception { void validateReturnsIdentityAndNodeAvailability() throws Exception {
when(publicService.getUserId("code")).thenReturn(new User(7, "code", "alice", null, true)); when(publicService.getUserId("code")).thenReturn(new User(7, "code", "alice", null, true));
@@ -97,11 +90,11 @@ class PublicControllerTest {
mockMvc.perform(post("/validate").param("AuthCode", "code")) mockMvc.perform(post("/validate").param("AuthCode", "code"))
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(content().string(org.hamcrest.Matchers.containsString("\\\"userId\\\":7"))) .andExpect(content().string(org.hamcrest.Matchers.containsString("\"userId\":7")))
.andExpect(content().string(org.hamcrest.Matchers.containsString("\\\"username\\\":\\\"alice\\\""))) .andExpect(content().string(org.hamcrest.Matchers.containsString("\"username\":\"alice\"")))
.andExpect(content().string(org.hamcrest.Matchers.containsString("\\\"isAvailable\\\":true"))) .andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAvailable\":true")))
// 普通用户 isAdmin=false,前端据此隐藏下载人信息。 // 普通用户 isAdmin=false,前端据此隐藏下载人信息。
.andExpect(content().string(org.hamcrest.Matchers.containsString("\\\"isAdmin\\\":false"))); .andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAdmin\":false")));
} }
/** 管理员(id=3)必须带上 isAdmin=true,前端据此显示下载人信息与筛选。 */ /** 管理员(id=3)必须带上 isAdmin=true,前端据此显示下载人信息与筛选。 */
@@ -112,7 +105,7 @@ class PublicControllerTest {
mockMvc.perform(post("/validate").param("AuthCode", "admin")) mockMvc.perform(post("/validate").param("AuthCode", "admin"))
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(content().string(org.hamcrest.Matchers.containsString("\\\"isAdmin\\\":true"))); .andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAdmin\":true")));
} }
/** /**
@@ -129,11 +122,10 @@ class PublicControllerTest {
.andExpect(status().isOk()) .andExpect(status().isOk())
.andReturn(); .andReturn();
// 取出外层 data(是内层 JSON 的字符串形式),必须能被 JSON 解析回原名。 // 直接读取结构化 data,特殊字符保持原样。
var outer = com.lion.lionwebsite.Util.CustomUtil.objectMapper var outer = com.lion.lionwebsite.Util.CustomUtil.objectMapper
.readTree(result.getResponse().getContentAsString()); .readTree(result.getResponse().getContentAsString());
var inner = com.lion.lionwebsite.Util.CustomUtil.objectMapper var inner = outer.get("data");
.readTree(outer.get("data").asText());
assertEquals("a\"b\\c", inner.get("username").asText(), "特殊字符必须原样保留且 JSON 合法"); assertEquals("a\"b\\c", inner.get("username").asText(), "特殊字符必须原样保留且 JSON 合法");
} }
@@ -145,7 +137,7 @@ class PublicControllerTest {
mockMvc.perform(post("/validate").param("AuthCode", "code")) mockMvc.perform(post("/validate").param("AuthCode", "code"))
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(content().string(org.hamcrest.Matchers.containsString("\\\"isAvailable\\\":false"))); .andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAvailable\":false")));
} }
@Test @Test
@@ -0,0 +1,219 @@
package com.lion.lionwebsite.Dao.cache;
import com.lion.lionwebsite.Domain.GidToKey;
import com.lion.lionwebsite.Domain.ImageKeyCache;
import org.apache.ibatis.datasource.unpooled.UnpooledDataSource;
import org.apache.ibatis.mapping.Environment;
import org.apache.ibatis.session.Configuration;
import org.apache.ibatis.session.SqlSession;
import org.apache.ibatis.session.SqlSessionFactory;
import org.apache.ibatis.session.SqlSessionFactoryBuilder;
import org.apache.ibatis.transaction.jdbc.JdbcTransactionFactory;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import static org.junit.jupiter.api.Assertions.*;
/**
* ImageCacheMapper 的实际 SQL 验证。
*
* <p>服务层测试用的是内存替身,无法覆盖注解 SQL 本身(尤其是批量 INSERT 的
* {@code <script>} 动态语句与唯一索引冲突)。这里用一个真实的 MyBatis
* SqlSessionFactory 打到临时 SQLite 文件上,不依赖 Spring 容器,也不碰生产配置。
*
* <p>表结构刻意保持与线上一致(两列条件唯一索引),这样「唯一约束确实拦住了重复写入」
* 也是在真实 SQLite 上被验证的。
*/
class ImageCacheMapperSqlTest {
private static final String GID = "1596929";
@TempDir
Path tempDir;
private Path database;
private SqlSessionFactory factory;
@BeforeEach
void setUp() throws Exception {
database = tempDir.resolve("cache.db");
try (Connection connection = DriverManager.getConnection("jdbc:sqlite:" + database);
Statement statement = connection.createStatement()) {
statement.execute("create table gidToKey (gid varchar(20), key varchar(20), pages integer)");
statement.execute("create table ImageKeyCache (gid varchar(20), imgkey varchar(20), page integer)");
// 与迁移脚本最终建立的一致:按 gid 唯一、按 (gid,page) 唯一。
statement.execute("create unique index ux_gidToKey_gid on gidToKey (gid)");
statement.execute("create unique index ux_ImageKeyCache_gid_page on ImageKeyCache (gid, page)");
}
UnpooledDataSource dataSource = new UnpooledDataSource("org.sqlite.JDBC", "jdbc:sqlite:" + database, null, null);
Environment environment = new Environment("cache-test", new JdbcTransactionFactory(), dataSource);
Configuration configuration = new Configuration(environment);
configuration.addMapper(ImageCacheMapper.class);
factory = new SqlSessionFactoryBuilder().build(configuration);
}
@AfterEach
void tearDown() {
factory = null;
}
private <T> T inSession(SessionWork<T> work) {
try (SqlSession session = factory.openSession(true)) {
return work.run(session.getMapper(ImageCacheMapper.class));
}
}
private interface SessionWork<T> {
T run(ImageCacheMapper mapper);
}
// ---------- 批量写入 ----------
@Test
void batchInsertWritesEveryRowIncludingMultipleBatches() {
List<ImageKeyCache> rows = new ArrayList<>();
for (int page = 1; page <= 250; page++)
rows.add(page(GID, page, "k" + page));
inSession(mapper -> {
mapper.insertGidToKey(header(GID, "f08534d87d", 250));
for (int start = 0; start < rows.size(); start += 200) {
int end = Math.min(start + 200, rows.size());
mapper.insertImageKeyCacheBatch(new ArrayList<>(rows.subList(start, end)));
}
return null;
});
inSession(mapper -> {
assertEquals(250, mapper.countImageKeyCacheByGid(GID));
assertEquals(250, mapper.selectPageNumbersByGid(GID).size());
assertEquals("k1", mapper.selectImageKeyCacheByGidAndPage(GID, 1).getImgkey());
assertEquals("k250", mapper.selectImageKeyCacheByGidAndPage(GID, 250).getImgkey());
return null;
});
}
@Test
void pageNumbersComeBackAscendingEvenWhenRowsWereInsertedOutOfOrder() {
inSession(mapper -> {
mapper.insertGidToKey(header(GID, "key", 3));
mapper.insertImageKeyCacheBatch(List.of(
page(GID, 3, "c"), page(GID, 1, "a"), page(GID, 2, "b")));
return null;
});
inSession(mapper -> {
assertEquals(List.of(1, 2, 3), mapper.selectPageNumbersByGid(GID));
return null;
});
}
// ---------- 完整性辅助查询 ----------
@Test
void invalidImageKeyCountCatchesNullAndBlank() {
inSession(mapper -> {
mapper.insertGidToKey(header(GID, "key", 3));
mapper.insertImageKeyCacheBatch(List.of(
page(GID, 1, "ok"), page(GID, 2, " "), page(GID, 3, "")));
assertEquals(2, mapper.countInvalidImageKeysByGid(GID),
"空白与空串都必须被算作无效 imgkey");
return null;
});
}
@Test
void pageNumbersForUnknownGidIsEmptyNotNull() {
inSession(mapper -> {
assertNotNull(mapper.selectPageNumbersByGid("999"));
assertTrue(mapper.selectPageNumbersByGid("999").isEmpty());
assertNull(mapper.selectKeyByGid("999"));
return null;
});
}
// ---------- 唯一约束 ----------
@Test
void uniqueIndexRejectsDuplicatePageAndDuplicateHeader() {
inSession(mapper -> {
mapper.insertGidToKey(header(GID, "key", 1));
mapper.insertImageKeyCacheBatch(List.of(page(GID, 1, "a")));
return null;
});
assertThrows(RuntimeException.class, () -> inSession(mapper -> {
mapper.insertImageKeyCacheBatch(List.of(page(GID, 1, "duplicate")));
return null;
}), "重复 (gid,page) 必须被唯一索引拒绝");
assertThrows(RuntimeException.class, () -> inSession(mapper -> {
mapper.insertGidToKey(header(GID, "other", 1));
return null;
}), "重复 gid 必须被唯一索引拒绝");
}
@Test
void deleteByGidRemovesBothHeaderAndPages() {
inSession(mapper -> {
mapper.insertGidToKey(header(GID, "key", 2));
mapper.insertImageKeyCacheBatch(List.of(page(GID, 1, "a"), page(GID, 2, "b")));
return null;
});
inSession(mapper -> {
mapper.deleteImageKeyCacheByGid(GID);
mapper.deleteGidToKey(GID);
return null;
});
inSession(mapper -> {
assertNull(mapper.selectKeyByGid(GID));
assertEquals(0, mapper.countImageKeyCacheByGid(GID));
return null;
});
}
@Test
void singleRowInsertStillWorksForLegacyCallers() {
inSession(mapper -> {
mapper.insertGidToKey(header(GID, "key", 1));
mapper.insertImageKeyCache(page(GID, 1, "legacy"));
return null;
});
inSession(mapper -> {
assertEquals("legacy", mapper.selectImageKeyCacheByGidAndPage(GID, 1).getImgkey());
return null;
});
}
// ---------- 辅助 ----------
private static GidToKey header(String gid, String key, int pages) {
GidToKey header = new GidToKey();
header.setGid(gid);
header.setKey(key);
header.setPages(pages);
return header;
}
private static ImageKeyCache page(String gid, int page, String imgkey) {
ImageKeyCache row = new ImageKeyCache();
row.setGid(gid);
row.setPage(page);
row.setImgkey(imgkey);
return row;
}
}
@@ -0,0 +1,249 @@
package com.lion.lionwebsite.Maintenance;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;
/**
* 离线体检与修复工具的行为约束。
*
* <p>用临时 SQLite 文件建出真实表结构,因此这里验证的是工具实际发出的 SQL
* (含判定条件与删除范围),而不是内存替身。
*/
class GalleryIndexMaintenanceTest {
private static final String SCHEMA = """
create table gidToKey (gid varchar(20), key varchar(20), pages integer);
create table ImageKeyCache (gid varchar(20), imgkey varchar(20), page integer);
""";
@TempDir
Path tempDir;
private Path database;
private Connection open() throws SQLException {
return DriverManager.getConnection("jdbc:sqlite:" + database);
}
private void exec(Connection connection, String sql) throws SQLException {
try (Statement statement = connection.createStatement()) {
statement.executeUpdate(sql);
}
}
/** 建一份「一好三坏」的缓存库:正常索引、半截索引、重复 gid、孤儿页 key。 */
private void seed() throws Exception {
database = tempDir.resolve("cache.db");
try (Connection connection = open()) {
for (String statement : SCHEMA.split(";"))
if (!statement.isBlank())
exec(connection, statement);
// 1. 正常:3 页,页码 1..3
exec(connection, "insert into gidToKey values ('100', 'sk100', 3)");
exec(connection, "insert into ImageKeyCache values ('100', 'a', 1)");
exec(connection, "insert into ImageKeyCache values ('100', 'b', 2)");
exec(connection, "insert into ImageKeyCache values ('100', 'c', 3)");
// 2. 半截:声称 3 页,只有 1、3 两页
exec(connection, "insert into gidToKey values ('200', 'sk200', 3)");
exec(connection, "insert into ImageKeyCache values ('200', 'x', 1)");
exec(connection, "insert into ImageKeyCache values ('200', 'y', 3)");
// 3. 重复 gid:两条头部记录
exec(connection, "insert into gidToKey values ('300', 'sk300', 1)");
exec(connection, "insert into gidToKey values ('300', 'sk300b', 1)");
exec(connection, "insert into ImageKeyCache values ('300', 'z', 1)");
// 4. 孤儿页 key:没有头部记录
exec(connection, "insert into ImageKeyCache values ('400', 'orphan', 1)");
}
}
private String scalar(Connection connection, String sql) throws SQLException {
try (Statement statement = connection.createStatement();
ResultSet rows = statement.executeQuery(sql)) {
return rows.next() ? rows.getString(1) : null;
}
}
@Test
void dryRunInspectReportsProblemsWithoutWriting() throws Exception {
seed();
String before;
try (Connection connection = open()) {
before = scalar(connection, "select count(*) || '/' || (select count(*) from ImageKeyCache) from gidToKey");
}
GalleryIndexMaintenance.Report report;
try (Connection connection = open()) {
report = GalleryIndexMaintenance.inspect(connection);
assertFalse(report.isHealthy());
assertEquals(4, report.headerRows, "gidToKey 应有 4 行(含重复 gid)");
assertEquals(7, report.pageRows);
assertEquals(1, report.duplicateHeaderGids, "gid=300 是重复组");
assertEquals(List.of("200", "300"), report.badGids, "半截 200 与重复 300 都应被判为待修复");
assertEquals(List.of("400"), report.orphanGids);
assertTrue(report.badReasons.get("200").contains("页 key 行数"),
"原因应说明行数不符,实际: " + report.badReasons.get("200"));
// gid=200 的页码是 1 和 3:区间端点看起来正常,必须靠「去重后个数」这条判断抓出来。
assertTrue(report.badReasons.get("200").contains("页码去重后"),
"原因应说明页码数量不足,实际: " + report.badReasons.get("200"));
assertTrue(report.badReasons.get("300").contains("页 key 行数")
|| report.badReasons.get("300").contains("重复"),
"重复 gid 必须给出可读原因,实际: " + report.badReasons.get("300"));
assertTrue(report.render("体检结果").contains("需要修复"));
}
try (Connection connection = open()) {
assertEquals(before, scalar(connection,
"select count(*) || '/' || (select count(*) from ImageKeyCache) from gidToKey"),
"体检必须是只读的");
}
}
@Test
void repairRemovesOnlyBrokenGroupsAndLeavesHealthyIndexIntact() throws Exception {
seed();
try (Connection connection = open()) {
GalleryIndexMaintenance.Report before = GalleryIndexMaintenance.inspect(connection);
connection.setAutoCommit(false);
int touched = GalleryIndexMaintenance.repair(connection, before.badGids, before.orphanGids);
connection.commit();
assertEquals(3, touched, "应处理 200、300、400 三个 gid");
GalleryIndexMaintenance.Report after = GalleryIndexMaintenance.inspect(connection);
assertTrue(after.isHealthy(), "修复后应健康,原因: " + after.render("修复后"));
assertEquals(1, after.headerRows, "只应剩下正常索引 gid=100");
assertEquals(3, after.pageRows);
assertTrue(after.badGids.isEmpty());
assertTrue(after.orphanGids.isEmpty());
// 正常索引必须逐行保留,不能被误删。
assertEquals("3", scalar(connection, "select pages from gidToKey where gid='100'"));
assertEquals("a", scalar(connection, "select imgkey from ImageKeyCache where gid='100' and page=1"));
assertEquals("c", scalar(connection, "select imgkey from ImageKeyCache where gid='100' and page=3"));
assertEquals("0", scalar(connection, "select count(*) from ImageKeyCache where gid in ('200','300','400')"));
}
}
@Test
void emptyDatabaseIsHealthy() throws Exception {
database = tempDir.resolve("empty.db");
try (Connection connection = open()) {
for (String statement : SCHEMA.split(";"))
if (!statement.isBlank())
exec(connection, statement);
GalleryIndexMaintenance.Report report = GalleryIndexMaintenance.inspect(connection);
assertTrue(report.isHealthy());
assertEquals(0, report.headerRows);
}
}
@Test
void missingTablesAreReportedInsteadOfCrashing() throws Exception {
database = tempDir.resolve("nothing.db");
try (Connection connection = open()) {
GalleryIndexMaintenance.Report report = GalleryIndexMaintenance.inspect(connection);
assertFalse(report.isHealthy());
assertFalse(report.hasHeaderTable);
assertFalse(report.hasPageTable);
assertTrue(report.render("体检结果").contains("缺少缓存表"));
}
}
@Test
void productionPathsAreRecognisedSoTheToolCanRefuseToWrite() {
assertTrue(GalleryIndexMaintenance.isProductionPath(Path.of("/home/lionwebsite/cache.db")));
assertTrue(GalleryIndexMaintenance.isProductionPath(Path.of("/home/lionwebsite/sub/deep/cache.db")));
assertFalse(GalleryIndexMaintenance.isProductionPath(Path.of("/tmp/cache.db")));
assertFalse(GalleryIndexMaintenance.isProductionPath(tempDir.resolve("cache.db")));
}
@Test
void gapInPageNumbersIsDetectedEvenWhenRowCountMatches() throws Exception {
// 2 行、page 分别是 1 和 3:只比行数会漏判。
database = tempDir.resolve("gap.db");
try (Connection connection = open()) {
for (String statement : SCHEMA.split(";"))
if (!statement.isBlank())
exec(connection, statement);
exec(connection, "insert into gidToKey values ('500', 'sk500', 2)");
exec(connection, "insert into ImageKeyCache values ('500', 'p1', 1)");
exec(connection, "insert into ImageKeyCache values ('500', 'p3', 3)");
GalleryIndexMaintenance.Report report = GalleryIndexMaintenance.inspect(connection);
assertFalse(report.isHealthy());
assertEquals(List.of("500"), report.badGids);
}
}
@Test
void blankImgKeyMakesGroupUnhealthy() throws Exception {
database = tempDir.resolve("blank.db");
try (Connection connection = open()) {
for (String statement : SCHEMA.split(";"))
if (!statement.isBlank())
exec(connection, statement);
exec(connection, "insert into gidToKey values ('600', 'sk600', 2)");
exec(connection, "insert into ImageKeyCache values ('600', 'ok', 1)");
exec(connection, "insert into ImageKeyCache values ('600', ' ', 2)");
GalleryIndexMaintenance.Report report = GalleryIndexMaintenance.inspect(connection);
assertEquals(List.of("600"), report.badGids);
assertTrue(report.badReasons.get("600").contains("imgkey 为空"),
"实际: " + report.badReasons.get("600"));
}
}
@Test
void reportRenderingIsReadableAndTruncatesLongLists() throws Exception {
database = tempDir.resolve("many.db");
try (Connection connection = open()) {
for (String statement : SCHEMA.split(";"))
if (!statement.isBlank())
exec(connection, statement);
for (int i = 0; i < 25; i++)
exec(connection, "insert into gidToKey values ('7" + i + "', 'k', 1)");
GalleryIndexMaintenance.Report report = GalleryIndexMaintenance.inspect(connection);
String rendered = report.render("体检结果");
assertEquals(25, report.badGids.size());
assertTrue(rendered.contains("其余 5 个略"), "应截断超长列表,实际:\n" + rendered);
}
}
/** 修复是幂等的:对已健康的库再跑一次不应改变任何东西。 */
@Test
void repairIsIdempotent() throws Exception {
seed();
try (Connection connection = open()) {
GalleryIndexMaintenance.Report first = GalleryIndexMaintenance.inspect(connection);
connection.setAutoCommit(false);
GalleryIndexMaintenance.repair(connection, first.badGids, first.orphanGids);
connection.commit();
GalleryIndexMaintenance.Report second = GalleryIndexMaintenance.inspect(connection);
connection.setAutoCommit(false);
int touched = GalleryIndexMaintenance.repair(connection, second.badGids, second.orphanGids);
connection.commit();
assertEquals(0, touched, "已健康的库再次修复不应删除任何东西");
assertTrue(GalleryIndexMaintenance.inspect(connection).isHealthy());
}
}
}
@@ -0,0 +1,39 @@
package com.lion.lionwebsite.Message;
import tools.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import java.util.Set;
import java.util.HashSet;
import static org.junit.jupiter.api.Assertions.*;
/** Both ends assert this exact wire shape; helper getters must not leak onto the wire. */
class ReliableWireContractTest {
private final ObjectMapper mapper = new ObjectMapper();
private void fields(Object message, Set<String> allowed) throws Exception {
var tree = mapper.readTree(mapper.writeValueAsString(message));
Set<String> actual = new HashSet<>();
tree.propertyNames().forEach(actual::add);
assertEquals(allowed, actual);
}
@Test void commandWireShapeAndGoldenInput() throws Exception {
var command = new TaskCommandMessage();
command.setMessageId(71); command.setOperationId("op-download");
command.setAction("DOWNLOAD"); command.setGid(17); command.setName("中文画廊");
fields(command, Set.of("messageType","messageId","operationId","action","gid","name"));
var input = mapper.readValue("{\"messageType\":11,\"messageId\":71,\"operationId\":\"op-download\",\"action\":\"DOWNLOAD\",\"gid\":17,\"name\":\"中文画廊\"}", TaskCommandMessage.class);
assertEquals(command.getOperationId(), input.getOperationId());
assertEquals(command.getGid(), input.getGid());
}
@Test void reportWireShapeAndGoldenInput() throws Exception {
var report = mapper.readValue("{\"messageType\":12,\"messageId\":72,\"reportId\":\"report-1\",\"revision\":9007199254,\"galleryTasks\":[{\"gid\":17,\"name\":\"中文画廊\",\"status\":4,\"proceeding\":20}],\"operationIds\":{\"17\":\"op-download\"}}", TaskReportMessage.class);
fields(report, Set.of("messageType","messageId","reportId","revision","galleryTasks","operationIds"));
assertEquals(9007199254L, report.getRevision());
assertEquals("op-download", report.getOperationIds().get(17));
assertEquals(20, report.getGalleryTasks()[0].getProceeding());
}
@Test void ackWireShapeAndGoldenInput() throws Exception {
var ack = mapper.readValue("{\"messageType\":13,\"messageId\":73,\"reportId\":\"report-1\",\"result\":0}", TaskReportAckMessage.class);
fields(ack, Set.of("messageType","messageId","reportId","result"));
assertEquals(0, ack.getResult());
}
}
@@ -0,0 +1,152 @@
package com.lion.lionwebsite.Message;
import com.lion.lionwebsite.Domain.GalleryTask;
import io.netty.buffer.ByteBuf;
import io.netty.channel.embedded.EmbeddedChannel;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;
/**
* 协议 v2(类型 10..13)的线格式。
*
* <p>旧类型 0..9 的字段布局不变(旧节点 Jackson 会拒绝未知字段),新类型沿用
* 「1 字节类型 + 4 字节大端长度 + JSON」的既有帧格式,因此这里同时锁住帧头与字段名,
* 作为与 storageNode 的契约样本。
*/
class TaskProtocolV2CodecTest {
private EmbeddedChannel channel;
@AfterEach
void tearDown() {
if (channel != null) channel.finishAndReleaseAll();
}
private EmbeddedChannel channel() {
channel = new EmbeddedChannel(new MessageCodec());
return channel;
}
private static String frameBody(ByteBuf frame) {
byte[] body = new byte[frame.getInt(1)];
frame.getBytes(5, body);
return new String(body, StandardCharsets.UTF_8);
}
private static <T> T roundTrip(EmbeddedChannel ch, AbstractMessage message, Class<T> type) {
assertTrue(ch.writeOutbound(message), "消息应被编码");
ByteBuf frame = ch.readOutbound();
assertEquals(message.messageType, frame.getByte(0), "帧首字节应为 messageType");
assertEquals(frame.getInt(1), frameBody(frame).getBytes(StandardCharsets.UTF_8).length,
"帧头长度须等于正文实际字节数");
assertTrue(ch.writeInbound(frame));
Object decoded = ch.readInbound();
assertInstanceOf(type, decoded);
return type.cast(decoded);
}
/** 能力协商:协议版本与特性列表按合同字段名传输。 */
@Test
void helloCarriesVersionAndFeatures() {
ProtocolHelloMessage hello = new ProtocolHelloMessage();
hello.setMessageId(7);
ProtocolHelloMessage decoded = roundTrip(channel(), hello, ProtocolHelloMessage.class);
assertEquals(2, decoded.getProtocolVersion());
assertTrue(decoded.getFeatures().contains("reliable-task-v2"));
assertTrue(decoded.supportsReliableTasks());
assertEquals(7, decoded.getMessageId(), "既有 messageId 字段必须保留");
}
/** 未声明 v2 的对端不得被当作可靠协议可用。 */
@Test
void helloWithoutV2FeatureDoesNotSupportReliableTasks() {
ProtocolHelloMessage legacy = new ProtocolHelloMessage();
legacy.setProtocolVersion(1);
legacy.setFeatures(java.util.List.of());
assertFalse(legacy.supportsReliableTasks());
}
/** 任务命令:稳定 operationId + 动作 + gid + 名称。 */
@Test
void taskCommandCarriesStableOperationIdAndAction() {
TaskCommandMessage command = new TaskCommandMessage();
command.setMessageId(11);
command.setOperationId("dl-abc");
command.setAction(TaskCommandMessage.ACTION_DELETE);
command.setGid(998);
command.setName("sample [998]");
TaskCommandMessage decoded = roundTrip(channel(), command, TaskCommandMessage.class);
assertEquals("dl-abc", decoded.getOperationId());
assertEquals("DELETE", decoded.getAction());
assertEquals(998, decoded.getGid());
assertEquals("sample [998]", decoded.getName());
}
/** 状态上报:reportId、revision、任务数组与 per-gid 的 operationId 映射。 */
@Test
void taskReportCarriesRevisionTasksAndOperationIds() {
GalleryTask task = new GalleryTask();
task.setGid(5);
task.setName("g [5]");
task.setStatus(GalleryTask.DOWNLOADING);
task.setProceeding(3);
Map<Integer, String> operationIds = new LinkedHashMap<>();
operationIds.put(5, "dl-5");
TaskReportMessage report = new TaskReportMessage();
report.setReportId("r-1");
report.setRevision(42);
report.setGalleryTasks(new GalleryTask[]{task});
report.setOperationIds(operationIds);
TaskReportMessage decoded = roundTrip(channel(), report, TaskReportMessage.class);
assertEquals("r-1", decoded.getReportId());
assertEquals(42L, decoded.getRevision());
assertEquals(1, decoded.getGalleryTasks().length);
assertEquals(5, decoded.getGalleryTasks()[0].getGid());
assertEquals(GalleryTask.DOWNLOADING, decoded.getGalleryTasks()[0].getStatus());
assertEquals("dl-5", decoded.getOperationIds().get(5));
}
/** 回执:只带 reportId 与结果码。 */
@Test
void reportAckCarriesReportIdAndResult() {
TaskReportAckMessage ack = new TaskReportAckMessage("r-9", TaskReportAckMessage.RESULT_SUCCESS);
ack.setMessageId(13);
TaskReportAckMessage decoded = roundTrip(channel(), ack, TaskReportAckMessage.class);
assertEquals("r-9", decoded.getReportId());
assertEquals((byte) 0, decoded.getResult());
assertEquals(13, decoded.getMessageId());
}
/** 旧类型(下载投递)必须保持原样:不能因为新增 v2 类型而改动它的字段。 */
@Test
void legacyDownloadPostMessageKeepsOriginalShape() {
GalleryTask task = new GalleryTask();
task.setGid(1);
task.setName("legacy");
DownloadPostMessage message = new DownloadPostMessage();
message.setGalleryTask(task);
assertTrue(channel().writeOutbound(message));
ByteBuf frame = channel.readOutbound();
assertEquals(AbstractMessage.DOWNLOAD_POST_MESSAGE, frame.getByte(0));
String json = frameBody(frame);
assertTrue(json.contains("galleryTask"), "旧字段名不能改: " + json);
assertFalse(json.contains("operationId"), "旧类型不得被塞入新字段: " + json);
}
}
@@ -0,0 +1,696 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.cache.ImageCacheMapper;
import com.lion.lionwebsite.Domain.GidToKey;
import com.lion.lionwebsite.Domain.ImageKeyCache;
import com.lion.lionwebsite.Exception.GalleryIndexException;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.transaction.support.SimpleTransactionStatus;
import org.springframework.transaction.support.TransactionCallback;
import org.springframework.transaction.support.TransactionOperations;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import static org.junit.jupiter.api.Assertions.*;
/**
* 图片索引服务的行为约束。
*
* <p>这些用例不依赖 Spring 容器、MyBatis 与网络:DB 用内存假实现,
* 事务用带快照回滚的假实现,上游用可控的假解析器。
* 因此它们验证的是服务自己的编排逻辑,而不是 SQLite 的行为。
*/
class GalleryIndexServiceTest {
private static final String URL = "https://exhentai.org/g/1596929/f08534d87d/";
private static final String GID = "1596929";
private static final String KEY = "f08534d87d";
private FakeMapper mapper;
/** 写事务(发布索引)探针。 */
private FakeTransactions transactions;
/** 读事务(快照读取)探针,独立计数,避免和写事务混淆。 */
private FakeTransactions reads;
private GalleryIndexService service;
@BeforeEach
void setUp() {
mapper = new FakeMapper();
transactions = new FakeTransactions(mapper);
reads = new FakeTransactions(mapper, true);
mapper.transactionsForProbe = transactions;
service = new GalleryIndexService(mapper, transactions, reads, url -> mapper.fetch());
}
// ---------- 正常路径 ----------
@Test
void buildsOnceThenServesFromCache() throws Exception {
GidToKey index = service.ensureIndex(URL);
assertEquals(GID, index.getGid());
assertEquals(KEY, index.getKey());
assertEquals(3, index.getPages());
assertEquals(1, mapper.fetchCount.get(), "首次访问应抓取上游");
assertEquals(1, transactions.commits.get(), "应只发布一次");
GidToKey again = service.ensureIndex(URL);
assertEquals(3, again.getPages());
assertEquals(1, mapper.fetchCount.get(), "二次访问必须命中缓存,不得再抓上游");
assertEquals(1, transactions.commits.get(), "命中缓存不应触发写入");
}
@Test
void concurrentCallersShareOneUpstreamFetch() throws Exception {
int callers = 24;
ExecutorService pool = Executors.newFixedThreadPool(callers);
CountDownLatch start = new CountDownLatch(1);
List<Future<GidToKey>> futures = new ArrayList<>();
for (int i = 0; i < callers; i++) {
futures.add(pool.submit(() -> {
start.await();
return service.ensureIndex(URL);
}));
}
start.countDown();
for (Future<GidToKey> future : futures) {
GidToKey index = future.get(20, TimeUnit.SECONDS);
assertEquals(GID, index.getGid());
assertEquals(3, index.getPages());
}
pool.shutdownNow();
assertEquals(1, mapper.fetchCount.get(), "同 gid 必须合并为一次上游抓取");
assertEquals(1, transactions.commits.get(), "同 gid 必须只发布一次");
}
@Test
void upstreamFetchRunsOutsideTransactionAndAllWritesInsideIt() throws Exception {
service.ensureIndex(URL);
assertEquals(0, mapper.upstreamSawTransaction.get(),
"抓取上游时不应有任何事务在执行,否则会长时间占住 SQLite 写锁");
assertTrue(mapper.insertSawTransaction.get() > 0, "写库必须在事务内");
assertEquals(0, mapper.mutationsOutsideTransaction.get(),
"任何一次删除或写入都不得落在事务之外,否则整组替换就不是原子的");
}
@Test
void wholeReplacementIsOneTransaction() throws Exception {
// 先放一份不完整索引,强制走一次完整重建。
mapper.putHeader(GID, KEY, 9);
mapper.putPage(GID, 1, "old");
service.ensureIndex(URL);
assertEquals(1, transactions.commits.get(), "删除加写入必须在同一个事务里提交");
assertEquals(1, mapper.transactionAttempts.get(), "不应出现多个独立写事务");
}
@Test
void cacheHitReadsInsideOneSnapshotAndOpensNoWriteTransaction() throws Exception {
service.ensureIndex(URL);
int writesAfterBuild = transactions.started.get();
int readsAfterBuild = reads.started.get();
service.ensureIndex(URL);
assertEquals(writesAfterBuild, transactions.started.get(),
"命中缓存不应创建任何写事务");
assertEquals(readsAfterBuild + 1, reads.started.get(),
"每次命中缓存都应恰好用一个读事务读取头部与页码");
}
@Test
void missFirstLooksUpUnderReadSnapshotThenRechecksInsideTheBuildCriticalSection() throws Exception {
service.ensureIndex(URL);
// 首次构建:先做一次一致性读确认未命中,进入临界区后再确认一次。
assertTrue(reads.started.get() >= 2,
"构建前与进入临界区后都应各自读一次快照,实际: " + reads.started.get());
assertEquals(1, transactions.started.get(),
"无论读了几次,写事务只能有一个");
}
// ---------- 索引完整性 ----------
@Test
void incompleteIndexIsRebuilt() throws Exception {
// 头部声称 3 页,实际只有 1、3 两页:半截且跳号。
mapper.putHeader(GID, KEY, 3);
mapper.putPage(GID, 1, "old1");
mapper.putPage(GID, 3, "old3");
GidToKey index = service.ensureIndex(URL);
assertEquals(3, index.getPages());
assertEquals(1, mapper.fetchCount.get(), "半截索引必须重新抓取");
assertEquals(List.of(1, 2, 3), mapper.pageNumbers(GID));
assertEquals("k1", mapper.pageKey(GID, 1));
}
@Test
void pageNumberGapIsTreatedAsIncompleteEvenWhenRowCountMatches() throws Exception {
// 行数刚好等于 pages,但页码是 1 和 3:只比行数会误判为完整。
mapper.putHeader(GID, KEY, 2);
mapper.putPage(GID, 1, "old1");
mapper.putPage(GID, 3, "old3");
mapper.pagesForFetch = List.of("n1", "n2");
GidToKey index = service.ensureIndex(URL);
assertEquals(1, mapper.fetchCount.get(), "页码跳号必须触发重建");
assertEquals(2, index.getPages());
assertEquals(List.of(1, 2), mapper.pageNumbers(GID));
}
@Test
void blankImgKeyIsTreatedAsIncomplete() throws Exception {
mapper.putHeader(GID, KEY, 3);
mapper.putPage(GID, 1, "ok");
mapper.putPage(GID, 2, " ");
mapper.putPage(GID, 3, "ok3");
service.ensureIndex(URL);
assertEquals(1, mapper.fetchCount.get(), "空 imgkey 必须触发重建");
assertEquals("k2", mapper.pageKey(GID, 2));
}
@Test
void headerWithoutPagesOrKeyIsTreatedAsIncomplete() throws Exception {
mapper.putHeader(GID, KEY, 0);
service.ensureIndex(URL);
assertEquals(1, mapper.fetchCount.get(), "pages 非法必须触发重建");
mapper.clear();
mapper.putHeader(GID, " ", 3);
service.ensureIndex(URL);
assertEquals(2, mapper.fetchCount.get(), "key 为空也必须触发重建");
}
// ---------- 上游畸形数据(必须在写事务之前拦下) ----------
@Test
void malformedUpstreamRowsAreRejectedBeforeAnyTransaction() throws Exception {
// 先放一份旧索引,用来断言畸形数据不会把它改掉。
mapper.putHeader(GID, KEY, 2);
mapper.putPage(GID, 1, "old1");
mapper.putPage(GID, 2, "old2");
mapper.forceIncomplete(GID);
Map<String, String> before = mapper.stateFingerprint();
List<List<ImageKeyCache>> malformedCases = List.of(
rows(row(GID, 1, "k"), row("999999", 2, "k")), // gid 不一致
rows(row(GID, 1, "k"), row(GID, 1, "k2")), // 页码重复
rows(row(GID, 1, "k"), row(GID, 3, "k2")), // 页码跳号
rows(row(GID, 1, "k"), row(GID, 2, " ")), // imgkey 为空
rows(row(GID, 0, "k"), row(GID, 1, "k2")), // 页码从 0 开始
rows(row(GID, 1, "k"), row(GID, 5, "k2"))); // 页码越界
for (List<ImageKeyCache> malformed : malformedCases) {
mapper.rawRowsForFetch = malformed;
GalleryIndexException error = assertThrows(GalleryIndexException.class,
() -> service.ensureIndex(URL), "应拒绝畸形数据: " + describe(malformed));
assertEquals(GalleryIndexException.Reason.MALFORMED, error.getReason(),
"应归为上游数据畸形: " + describe(malformed));
assertFalse(error.isPermanent(), "上游数据畸形可以重试");
assertEquals(0, transactions.commits.get(), "畸形数据不得写入任何东西");
assertEquals(0, transactions.rollbacks.get(), "应在开启写事务之前就失败");
assertEquals(before, mapper.stateFingerprint(), "旧索引必须原样保留: " + describe(malformed));
}
}
@Test
void validationRunsBeforeTheWriteTransactionOpens() throws Exception {
mapper.rawRowsForFetch = rows(row(GID, 1, "k"), row(GID, 1, "k2"));
assertThrows(GalleryIndexException.class, () -> service.ensureIndex(URL));
assertEquals(0, mapper.transactionAttempts.get(),
"畸形数据应在事务创建之前就被拒绝,不产生任何写事务");
assertEquals(0, mapper.mutationsOutsideTransaction.get(),
"更不允许事务外写入");
}
@Test
void wellFormedButOutOfOrderRowsAreAccepted() throws Exception {
// 上游行序无保证:页码乱序但集合正确时应当接受。
mapper.rawRowsForFetch = rows(row(GID, 3, "c"), row(GID, 1, "a"), row(GID, 2, "b"));
GidToKey index = service.ensureIndex(URL);
assertEquals(3, index.getPages());
assertEquals(1, mapper.headerCount(GID));
assertEquals("b", mapper.pageKey(GID, 2));
}
@Test
void failedPublishRollsBackAndLeavesExistingIndexUntouched() throws Exception {
// 旧的半截索引:头部 3 页,实际只有 1、2 两页。它会触发重建。
mapper.putHeader(GID, KEY, 3);
mapper.putPage(GID, 1, "old1");
mapper.putPage(GID, 2, "old2");
Map<String, String> before = mapper.stateFingerprint();
// 重建时上游给了新 key,但第一次批量写入就失败。
mapper.pagesForFetch = List.of("n1", "n2", "n3");
mapper.failBatchCallAt.set(1);
GalleryIndexException error = assertThrows(GalleryIndexException.class,
() -> service.ensureIndex(URL));
assertEquals(GalleryIndexException.Reason.TRANSIENT, error.getReason(),
"写库失败应被归为可重试的暂时错误");
assertEquals(1, transactions.rollbacks.get(), "失败必须回滚");
assertEquals(0, transactions.commits.get(), "失败不应留下已提交事务");
assertEquals(before, mapper.stateFingerprint(),
"回滚后必须与失败前完全一致,既不能被清空,也不能只写一半");
}
@Test
void upstreamFailureIsReportedAsTransientAndWritesNothing() {
mapper.upstreamError = new RuntimeException("network down");
GalleryIndexException error = assertThrows(GalleryIndexException.class,
() -> service.ensureIndex(URL));
assertEquals(GalleryIndexException.Reason.TRANSIENT, error.getReason());
assertEquals(0, transactions.commits.get());
assertEquals(0, transactions.rollbacks.get(), "尚未开始写入,不应有事务");
assertNull(mapper.header(GID));
}
@Test
void unavailableGalleryIsReportedAsPermanent() {
mapper.pagesForFetch = List.of();
GalleryIndexException error = assertThrows(GalleryIndexException.class,
() -> service.ensureIndex(URL));
assertEquals(GalleryIndexException.Reason.UNAVAILABLE, error.getReason());
assertTrue(error.isPermanent(), "画廊已下架不应被上层当成可重试错误");
assertEquals(0, transactions.commits.get());
}
@Test
void malformedLinkIsRejectedBeforeAnyWork() {
for (String bad : new String[]{"abc", "https://exhentai.org/g/1/", null, "https://exhentai.org/g/x/y/"}) {
GalleryIndexException error = assertThrows(GalleryIndexException.class,
() -> service.ensureIndex(bad), "输入: " + bad);
assertEquals(GalleryIndexException.Reason.INVALID_LINK, error.getReason(), "输入: " + bad);
}
assertEquals(0, mapper.fetchCount.get(), "链接错误不得触发上游抓取");
assertEquals(0, transactions.commits.get());
}
@Test
void failureIsSharedByConcurrentWaiters() throws Exception {
mapper.upstreamError = new RuntimeException("network down");
ExecutorService pool = Executors.newFixedThreadPool(8);
CountDownLatch start = new CountDownLatch(1);
List<Future<String>> outcomes = new ArrayList<>();
for (int i = 0; i < 8; i++) {
outcomes.add(pool.submit(() -> {
start.await();
try {
service.ensureIndex(URL);
return "SUCCESS";
} catch (GalleryIndexException e) {
return e.getReason().name();
}
}));
}
start.countDown();
for (Future<String> outcome : outcomes)
assertEquals("TRANSIENT", outcome.get(20, TimeUnit.SECONDS),
"失败的构建结果也必须被等待者共享,而不是各自重试");
pool.shutdownNow();
}
/** 重试语义:同一 gid 失败后再次调用应真的重试,而不是永久返回失败。 */
@Test
void retryAfterFailureFetchesAgain() throws Exception {
mapper.upstreamError = new RuntimeException("network down");
assertThrows(GalleryIndexException.class, () -> service.ensureIndex(URL));
assertEquals(1, mapper.fetchCount.get());
mapper.upstreamError = null;
GidToKey index = service.ensureIndex(URL);
assertEquals(2, mapper.fetchCount.get(), "失败不应被永久缓存,必须允许重试");
assertEquals(3, index.getPages());
assertEquals(3, mapper.pageNumbers(GID).size());
}
// ---------- 批量写入分片 ----------
@Test
void largeIndexIsWrittenInBatches() throws Exception {
int pages = GalleryIndexService.BATCH_SIZE * 2 + 7;
List<String> keys = new ArrayList<>();
for (int i = 0; i < pages; i++) keys.add("k" + i);
mapper.pagesForFetch = keys;
GidToKey index = service.ensureIndex(URL);
assertEquals(pages, index.getPages());
assertEquals(3, mapper.batchCalls.get(), "应按 BATCH_SIZE 分片写入");
assertEquals(pages, mapper.pageNumbers(GID).size());
assertEquals("k0", mapper.pageKey(GID, 1));
}
@Test
void republishReplacesPreviousRowsInsteadOfAccumulating() throws Exception {
service.ensureIndex(URL);
assertEquals(3, mapper.pageNumbers(GID).size());
// 上游改版:同一 gid 变成 5 页。
mapper.pagesForFetch = List.of("b1", "b2", "b3", "b4", "b5");
mapper.forceIncomplete(GID);
GidToKey index = service.ensureIndex(URL);
assertEquals(5, index.getPages());
assertEquals(5, mapper.pageNumbers(GID).size(), "旧页 key 必须被替换掉,不能残留");
assertEquals(1, mapper.headerCount(GID), "gidToKey 必须仍只有一条记录");
assertEquals(List.of(1, 2, 3, 4, 5), mapper.pageNumbers(GID));
}
/** 构造一行上游页 key,便于在用例里拼畸形数据。 */
private static ImageKeyCache row(String gid, int page, String imgkey) {
ImageKeyCache row = new ImageKeyCache();
row.setGid(gid);
row.setPage(page);
row.setImgkey(imgkey);
return row;
}
private static List<ImageKeyCache> rows(ImageKeyCache... items) {
return new ArrayList<>(Arrays.asList(items));
}
private static String describe(List<ImageKeyCache> rows) {
StringBuilder out = new StringBuilder();
for (ImageKeyCache row : rows) {
if (out.length() > 0) out.append(", ");
out.append(row.getGid()).append('/').append(row.getPage())
.append('/').append(row.getImgkey());
}
return "[" + out + "]";
}
// ---------- 测试替身 ----------
/**
* 记录事务边界与写入所在状态的假事务。
*
* <p>写事务会做真实快照并支持回滚;读事务({@code readOnly=true})只标记一个快照区间,
* 不计入提交/回滚次数,否则无法把「发布事务」和「读快照」分开断言。
*/
static final class FakeTransactions implements TransactionOperations {
private final FakeMapper mapper;
final AtomicInteger active = new AtomicInteger();
final AtomicInteger commits = new AtomicInteger();
final AtomicInteger rollbacks = new AtomicInteger();
final AtomicInteger started = new AtomicInteger();
private final boolean readOnly;
FakeTransactions(FakeMapper mapper) {
this(mapper, false);
}
FakeTransactions(FakeMapper mapper, boolean readOnly) {
this.mapper = mapper;
this.readOnly = readOnly;
}
@Override
public <T> T execute(TransactionCallback<T> action) {
started.incrementAndGet();
active.incrementAndGet();
if (readOnly) {
try {
return action.doInTransaction(new SimpleTransactionStatus());
} finally {
active.decrementAndGet();
}
}
mapper.transactionAttempts.incrementAndGet();
Map<String, Object> before = mapper.snapshot();
try {
T result = action.doInTransaction(new SimpleTransactionStatus());
commits.incrementAndGet();
return result;
} catch (RuntimeException | Error e) {
mapper.restore(before);
rollbacks.incrementAndGet();
throw e;
} finally {
active.decrementAndGet();
}
}
}
/** 内存版 ImageCacheMapper,无需数据库,且对并发访问安全。 */
static final class FakeMapper implements ImageCacheMapper {
private final Map<String, GidToKey> headers = Collections.synchronizedMap(new LinkedHashMap<>());
private final List<ImageKeyCache> pages = Collections.synchronizedList(new ArrayList<>());
final AtomicInteger fetchCount = new AtomicInteger();
final AtomicInteger batchCalls = new AtomicInteger();
final AtomicInteger transactionAttempts = new AtomicInteger();
final AtomicInteger upstreamSawTransaction = new AtomicInteger();
final AtomicInteger insertSawTransaction = new AtomicInteger();
final AtomicInteger mutationsOutsideTransaction = new AtomicInteger();
List<String> pagesForFetch = List.of("k1", "k2", "k3");
/** 设置后 fetch 直接返回这些行,用于构造上游畸形数据。 */
List<ImageKeyCache> rawRowsForFetch;
RuntimeException upstreamError;
/** 第几次批量写入时抛异常;小于等于 0 表示不失败。 */
final AtomicInteger failBatchCallAt = new AtomicInteger(-1);
FakeTransactions transactionsForProbe;
List<ImageKeyCache> fetch() {
fetchCount.incrementAndGet();
if (transactionsForProbe != null)
upstreamSawTransaction.set(transactionsForProbe.active.get());
if (upstreamError != null)
throw upstreamError;
if (rawRowsForFetch != null)
return new ArrayList<>(rawRowsForFetch);
List<ImageKeyCache> result = new ArrayList<>();
int page = 1;
for (String key : pagesForFetch) {
ImageKeyCache row = new ImageKeyCache();
row.setGid(GID);
row.setPage(page++);
row.setImgkey(key);
result.add(row);
}
return result;
}
/** 任何写操作都调用它:事务外写入会被计数,用于断言原子性。 */
private void noteWrite() {
if (transactionsForProbe == null)
return;
if (transactionsForProbe.active.get() == 0)
mutationsOutsideTransaction.incrementAndGet();
else
insertSawTransaction.set(transactionsForProbe.active.get());
}
void putHeader(String gid, String key, int pageCount) {
GidToKey header = new GidToKey();
header.setGid(gid);
header.setKey(key);
header.setPages(pageCount);
headers.put(gid, header);
}
void putPage(String gid, int page, String imgkey) {
ImageKeyCache row = new ImageKeyCache();
row.setGid(gid);
row.setPage(page);
row.setImgkey(imgkey);
pages.add(row);
}
/** 把头部页数改成与现有页 key 不符,使下一次调用必定重建。 */
void forceIncomplete(String gid) {
GidToKey header = headers.get(gid);
if (header != null)
header.setPages(header.getPages() + 99);
}
GidToKey header(String gid) {
return headers.get(gid);
}
int headerCount(String gid) {
return headers.containsKey(gid) ? 1 : 0;
}
String pageKey(String gid, int page) {
return pageSnapshot().stream()
.filter(row -> row.getGid().equals(gid) && row.getPage() == page)
.map(ImageKeyCache::getImgkey)
.findFirst().orElse(null);
}
List<Integer> pageNumbers(String gid) {
List<Integer> numbers = new ArrayList<>();
pageSnapshot().stream().filter(row -> row.getGid().equals(gid))
.forEach(row -> numbers.add(row.getPage()));
numbers.sort(Integer::compareTo);
return numbers;
}
void clear() {
headers.clear();
pages.clear();
}
/** 状态指纹:用于断言回滚后与失败前完全一致。 */
Map<String, String> stateFingerprint() {
Map<String, String> state = new LinkedHashMap<>();
synchronized (headers) {
headers.forEach((gid, header) ->
state.put("h:" + gid, header.getKey() + "/" + header.getPages()));
}
for (ImageKeyCache row : pageSnapshot())
state.put("p:" + row.getGid() + ":" + row.getPage(), String.valueOf(row.getImgkey()));
return state;
}
Map<String, Object> snapshot() {
Map<String, Object> state = new LinkedHashMap<>();
synchronized (headers) {
headers.forEach((gid, header) -> {
GidToKey copy = new GidToKey();
copy.setGid(header.getGid());
copy.setKey(header.getKey());
copy.setPages(header.getPages());
state.put("h:" + gid, copy);
});
}
state.put("pages", new ArrayList<>(pageSnapshot()));
return state;
}
@SuppressWarnings("unchecked")
void restore(Map<String, Object> state) {
headers.clear();
pages.clear();
for (Map.Entry<String, Object> entry : state.entrySet()) {
if (entry.getKey().startsWith("h:"))
headers.put(entry.getKey().substring(2), (GidToKey) entry.getValue());
}
pages.addAll((List<ImageKeyCache>) state.get("pages"));
}
/** 加锁复制一份页 key,避免在并发流操作时读到中间状态。 */
private List<ImageKeyCache> pageSnapshot() {
synchronized (pages) {
List<ImageKeyCache> copy = new ArrayList<>(pages.size());
for (ImageKeyCache row : pages)
copy.add(copyOf(row));
return copy;
}
}
private static ImageKeyCache copyOf(ImageKeyCache row) {
ImageKeyCache copy = new ImageKeyCache();
copy.setGid(row.getGid());
copy.setPage(row.getPage());
copy.setImgkey(row.getImgkey());
return copy;
}
@Override
public void insertImageKeyCache(ImageKeyCache imageKeyCache) {
noteWrite();
pages.add(imageKeyCache);
}
@Override
public void insertGidToKey(GidToKey gidToKey) {
noteWrite();
headers.put(gidToKey.getGid(), gidToKey);
}
@Override
public ImageKeyCache selectImageKeyCacheByGidAndPage(String gid, int page) {
return pageSnapshot().stream()
.filter(row -> row.getGid().equals(gid) && row.getPage() == page)
.findFirst().orElse(null);
}
@Override
public GidToKey selectKeyByGid(String gid) {
return headers.get(gid);
}
@Override
public int countImageKeyCacheByGid(String gid) {
return (int) pageSnapshot().stream().filter(row -> row.getGid().equals(gid)).count();
}
@Override
public void deleteImageKeyCacheByGid(String gid) {
noteWrite();
synchronized (pages) {
pages.removeIf(row -> row.getGid().equals(gid));
}
}
@Override
public void deleteGidToKey(String gid) {
noteWrite();
headers.remove(gid);
}
@Override
public List<Integer> selectPageNumbersByGid(String gid) {
List<Integer> numbers = new ArrayList<>();
pageSnapshot().stream().filter(row -> row.getGid().equals(gid))
.forEach(row -> numbers.add(row.getPage()));
numbers.sort(Integer::compareTo);
return numbers;
}
@Override
public int countInvalidImageKeysByGid(String gid) {
return (int) pageSnapshot().stream()
.filter(row -> row.getGid().equals(gid))
.filter(row -> row.getImgkey() == null || row.getImgkey().isBlank())
.count();
}
@Override
public int insertImageKeyCacheBatch(List<ImageKeyCache> batch) {
noteWrite();
int call = batchCalls.incrementAndGet();
if (failBatchCallAt.get() > 0 && call == failBatchCallAt.get())
throw new RuntimeException("db down");
pages.addAll(batch);
return batch.size();
}
}
}
@@ -44,170 +44,29 @@ class GalleryManageServiceTest {
when(users.selectUserByAuthCode("code")).thenReturn(user); when(users.selectUserByAuthCode("code")).thenReturn(user);
} }
// ---------- cacheImagesKey 的异常兜底 ----------
/**
* 回归修复验证(缺陷于 2026-09-14 Jackson 2→3 迁移引入,2026-09-15 修复)。
*
* 线上真实 mpv 页的 imagelist 是 JS 语句、行尾带分号;解析失败曾以
* `JacksonException`(Jackson 3 中继承 RuntimeException,不再是 IOException)穿透
* `catch (IOException)`,导致新画廊在线看图 500。
* 修复后:解析成功并落库,不再抛异常。
*/
@Test @Test
void cacheImagesKeyParsesRealPageFormatAndCachesKeys() throws Exception { void cacheApiReturnsStructuredIndexWithoutDoubleEncoding() throws Exception {
ImageCacheMapper imageCacheMapper = mock(ImageCacheMapper.class); service.galleryIndexService = mock(GalleryIndexService.class);
when(imageCacheMapper.selectKeyByGid(anyString())).thenReturn(null); // 缓存未命中 var index = new com.lion.lionwebsite.Domain.GidToKey();
GalleryManageService svc = new GalleryManageService(galleries, collectMapper, index.setGid("123"); index.setKey("key"); index.setPages(20);
configurationMapper, users, when(service.galleryIndexService.ensureIndex("link")).thenReturn(index);
imageCacheMapper, remote, push); var response = com.lion.lionwebsite.Util.CustomUtil.objectMapper.readTree(service.cacheImagesKey("link"));
assertEquals("success", response.get("result").asText());
// imagelist 行以分号结尾——即线上真实页面格式 assertTrue(response.get("data").isObject());
String realMpvPage = "<html><body><script>x</script><script>\n" assertEquals(20, response.get("data").get("pages").asInt());
+ "var gid=1596929;\n"
+ "var mpvkey = \"nfa9l8ianjg\";\n"
+ "var imagelist = [{\"n\":\"a.png\",\"k\":\"bd9015\",\"t\":\"(x) -0px 0\"}];\n"
+ "</script></body></html>";
try (var parser = mockStatic(com.lion.lionwebsite.Util.GalleryUtil.class)) {
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil.parseImageKeys(anyString()))
.thenCallRealMethod();
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil
.requests(anyString(), anyString(), any(), any()))
.thenReturn(realMpvPage);
String json = svc.cacheImagesKey("https://exhentai.org/g/1596929/f08534d87d/");
assertTrue(json.contains("\"result\":\"success\""), "应缓存成功,实际: " + json);
} }
verify(imageCacheMapper).insertGidToKey(any());
verify(imageCacheMapper).insertImageKeyCache(any());
}
/**
* 第三方页面格式异常时必须回业务失败,不能再穿透成 500。
* 这里让 requests 返回畸形 JSON,验证 catch 兜住解析异常。
*/
@Test @Test
void cacheImagesKeyConvertsParseFailureToBusinessFailure() throws Exception { void cacheApiMapsPermanentAndTransientFailures() throws Exception {
ImageCacheMapper imageCacheMapper = mock(ImageCacheMapper.class); service.galleryIndexService = mock(GalleryIndexService.class);
when(imageCacheMapper.selectKeyByGid(anyString())).thenReturn(null); var invalid = new com.lion.lionwebsite.Exception.GalleryIndexException(
GalleryManageService svc = new GalleryManageService(galleries, collectMapper, com.lion.lionwebsite.Exception.GalleryIndexException.Reason.INVALID_LINK, "invalid");
configurationMapper, users, when(service.galleryIndexService.ensureIndex("bad")).thenThrow(invalid);
imageCacheMapper, remote, push); assertTrue(service.cacheImagesKey("bad").contains("链接错误"));
var failed = new com.lion.lionwebsite.Exception.GalleryIndexException(
String brokenPage = "<html><body><script>x</script><script>\n" com.lion.lionwebsite.Exception.GalleryIndexException.Reason.TRANSIENT, "retry");
+ "var gid=1;\n" when(service.galleryIndexService.ensureIndex("retry")).thenThrow(failed);
+ "var mpvkey = \"k\";\n" assertTrue(service.cacheImagesKey("retry").contains("网络波动或其他异常"));
+ "var imagelist = [{{not valid json;\n"
+ "</script></body></html>";
try (var parser = mockStatic(com.lion.lionwebsite.Util.GalleryUtil.class)) {
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil.parseImageKeys(anyString()))
.thenCallRealMethod();
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil
.requests(anyString(), anyString(), any(), any()))
.thenReturn(brokenPage);
String json = assertDoesNotThrow(() ->
svc.cacheImagesKey("https://exhentai.org/g/1596929/f08534d87d/"));
assertFalse(json.contains("\"result\":\"success\""), "不应报成功: " + json);
assertTrue(json.contains("网络波动或其他异常"), "实际: " + json);
}
verify(imageCacheMapper, never()).insertGidToKey(any());
verify(imageCacheMapper, never()).insertImageKeyCache(any());
}
/** 畸形链接(段数不足)必须回业务失败,不得抛 ArrayIndexOutOfBoundsException。 */
@Test
void cacheImagesKeyRejectsMalformedLink() {
ImageCacheMapper imageCacheMapper = mock(ImageCacheMapper.class);
GalleryManageService svc = new GalleryManageService(galleries, collectMapper,
configurationMapper, users,
imageCacheMapper, remote, push);
for (String bad : new String[]{"abc", "https://exhentai.org/g/1/", null}) {
String json = assertDoesNotThrow(() -> svc.cacheImagesKey(bad),
"畸形链接不应抛异常,实际输入: " + bad);
assertTrue(json.contains("链接错误"), "实际输出: " + json);
}
verify(imageCacheMapper, never()).insertGidToKey(any());
}
/**
* 回归:索引写入不是原子的。若 gidToKey 写入成功、逐页写 ImageKeyCache 时失败,
* 会留下「gidToKey 命中但页 key 缺失」的半截缓存,后续请求直接返回已缓存而永远取不到图。
* 现在任何异常都要把已写入的部分清掉再回业务失败。
*/
@Test
void cacheImagesKeyRollsBackWhenPageInsertFails() throws Exception {
ImageCacheMapper imageCacheMapper = mock(ImageCacheMapper.class);
when(imageCacheMapper.selectKeyByGid(anyString())).thenReturn(null);
doThrow(new RuntimeException("db down")).when(imageCacheMapper).insertImageKeyCache(any());
GalleryManageService svc = new GalleryManageService(galleries, collectMapper,
configurationMapper, users,
imageCacheMapper, remote, push);
String realMpvPage = "<html><body><script>x</script><script>\n"
+ "var gid=1596929;\n"
+ "var mpvkey = \"nfa9l8ianjg\";\n"
+ "var imagelist = [{\"n\":\"a.png\",\"k\":\"bd9015\",\"t\":\"(x) -0px 0\"}];\n"
+ "</script></body></html>";
try (var parser = mockStatic(com.lion.lionwebsite.Util.GalleryUtil.class)) {
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil.parseImageKeys(anyString()))
.thenCallRealMethod();
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil
.requests(anyString(), anyString(), any(), any()))
.thenReturn(realMpvPage);
String json = assertDoesNotThrow(() ->
svc.cacheImagesKey("https://exhentai.org/g/1596929/f08534d87d/"));
assertFalse(json.contains("\"result\":\"success\""), "写入失败不应报成功: " + json);
}
verify(imageCacheMapper).insertGidToKey(any());
verify(imageCacheMapper).deleteImageKeyCacheByGid(anyString());
verify(imageCacheMapper).deleteGidToKey(anyString());
}
/** 已缓存的索引若页 key 数对不上(半截缓存),必须重建而不是直接返回。 */
@Test
void cacheImagesKeyRebuildsIncompleteCache() throws Exception {
ImageCacheMapper imageCacheMapper = mock(ImageCacheMapper.class);
var stale = new com.lion.lionwebsite.Domain.GidToKey();
stale.setGid("1596929");
stale.setKey("f08534d87d");
stale.setPages(3);
when(imageCacheMapper.selectKeyByGid("1596929")).thenReturn(stale);
when(imageCacheMapper.countImageKeyCacheByGid("1596929")).thenReturn(1); // 只剩 1 页
GalleryManageService svc = new GalleryManageService(galleries, collectMapper,
configurationMapper, users,
imageCacheMapper, remote, push);
String realMpvPage = "<html><body><script>x</script><script>\n"
+ "var gid=1596929;\n"
+ "var mpvkey = \"nfa9l8ianjg\";\n"
+ "var imagelist = [{\"n\":\"a.png\",\"k\":\"bd9015\",\"t\":\"(x) -0px 0\"}];\n"
+ "</script></body></html>";
try (var parser = mockStatic(com.lion.lionwebsite.Util.GalleryUtil.class)) {
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil.parseImageKeys(anyString()))
.thenCallRealMethod();
parser.when(() -> com.lion.lionwebsite.Util.GalleryUtil
.requests(anyString(), anyString(), any(), any()))
.thenReturn(realMpvPage);
String json = svc.cacheImagesKey("https://exhentai.org/g/1596929/f08534d87d/");
assertTrue(json.contains("\"result\":\"success\""), "应重建成功: " + json);
}
verify(imageCacheMapper).deleteImageKeyCacheByGid("1596929");
verify(imageCacheMapper).deleteGidToKey("1596929");
verify(imageCacheMapper).insertGidToKey(any());
} }
// ---------- createTask 输入校验 ---------- // ---------- createTask 输入校验 ----------
@@ -553,6 +412,72 @@ class GalleryManageServiceTest {
assertFalse(service.retryGallery(2).contains("\"result\":\"success\"")); assertFalse(service.retryGallery(2).contains("\"result\":\"success\""));
} }
// ---------- 生命周期拒绝路径 ----------
/** beginDelete 被拒(任务已在删除中/提交中)时必须回业务失败,且不得退避到裸删除。 */
@Test
void deleteRefusedByLifecycleDoesNotFallBackToRawDelete() {
Gallery gallery = new Gallery();
gallery.setGid(897);
gallery.setName("mine [897]");
gallery.setDownloader(7);
when(galleries.selectGalleryByGid(897)).thenReturn(gallery);
when(collectMapper.selectCollectorByGid(897)).thenReturn(new java.util.ArrayList<>());
service.lifecycleService = mock(GalleryTaskLifecycleService.class);
when(service.lifecycleService.beginDelete(897))
.thenReturn(GalleryTaskLifecycleService.DeleteIntent.refused("任务正在提交中,请等待提交结果明确后再删除"));
String response = service.deleteGalleryByGid(897, "code");
assertFalse(response.contains("\"result\":\"success\""), "被拒时必须回失败: " + response);
assertTrue(response.contains("提交中"), response);
verify(remote, never()).deleteGallery(any());
verify(service.lifecycleService, never()).completeDelete(anyInt(), anyString());
}
/** reset 只重投可安全下发的任务,跳过需要人工对账的 UNKNOWN 等。 */
@Test
void resetSkipsTasksThatMustNotBeDispatched() {
when(remote.isDead()).thenReturn(false);
Gallery safe = new Gallery();
safe.setGid(900);
safe.setName("safe [900]");
Gallery unknown = new Gallery();
unknown.setGid(901);
unknown.setName("unknown [901]");
when(galleries.selectUnDoneGalleries()).thenReturn(new Gallery[]{safe, unknown});
service.lifecycleService = mock(GalleryTaskLifecycleService.class);
when(service.lifecycleService.resolveDispatch(900))
.thenReturn(GalleryTaskLifecycleService.DispatchDecision.ready("dl-900"));
when(service.lifecycleService.resolveDispatch(901))
.thenReturn(GalleryTaskLifecycleService.DispatchDecision.refuse());
String response = service.resetUndone();
assertTrue(response.contains("\"result\":\"success\""), response);
verify(remote).addGalleryToQueue(safe);
verify(remote, never()).addGalleryToQueue(unknown);
}
/** 提交未知的任务重试必须被拒,提示人工对账,且不触碰节点。 */
@Test
void retryRefusesWhenLifecycleRejects() {
Gallery gallery = new Gallery();
gallery.setGid(902);
gallery.setName("unknown [902]");
gallery.setStatus("提交未知");
when(galleries.selectGalleryByGid(902)).thenReturn(gallery);
service.lifecycleService = mock(GalleryTaskLifecycleService.class);
when(service.lifecycleService.resolveDispatch(902))
.thenReturn(GalleryTaskLifecycleService.DispatchDecision.refuse());
String response = service.retryGallery(902);
assertFalse(response.contains("\"result\":\"success\""), response);
assertTrue(response.contains("人工对账"), response);
verify(remote, never()).retryGallery(any());
}
// ---------- 列表 ---------- // ---------- 列表 ----------
/** 用量查询应返回格式化后的已用量与上次重置时间。 */ /** 用量查询应返回格式化后的已用量与上次重置时间。 */
@@ -65,9 +65,9 @@ class QueryServiceTest {
assertTrue(json.contains("Title Here"), "应解析出画廊名"); assertTrue(json.contains("Title Here"), "应解析出画廊名");
assertTrue(json.contains("1234567"), "应解析出 gid"); assertTrue(json.contains("1234567"), "应解析出 gid");
assertTrue(json.contains("Manga"), "应解析出类型"); assertTrue(json.contains("Manga"), "应解析出类型");
// data 是「数组序列化后的字符串」,字段因此带转义,这里按转义后的形态断言 // data 直接为结构化数组。
assertTrue(json.contains("\\\"page\\\":42"), "应解析出页数: " + json); assertTrue(json.contains("\"page\":42"), "应解析出页数: " + json);
assertTrue(json.contains("\\\"gid\\\":\\\"1234567\\\""), "应解析出 gid: " + json); assertTrue(json.contains("\"gid\":\"1234567\""), "应解析出 gid: " + json);
// 缩略图地址去掉了站点前缀,前端要拿相对路径 // 缩略图地址去掉了站点前缀,前端要拿相对路径
assertTrue(json.contains("/t/thumb.jpg")); assertTrue(json.contains("/t/thumb.jpg"));
assertFalse(json.contains("s.exhentai.org")); assertFalse(json.contains("s.exhentai.org"));
@@ -0,0 +1,350 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.GalleryMapper;
import com.lion.lionwebsite.Domain.Gallery;
import com.lion.lionwebsite.Domain.GalleryTask;
import com.lion.lionwebsite.Message.ProtocolHelloMessage;
import com.lion.lionwebsite.Message.TaskCommandMessage;
import com.lion.lionwebsite.Message.TaskReportAckMessage;
import com.lion.lionwebsite.Message.TaskReportMessage;
import io.netty.channel.embedded.EmbeddedChannel;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.CopyOnWriteArrayList;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
/**
* 节点下发的安全边界:不是「能解析出 operationId 就发」。
*
* <p>UNKNOWN / 预占中 / 待删除的任务一旦被重投,未计量的未知任务会在节点侧变成
* 「下载完成却 0 额度」;协商完成前与启动恢复不安全时也不允许批量重投。
*/
class RemoteServiceEligibilityTest {
private GalleryMapper galleryMapper;
private PushService pushService;
private WebSocketService webSocketService;
private RemoteService service;
private EmbeddedChannel channel;
private ExecutorService retryExecutor;
private final CopyOnWriteArrayList<Object> written = new CopyOnWriteArrayList<>();
private final java.util.concurrent.CountDownLatch helloWritten = new java.util.concurrent.CountDownLatch(1);
private volatile boolean failAckWrites;
@BeforeEach
void setUp() {
galleryMapper = mock(GalleryMapper.class);
pushService = mock(PushService.class);
webSocketService = mock(WebSocketService.class);
service = new RemoteService(galleryMapper, pushService,
webSocketService, mock(SubscriptionStandbySnapshotService.class));
channel = new EmbeddedChannel(new io.netty.channel.ChannelOutboundHandlerAdapter() {
@Override
public void write(io.netty.channel.ChannelHandlerContext ctx, Object msg,
io.netty.channel.ChannelPromise promise) {
if (failAckWrites && msg instanceof TaskReportAckMessage)
throw new io.netty.channel.ChannelException("ACK 写通道失败");
io.netty.util.concurrent.Promise<com.lion.lionwebsite.Message.AbstractMessage> inner =
service.promiseHashMap.get(((com.lion.lionwebsite.Message.AbstractMessage) msg).messageId);
if (inner != null)
inner.trySuccess(new com.lion.lionwebsite.Message.ResponseMessage());
written.add(msg);
if (msg instanceof ProtocolHelloMessage)
helloWritten.countDown();
promise.setSuccess();
}
}, service.new MyChannelInboundHandlerAdapter());
service.channel = channel;
io.netty.channel.ChannelFuture channelFuture = mock(io.netty.channel.ChannelFuture.class);
when(channelFuture.channel()).thenReturn(channel);
service.channelFuture = channelFuture;
retryExecutor = Executors.newCachedThreadPool();
}
@AfterEach
void tearDown() {
service.shutdownResources();
retryExecutor.shutdownNow();
channel.finishAndReleaseAll();
}
private static Gallery gallery(int gid) {
Gallery gallery = new Gallery();
gallery.setGid(gid);
gallery.setName("任务 [" + gid + "]");
gallery.setStatus("已提交");
return gallery;
}
private static GalleryTask task(int gid, byte status, int proceeding) {
GalleryTask task = new GalleryTask();
task.setGid(gid);
task.setName("任务 [" + gid + "]");
task.setStatus(status);
task.setProceeding(proceeding);
return task;
}
private static TaskReportMessage report(int gid, String reportId) {
TaskReportMessage report = new TaskReportMessage();
report.setReportId(reportId);
report.setRevision(7);
report.setOperationIds(Map.of(gid, "op-" + gid));
return report;
}
/** 让节点命令立刻拿到成功的 ResponseMessage(沿用既有 setup 的自动应答行为)。 */
private void enableReadyDispatch() {
service.setDispatchResolver(gid -> GalleryTaskLifecycleService.DispatchDecision.ready("op-" + gid));
service.v2Negotiated = true;
}
private Future<RemoteService.RetryResult> retryInBackground(int gid) {
return retryExecutor.submit(() -> service.retryGallery(gallery(gid)));
}
private void awaitWaiterRegistered(int gid) throws InterruptedException {
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(3);
while (System.nanoTime() < deadline) {
var waiters = service.retryStatusWaiters.get(gid);
if (waiters != null && !waiters.isEmpty())
return;
Thread.sleep(5);
}
fail("重试等待者未注册 gid=" + gid);
}
private List<String> ackedReportIds() {
List<String> ids = new java.util.ArrayList<>();
for (Object message : written)
if (message instanceof TaskReportAckMessage ack)
ids.add(ack.getReportId());
return ids;
}
/** 未协商 v2 / 装配未就绪时绝不批量重投未完成任务。 */
@Test
void resetUndoneSkipsBeforeNegotiationAndWiring() {
service.v2Negotiated = false;
service.resetUndone();
service.v2Negotiated = true;
service.lifecycleReady = false;
service.resetUndone();
service.lifecycleReady = true;
service.recoverySafe = false;
service.resetUndone();
verify(galleryMapper, never()).selectUnDoneGalleries();
// 四项条件都满足时才真正重投。
service.recoverySafe = true;
when(galleryMapper.selectUnDoneGalleries()).thenReturn(new Gallery[0]);
service.resetUndone();
verify(galleryMapper).selectUnDoneGalleries();
}
/** 判定解析器明确拒绝时,绝不能退化为无 operationId 的旧命令。 */
@Test
void refusedDecisionNeverFallsBackToLegacyCommand() {
service.setDispatchResolver(gid -> GalleryTaskLifecycleService.DispatchDecision.refuse());
byte result = service.addGalleryToQueue(gallery(77));
assertEquals(RemoteService.REFUSED, result);
assertTrue(written.isEmpty(), "被拒的任务不得写入任何命令");
}
/** 判定为可下发时必须使用解析到的稳定 operationId。 */
@Test
void readyDecisionSendsStableOperationId() {
service.setDispatchResolver(gid -> GalleryTaskLifecycleService.DispatchDecision.ready("dl-stable"));
service.v2Negotiated = true;
byte result = service.addGalleryToQueue(gallery(78));
assertEquals(0, result);
assertEquals(1, written.size(), "应写入一条任务命令");
Object sent = written.get(0);
assertInstanceOf(TaskCommandMessage.class, sent);
TaskCommandMessage command = (TaskCommandMessage) sent;
assertEquals("dl-stable", command.getOperationId());
assertEquals(TaskCommandMessage.ACTION_DOWNLOAD, command.getAction());
}
/** UNKNOWN 任务的重试必须被拒,并给出人工对账提示。 */
@Test
void retryRefusesWhenDecisionRejects() {
service.setDispatchResolver(gid -> GalleryTaskLifecycleService.DispatchDecision.refuse());
RemoteService.RetryResult result = service.retryGallery(gallery(79));
assertFalse(result.success());
assertTrue(result.message().contains("人工对账"), result.message());
}
/** 节点回 Hello(不是 ResponseMessage)即完成协商 future,不再白等 10 秒。 */
@Test
void nodeHelloCompletesNegotiationFuture() throws Exception {
CompletableFuture<Boolean> negotiation = new CompletableFuture<>();
service.helloNegotiation = negotiation;
channel.writeInbound(new ProtocolHelloMessage());
assertTrue(negotiation.get(2, TimeUnit.SECONDS), "节点声明的 v2 能力应完成协商");
assertTrue(service.isReliableProtocolReady());
}
@Test
@SuppressWarnings("unchecked")
void realHandshakeWaitsForHelloWithoutResponseMessagePromise() throws Exception {
var method = RemoteService.class.getDeclaredMethod("queueHelloHandshake");
method.setAccessible(true);
var negotiation = (CompletableFuture<Boolean>) method.invoke(service);
assertTrue(helloWritten.await(2, TimeUnit.SECONDS), "应实际发出 Hello");
assertTrue(service.promiseHashMap.isEmpty(), "Hello 不应等待 ResponseMessage");
assertFalse(negotiation.isDone(), "尚未收到 Hello 时不能宣告协商结束");
channel.writeInbound(new ProtocolHelloMessage());
assertTrue(negotiation.get(2, TimeUnit.SECONDS));
}
/** 正常 v2 重试:节点上报提交成功后,等待者必须在短 deadline 内被唤醒,而不是拖到 10 秒超时。 */
@Test
void v2RetryCompletesWithinShortDeadlineThroughRealHandler() throws Exception {
enableReadyDispatch();
service.setTaskReportHandler(report -> new TaskReportHandler.CommittedReport(
report.getReportId(), List.of(), List.of(task(501, GalleryTask.COMPRESS_COMPLETE, 40))));
Future<RemoteService.RetryResult> pending = retryInBackground(501);
awaitWaiterRegistered(501);
TaskReportMessage message = report(501, "rep-501");
message.setGalleryTasks(new GalleryTask[]{task(501, GalleryTask.COMPRESS_COMPLETE, 40)});
long start = System.nanoTime();
channel.writeInbound(message);
RemoteService.RetryResult result = pending.get(3, TimeUnit.SECONDS);
long elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start);
assertTrue(result.success(), result.message());
assertEquals(TaskStatus.COMPLETE, result.message());
assertTrue(elapsedMs < 2_000, "正常 v2 上报不应退化为 10 秒超时,实测 " + elapsedMs + "ms");
}
/** 未提交(apply 返回 null)不回执也不完成;空了 acceptedTasks 的任务同样不能完成等待者。 */
@Test
void uncommittedOrFilteredReportNeverCompletesWaiter() throws Exception {
enableReadyDispatch();
service.setTaskReportHandler(rejectedHandler -> null);
Future<RemoteService.RetryResult> uncommitted = retryInBackground(502);
awaitWaiterRegistered(502);
TaskReportMessage rejected = report(502, "rep-502");
rejected.setGalleryTasks(new GalleryTask[]{task(502, GalleryTask.COMPRESS_COMPLETE, 40)});
channel.writeInbound(rejected);
service.awaitStatusApplied();
assertFalse(uncommitted.isDone(), "未提交的上报不能完成重试等待者");
assertTrue(ackedReportIds().isEmpty(), "未提交的上报不能回执");
// 已被事务过滤掉(过期/旧 operation/待删除):只 ACK,不完成等待者。
service.setTaskReportHandler(filteredHandler -> new TaskReportHandler.CommittedReport(
filteredHandler.getReportId(), List.of(), List.of()));
Future<RemoteService.RetryResult> filtered = retryInBackground(503);
awaitWaiterRegistered(503);
TaskReportMessage filteredMessage = report(503, "rep-503");
filteredMessage.setGalleryTasks(new GalleryTask[]{task(503, GalleryTask.COMPRESS_COMPLETE, 40)});
channel.writeInbound(filteredMessage);
service.awaitStatusApplied();
assertFalse(filtered.isDone(), "被过滤掉的更新不能完成重试等待者");
assertEquals(List.of("rep-503"), ackedReportIds(), "已提交的上报仍应回执,让节点清理快照");
}
/** 通知线程异常或通知队列拒绝都不得影响 ACK 与等待者完成。 */
@Test
void notificationFailureDoesNotAffectAckOrWaiterCompletion() throws Exception {
enableReadyDispatch();
doThrow(new RuntimeException("推送失败")).when(webSocketService)
.updateTaskProcessing(any(GalleryTask[].class));
service.setTaskReportHandler(report -> new TaskReportHandler.CommittedReport(
report.getReportId(), List.of(), List.of(report.getGalleryTasks())));
Future<RemoteService.RetryResult> pending = retryInBackground(504);
awaitWaiterRegistered(504);
TaskReportMessage message = report(504, "rep-504");
message.setGalleryTasks(new GalleryTask[]{task(504, GalleryTask.COMPRESS_COMPLETE, 40)});
channel.writeInbound(message);
RemoteService.RetryResult result = pending.get(3, TimeUnit.SECONDS);
assertTrue(result.success(), result.message());
assertEquals(TaskStatus.COMPLETE, result.message());
assertTrue(ackedReportIds().contains("rep-504"));
// 通知队列已拒绝:提交成功仍必须回执并完成等待者。
service.notificationExecutor.shutdownNow();
Future<RemoteService.RetryResult> afterShutdown = retryInBackground(505);
awaitWaiterRegistered(505);
TaskReportMessage queued = report(505, "rep-505");
queued.setGalleryTasks(new GalleryTask[]{task(505, GalleryTask.COMPRESS_COMPLETE, 40)});
channel.writeInbound(queued);
service.awaitStatusApplied();
RemoteService.RetryResult queuedResult = afterShutdown.get(3, TimeUnit.SECONDS);
assertTrue(queuedResult.success(), queuedResult.message());
assertTrue(ackedReportIds().contains("rep-505"));
}
/**
* 节点 accepted 后刚清完队列、当前 DB 已是下载完成:必须用重新读到的完成状态收尾,
* 不能拿调用方传入的旧 Gallery(旧「下载中」)当成新确认。
*/
@Test
void retryShortCircuitsOnFreshlyCompletedDbState() throws Exception {
enableReadyDispatch();
service.setTaskReportHandler(report -> null);
Gallery completed = gallery(506);
completed.setStatus(TaskStatus.COMPLETE);
when(galleryMapper.selectGalleryByGid(506)).thenReturn(completed);
Gallery stale = gallery(506);
stale.setStatus(TaskStatus.DOWNLOADING);
long start = System.nanoTime();
RemoteService.RetryResult result = service.retryGallery(stale);
long elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start);
assertTrue(result.success(), result.message());
assertEquals(TaskStatus.COMPLETE, result.message(), "必须返回重新读到的完成状态");
assertTrue(elapsedMs < 2_000, "已完成的库状态应立刻返回,实测 " + elapsedMs + "ms");
verify(galleryMapper).selectGalleryByGid(506);
}
/** ACK 写通道抛异常也不得吞掉已提交的重试状态:等待者仍须拿到真实状态,而非 10 秒超时。 */
@Test
void ackWriteFailureStillCompletesRetryWaiter() throws Exception {
enableReadyDispatch();
service.setTaskReportHandler(report -> new TaskReportHandler.CommittedReport(
report.getReportId(), List.of(), List.of(report.getGalleryTasks())));
failAckWrites = true;
Future<RemoteService.RetryResult> pending = retryInBackground(507);
awaitWaiterRegistered(507);
TaskReportMessage message = report(507, "rep-507");
message.setGalleryTasks(new GalleryTask[]{task(507, GalleryTask.COMPRESS_COMPLETE, 40)});
channel.writeInbound(message);
RemoteService.RetryResult result = pending.get(3, TimeUnit.SECONDS);
assertTrue(result.success(), result.message());
assertEquals(TaskStatus.COMPLETE, result.message());
}
}
@@ -337,6 +337,32 @@ class SubServiceTest {
verify(response).sendError(eq(503), anyString()); verify(response).sendError(eq(503), anyString());
} }
/**
* 半份缓存(旧结构只写了 v2ray.txt)也必须回 503,而不是把残缺内容当订阅发出。
*
* <p>这里用真实的 {@link SubscriptionRefreshService} 而不是 mock,才能真正覆盖
* {@code cachedPath} 的整套配对契约:只有两份齐全才解析出真实文件。
*/
@Test
void publicSubReturns503ForIncompleteCacheFiles(@org.junit.jupiter.api.io.TempDir java.nio.file.Path cacheRoot)
throws Exception {
var realResolver = new SubscriptionRefreshService(subMapper, new SubscriptionStateCoordinator());
realResolver.cacheRoot = cacheRoot.toString();
var boundService = new SubService(subMapper, userMapper, realResolver, remoteService,
new SubscriptionStateCoordinator());
java.nio.file.Files.createDirectories(cacheRoot.resolve("1"));
java.nio.file.Files.writeString(cacheRoot.resolve("1/v2ray.txt"), "half");
when(subMapper.selectSubBind("key1")).thenReturn(new SubBind("key1", "alice", 1, "name", true, false));
var request = mock(jakarta.servlet.http.HttpServletRequest.class);
when(request.getHeader("User-Agent")).thenReturn("Mozilla/5.0");
when(request.getRemoteAddr()).thenReturn("203.0.113.5");
var response = mock(jakarta.servlet.http.HttpServletResponse.class);
boundService.updateSub(response, request, "v2", "key1");
verify(response).sendError(eq(503), anyString());
}
@Test @Test
void publicSubIgnoresNullKeyOrClient() throws Exception { void publicSubIgnoresNullKeyOrClient() throws Exception {
var response = mock(jakarta.servlet.http.HttpServletResponse.class); var response = mock(jakarta.servlet.http.HttpServletResponse.class);
@@ -366,6 +392,7 @@ class SubServiceTest {
// 也避免单测真的去请求 ip138。 // 也避免单测真的去请求 ip138。
var recorded = new java.util.concurrent.atomic.AtomicReference<Runnable>(); var recorded = new java.util.concurrent.atomic.AtomicReference<Runnable>();
service.locationExecutor = command -> recorded.set(command); service.locationExecutor = command -> recorded.set(command);
doAnswer(call -> { call.getArgument(0, com.lion.lionwebsite.Domain.SubUpdateRecord.class).setId(81); return null; }).when(subMapper).insertSubUpdateRecord(any());
long start = System.nanoTime(); long start = System.nanoTime();
service.updateSub(response, request, "v2", "key1"); service.updateSub(response, request, "v2", "key1");
@@ -389,4 +416,46 @@ class SubServiceTest {
assertEquals("unknown", SubService.resolveLocation("203.0.113.5")); assertEquals("unknown", SubService.resolveLocation("203.0.113.5"));
} }
} }
@Test
void outOfOrderLocationLookupsUpdateTheirOwnGeneratedRows() {
when(subMapper.selectSubBind("key1")).thenReturn(new SubBind("key1", "alice", 1, "name", true, false));
when(refreshService.cachedPath(eq(1), anyString())).thenReturn(java.nio.file.Path.of("/nonexistent/cache/v2.txt"));
var request = mock(jakarta.servlet.http.HttpServletRequest.class);
when(request.getHeader("User-Agent")).thenReturn("test");
when(request.getRemoteAddr()).thenReturn("203.0.113.1", "203.0.113.2");
var nextId = new java.util.concurrent.atomic.AtomicInteger(100);
doAnswer(call -> { call.getArgument(0, com.lion.lionwebsite.Domain.SubUpdateRecord.class).setId(nextId.getAndIncrement()); return null; })
.when(subMapper).insertSubUpdateRecord(any());
var lookups = new java.util.ArrayList<Runnable>();
service.locationExecutor = lookups::add;
var response = mock(jakarta.servlet.http.HttpServletResponse.class);
service.updateSub(response, request, "v2", "key1");
service.updateSub(response, request, "v2", "key1");
try (var location = mockStatic(SubService.class, CALLS_REAL_METHODS)) {
location.when(() -> SubService.resolveLocation("203.0.113.1")).thenReturn("first-location");
location.when(() -> SubService.resolveLocation("203.0.113.2")).thenReturn("second-location");
lookups.get(1).run();
lookups.get(0).run();
}
verify(subMapper).updateSubUpdateRecordLocation(100, "first-location");
verify(subMapper).updateSubUpdateRecordLocation(101, "second-location");
verify(subMapper, never()).updateLatestSubUpdateRecordLocation(anyString(), anyString());
}
@Test
void busyLocationQueueDoesNotBreakSubscriptionDistribution() throws Exception {
when(subMapper.selectSubBind("key1")).thenReturn(new SubBind("key1", "alice", 1, "name", true, false));
when(refreshService.cachedPath(eq(1), anyString())).thenReturn(java.nio.file.Path.of("/nonexistent/cache/v2.txt"));
doAnswer(call -> { call.getArgument(0, com.lion.lionwebsite.Domain.SubUpdateRecord.class).setId(91); return null; })
.when(subMapper).insertSubUpdateRecord(any());
var request = mock(jakarta.servlet.http.HttpServletRequest.class);
when(request.getHeader("User-Agent")).thenReturn("test");
when(request.getRemoteAddr()).thenReturn("203.0.113.1");
service.locationExecutor = command -> { throw new java.util.concurrent.RejectedExecutionException("busy"); };
var response = mock(jakarta.servlet.http.HttpServletResponse.class);
assertDoesNotThrow(() -> service.updateSub(response, request, "v2", "key1"));
verify(subMapper).insertSubUpdateRecord(any());
verify(response).sendError(jakarta.servlet.http.HttpServletResponse.SC_SERVICE_UNAVAILABLE, "subscription unavailable");
}
} }
@@ -0,0 +1,165 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Util.SubscriptionClientProfile;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Base64;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
/**
* 跨刷新的版本完整性:一次刷新产出的两份文件必须同版本,且过期的在途刷新不得覆盖更新的版本。
*/
class SubscriptionCacheGenerationTest {
private static SubscriptionAccount account(int id, String key) {
SubscriptionAccount a = new SubscriptionAccount();
a.setId(id);
a.setName("acc-" + id);
a.setUpstreamKey(key);
a.setEnabled(true);
return a;
}
private static List<String> v2(String node) {
return List.of(Base64.getEncoder().encodeToString(node.getBytes(StandardCharsets.UTF_8)));
}
private static String decodedV2(SubscriptionRefreshService service, int accountId) throws IOException {
Path path = service.cachedPath(accountId, "v2");
return new String(Base64.getMimeDecoder().decode(Files.readString(path, StandardCharsets.UTF_8)),
StandardCharsets.UTF_8);
}
/**
* 在途的旧刷新在失效并发布新版本后才返回:必须被丢弃,不能把旧内容写回。
*/
@Test
void staleInFlightRefreshCannotOverwriteNewerGeneration(@TempDir Path cacheRoot) throws Exception {
SubMapper mapper = mock(SubMapper.class);
SubscriptionAccount stale = account(1, "old-key");
SubscriptionAccount fresh = account(1, "new-key");
when(mapper.selectSubscriptionAccount(1)).thenReturn(stale);
SubscriptionStateCoordinator coordinator = new SubscriptionStateCoordinator();
CountDownLatch blocked = new CountDownLatch(1);
CountDownLatch release = new CountDownLatch(1);
SubscriptionRefreshService service = new SubscriptionRefreshService(mapper, coordinator) {
@Override
List<String> download(String url, SubscriptionClientProfile profile) throws IOException {
if (url.contains("old-key")) {
blocked.countDown();
try {
if (!release.await(10, TimeUnit.SECONDS))
throw new IOException("test timed out");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException(e);
}
return v2("vmess://stale-node");
}
return url.contains("client=v2") ? v2("vmess://fresh-node") : List.of("proxies: []");
}
};
service.cacheRoot = cacheRoot.toString();
service.v2UrlTemplate = "https://upstream.invalid/{key}?client=v2";
service.clashUrlTemplate = "https://upstream.invalid/{key}?client=clashmeta";
ExecutorService worker = Executors.newSingleThreadExecutor();
try {
Future<Boolean> inFlight = worker.submit(() -> service.refresh(1));
assertTrue(blocked.await(5, TimeUnit.SECONDS), "旧刷新应已进入下载阶段");
// 上游 key 被改掉并失效缓存,随后新刷新发布新版本。
when(mapper.selectSubscriptionAccount(1)).thenReturn(fresh);
service.invalidateCache(1);
assertFalse(service.hasCompleteCache(1), "失效后不应还有可分发版本");
release.countDown();
assertFalse(inFlight.get(10, TimeUnit.SECONDS), "过期的在途刷新必须被丢弃");
assertTrue(service.refresh(1));
assertEquals("vmess://fresh-node\n", decodedV2(service, 1), "应只保留新版本内容");
assertFalse(decodedV2(service, 1).contains("stale-node"), "过期刷新内容不得出现");
// 只有新的那次刷新可以记成功;过期刷新必须完全没有留下成功记录。
verify(mapper, times(1)).markSubscriptionRefreshSuccess(eq(1), anyLong());
} finally {
release.countDown();
worker.shutdownNow();
}
}
/**
* 过期刷新被丢弃后不得留下任何半成品:缓存要么是上一次的完整版本,要么为空。
*/
@Test
void discardedRefreshLeavesNoPartialCache(@TempDir Path cacheRoot) throws Exception {
SubMapper mapper = mock(SubMapper.class);
SubscriptionAccount stale = account(2, "k-old");
when(mapper.selectSubscriptionAccount(2)).thenReturn(stale);
SubscriptionStateCoordinator coordinator = new SubscriptionStateCoordinator();
CountDownLatch blocked = new CountDownLatch(1);
CountDownLatch release = new CountDownLatch(1);
SubscriptionRefreshService service = new SubscriptionRefreshService(mapper, coordinator) {
@Override
List<String> download(String url, SubscriptionClientProfile profile) throws IOException {
if (url.contains("k-old")) {
blocked.countDown();
try {
if (!release.await(10, TimeUnit.SECONDS))
throw new IOException("test timed out");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException(e);
}
return v2("vmess://stale");
}
return url.contains("client=v2") ? v2("vmess://x") : List.of("proxies: []");
}
};
service.cacheRoot = cacheRoot.toString();
service.v2UrlTemplate = "https://upstream.invalid/{key}?client=v2";
service.clashUrlTemplate = "https://upstream.invalid/{key}?client=clashmeta";
// 先成功发布一版,作为「上一次完整版本」。
when(mapper.selectSubscriptionAccount(2)).thenReturn(account(2, "k-new"));
assertTrue(service.refresh(2));
String before = decodedV2(service, 2);
when(mapper.selectSubscriptionAccount(2)).thenReturn(stale);
ExecutorService worker = Executors.newSingleThreadExecutor();
try {
Future<Boolean> inFlight = worker.submit(() -> service.refresh(2));
assertTrue(blocked.await(5, TimeUnit.SECONDS));
when(mapper.selectSubscriptionAccount(2)).thenReturn(account(2, "k-newer"));
service.invalidateCache(2);
release.countDown();
assertFalse(inFlight.get(10, TimeUnit.SECONDS));
assertEquals("vmess://x\n", before);
// 失效已清掉旧版本,而被丢弃的刷新又不得重新发布,因此当前应无可分发缓存。
assertFalse(service.hasCompleteCache(2), "被丢弃的刷新不得留下缓存");
// 只有首次成功刷新记过成功,被丢弃的那次不得再记。
verify(mapper, times(1)).markSubscriptionRefreshSuccess(eq(2), anyLong());
} finally {
release.countDown();
worker.shutdownNow();
}
}
}
@@ -0,0 +1,340 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Controller.SubscriptionEventController;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.SubscriptionEvent;
import com.lion.lionwebsite.Util.CustomUtil;
import org.junit.jupiter.api.*;
import org.junit.jupiter.api.io.TempDir;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.mock.web.MockHttpServletRequest;
import org.springframework.mock.web.MockHttpServletResponse;
import java.nio.file.Files;
import static org.mockito.Mockito.*;
import org.springframework.transaction.support.TransactionTemplate;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.*;
import static org.junit.jupiter.api.Assertions.*;
/** Real SQLite/MyBatis/Spring transactions: these assertions cover committed database state. */
class SubscriptionEventIntegrationTest {
@TempDir Path directory;
private JdbcTemplate db;
private SubscriptionEventService service;
private SubMapper subscriptions;
private static final long NOW = 1_792_555_200_000L;
private static final String SECRET = "integration-test-only-secret";
private static final String KEY = "example-key";
@BeforeEach void setUp() throws Exception {
var source = new DriverManagerDataSource("jdbc:sqlite:" + directory.resolve("events.db") + "?busy_timeout=10000");
db = new JdbcTemplate(source);
db.execute("create table User(id integer primary key,username text,AuthCode text,isEnable integer,lastAccessTime text)");
db.execute("create table subscription_account(id integer primary key,name text,enabled integer,filter_high_multiplier integer)");
db.execute("create table sub_bind(key text primary key,user text,user_id integer,subscription_account_id integer)");
db.execute("create table sub_update_record(id integer primary key autoincrement,user text,ip text,UA text,time integer,location text,user_id integer)");
db.execute("create table subscription_event_receipt(event_id text primary key,key_hash text,client text,occurred_at integer,source text,ingested_at integer,fingerprint text,outcome text)");
db.execute("create table subscription_event_binding(key_hash text primary key,user_id integer,user_name text,first_seen_at integer,last_seen_at integer)");
db.update("insert into User(id,username,isEnable) values(5,'alice',1)");
db.update("insert into subscription_account values(1,'account',1,1)");
db.update("insert into sub_bind values(?,'alice',5,1)", KEY);
var bean = new SqlSessionFactoryBean();
bean.setDataSource(source);
var factory = bean.getObject();
factory.getConfiguration().addMapper(SubMapper.class);
factory.getConfiguration().addMapper(UserMapper.class);
factory.getConfiguration().addMapper(SubscriptionEventMapper.class);
var session = new SqlSessionTemplate(factory);
subscriptions = session.getMapper(SubMapper.class);
service = new SubscriptionEventService(session.getMapper(SubscriptionEventMapper.class), subscriptions,
session.getMapper(UserMapper.class), new TransactionTemplate(new DataSourceTransactionManager(source)),
new SubscriptionStateCoordinator());
service.syncSecret = SECRET;
service.clock = Clock.fixed(Instant.ofEpochMilli(NOW), ZoneOffset.UTC);
service.close(); // Reject optional geolocation work; never make network calls from these tests.
}
@Test void hmacRejectsInvalidExpiredModifiedAndOversizedInputs() throws Exception {
byte[] body = batch(event(NOW - 60_000));
String timestamp = Long.toString(NOW / 1000);
assertTrue(service.verify(body, timestamp, sign(body, timestamp)));
assertFalse(service.verify(body, timestamp, "0".repeat(64)));
assertFalse(service.verify("changed".getBytes(StandardCharsets.UTF_8), timestamp, sign(body, timestamp)));
String expired = Long.toString(NOW / 1000 - 301);
assertFalse(service.verify(body, expired, sign(body, expired)));
assertFalse(service.verify(body, "9".repeat(1024), "0".repeat(64)));
assertFalse(service.verify(new byte[SubscriptionEventService.MAX_BODY + 1], timestamp, "0".repeat(64)));
}
@Test void primaryAndStandbySameEventProduceOneRecordAndPreserveOriginalMetadata() {
SubscriptionEvent event = event(NOW - 86_400_000);
service.recordPrimary(event, subscriptions.selectSubBind(KEY));
assertEquals(List.of(event.getEventId()), service.ingest(List.of(event)).get("acknowledged"));
assertEquals(1, count("sub_update_record"));
assertEquals(1, count("subscription_event_receipt"));
assertEquals("primary", db.queryForObject("select source from subscription_event_receipt", String.class));
Map<String, Object> row = db.queryForMap("select * from sub_update_record");
assertEquals(event.getOccurredAt(), ((Number) row.get("time")).longValue());
assertEquals("2001:db8::7", row.get("ip"));
assertEquals("client/1.2", row.get("UA"));
assertEquals(5, ((Number) row.get("user_id")).intValue());
}
@Test void concurrentPrimaryAndReplayTransactionsReserveTheWriterBeforeReading() throws Exception {
var start = new java.util.concurrent.CountDownLatch(1);
var executor = java.util.concurrent.Executors.newFixedThreadPool(8);
var binding = subscriptions.selectSubBind(KEY);
try {
List<java.util.concurrent.Future<?>> futures = new ArrayList<>();
for (int i = 0; i < 32; i++) {
SubscriptionEvent event = event(NOW - i);
boolean primary = i % 2 == 0;
futures.add(executor.submit(() -> {
try { start.await(); }
catch (InterruptedException interrupted) { Thread.currentThread().interrupt(); throw new RuntimeException(interrupted); }
if (primary) service.recordPrimary(event, binding);
else assertEquals(List.of(event.getEventId()), service.ingest(List.of(event)).get("acknowledged"));
}));
}
start.countDown();
for (var future : futures) future.get(30, java.util.concurrent.TimeUnit.SECONDS);
assertEquals(32, count("subscription_event_receipt"));
assertEquals(10, count("sub_update_record"));
} finally { executor.shutdownNow(); }
}
@Test void keyRotationAndRenameRetainStableIdentity() {
service.rememberBindings(subscriptions.selectAllSubBind());
db.update("update sub_bind set key='rotated-key'");
db.update("update User set username='renamed' where id=5");
SubscriptionEvent old = event(NOW - 30_000);
assertEquals(List.of(old.getEventId()), service.ingest(List.of(old)).get("acknowledged"));
assertEquals("renamed", db.queryForObject("select user from sub_update_record", String.class));
assertEquals(5, db.queryForObject("select user_id from sub_update_record", Integer.class));
assertEquals(SubscriptionEventService.sha256(KEY.getBytes(StandardCharsets.UTF_8)),
db.queryForObject("select key_hash from subscription_event_binding", String.class));
}
@Test void legacyLabelWithoutUserHasPrimaryStandbyReplayAndRotatedKeyHistory() {
db.update("update sub_bind set user='legacy-label',user_id=NULL");
db.update("delete from User");
SubscriptionEvent primary = event(NOW - 10_000);
service.recordPrimary(primary, subscriptions.selectSubBind(KEY));
assertEquals(List.of(primary.getEventId()), service.ingest(List.of(primary)).get("acknowledged"));
assertEquals(1, count("sub_update_record"));
assertEquals("legacy-label", db.queryForObject("select user from sub_update_record", String.class));
assertNull(db.queryForObject("select user_id from sub_update_record", Integer.class));
service.rememberBindings(subscriptions.selectAllSubBind());
db.update("update sub_bind set key='rotated-legacy-key'");
SubscriptionEvent offline = event(NOW - 5_000);
assertEquals(List.of(offline.getEventId()), service.ingest(List.of(offline)).get("acknowledged"));
assertEquals(2, count("sub_update_record"));
assertEquals(0, count("User"));
assertEquals(2, db.queryForObject("select count(*) from sub_update_record where user='legacy-label' and user_id is null", Integer.class));
for (int i = 0; i < 12; i++) service.ingest(List.of(event(NOW - i)));
assertEquals(10, count("sub_update_record"));
assertEquals(14, count("subscription_event_receipt"));
service.ingest(List.of(primary));
assertEquals(10, count("sub_update_record"));
}
@Test void legacyLabelDoesNotAdoptNewSameNamedUser() {
db.update("update sub_bind set user='legacy-label',user_id=NULL");
service.rememberBindings(subscriptions.selectAllSubBind());
db.update("delete from sub_bind");
db.update("insert into User(id,username,isEnable) values(8,'legacy-label',1)");
SubscriptionEvent event = event(NOW);
assertEquals(List.of(event.getEventId()), service.ingest(List.of(event)).get("acknowledged"));
assertNull(db.queryForObject("select user_id from sub_update_record", Integer.class));
}
@Test void unknownBindingIsNotAcknowledgedAndDeletedUserIsTerminal() {
SubscriptionEvent unknown = event(NOW);
unknown.setKeyHash("0".repeat(64));
assertEquals(List.of(), service.ingest(List.of(unknown)).get("acknowledged"));
assertEquals(0, count("subscription_event_receipt"));
service.rememberBindings(subscriptions.selectAllSubBind());
db.update("delete from sub_bind");
db.update("delete from User");
SubscriptionEvent deleted = event(NOW - 1_000);
assertEquals(List.of(deleted.getEventId()), service.ingest(List.of(deleted)).get("acknowledged"));
assertEquals("user_deleted", db.queryForObject("select outcome from subscription_event_receipt", String.class));
assertEquals(0, count("sub_update_record"));
}
@Test void transactionFailureRollsBackWholeBatchAndControllerNeverAcknowledges() throws Exception {
db.execute("create trigger fail_receipt before insert on subscription_event_receipt begin select raise(ABORT,'test rollback'); end");
SubscriptionEvent event = event(NOW);
assertThrows(RuntimeException.class, () -> service.ingest(List.of(event)));
assertEquals(0, count("sub_update_record"));
assertEquals(0, count("subscription_event_receipt"));
var controller = new SubscriptionEventController(service);
assertThrows(RuntimeException.class, () -> controller.ingest(signedPost(batch(event))));
assertEquals(0, count("sub_update_record"));
}
@Test void pruningUsesEventTimeAndReceiptsSurviveVisibleRecordDeletion() {
SubscriptionEvent oldest = event(NOW - 100_000);
service.ingest(List.of(oldest));
List<SubscriptionEvent> newer = new ArrayList<>();
for (int i = 0; i < 12; i++) newer.add(event(NOW - i * 1_000));
service.ingest(newer);
assertEquals(10, count("sub_update_record"));
assertEquals(13, count("subscription_event_receipt"));
assertEquals(NOW - 9_000, db.queryForObject("select min(time) from sub_update_record", Long.class));
assertEquals(List.of(oldest.getEventId()), service.ingest(List.of(oldest)).get("acknowledged"));
assertEquals(10, count("sub_update_record"));
db.update("delete from sub_update_record");
service.recordPrimary(newer.getFirst(), subscriptions.selectSubBind(KEY));
assertEquals(0, count("sub_update_record"));
assertEquals(13, count("subscription_event_receipt"));
}
@Test void signedGetVerifiesKeyClientUaAndUsesAuthenticatedIp() throws Exception {
SubscriptionEvent event = event(NOW);
byte[] body = CustomUtil.objectMapper.writeValueAsBytes(asMap(event));
var request = new MockHttpServletRequest("GET", "/sub/cat/" + KEY);
request.setRemoteAddr("127.0.0.1");
request.addHeader("X-Forwarded-For", "198.51.100.123");
request.addHeader("User-Agent", event.getUserAgent());
request.addHeader("X-Subscription-Event", Base64.getUrlEncoder().withoutPadding().encodeToString(body));
String timestamp = Long.toString(NOW / 1000);
request.addHeader("X-Subscription-Timestamp", timestamp);
request.addHeader("X-Subscription-Signature", sign(body, timestamp));
SubscriptionEvent parsed = service.requestEvent(request, KEY, "cat");
assertNotNull(parsed);
assertEquals("2001:db8::7", parsed.getClientIp());
assertNull(service.requestEvent(request, "wrong-key", "cat"));
assertNull(service.requestEvent(request, KEY, "v2"));
request.removeHeader("User-Agent");
request.addHeader("User-Agent", "different-client");
assertNull(service.requestEvent(request, KEY, "cat"));
}
@Test void signedHeadAndUnavailableCacheDoNotWriteAndGetReplayWritesOnce() throws Exception {
var refresh = mock(SubscriptionRefreshService.class);
var sub = new SubService(subscriptions, mock(UserMapper.class), refresh, mock(RemoteService.class),
new SubscriptionStateCoordinator());
sub.subscriptionEvents = service;
sub.locationExecutor = runnable -> { };
Path cache = directory.resolve("subscription.txt");
when(refresh.cachedPath(1, "cat")).thenReturn(cache);
SubscriptionEvent event = event(NOW);
var unavailable = new MockHttpServletResponse();
sub.updateSub(unavailable, signedGet(event, "GET"), "cat", KEY);
assertEquals(503, unavailable.getStatus());
assertEquals(0, count("sub_update_record"));
Files.writeString(cache, "subscription-content");
var head = new MockHttpServletResponse();
sub.updateSub(head, signedGet(event, "HEAD"), "cat", KEY);
assertEquals(200, head.getStatus());
assertEquals(0, head.getContentAsByteArray().length);
assertEquals(0, count("sub_update_record"));
assertEquals(0, count("subscription_event_receipt"));
for (int i = 0; i < 2; i++) {
var get = new MockHttpServletResponse();
sub.updateSub(get, signedGet(event, "GET"), "cat", KEY);
assertEquals(200, get.getStatus());
assertEquals("subscription-content", get.getContentAsString());
}
assertEquals(1, count("sub_update_record"));
assertEquals(1, count("subscription_event_receipt"));
assertEquals("2001:db8::7", db.queryForObject("select ip from sub_update_record", String.class));
sub.shutdownLocationResolver();
}
@Test void invalidSignedGetStillUsesOrdinaryLogging() throws Exception {
var refresh = mock(SubscriptionRefreshService.class);
var sub = new SubService(subscriptions, mock(UserMapper.class), refresh, mock(RemoteService.class),
new SubscriptionStateCoordinator());
sub.subscriptionEvents = service;
sub.locationExecutor = runnable -> { };
Path cache = directory.resolve("subscription.txt");
Files.writeString(cache, "subscription-content");
when(refresh.cachedPath(1, "cat")).thenReturn(cache);
var request = signedGet(event(NOW), "GET");
request.removeHeader("X-Subscription-Signature");
request.addHeader("X-Subscription-Signature", "0".repeat(64));
request.setRemoteAddr("198.51.100.10");
sub.updateSub(new MockHttpServletResponse(), request, "cat", KEY);
assertEquals(1, count("sub_update_record"));
assertEquals(0, count("subscription_event_receipt"));
assertEquals("198.51.100.10", db.queryForObject("select ip from sub_update_record", String.class));
sub.shutdownLocationResolver();
}
@Test void invalidInputAndAuthenticationReturnNoAcknowledgements() throws Exception {
var controller = new SubscriptionEventController(service);
var unauthenticated = new MockHttpServletRequest("POST", "/internal/subscription-events");
unauthenticated.setContent(batch(event(NOW)));
assertEquals(403, controller.ingest(unauthenticated).getStatusCode().value());
assertEquals(400, controller.ingest(signedPost("{}".getBytes(StandardCharsets.UTF_8))).getStatusCode().value());
assertEquals(400, controller.ingest(signedPost(new byte[SubscriptionEventService.MAX_BODY + 1])).getStatusCode().value());
SubscriptionEvent future = event(NOW + 300_001);
assertThrows(IllegalArgumentException.class, () -> service.parseBatch(batch(future)));
SubscriptionEvent tooLong = event(NOW);
tooLong.setUserAgent("a".repeat(2049));
assertThrows(IllegalArgumentException.class, () -> service.parseBatch(batch(tooLong)));
assertEquals(0, count("subscription_event_receipt"));
}
@Test void conflictingIdCannotAcknowledgeOrWriteChangedEvent() {
SubscriptionEvent original = event(NOW);
service.ingest(List.of(original));
original.setClientIp("192.0.2.7");
assertThrows(IllegalArgumentException.class, () -> service.ingest(List.of(original)));
assertEquals(1, count("sub_update_record"));
}
private int count(String table) { return db.queryForObject("select count(*) from " + table, Integer.class); }
private SubscriptionEvent event(long time) {
return new SubscriptionEvent(UUID.randomUUID().toString(), SubscriptionEventService.sha256(KEY.getBytes(StandardCharsets.UTF_8)),
"cat", "2001:db8::7", "client/1.2", time);
}
private static Map<String, Object> asMap(SubscriptionEvent event) {
return Map.of("eventId", event.getEventId(), "keyHash", event.getKeyHash(), "client", event.getClient(),
"clientIp", event.getClientIp(), "userAgent", event.getUserAgent(), "occurredAt", event.getOccurredAt());
}
private static byte[] batch(SubscriptionEvent event) {
return CustomUtil.objectMapper.writeValueAsBytes(Map.of("events", List.of(asMap(event))));
}
private static String sign(byte[] body, String timestamp) throws Exception {
var mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
return HexFormat.of().formatHex(mac.doFinal(("v1\n" + timestamp + "\n" + SubscriptionEventService.sha256(body)).getBytes(StandardCharsets.UTF_8)));
}
private MockHttpServletRequest signedGet(SubscriptionEvent event, String method) throws Exception {
byte[] body = CustomUtil.objectMapper.writeValueAsBytes(asMap(event));
var request = new MockHttpServletRequest(method, "/sub/cat/" + KEY);
request.setRemoteAddr("127.0.0.1");
request.addHeader("User-Agent", event.getUserAgent());
request.addHeader("X-Subscription-Event", Base64.getUrlEncoder().withoutPadding().encodeToString(body));
String timestamp = Long.toString(NOW / 1000);
request.addHeader("X-Subscription-Timestamp", timestamp);
request.addHeader("X-Subscription-Signature", sign(body, timestamp));
return request;
}
private MockHttpServletRequest signedPost(byte[] body) throws Exception {
var request = new MockHttpServletRequest("POST", "/internal/subscription-events");
request.setContent(body);
String timestamp = Long.toString(NOW / 1000);
request.addHeader("X-Subscription-Timestamp", timestamp);
request.addHeader("X-Subscription-Signature", sign(body, timestamp));
return request;
}
}
@@ -67,6 +67,25 @@ class SubscriptionFilteringTest {
doReturn(clash).when(service).download(contains("client=clashmeta"), any()); doReturn(clash).when(service).download(contains("client=clashmeta"), any());
} }
/** 读取该账号当前发布分代里的 v2 内容并解回明文。 */
private String readV2(int accountId) throws Exception {
Path path = service.cachedPath(accountId, "v2");
assertTrue(Files.isRegularFile(path), "应存在已发布的 v2 缓存: " + path);
return decodeV2(List.of(Files.readString(path, StandardCharsets.UTF_8)));
}
/** 读取该账号当前发布分代里的 clash 内容。 */
private String readClash(int accountId) throws Exception {
Path path = service.cachedPath(accountId, "cat");
assertTrue(Files.isRegularFile(path), "应存在已发布的 clash 缓存: " + path);
return Files.readString(path, StandardCharsets.UTF_8);
}
/** 空的合法 Clash 上游,用于只关心 v2 侧行为的用例。 */
private static List<String> emptyClash() {
return List.of("proxies: []");
}
// ---------- URL 模板 ---------- // ---------- URL 模板 ----------
/** {key} 必须被 URL 编码后替换,避免特殊字符破坏 URL。 */ /** {key} 必须被 URL 编码后替换,避免特殊字符破坏 URL。 */
@@ -100,10 +119,12 @@ class SubscriptionFilteringTest {
assertTrue(service.refresh(1)); assertTrue(service.refresh(1));
Path v2 = cacheRoot.resolve("1/v2ray.txt"); Path v2 = service.cachedPath(1, "v2");
Path clash = cacheRoot.resolve("1/clash.yaml"); Path clash = service.cachedPath(1, "cat");
assertTrue(Files.isRegularFile(v2), "应写入 v2 缓存"); assertTrue(Files.isRegularFile(v2), "应写入 v2 缓存");
assertTrue(Files.isRegularFile(clash), "应写入 clash 缓存"); assertTrue(Files.isRegularFile(clash), "应写入 clash 缓存");
// 两份文件位于同一分代目录,才能保证读取到的内容同版本。
assertEquals(v2.getParent(), clash.getParent(), "两份缓存必须来自同一分代");
// v2 缓存本身是 Base64 文本,需解码后再断言节点内容 // v2 缓存本身是 Base64 文本,需解码后再断言节点内容
String decodedV2 = new String(Base64.getMimeDecoder() String decodedV2 = new String(Base64.getMimeDecoder()
.decode(Files.readString(v2)), StandardCharsets.UTF_8); .decode(Files.readString(v2)), StandardCharsets.UTF_8);
@@ -111,7 +132,9 @@ class SubscriptionFilteringTest {
verify(subMapper).markSubscriptionRefreshSuccess(eq(1), anyLong()); verify(subMapper).markSubscriptionRefreshSuccess(eq(1), anyLong());
verify(subMapper, never()).markSubscriptionRefreshFailure(anyInt(), anyString()); verify(subMapper, never()).markSubscriptionRefreshFailure(anyInt(), anyString());
// 临时文件不应残留 // 临时文件不应残留
assertFalse(Files.exists(cacheRoot.resolve("1/clash.yaml.tmp"))); try (var files = Files.walk(cacheRoot)) {
assertTrue(files.noneMatch(path -> path.toString().endsWith(".tmp")), "临时文件不应残留");
}
} }
/** 停用账号或不存在账号直接返回 false,且不发任何网络请求。 */ /** 停用账号或不存在账号直接返回 false,且不发任何网络请求。 */
@@ -136,12 +159,11 @@ class SubscriptionFilteringTest {
stubUpstream(v2Upstream( stubUpstream(v2Upstream(
"vmess://aaa#keep-1", "vmess://aaa#keep-1",
"vmess://bbb#香港 5x", "vmess://bbb#香港 5x",
"vmess://ccc#keep-2"), List.of("proxies:")); "vmess://ccc#keep-2"), emptyClash());
assertTrue(service.refresh(5)); assertTrue(service.refresh(5));
String content = Files.readString(cacheRoot.resolve("5/v2ray.txt"), StandardCharsets.UTF_8); String decoded = readV2(5);
String decoded = new String(Base64.getMimeDecoder().decode(content), StandardCharsets.UTF_8);
assertTrue(decoded.contains("keep-1"), "实际内容: " + decoded); assertTrue(decoded.contains("keep-1"), "实际内容: " + decoded);
assertTrue(decoded.contains("keep-2"), "实际内容: " + decoded); assertTrue(decoded.contains("keep-2"), "实际内容: " + decoded);
assertFalse(decoded.contains("5x"), "高倍率节点应被剔除: " + decoded); assertFalse(decoded.contains("5x"), "高倍率节点应被剔除: " + decoded);
@@ -152,12 +174,11 @@ class SubscriptionFilteringTest {
void v2WithoutFilterKeepsHighMultiplierNodes() throws Exception { void v2WithoutFilterKeepsHighMultiplierNodes() throws Exception {
SubscriptionAccount acc = account(6, "key6", true, false); SubscriptionAccount acc = account(6, "key6", true, false);
when(subMapper.selectSubscriptionAccount(6)).thenReturn(acc); when(subMapper.selectSubscriptionAccount(6)).thenReturn(acc);
stubUpstream(v2Upstream("vmess://bbb#香港 5x"), List.of("proxies:")); stubUpstream(v2Upstream("vmess://bbb#香港 5x"), emptyClash());
assertTrue(service.refresh(6)); assertTrue(service.refresh(6));
String content = Files.readString(cacheRoot.resolve("6/v2ray.txt"), StandardCharsets.UTF_8); String decoded = readV2(6);
String decoded = new String(Base64.getMimeDecoder().decode(content), StandardCharsets.UTF_8);
assertTrue(decoded.contains("5x"), "未开启过滤时应保留: " + decoded); assertTrue(decoded.contains("5x"), "未开启过滤时应保留: " + decoded);
} }
@@ -166,11 +187,11 @@ class SubscriptionFilteringTest {
void v2FilterKeepsNodesWithoutMultiplierMarker() throws Exception { void v2FilterKeepsNodesWithoutMultiplierMarker() throws Exception {
SubscriptionAccount acc = account(7, "key7", true, true); SubscriptionAccount acc = account(7, "key7", true, true);
when(subMapper.selectSubscriptionAccount(7)).thenReturn(acc); when(subMapper.selectSubscriptionAccount(7)).thenReturn(acc);
stubUpstream(v2Upstream("vmess://aaa#plain-node", "trojan://bbb#another"), List.of("proxies:")); stubUpstream(v2Upstream("vmess://aaa#plain-node", "trojan://bbb#another"), emptyClash());
assertTrue(service.refresh(7)); assertTrue(service.refresh(7));
String decoded = decodeV2(List.of(Files.readString(cacheRoot.resolve("7/v2ray.txt")))); String decoded = readV2(7);
assertTrue(decoded.contains("plain-node")); assertTrue(decoded.contains("plain-node"));
assertTrue(decoded.contains("another")); assertTrue(decoded.contains("another"));
} }
@@ -180,11 +201,11 @@ class SubscriptionFilteringTest {
void v2FilterTreatsThresholdAsExclusive() throws Exception { void v2FilterTreatsThresholdAsExclusive() throws Exception {
SubscriptionAccount acc = account(8, "key8", true, true); SubscriptionAccount acc = account(8, "key8", true, true);
when(subMapper.selectSubscriptionAccount(8)).thenReturn(acc); when(subMapper.selectSubscriptionAccount(8)).thenReturn(acc);
stubUpstream(v2Upstream("vmess://aaa#节点 2x", "vmess://bbb#节点 2.1x"), List.of("proxies:")); stubUpstream(v2Upstream("vmess://aaa#节点 2x", "vmess://bbb#节点 2.1x"), emptyClash());
assertTrue(service.refresh(8)); assertTrue(service.refresh(8));
String decoded = decodeV2(List.of(Files.readString(cacheRoot.resolve("8/v2ray.txt")))); String decoded = readV2(8);
assertTrue(decoded.contains("节点 2x"), "等于阈值应保留: " + decoded); assertTrue(decoded.contains("节点 2x"), "等于阈值应保留: " + decoded);
assertFalse(decoded.contains("2.1x"), "超过阈值应剔除: " + decoded); assertFalse(decoded.contains("2.1x"), "超过阈值应剔除: " + decoded);
} }
@@ -212,7 +233,7 @@ class SubscriptionFilteringTest {
assertTrue(service.refresh(9)); assertTrue(service.refresh(9));
String yaml = Files.readString(cacheRoot.resolve("9/clash.yaml"), StandardCharsets.UTF_8); String yaml = readClash(9);
assertTrue(yaml.contains("keep"), "保留节点应在: " + yaml); assertTrue(yaml.contains("keep"), "保留节点应在: " + yaml);
assertFalse(yaml.contains("drop 5x"), "高倍率节点不应出现(含分组引用): " + yaml); assertFalse(yaml.contains("drop 5x"), "高倍率节点不应出现(含分组引用): " + yaml);
} }
@@ -234,7 +255,7 @@ class SubscriptionFilteringTest {
assertTrue(service.refresh(10)); assertTrue(service.refresh(10));
String yaml = Files.readString(cacheRoot.resolve("10/clash.yaml"), StandardCharsets.UTF_8); String yaml = readClash(10);
assertTrue(yaml.contains("keep 5x"), "未过滤时应保留: " + yaml); assertTrue(yaml.contains("keep 5x"), "未过滤时应保留: " + yaml);
} }
@@ -253,7 +274,7 @@ class SubscriptionFilteringTest {
assertTrue(service.refresh(11)); assertTrue(service.refresh(11));
String yaml = Files.readString(cacheRoot.resolve("11/clash.yaml"), StandardCharsets.UTF_8); String yaml = readClash(11);
assertTrue(yaml.contains("group 9x"), "proxy-groups 段不应被过滤: " + yaml); assertTrue(yaml.contains("group 9x"), "proxy-groups 段不应被过滤: " + yaml);
} }
@@ -270,7 +291,7 @@ class SubscriptionFilteringTest {
verify(subMapper).markSubscriptionRefreshFailure(eq(12), contains("为空")); verify(subMapper).markSubscriptionRefreshFailure(eq(12), contains("为空"));
verify(subMapper, never()).markSubscriptionRefreshSuccess(anyInt(), anyLong()); verify(subMapper, never()).markSubscriptionRefreshSuccess(anyInt(), anyLong());
assertFalse(Files.exists(cacheRoot.resolve("12/v2ray.txt")), "失败时不应产出缓存"); assertFalse(service.hasCompleteCache(12), "失败时不应产出缓存");
} }
/** 上游不是合法 Base64 时必须失败并记录原因。 */ /** 上游不是合法 Base64 时必须失败并记录原因。 */
@@ -283,7 +304,7 @@ class SubscriptionFilteringTest {
assertFalse(service.refresh(13)); assertFalse(service.refresh(13));
verify(subMapper).markSubscriptionRefreshFailure(eq(13), contains("Base64")); verify(subMapper).markSubscriptionRefreshFailure(eq(13), contains("Base64"));
assertFalse(Files.exists(cacheRoot.resolve("13/v2ray.txt"))); assertFalse(service.hasCompleteCache(13));
} }
/** 网络异常同样必须被兜住并记账,不能向上抛。 */ /** 网络异常同样必须被兜住并记账,不能向上抛。 */
@@ -334,7 +355,7 @@ class SubscriptionFilteringTest {
throw new java.io.IOException("account 22 upstream down"); throw new java.io.IOException("account 22 upstream down");
if (url.contains("client=v2")) if (url.contains("client=v2"))
return List.of(Base64.getEncoder().encodeToString("vmess://a".getBytes(StandardCharsets.UTF_8))); return List.of(Base64.getEncoder().encodeToString("vmess://a".getBytes(StandardCharsets.UTF_8)));
return List.of("proxies:"); return emptyClash();
}).when(service).download(anyString(), any()); }).when(service).download(anyString(), any());
assertTrue(service.refresh(20)); assertTrue(service.refresh(20));
@@ -342,8 +363,8 @@ class SubscriptionFilteringTest {
verify(subMapper).markSubscriptionRefreshSuccess(eq(20), anyLong()); verify(subMapper).markSubscriptionRefreshSuccess(eq(20), anyLong());
verify(subMapper).markSubscriptionRefreshFailure(eq(22), anyString()); verify(subMapper).markSubscriptionRefreshFailure(eq(22), anyString());
assertTrue(Files.exists(cacheRoot.resolve("20/v2ray.txt")), "成功账号应落盘"); assertTrue(service.hasCompleteCache(20), "成功账号应落盘");
assertFalse(Files.exists(cacheRoot.resolve("22/v2ray.txt")), "失败账号不应落盘"); assertFalse(service.hasCompleteCache(22), "失败账号不应落盘");
} }
// ---------- 缓存状态 ---------- // ---------- 缓存状态 ----------
@@ -358,9 +379,12 @@ class SubscriptionFilteringTest {
Files.writeString(cacheRoot.resolve("40/clash.yaml"), "y"); Files.writeString(cacheRoot.resolve("40/clash.yaml"), "y");
assertTrue(service.hasCompleteCache(40)); assertTrue(service.hasCompleteCache(40));
// 旧结构两份齐全时也应能定位到可分发路径。
assertEquals(cacheRoot.resolve("40/v2ray.txt"), service.cachedPath(40, "v2"));
assertEquals(cacheRoot.resolve("40/clash.yaml"), service.cachedPath(40, "cat"));
} }
/** 失效缓存应删除两份文件,使同一账号的被拒请求不再拿到旧内容。 */ /** 失效缓存应清掉该账号的全部缓存,使同一账号的被拒请求不再拿到旧内容。 */
@Test @Test
void invalidateCacheRemovesBothFiles() throws Exception { void invalidateCacheRemovesBothFiles() throws Exception {
Files.createDirectories(cacheRoot.resolve("41")); Files.createDirectories(cacheRoot.resolve("41"));
@@ -371,6 +395,7 @@ class SubscriptionFilteringTest {
assertFalse(Files.exists(cacheRoot.resolve("41/v2ray.txt"))); assertFalse(Files.exists(cacheRoot.resolve("41/v2ray.txt")));
assertFalse(Files.exists(cacheRoot.resolve("41/clash.yaml"))); assertFalse(Files.exists(cacheRoot.resolve("41/clash.yaml")));
assertFalse(service.hasCompleteCache(41));
} }
/** 缓存文件本就不存在时不应抛异常。 */ /** 缓存文件本就不存在时不应抛异常。 */
@@ -378,4 +403,43 @@ class SubscriptionFilteringTest {
void invalidateCacheIsIdempotent() { void invalidateCacheIsIdempotent() {
assertDoesNotThrow(() -> service.invalidateCache(42)); assertDoesNotThrow(() -> service.invalidateCache(42));
} }
/**
* 分发取用的路径必须来自完整版本:半缓存不能解析成真实文件,
* 否则 SubService 的 {@code Files.isRegularFile} 会放行残缺内容。
*/
@Test
void cachedPathNeverPointsAtIncompleteLegacyCache() throws Exception {
Files.createDirectories(cacheRoot.resolve("43"));
Files.writeString(cacheRoot.resolve("43/v2ray.txt"), "half");
assertFalse(Files.isRegularFile(service.cachedPath(43, "v2")),
"只有一份旧文件时 cachedPath 不得指向真实文件");
assertFalse(Files.isRegularFile(service.cachedPath(43, "cat")));
assertFalse(service.hasCompleteCache(43));
// 补齐第二份后同一契约应立刻放行,且指向 legacy 文件本身。
Files.writeString(cacheRoot.resolve("43/clash.yaml"), "half-yaml");
assertTrue(service.hasCompleteCache(43));
assertEquals(cacheRoot.resolve("43/v2ray.txt"), service.cachedPath(43, "v2"));
assertEquals(cacheRoot.resolve("43/clash.yaml"), service.cachedPath(43, "cat"));
}
/** 抓取失败时必须保留上一份完整版本,而不是留下半截状态。 */
@Test
void failedRefreshKeepsPreviousCompleteGeneration() throws Exception {
SubscriptionAccount acc = account(50, "key50", true, true);
when(subMapper.selectSubscriptionAccount(50)).thenReturn(acc);
stubUpstream(v2Upstream("vmess://good#node"), emptyClash());
assertTrue(service.refresh(50));
String goodV2 = readV2(50);
assertTrue(goodV2.contains("good"));
doThrow(new java.io.IOException("upstream 500")).when(service).download(anyString(), any());
assertFalse(service.refresh(50));
assertTrue(service.hasCompleteCache(50), "失败后仍应保留上次完整版本");
assertEquals(goodV2, readV2(50));
}
} }
@@ -41,12 +41,11 @@ class SubscriptionStandbySnapshotServiceTest {
default -> throw new UnsupportedOperationException(method.getName()); default -> throw new UnsupportedOperationException(method.getName());
}); });
SubscriptionStateCoordinator coordinator = new SubscriptionStateCoordinator(); SubscriptionStateCoordinator coordinator = new SubscriptionStateCoordinator();
// 旧结构(账号根下两份文件)仍应被读取为 legacy 版本。
Path accountDirectory = directory.resolve("1"); Path accountDirectory = directory.resolve("1");
Files.createDirectories(accountDirectory); Files.createDirectories(accountDirectory);
Path v2 = accountDirectory.resolve("v2ray.txt"); Files.writeString(accountDirectory.resolve("v2ray.txt"), "v2-content", StandardCharsets.UTF_8);
Path clash = accountDirectory.resolve("clash.yaml"); Files.writeString(accountDirectory.resolve("clash.yaml"), "clash-content", StandardCharsets.UTF_8);
Files.writeString(v2, "v2-content", StandardCharsets.UTF_8);
Files.writeString(clash, "clash-content", StandardCharsets.UTF_8);
SubscriptionStandbySnapshotService service = new SubscriptionStandbySnapshotService(mapper, coordinator); SubscriptionStandbySnapshotService service = new SubscriptionStandbySnapshotService(mapper, coordinator);
service.syncSecret = "test-sync-secret"; service.syncSecret = "test-sync-secret";
@@ -69,4 +68,79 @@ class SubscriptionStandbySnapshotServiceTest {
assertEquals(2, payload.getBindings().size()); assertEquals(2, payload.getBindings().size());
assertTrue(payload.getAccounts().getFirst().isFilterHighMultiplier()); assertTrue(payload.getAccounts().getFirst().isFilterHighMultiplier());
} }
/**
* 快照必须从同一分代读出成对内容:即使分代目录里另有一份更新(未发布)的内容,
* 也只能取到指针指向的那一对,不能出现「新 v2 + 旧 clash」的组合。
*/
@Test
void snapshotReadsPairedContentFromOneGeneration(@TempDir Path directory) throws Exception {
SubscriptionAccount account = new SubscriptionAccount();
account.setId(5);
account.setName("paired");
account.setUpstreamKey("upstream-secret");
account.setEnabled(true);
ArrayList<SubscriptionAccount> accounts = new ArrayList<>(java.util.List.of(account));
SubMapper mapper = (SubMapper) Proxy.newProxyInstance(SubMapper.class.getClassLoader(),
new Class<?>[]{SubMapper.class}, (proxy, method, args) -> switch (method.getName()) {
case "selectAllSubscriptionAccounts" -> accounts;
case "selectAllSubBind" -> new ArrayList<SubBind>();
default -> throw new UnsupportedOperationException(method.getName());
});
var store = new com.lion.lionwebsite.Util.SubscriptionCacheStore(directory);
store.publish(5, "v2-published".getBytes(StandardCharsets.UTF_8),
"clash-published".getBytes(StandardCharsets.UTF_8));
SubscriptionStandbySnapshotService service =
new SubscriptionStandbySnapshotService(mapper, new SubscriptionStateCoordinator());
service.syncSecret = "test-sync-secret";
service.cacheRoot = directory.toString();
service.cacheStore = store;
SubscriptionSnapshotMessage message = service.build();
byte[] compressed = Base64.getDecoder().decode(message.getPayloadBase64());
byte[] json;
try (GZIPInputStream gzip = new GZIPInputStream(new ByteArrayInputStream(compressed))) {
json = gzip.readAllBytes();
}
SubscriptionSnapshotPayload payload = CustomUtil.objectMapper.readValue(json, SubscriptionSnapshotPayload.class);
assertEquals(1, payload.getAccounts().size());
var sent = payload.getAccounts().getFirst();
assertEquals("v2-published", new String(Base64.getDecoder().decode(sent.getV2ContentBase64()), StandardCharsets.UTF_8));
assertEquals("clash-published", new String(Base64.getDecoder().decode(sent.getClashContentBase64()), StandardCharsets.UTF_8));
}
/** 只有一份 legacy 文件(不完整)的账号不得进入快照。 */
@Test
void snapshotSkipsIncompleteLegacyAccount(@TempDir Path directory) throws Exception {
SubscriptionAccount account = new SubscriptionAccount();
account.setId(6);
account.setName("incomplete");
account.setUpstreamKey("upstream-secret");
account.setEnabled(true);
ArrayList<SubscriptionAccount> accounts = new ArrayList<>(java.util.List.of(account));
SubMapper mapper = (SubMapper) Proxy.newProxyInstance(SubMapper.class.getClassLoader(),
new Class<?>[]{SubMapper.class}, (proxy, method, args) -> switch (method.getName()) {
case "selectAllSubscriptionAccounts" -> accounts;
case "selectAllSubBind" -> new ArrayList<SubBind>();
default -> throw new UnsupportedOperationException(method.getName());
});
Path accountDirectory = directory.resolve("6");
Files.createDirectories(accountDirectory);
Files.writeString(accountDirectory.resolve("v2ray.txt"), "only-v2", StandardCharsets.UTF_8);
SubscriptionStandbySnapshotService service =
new SubscriptionStandbySnapshotService(mapper, new SubscriptionStateCoordinator());
service.syncSecret = "test-sync-secret";
service.cacheRoot = directory.toString();
SubscriptionSnapshotMessage message = service.build();
byte[] compressed = Base64.getDecoder().decode(message.getPayloadBase64());
byte[] json;
try (GZIPInputStream gzip = new GZIPInputStream(new ByteArrayInputStream(compressed))) {
json = gzip.readAllBytes();
}
SubscriptionSnapshotPayload payload = CustomUtil.objectMapper.readValue(json, SubscriptionSnapshotPayload.class);
assertTrue(payload.getAccounts().isEmpty(), "不完整的旧缓存不应进入快照");
}
} }
@@ -0,0 +1,818 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.*;
import com.lion.lionwebsite.Domain.*;
import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import com.lion.lionwebsite.Message.TaskCommandMessage;
import com.lion.lionwebsite.Message.TaskReportMessage;
import com.lion.lionwebsite.Util.PreparedGallery;
import com.lion.lionwebsite.Util.Response;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.transaction.support.TransactionTemplate;
import java.io.IOException;
import java.nio.file.Path;
import java.sql.Connection;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
/**
* 任务生命周期在真实 SQLite 上的故障行为。
*
* <p>这里刻意不用 mock 的 mapper:只有真实事务、真实唯一约束与真实回滚才能验证
* 「先持久化意图再 POST」「提交未知不重发」「任务+额度同事务」「删除失败保留记录」。
* 数据库是 /tmp 下的临时副本,不接触生产库。
*/
class TaskLifecycleIntegrationTest {
@TempDir
Path directory;
private SqlSessionTemplate template;
private DriverManagerDataSource source;
private GalleryMapper galleries;
private CollectMapper collects;
private CustomConfigurationMapper configurations;
private TaskOperationRepository operations;
private RemoteService remote;
private GalleryTaskLifecycleService service;
private FakeGateway gateway;
@BeforeEach
void setUp() throws Exception {
// 主数据源刻意带上 busy_timeout:生产库同样是多线程共用一个 SQLite 文件,
// 没有 busy 重试时并发事务会直接抛 SQLITE_BUSY。
source = new DriverManagerDataSource("jdbc:sqlite:" + directory.resolve("lifecycle.db") + "?busy_timeout=10000");
try (Connection connection = source.getConnection(); var statement = connection.createStatement()) {
statement.executeUpdate("create table gallery (gid integer not null primary key, name varchar(100),"
+ " link varchar(100), language varchar(20), pages integer, status varchar(20), fileSize long,"
+ " createTime double, proceeding integer, resolution varchar(20), displayFileSize varchar(30),"
+ " downloader integer, thumb_link varchar(255), is_download boolean)");
statement.executeUpdate("create table collect (id integer primary key autoincrement, gid integer not null,"
+ " collector integer not null)");
statement.executeUpdate("create table customConfiguration (parameter varchar(30), value varchar(100))");
statement.executeUpdate("create table task_operation (gid integer not null primary key, operationId varchar(64),"
+ " deleteOperationId varchar(64), state varchar(20) not null, lastRevision integer not null default 0,"
+ " lastReportId varchar(64), updateTime long not null default 0)");
statement.executeUpdate("insert into customConfiguration values('weekUsedAmount','0')");
statement.executeUpdate("insert into customConfiguration values('lastResetAmountTime','2026-10-01 00:00:00')");
// 任务生命周期会并发开启事务写同一个 SQLite 文件。WAL 让读不阻塞写,
// 并让写-写冲突走 busy 重试而不是直接 SQLITE_BUSY(与生产库要求的配置一致)。
statement.execute("PRAGMA journal_mode=WAL");
}
var bean = new SqlSessionFactoryBean();
bean.setDataSource(source);
var factory = bean.getObject();
factory.getConfiguration().addMapper(GalleryMapper.class);
factory.getConfiguration().addMapper(CollectMapper.class);
factory.getConfiguration().addMapper(CustomConfigurationMapper.class);
factory.getConfiguration().addMapper(TaskOperationRepository.class);
template = new SqlSessionTemplate(factory);
galleries = template.getMapper(GalleryMapper.class);
collects = template.getMapper(CollectMapper.class);
configurations = template.getMapper(CustomConfigurationMapper.class);
operations = template.getMapper(TaskOperationRepository.class);
remote = mock(RemoteService.class);
gateway = new FakeGateway();
service = new GalleryTaskLifecycleService(galleries, collects, configurations, mock(UserMapper.class),
operations, gateway, remote, new TransactionTemplate(new DataSourceTransactionManager(source)));
}
/** 可编程的假上游:记录调用顺序,按脚本决定 prepare/submit 的结果。 */
private class FakeGateway implements UpstreamGateway {
final List<String> calls = new CopyOnWriteArrayList<>();
PreparedGallery prepared;
boolean prepareReturnsNull;
IOException prepareFailure;
IOException submitFailure;
String submitStatus = TaskStatus.SUBMITTED;
long fileSize = 4096L;
Runnable duringPrepare = () -> { };
Runnable duringSubmit = () -> { };
@Override
public PreparedGallery prepare(String link) throws IOException {
calls.add("prepare");
duringPrepare.run();
if (prepareFailure != null)
throw prepareFailure;
if (prepareReturnsNull)
return null;
if (prepared != null)
return prepared;
Gallery gallery = new Gallery();
gallery.setStatus("等待确认下载");
gallery.setFileSize(fileSize);
gallery.setName("sample [" + 123 + "]");
gallery.setPages(10);
return new PreparedGallery(gallery, "https://example.org/dl", "https://example.org");
}
@Override
public Gallery submit(PreparedGallery prepared, String targetResolution) throws IOException {
calls.add("submit");
duringSubmit.run();
if (submitFailure != null)
throw submitFailure;
Gallery gallery = prepared.gallery();
gallery.setStatus(submitStatus);
gallery.setResolution(targetResolution);
return gallery;
}
}
private static Gallery gallery(int gid, String name, String status) {
Gallery gallery = new Gallery();
gallery.setGid(gid);
gallery.setName(name);
gallery.setStatus(status);
return gallery;
}
private TaskOperation operation(int gid) {
return operations.selectByGid(gid);
}
/** 与主服务相同的真实事务模板;用于构造额外的服务实例。 */
private TransactionTemplate sasTransactions() {
return new TransactionTemplate(new DataSourceTransactionManager(source));
}
/** 读取任务行;对 SQLITE_BUSY 做有限重试,避免并发用例里的瞬时锁冲突影响断言。 */
private Gallery storedGallery(int gid) {
for (int attempt = 0; ; attempt++) {
try {
return galleries.selectGalleryByGid(gid);
} catch (RuntimeException e) {
if (attempt >= 4 || !String.valueOf(e.getMessage()).contains("locked"))
throw e;
try {
Thread.sleep(200);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
throw e;
}
}
}
}
private long weekUsed() {
return Long.parseLong(configurations.selectConfiguration(CustomConfiguration.WEEK_USED_AMOUNT).getValue());
}
/** 直接用 JDBC 数行,避免依赖 MyBatis 的动态 SQL 语句 ID。 */
private int rawInt(String sql) {
try (Connection connection = new DriverManagerDataSource(dbUrl()).getConnection();
var statement = connection.createStatement();
var rows = statement.executeQuery(sql)) {
return rows.next() ? rows.getInt(1) : 0;
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
private String dbUrl() {
return "jdbc:sqlite:" + directory.resolve("lifecycle.db") + "?busy_timeout=10000";
}
// ---------- 意图先于外部副作用 ----------
/** CREATE 意图(含可见占位行)必须在上游 POST 之前已经落库。 */
@Test
void createIntentIsPersistedBeforeUpstreamSubmit() {
gateway.duringPrepare = () -> {
TaskOperation reserved = operation(123);
assertNotNull(reserved, "prepare 之前必须有唯一 CREATE 意图");
assertEquals(TaskOperation.STATE_RESERVED, reserved.getState());
Gallery placeholder = galleries.selectGalleryByGid(123);
assertNotNull(placeholder, "提交期间任务必须在列表里可见");
assertEquals(TaskStatus.PENDING_SUBMIT, placeholder.getStatus());
};
gateway.duringSubmit = () -> {
assertEquals(TaskOperation.STATE_SUBMITTING, operation(123).getState(),
"真正 POST 之前必须已持久化 SUBMITTING");
};
Response response = service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
assertTrue(response.isSuccess(), response.toJSONString());
assertEquals(List.of("prepare", "submit"), gateway.calls);
assertEquals(TaskOperation.STATE_SUBMITTED, operation(123).getState());
assertEquals(TaskStatus.SUBMITTED, galleries.selectGalleryByGid(123).getStatus());
assertEquals(7, galleries.selectGalleryByGid(123).getDownloader());
verify(remote).sendTaskCommand(eq(TaskCommandMessage.ACTION_DOWNLOAD), anyString(), eq(123), anyString());
}
/**
* 并发提交同一 gid(真实 SQLite + 真实唯一键):恰好一个赢家、上游只 POST 一次,
* 其余全部返回业务失败,绝不抛异常穿透成 500。
*/
@Test
void concurrentCreateForSameGidSubmitsUpstreamOnce() throws Exception {
int threads = 6;
var start = new CountDownLatch(1);
var pool = Executors.newFixedThreadPool(threads);
var accepted = new AtomicInteger();
var failed = new AtomicInteger();
List<Future<?>> futures = new ArrayList<>();
for (int i = 0; i < threads; i++) {
futures.add(pool.submit(() -> {
start.await();
Response response = service.createTask(4242, "任务 [4242]",
"https://example.org/g/4242/key/", "original", 7);
if (response.isSuccess())
accepted.incrementAndGet();
else {
// 输家必须拿到结构化业务失败(锁竞争不得逃逸成异常)。
assertNotNull(response.getData());
failed.incrementAndGet();
}
return null;
}));
}
start.countDown();
for (Future<?> future : futures)
future.get(30, TimeUnit.SECONDS);
pool.shutdownNow();
assertEquals(1, accepted.get(), "同一 gid 必须恰好有一个成功创建者");
assertEquals(threads - 1, failed.get(), "其余请求必须全部返回业务失败");
assertEquals(List.of("prepare", "submit"), gateway.calls, "上游只应被提交一次");
assertEquals(1, (int) rawInt("select count(*) from gallery where gid=4242"),
"同一 gid 不允许出现两条任务行");
assertEquals(1, (int) rawInt("select count(*) from task_operation where gid=4242"));
assertNotNull(operation(4242), "唯一意图必须存在");
}
/** 预占阶段的数据库失败必须转成业务失败且绝不 POST 上游,不能穿透成 500。 */
@Test
void reservationDatabaseFailureBecomesBusinessFailureWithoutSubmit() {
GalleryMapper failing = mock(GalleryMapper.class);
when(failing.selectGalleryByGid(anyInt())).thenReturn(null);
doThrow(new org.springframework.dao.DataIntegrityViolationException("db down"))
.when(failing).insertGallery(any());
GalleryTaskLifecycleService failingService = new GalleryTaskLifecycleService(failing, collects,
configurations, mock(UserMapper.class), operations, gateway, remote, sasTransactions());
Response response = assertDoesNotThrow(
() -> failingService.createTask(555, "任务 [555]", "https://example.org/g/555/key/", "original", 7));
assertFalse(response.isSuccess());
assertTrue(gateway.calls.isEmpty(), "预占失败绝不能产生上游副作用");
}
/** 解析阶段失败(读取异常,未产生上游副作用):预占被释放,可重新提交。 */
@Test
void prepareFailureReleasesIntentSoResubmissionIsPossible() {
gateway.prepareFailure = new IOException("network");
Response first = service.createTask(55, "任务 [55]", "https://example.org/g/55/key/", "original", 7);
assertFalse(first.isSuccess());
assertNull(operation(55), "没有副作用时预占应被释放");
assertNull(galleries.selectGalleryByGid(55), "占位行也应一并释放");
gateway.prepareFailure = null;
Response second = service.createTask(55, "任务 [55]", "https://example.org/g/55/key/", "original", 7);
assertTrue(second.isSuccess(), "释放后应能重新提交");
}
/**
* 释放预占必须严格匹配世代:若解析期间该 gid 已换代(出现新的 RESERVED 记录),
* 旧请求的清理绝不能删掉新世代的意图与占位行。
*/
@Test
void releaseReservationDoesNotTouchNewerGeneration() {
gateway.prepareFailure = new IOException("network");
service.beforePrepare = () -> {
String staleGeneration = operation(123).getOperationId();
assertNotNull(staleGeneration);
// 模拟该 gid 已被新的创建请求接管:换成新 operationId,占位行仍在。
operations.updateOperationId(123, "dl-newer-generation", TaskOperation.STATE_RESERVED, 1);
};
Response response = service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
assertFalse(response.isSuccess());
assertNotNull(operation(123), "新世代的意图不得被旧请求释放");
assertEquals("dl-newer-generation", operation(123).getOperationId());
assertNotNull(galleries.selectGalleryByGid(123), "新世代的占位行不得被旧请求删除");
}
// ---------- 提交未知绝不自动重发 ----------
/** 上游提交超时/断连:落 UNKNOWN,任务行保留,后续同 gid 创建不得再次 POST 上游。 */
@Test
void unknownSubmitNeverResendsUpstream() {
gateway.submitFailure = new IOException("timeout after request was sent");
Response first = service.createTask(77, "任务 [77]", "https://example.org/g/77/key/", "original", 7);
assertFalse(first.isSuccess(), "结果未知不能报成功");
assertEquals(TaskOperation.STATE_UNKNOWN, operation(77).getState());
assertNotNull(galleries.selectGalleryByGid(77), "未知任务必须留在列表里可对账");
// Gallery 与 TaskOperation 必须一致表达「提交未知」:否则运营侧会以为还能自动重试。
assertEquals(TaskStatus.UNKNOWN, galleries.selectGalleryByGid(77).getStatus());
assertEquals(List.of("prepare", "submit"), gateway.calls);
// 用户再点一次:只能拿到「已有此任务」,绝不能第二次 POST 上游。
gateway.submitFailure = null;
Response second = service.createTask(77, "任务 [77]", "https://example.org/g/77/key/", "original", 7);
assertFalse(second.isSuccess());
assertEquals(List.of("prepare", "submit"), gateway.calls, "未知结果不得自动重发上游");
assertEquals(TaskOperation.STATE_UNKNOWN, operation(77).getState());
}
/** 重启恢复:中断在 SUBMITTING 的意图被判为 UNKNOWN,而不是重新提交。 */
@Test
void restartRecoversInterruptedSubmittingToUnknown() {
TaskOperation interrupted = new TaskOperation();
interrupted.setGid(88);
interrupted.setOperationId("dl-interrupted");
interrupted.setState(TaskOperation.STATE_SUBMITTING);
interrupted.setUpdateTime(1);
operations.insert(interrupted);
galleries.insertGallery(gallery(88, "任务 [88]", TaskStatus.PENDING_SUBMIT));
assertEquals(1, service.recoverInterruptedSubmissions());
assertEquals(TaskOperation.STATE_UNKNOWN, operation(88).getState());
assertEquals("dl-interrupted", operation(88).getOperationId(), "恢复不得改写 operationId");
assertEquals(TaskStatus.UNKNOWN, galleries.selectGalleryByGid(88).getStatus(),
"启动恢复后任务行也必须显示提交未知,供人工对账");
assertTrue(gateway.calls.isEmpty(), "恢复过程不得再 POST 上游");
}
/** 重启恢复:还没产生上游副作用的 RESERVED 预占应被安全释放,让用户能重新提交。 */
@Test
void restartReleasesReservedPlaceholders() {
TaskOperation reserved = new TaskOperation();
reserved.setGid(89);
reserved.setOperationId("dl-reserved");
reserved.setState(TaskOperation.STATE_RESERVED);
reserved.setUpdateTime(1);
operations.insert(reserved);
galleries.insertGallery(gallery(89, "任务 [89]", TaskStatus.PENDING_SUBMIT));
assertEquals(1, service.recoverInterruptedSubmissions());
assertNull(operation(89), "RESERVED 尚未 POST,重启应释放占位意图");
assertNull(galleries.selectGalleryByGid(89), "占位行也应一并释放");
assertTrue(gateway.calls.isEmpty(), "释放预占不得 POST 上游");
}
/** 释放预占只有确实删掉匹配记录时才删占位行:不匹配时任务行必须保留。 */
@Test
void restartKeepsPlaceholderWhenReservationRowDoesNotMatch() {
TaskOperation mismatched = new TaskOperation();
mismatched.setGid(90);
mismatched.setOperationId(null);
mismatched.setState(TaskOperation.STATE_RESERVED);
mismatched.setUpdateTime(1);
operations.insert(mismatched);
galleries.insertGallery(gallery(90, "任务 [90]", TaskStatus.PENDING_SUBMIT));
service.recoverInterruptedSubmissions();
assertNotNull(galleries.selectGalleryByGid(90), "没有删掉匹配意图时不得删占位行");
}
// ---------- 任务行与额度同一短事务 ----------
/** 提交成功:任务行与周用量一起写入;重复落库不重复计量。 */
@Test
void submittedTaskAndQuotaCommitTogetherAndQuotaIsIdempotent() {
Response response = service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
assertTrue(response.isSuccess(), response.toJSONString());
assertEquals(4096L, weekUsed(), "用量应按文件大小累计一次");
// 模拟上游已提交后重跑同一意图:不得再计一次用量。
service.persistSubmitted(galleries.selectGalleryByGid(123), 7, operation(123).getOperationId());
assertEquals(4096L, weekUsed(), "重复落库不得重复计用量");
}
/** 落库必须严格匹配本次创建世代:被删除/重建后不得复活已删 gid 或污染新世代。 */
@Test
void persistSubmittedRejectsStaleGeneration() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String liveGeneration = operation(123).getOperationId();
assertThrows(IllegalStateException.class,
() -> service.persistSubmitted(galleries.selectGalleryByGid(123), 7, "dl-stale-generation"),
"operationId 不匹配时禁止落库");
assertEquals(4096L, weekUsed(), "被拒的落库不得重复计用量");
assertNotNull(operation(123), "被拒后原世代记录保持不动");
assertEquals(liveGeneration, operation(123).getOperationId());
service.beginDelete(123);
assertThrows(IllegalStateException.class,
() -> service.persistSubmitted(galleries.selectGalleryByGid(123), 7, liveGeneration),
"待删除/已删除的任务禁止落库复活");
}
/** 用量写入失败时任务行必须回滚,不能留下「有任务没计量」的半截状态。 */
@Test
void quotaFailureRollsBackTaskRow() throws Exception {
// 用真实触发器让 customConfiguration 更新失败,验证事务真的回滚任务行。
try (Connection connection = new DriverManagerDataSource("jdbc:sqlite:" + directory.resolve("lifecycle.db") + "?busy_timeout=10000")
.getConnection(); var statement = connection.createStatement()) {
statement.executeUpdate("create trigger reject_quota before update on customConfiguration"
+ " when new.parameter='weekUsedAmount' begin select raise(ABORT,'simulated quota failure'); end");
}
Response response = service.createTask(124, "任务 [124]", "https://example.org/g/124/key/", "original", 7);
assertFalse(response.isSuccess(), "额度写失败必须回业务失败");
// 提交后的落库事务失败时要回滚本次写入,但「上游可能已受理」这件事不能丢:
// 占位行仍在列表里可见,操作记录标为提交未知,二者一起支持后续人工对账,
// 且绝不允许自动重发上游。
assertNotNull(storedGallery(124), "上游已受理的任务必须保持可见,供对账");
assertEquals(TaskStatus.UNKNOWN, storedGallery(124).getStatus(), "落库失败也必须明确标为提交未知");
assertNull(storedGallery(124).getResolution(), "失败的落库不得留下分辨率等半截字段");
assertEquals(TaskOperation.STATE_UNKNOWN, operation(124).getState(),
"上游已受理但本地未落库:只能记 UNKNOWN 等对账");
}
// ---------- 稳定 operationId ----------
/** 重投/重试同一任务必须复用同一个下载 operationId。 */
@Test
void redispatchReusesStableOperationId() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String first = operation(123).getOperationId();
service.dispatchDownload(123, "任务 [123]");
service.dispatchDownload(123, "任务 [123]");
assertEquals(first, operation(123).getOperationId(), "重投不得更换 operationId");
verify(remote, times(3)).sendTaskCommand(eq(TaskCommandMessage.ACTION_DOWNLOAD), eq(first), eq(123), anyString());
}
// ---------- 删除:失败保留、成功留墓碑 ----------
/** 删除失败:任务行与待删除意图都保留,可沿原记录重试并复用同一个删除 ID。 */
@Test
void deleteFailureRetainsTaskAndReusesDeleteOperationId() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
collects.collect(123, 7);
GalleryTaskLifecycleService.DeleteIntent intent = service.beginDelete(123);
assertTrue(intent.accepted(), intent.message());
String deleteId = intent.deleteOperationId();
assertNotNull(deleteId);
assertEquals(TaskOperation.STATE_DELETE_PENDING, operation(123).getState());
assertEquals(TaskStatus.DELETE_PENDING, galleries.selectGalleryByGid(123).getStatus());
service.failDelete(123);
assertNotNull(galleries.selectGalleryByGid(123), "删除失败必须保留任务记录");
assertEquals(TaskStatus.DELETE_FAILED, galleries.selectGalleryByGid(123).getStatus());
assertTrue(service.beginDelete(123).accepted());
assertEquals(deleteId, service.beginDelete(123).deleteOperationId(),
"重试必须复用同一个删除 operationId");
assertEquals(1, (int) rawInt("select count(*) from collect where gid=123"),
"删除未完成前收藏记录保持不动");
}
/** 删除完成:任务行、收藏与状态在一个事务里落定并留墓碑。 */
@Test
void completedDeleteRemovesRowAndCollectAndLeavesTombstone() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
collects.collect(123, 7);
GalleryTaskLifecycleService.DeleteIntent intent = service.beginDelete(123);
assertTrue(service.completeDelete(123, intent.deleteOperationId()));
assertNull(galleries.selectGalleryByGid(123));
assertEquals(0, (int) rawInt("select count(*) from collect where gid=123"));
assertEquals(TaskOperation.STATE_DELETED, operation(123).getState());
assertNull(operation(123).getDeleteOperationId(), "墓碑不再持有删除 ID");
}
/** 删除意图 ID 不匹配时不得完成删除:防止并发换代把别的一代抹掉。 */
@Test
void completeDeleteRequiresMatchingDeleteIntent() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
service.beginDelete(123);
assertFalse(service.completeDelete(123, "del-not-mine"), "不匹配的删除意图必须被拒");
assertNotNull(galleries.selectGalleryByGid(123), "被拒时任务行必须保留");
assertEquals(TaskOperation.STATE_DELETE_PENDING, operation(123).getState());
}
/** 正在提交(RESERVED/SUBMITTING)的任务不能开始删除:否则 DELETE 会与创建 POST 同时发生。 */
@Test
void beginDeleteRefusesWhileCreateIsInFlight() {
service.beforePrepare = () -> {
GalleryTaskLifecycleService.DeleteIntent duringPrepare = service.beginDelete(123);
assertFalse(duringPrepare.accepted(), "提交中的任务必须拒绝删除");
assertEquals("任务正在提交中,请等待提交结果明确后再删除", duringPrepare.message());
};
service.afterMarkSubmitting = () -> {
GalleryTaskLifecycleService.DeleteIntent duringSubmitting = service.beginDelete(123);
assertFalse(duringSubmitting.accepted(), "SUBMITTING 阶段仍必须拒绝删除");
};
Response response = service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
assertTrue(response.isSuccess(), response.toJSONString());
assertEquals(TaskOperation.STATE_SUBMITTED, operation(123).getState());
assertEquals(TaskStatus.SUBMITTED, galleries.selectGalleryByGid(123).getStatus());
}
/**
* latch 控制的跨线程并发删除:删除线程在创建线程处于 RESERVED(prepare 期间)时进入,
* 必须被拒;创建线程随后正常完成,两者都不产生「删了又回写」。
*/
@Test
void concurrentDeleteDuringPrepareIsRejectedAndCreateCompletes() throws Exception {
var prepared = new CountDownLatch(1);
var deleteAttempted = new CountDownLatch(1);
var deleteAccepted = new java.util.concurrent.atomic.AtomicBoolean();
service.beforePrepare = () -> {
prepared.countDown();
Thread deleter = new Thread(() -> {
try {
prepared.await();
deleteAccepted.set(service.beginDelete(123).accepted());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
deleteAttempted.countDown();
}
});
deleter.start();
};
Response response = service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
assertTrue(deleteAttempted.await(10, TimeUnit.SECONDS), "删除线程应已给出结论");
assertTrue(response.isSuccess(), response.toJSONString());
assertFalse(deleteAccepted.get(), "创建意图未决时并发删除必须被拒");
assertEquals(TaskOperation.STATE_SUBMITTED, operation(123).getState());
assertEquals(TaskStatus.SUBMITTED, galleries.selectGalleryByGid(123).getStatus());
}
/** 墓碑允许用全新 operationId 重新创建;旧世代的迟到上报仍被丢弃。 */
@Test
void tombstoneAllowsFreshCreateWithNewGeneration() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String oldOperationId = operation(123).getOperationId();
String deleteId = service.beginDelete(123).deleteOperationId();
service.completeDelete(123, deleteId);
Response recreated = service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
assertTrue(recreated.isSuccess(), "墓碑之后应能用新 operationId 重新创建");
assertNotEquals(oldOperationId, operation(123).getOperationId());
assertEquals(0L, operation(123).getLastRevision(), "新世代 revision 必须清零");
// 旧世代的迟到上报:operationId 不匹配 → 丢弃,且不得把新任务改回完成。
TaskReportMessage stale = report("stale-report", 5, Map.of(123, oldOperationId),
task(123, TaskStatus.COMPLETE, 10));
TaskReportHandler.CommittedReport committed = service.apply(stale);
assertNotNull(committed, "应当回执以停止重放");
assertTrue(committed.acceptedTasks().isEmpty(), "旧世代上报不得被接受");
assertNotEquals(TaskStatus.COMPLETE, galleries.selectGalleryByGid(123).getStatus());
assertEquals(0L, operation(123).getLastRevision());
}
/** 墓碑必须永久忽略旧上报:不管上报里的 operationId 是否等于已删除的那一代。 */
@Test
void tombstoneIgnoresMatchingGenerationReport() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String deletedOperationId = operation(123).getOperationId();
String deleteId = service.beginDelete(123).deleteOperationId();
service.completeDelete(123, deleteId);
TaskReportHandler.CommittedReport committed = service.apply(
report("r-tomb", 9, Map.of(123, deletedOperationId), task(123, TaskStatus.COMPLETE, 10)));
assertNotNull(committed, "墓碑上报仍要回执以停止重放");
assertTrue(committed.acceptedTasks().isEmpty(), "已删除任务不得被上报复活");
assertNull(galleries.selectGalleryByGid(123), "墓碑不得重新插入任务行");
assertEquals(TaskOperation.STATE_DELETED, operation(123).getState(), "墓碑状态必须保持");
}
// ---------- v2 状态上报 ----------
private static GalleryTask task(int gid, String status, int proceeding) {
GalleryTask task = new GalleryTask();
task.setGid(gid);
task.setName("任务 [" + gid + "]");
task.setStatus(switch (status) {
case TaskStatus.DOWNLOADING -> GalleryTask.DOWNLOADING;
case TaskStatus.WAITING_COMPRESS -> GalleryTask.DOWNLOAD_COMPLETE;
case TaskStatus.COMPRESSING -> GalleryTask.COMPRESSING;
default -> GalleryTask.COMPRESS_COMPLETE;
});
task.setProceeding(proceeding);
return task;
}
private static TaskReportMessage report(String reportId, long revision,
Map<Integer, String> operationIds, GalleryTask... tasks) {
TaskReportMessage report = new TaskReportMessage();
report.setReportId(reportId);
report.setRevision(revision);
report.setGalleryTasks(tasks);
report.setOperationIds(new LinkedHashMap<>(operationIds));
return report;
}
/** 正常上报:落库、记录回执、并把下载中这类中间状态回给前端。 */
@Test
void reportAppliesProgressAndRecordsReceipt() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String operationId = operation(123).getOperationId();
TaskReportHandler.CommittedReport committed = service.apply(
report("r-1", 10, Map.of(123, operationId), task(123, TaskStatus.DOWNLOADING, 4)));
assertNotNull(committed);
assertEquals("r-1", committed.reportId());
assertEquals(1, committed.acceptedTasks().size(), "中间状态也要回给前端");
assertEquals(TaskStatus.DOWNLOADING, galleries.selectGalleryByGid(123).getStatus());
assertEquals(4, galleries.selectGalleryByGid(123).getProceeding());
assertEquals(10L, operation(123).getLastRevision());
assertEquals("r-1", operation(123).getLastReportId());
}
/** 乱序/过期 revision:丢弃但不影响已提交状态,且仍需回执停止重放。 */
@Test
void staleRevisionIsIgnoredButStillAcked() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String operationId = operation(123).getOperationId();
service.apply(report("r-5", 50, Map.of(123, operationId), task(123, TaskStatus.DOWNLOADING, 7)));
TaskReportHandler.CommittedReport committed = service.apply(
report("r-old", 20, Map.of(123, operationId), task(123, TaskStatus.DOWNLOADING, 1)));
assertNotNull(committed);
assertTrue(committed.acceptedTasks().isEmpty());
assertEquals(7, galleries.selectGalleryByGid(123).getProceeding(), "过期上报不得回退进度");
assertEquals(50L, operation(123).getLastRevision());
}
/** 重复上报同一份快照:仍需回执(让节点停止重放),但不得重复通知。 */
@Test
void duplicateReportAcksWithoutDuplicateCompletion() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String operationId = operation(123).getOperationId();
TaskReportMessage completion = report("r-done", 30, Map.of(123, operationId),
task(123, TaskStatus.COMPLETE, 10));
TaskReportHandler.CommittedReport first = service.apply(completion);
TaskReportHandler.CommittedReport repeated = service.apply(completion);
assertNotNull(first);
assertEquals(1, first.newlyCompleted().size(), "首次完成应触发一次通知");
assertNotNull(repeated, "重复上报仍要回执");
assertTrue(repeated.newlyCompleted().isEmpty(), "重复上报不得重复通知");
assertEquals(TaskStatus.COMPLETE, galleries.selectGalleryByGid(123).getStatus());
}
/** 待删除任务的迟到完成上报:忽略,不得把它改回完成。 */
@Test
void reportForPendingDeleteTaskIsIgnored() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String operationId = operation(123).getOperationId();
service.beginDelete(123);
TaskReportHandler.CommittedReport committed = service.apply(
report("r-late", 40, Map.of(123, operationId), task(123, TaskStatus.COMPLETE, 10)));
assertNotNull(committed);
assertTrue(committed.acceptedTasks().isEmpty());
assertEquals(TaskStatus.DELETE_PENDING, galleries.selectGalleryByGid(123).getStatus());
}
/** 未知 gid(任务已删除):忽略但回执,避免节点无限重放。 */
@Test
void unknownGidIsIgnoredButAcked() {
TaskReportHandler.CommittedReport committed = service.apply(
report("r-ghost", 1, Map.of(999, "dl-ghost"), task(999, TaskStatus.DOWNLOADING, 1)));
assertNotNull(committed, "未知 gid 也要回执");
assertTrue(committed.acceptedTasks().isEmpty());
assertNull(galleries.selectGalleryByGid(999));
}
/** v2 任务缺 operationId 的上报:无法归属,必须丢弃而不是改当前任务。 */
@Test
void reportWithoutOperationIdForV2TaskIsRejected() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
TaskReportHandler.CommittedReport committed = service.apply(
report("r-noop", 60, Map.of(), task(123, TaskStatus.COMPLETE, 10)));
assertNotNull(committed);
assertTrue(committed.acceptedTasks().isEmpty(), "缺 operationId 不得改当前任务");
assertNotEquals(TaskStatus.COMPLETE, galleries.selectGalleryByGid(123).getStatus());
assertEquals(0L, operation(123).getLastRevision());
}
/** 迁移前的历史任务行:首次上报按显式策略建立世代后正常接受。 */
@Test
void migratedLegacyTaskAdoptsGenerationOnFirstReport() {
galleries.insertGallery(gallery(321, "legacy [321]", TaskStatus.DOWNLOADING));
TaskReportHandler.CommittedReport committed = service.apply(
report("r-legacy", 3, Map.of(321, "dl-legacy"), task(321, TaskStatus.COMPRESSING, 9)));
assertNotNull(committed);
assertEquals(1, committed.acceptedTasks().size());
assertEquals("dl-legacy", operation(321).getOperationId(), "应采纳上报里的世代 ID");
assertEquals(TaskStatus.COMPRESSING, galleries.selectGalleryByGid(321).getStatus());
}
/**
* 旧节点不带 operationId 的两份连续上报都必须被接受:
* 第一份补建世代后,后续上报不能因为「缺 operationId」而被全部丢弃。
*/
@Test
void legacyReportWithoutOperationIdKeepsAcceptingLaterReports() {
galleries.insertGallery(gallery(322, "legacy [322]", TaskStatus.DOWNLOADING));
TaskReportHandler.CommittedReport first = service.apply(
report("r-legacy-1", 1, Map.of(), task(322, TaskStatus.DOWNLOADING, 3)));
assertNotNull(first);
assertEquals(1, first.acceptedTasks().size(), "旧节点首份无 operationId 上报必须接受");
assertNull(operation(322).getOperationId(),
"旧节点不带 operationId 时先保留 null,稳定 ID 只在首次下发时落库");
assertEquals(TaskStatus.DOWNLOADING, galleries.selectGalleryByGid(322).getStatus());
TaskReportHandler.CommittedReport second = service.apply(
report("r-legacy-2", 2, Map.of(), task(322, TaskStatus.COMPRESSING, 8)));
assertNotNull(second);
assertEquals(1, second.acceptedTasks().size(), "同一旧世代的后续上报不得被丢弃");
assertEquals(TaskStatus.COMPRESSING, galleries.selectGalleryByGid(322).getStatus());
assertEquals(2L, operation(322).getLastRevision());
}
/** 新 v2 任务(已知 operationId)缺归属的上报仍必须被拒,不能污染当前任务。 */
@Test
void knownOperationStillRejectsReportWithoutOperationId() {
service.createTask(323, "任务 [323]", "https://example.org/g/323/key/", "original", 7);
String operationId = operation(323).getOperationId();
service.apply(report("r-known-1", 1, Map.of(323, operationId), task(323, TaskStatus.DOWNLOADING, 4)));
TaskReportHandler.CommittedReport committed = service.apply(
report("r-known-2", 2, Map.of(), task(323, TaskStatus.COMPLETE, 10)));
assertNotNull(committed);
assertTrue(committed.acceptedTasks().isEmpty(), "缺归属不得改已知世代任务");
assertNotEquals(TaskStatus.COMPLETE, galleries.selectGalleryByGid(323).getStatus());
assertEquals(1L, operation(323).getLastRevision(), "被拒上报不得推进 revision");
}
/**
* 迁移前历史任务的首份上报已采纳世代 ID 时,后续同 ID 上报继续被接受(不走 null 兼容分支)。
*/
@Test
void legacyReportWithAdoptedIdStaysAccepted() {
galleries.insertGallery(gallery(324, "legacy [324]", TaskStatus.DOWNLOADING));
service.apply(report("r324-1", 1, Map.of(324, "dl-legacy-324"), task(324, TaskStatus.DOWNLOADING, 2)));
assertEquals("dl-legacy-324", operation(324).getOperationId());
TaskReportHandler.CommittedReport second = service.apply(
report("r324-2", 2, Map.of(324, "dl-legacy-324"), task(324, TaskStatus.COMPRESSING, 7)));
assertNotNull(second);
assertEquals(1, second.acceptedTasks().size(), "同 ID 的后续旧上报必须继续被接受");
assertEquals(TaskStatus.COMPRESSING, galleries.selectGalleryByGid(324).getStatus());
}
/** 数据库失败:apply 返回 null(不可回 ACK),节点会保留快照重投。 */
@Test
void databaseFailureOnReportReturnsNullSoNodeRetainsSnapshot() {
service.createTask(123, "任务 [123]", "https://example.org/g/123/key/", "original", 7);
String operationId = operation(123).getOperationId();
TaskReportMessage invalid = report("r-bad", 70, Map.of(123, operationId), task(123, TaskStatus.DOWNLOADING, 2));
// 用真实触发器让回执写入失败,模拟提交阶段的数据库故障。
try (Connection connection = new DriverManagerDataSource(dbUrl()).getConnection();
var statement = connection.createStatement()) {
statement.executeUpdate("create trigger reject_receipt before update on task_operation"
+ " when new.lastReportId='r-bad' begin select raise(ABORT,'simulated receipt failure'); end");
} catch (Exception e) {
fail("准备失败条件时出错: " + e.getMessage());
}
assertNull(service.apply(invalid), "数据库失败不得返回可回执结果");
assertEquals(0L, operation(123).getLastRevision(), "失败的回执不得留下半截 revision");
assertNotEquals(TaskStatus.DOWNLOADING, galleries.selectGalleryByGid(123).getStatus(),
"整个上报事务必须回滚,任务状态不得被改成下载中");
}
}
@@ -0,0 +1,52 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.*;
import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.transaction.support.TransactionTemplate;
import java.nio.file.Path;
import java.sql.Connection;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
class UserIdentityIntegrationTest {
@TempDir Path directory;
@Test
void renameAndBindingAreAtomicWhenBindingUpdateFails() throws Exception {
var source = new DriverManagerDataSource("jdbc:sqlite:" + directory.resolve("identity.db"));
try (Connection connection = source.getConnection(); var statement = connection.createStatement()) {
statement.executeUpdate("create table User(id integer primary key, AuthCode text, username text, lastAccessTime text, isEnable integer)");
statement.executeUpdate("create table sub_bind(key text primary key, user text unique, subscription_account_id integer, user_id integer)");
statement.executeUpdate("insert into User values(5,'example-code','alice',null,1)");
statement.executeUpdate("insert into sub_bind values('example-key','alice',1,5)");
statement.executeUpdate("create trigger reject_binding_rename before update on sub_bind when new.user='bob' begin select raise(ABORT,'simulated failure'); end");
}
var bean = new SqlSessionFactoryBean();
bean.setDataSource(source);
var factory = bean.getObject();
factory.getConfiguration().addMapper(UserMapper.class);
factory.getConfiguration().addMapper(SubMapper.class);
var template = new SqlSessionTemplate(factory);
var service = new UserService(template.getMapper(UserMapper.class), mock(GalleryMapper.class),
mock(CollectMapper.class), mock(TaskHandlerInterceptor.class));
service.subMapper = template.getMapper(SubMapper.class);
service.subscriptionState = new SubscriptionStateCoordinator();
service.transactions = new TransactionTemplate(new DataSourceTransactionManager(source));
String result = service.alterUsername("example-code", "bob");
assertTrue(result.contains("failure"));
assertEquals("alice", template.getMapper(UserMapper.class).selectUserById(5).getUsername());
try (Connection connection = source.getConnection(); var statement = connection.createStatement();
var row = statement.executeQuery("select user,user_id from sub_bind")) {
assertTrue(row.next());
assertEquals("alice", row.getString(1));
assertEquals(5, row.getInt(2));
}
}
}
@@ -214,14 +214,14 @@ class UserServiceTest {
assertTrue(json.contains("bob")); assertTrue(json.contains("bob"));
} }
/** 空表也要返回成功,且 data 是「数组序列化后的字符串」(历史契约,前端按字符串解析)。 */ /** 空表返回结构化空数组。 */
@Test @Test
void getAllUserHandlesEmptyTable() { void getAllUserHandlesEmptyTable() {
when(userMapper.selectAllUser()).thenReturn(new User[0]); when(userMapper.selectAllUser()).thenReturn(new User[0]);
String json = service.getAllUser(); String json = service.getAllUser();
assertTrue(ok(json)); assertTrue(ok(json));
assertTrue(json.contains("\"data\":\"[]\""), "实际输出: " + json); assertTrue(json.contains("\"data\":[]"), "实际输出: " + json);
} }
@Test @Test
@@ -239,4 +239,22 @@ class UserServiceTest {
when(userMapper.selectUserByAuthCode("ghost")).thenReturn(null); when(userMapper.selectUserByAuthCode("ghost")).thenReturn(null);
assertThrows(NullPointerException.class, () -> service.getUserId("ghost")); assertThrows(NullPointerException.class, () -> service.getUserId("ghost"));
} }
@Test
void enablingUserDoesNotTransferExistingDownloads() {
when(userMapper.isExist("code")).thenReturn(1);
when(userMapper.selectUserByAuthCode("code")).thenReturn(user(5, "code", "alice"));
assertTrue(ok(service.alterStatus("code", true)));
verify(userMapper).updateIsEnableById(5, true);
verify(galleryMapper, never()).updateGalleryDownloader(anyInt(), anyInt());
}
@Test
void renameKeepsSubscriptionBindingLinkedToUserId() {
service.subMapper = mock(com.lion.lionwebsite.Dao.normal.SubMapper.class);
when(userMapper.isExist("code")).thenReturn(1);
when(userMapper.selectUserByAuthCode("code")).thenReturn(user(5, "code", "alice"));
assertTrue(ok(service.alterUsername("code", "bob")));
verify(service.subMapper).renameBoundUser(5, "alice", "bob");
}
} }
@@ -0,0 +1,25 @@
package com.lion.lionwebsite.Util;
import org.junit.jupiter.api.Test;
import org.yaml.snakeyaml.Yaml;
import java.util.*;
import static org.junit.jupiter.api.Assertions.*;
class ClashSubscriptionParserTest {
@Test void inlineNodesAndReferencesAreFilteredRegardlessOfIndentation() {
String yaml = "proxies: [{name: 'Slow 1x', type: ss, server: a}, {name: 'Fast 3x', type: ss, server: b}]\n"
+ "proxy-groups: [{name: Choice, type: select, proxies: ['Fast 3x', 'Slow 1x', DIRECT]}]\n"
+ "rules: ['MATCH,Choice']\n";
Map<?,?> result = new Yaml().load(ClashSubscriptionParser.transform(yaml.lines().toList(), true, 2));
var proxies = (List<?>) result.get("proxies");
assertEquals(1, proxies.size());
assertEquals("Slow 1x", ((Map<?,?>) proxies.getFirst()).get("name"));
var group = (Map<?,?>) ((List<?>)result.get("proxy-groups")).getFirst();
assertEquals(List.of("Slow 1x", "DIRECT"), group.get("proxies"));
assertEquals(List.of("MATCH,Choice"), result.get("rules"));
}
@Test void malformedBodiesCannotReplaceLastGoodSubscription() {
assertThrows(IllegalArgumentException.class, () -> ClashSubscriptionParser.transform(List.of("error page"), false, 2));
assertThrows(IllegalArgumentException.class, () -> ClashSubscriptionParser.transform(List.of("proxies: [{type: ss}]"), false, 2));
}
}
@@ -75,7 +75,7 @@ class GalleryParsingTest {
/** 仅解析(不下载):应填好基本信息与可选分辨率,状态停在等待确认。 */ /** 仅解析(不下载):应填好基本信息与可选分辨率,状态停在等待确认。 */
@Test @Test
void parseCollectsMetadataWithoutSubmitting() throws Exception { void parseCollectsMetadataWithoutSubmitting() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
stubPages(ms); stubPages(ms);
Gallery g = GalleryUtil.parse(URL, false, ""); Gallery g = GalleryUtil.parse(URL, false, "");
@@ -98,7 +98,7 @@ class GalleryParsingTest {
/** 下载且分辨率可用:应设定分辨率与体积,并把提交结果记为「已提交」。 */ /** 下载且分辨率可用:应设定分辨率与体积,并把提交结果记为「已提交」。 */
@Test @Test
void parseSubmitsDownloadForAvailableResolution() throws Exception { void parseSubmitsDownloadForAvailableResolution() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
stubPages(ms); stubPages(ms);
Gallery g = GalleryUtil.parse(URL, true, "1280x"); Gallery g = GalleryUtil.parse(URL, true, "1280x");
@@ -112,7 +112,7 @@ class GalleryParsingTest {
/** 请求的分辨率不在可选列表里必须抛业务异常,而不是提交一个无效任务。 */ /** 请求的分辨率不在可选列表里必须抛业务异常,而不是提交一个无效任务。 */
@Test @Test
void parseThrowsWhenResolutionUnavailable() throws Exception { void parseThrowsWhenResolutionUnavailable() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
stubPages(ms); stubPages(ms);
var thrown = assertThrows(ResolutionNotMatchException.class, var thrown = assertThrows(ResolutionNotMatchException.class,
@@ -128,7 +128,7 @@ class GalleryParsingTest {
@Test @Test
void parseMarksFailureWhenSubmissionNotConfirmed() throws Exception { void parseMarksFailureWhenSubmissionNotConfirmed() throws Exception {
String failurePage = "<html><body><div id='db'><p>a</p><p><strong>Error occurred</strong></p></div></body></html>"; String failurePage = "<html><body><div id='db'><p>a</p><p><strong>Error occurred</strong></p></div></body></html>";
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenAnswer(inv -> { ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenAnswer(inv -> {
String u = inv.getArgument(0); String u = inv.getArgument(0);
if (u.contains("/g/")) return galleryPage(); if (u.contains("/g/")) return galleryPage();
@@ -147,7 +147,7 @@ class GalleryParsingTest {
/** 非法链接应直接返回 null,不产生任何网络请求。 */ /** 非法链接应直接返回 null,不产生任何网络请求。 */
@Test @Test
void parseReturnsNullForInvalidLink() throws Exception { void parseReturnsNullForInvalidLink() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.verifyLink(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.verifyLink(anyString())).thenCallRealMethod();
ms.when(() -> GalleryUtil.parse(anyString(), anyBoolean(), any())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parse(anyString(), anyBoolean(), any())).thenCallRealMethod();
@@ -164,7 +164,7 @@ class GalleryParsingTest {
+ "<tr><td><p>Original 30.2 MiB</p></td></tr>" + "<tr><td><p>Original 30.2 MiB</p></td></tr>"
+ "</tbody></table></div></div>" + "</tbody></table></div></div>"
+ "<form id='hathdl_form' action='https://exhentai.org/archiveresolve'></form></body></html>"; + "<form id='hathdl_form' action='https://exhentai.org/archiveresolve'></form></body></html>";
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenAnswer(inv -> { ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenAnswer(inv -> {
String u = inv.getArgument(0); String u = inv.getArgument(0);
if (u.contains("/g/")) return galleryPage(); if (u.contains("/g/")) return galleryPage();
@@ -206,7 +206,7 @@ class GalleryParsingTest {
*/ */
@Test @Test
void parseImageKeysAcceptsRealPageFormatWithTrailingSemicolon() throws Exception { void parseImageKeysAcceptsRealPageFormatWithTrailingSemicolon() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(mpvPage()); ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(mpvPage());
ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod();
@@ -225,7 +225,7 @@ class GalleryParsingTest {
@Test @Test
void parseImageKeysAcceptsVariantWithoutTrailingSemicolon() throws Exception { void parseImageKeysAcceptsVariantWithoutTrailingSemicolon() throws Exception {
String noSemicolon = mpvPage().replace("\"k\":\"key3\"}];", "\"k\":\"key3\"}]"); String noSemicolon = mpvPage().replace("\"k\":\"key3\"}];", "\"k\":\"key3\"}]");
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(noSemicolon); ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(noSemicolon);
ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod();
@@ -238,7 +238,7 @@ class GalleryParsingTest {
/** 页码必须从 1 开始逐页递增(前端按页码取图)。 */ /** 页码必须从 1 开始逐页递增(前端按页码取图)。 */
@Test @Test
void parseImageKeysNumbersPagesFromOne() throws Exception { void parseImageKeysNumbersPagesFromOne() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(mpvPage()); ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(mpvPage());
ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod();
@@ -252,7 +252,7 @@ class GalleryParsingTest {
/** 空响应表示画廊已下架/被删,应返回 null 交由上层提示。 */ /** 空响应表示画廊已下架/被删,应返回 null 交由上层提示。 */
@Test @Test
void parseImageKeysReturnsNullForEmptyResponse() throws Exception { void parseImageKeysReturnsNullForEmptyResponse() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(" "); ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(" ");
ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod();
@@ -263,7 +263,7 @@ class GalleryParsingTest {
/** 页面缺少预期 script 标签时返回 null,而不是抛索引越界。 */ /** 页面缺少预期 script 标签时返回 null,而不是抛索引越界。 */
@Test @Test
void parseImageKeysReturnsNullWhenScriptsMissing() throws Exception { void parseImageKeysReturnsNullWhenScriptsMissing() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())) ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any()))
.thenReturn("<html><body><script>var only = 1;</script></body></html>"); .thenReturn("<html><body><script>var only = 1;</script></body></html>");
ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod();
@@ -275,7 +275,7 @@ class GalleryParsingTest {
/** 解析成功后 mpvKey 应进入缓存,后续 getMpvKey 可直接命中。 */ /** 解析成功后 mpvKey 应进入缓存,后续 getMpvKey 可直接命中。 */
@Test @Test
void parseImageKeysCachesMpvKey() throws Exception { void parseImageKeysCachesMpvKey() throws Exception {
try (var ms = mockStatic(GalleryUtil.class)) { try (var ms = mockStatic(GalleryUtil.class, CALLS_REAL_METHODS)) {
ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(mpvPage()); ms.when(() -> GalleryUtil.requests(anyString(), anyString(), any(), any())).thenReturn(mpvPage());
ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.parseImageKeys(anyString())).thenCallRealMethod();
ms.when(() -> GalleryUtil.getMpvKey(anyString())).thenCallRealMethod(); ms.when(() -> GalleryUtil.getMpvKey(anyString())).thenCallRealMethod();
@@ -0,0 +1,58 @@
package com.lion.lionwebsite.Util;
import org.jsoup.Jsoup;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class GallerySearchParserRegressionTest {
@Test
void thumbnailModeUsesGalleryCards() throws Exception {
var document = Jsoup.parse("<div class=itg><div class=gl1t>"
+ "<a href='https://exhentai.org/g/12/key/'><img src='/cover.jpg'></a>"
+ "<div class=gl4t><a href='https://exhentai.org/g/12/key/'>"
+ "<div class=glink>Title [100 pages]</div></a></div>"
+ "<div class=cn>Manga</div><div class=gl5t>42 pages</div>"
+ "<div id=posted_12>2026-10-07 12:00</div></div></div>");
var galleries = GallerySearchParser.parse(document);
assertEquals(1, galleries.size());
assertEquals("Title [100 pages]", galleries.get(0).getName());
assertEquals(42, galleries.get(0).getPage());
assertEquals("Manga", galleries.get(0).getType());
assertEquals("/cover.jpg", galleries.get(0).getThumbnailUrl());
}
@Test
void extendedModeTakesTitleFromNameRatherThanCoverLink() throws Exception {
var document = Jsoup.parse("<table><tr><td class=gl1e><div>"
+ "<a href='https://exhentai.org/g/12/key/'><img src='/cover.jpg'></a>"
+ "</div></td><td class='gl2e glname'>"
+ "<a href='https://exhentai.org/g/12/key/'><div class=glink>Title</div></a>"
+ "<div class=cn>Manga</div><div>42 pages</div></td></tr></table>");
var galleries = GallerySearchParser.parse(document);
assertEquals(1, galleries.size());
assertEquals("Title", galleries.get(0).getName());
assertEquals(42, galleries.get(0).getPage());
}
@Test
void titleAndTagsCannotOverridePageCount() throws Exception {
var document = Jsoup.parse("<table><tr><td class=cn>Manga</td>"
+ "<td><img src='/c.jpg'></td><td class=glname>"
+ "<a href='https://exhentai.org/g/12/key/'>"
+ "<div class=glink>Title [100 pages]</div></a>"
+ "<div class=gt title='misc:200 pages'>200 pages</div></td>"
+ "<td class=gl4c><div>42 pages</div></td></tr></table>");
var galleries = GallerySearchParser.parse(document);
assertEquals(1, galleries.size());
assertEquals(42, galleries.get(0).getPage());
}
@Test
void pageCountRequiresMetadataEvenIfTitleContainsPages() {
var document = Jsoup.parse("<table><tr><td class=cn>Manga</td><td class=glname>"
+ "<a href='https://exhentai.org/g/12/key/'>"
+ "<div class=glink>Title [100 pages]</div></a></td></tr></table>");
assertThrows(java.io.IOException.class, () -> GallerySearchParser.parse(document));
}
}
@@ -0,0 +1,113 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.Gallery;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.*;
/** Exercise actual HTTP bytes against loopback, without contacting or queuing upstream. */
class GallerySubmissionHttpTest {
private HttpServer server;
private String endpoint;
private int responseStatus = 200;
private String responseBody = "<div id='db'><p>Queued</p><p><strong>#12345</strong></p></div>";
private final AtomicInteger requestCount = new AtomicInteger();
private final AtomicReference<Received> received = new AtomicReference<>();
record Received(String method, String origin, String referer, String contentType, String body) { }
@BeforeEach
void startServer() throws IOException {
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/submit", exchange -> {
requestCount.incrementAndGet();
var headers = exchange.getRequestHeaders();
received.set(new Received(exchange.getRequestMethod(), headers.getFirst("Origin"),
headers.getFirst("Referer"), headers.getFirst("Content-Type"),
new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8)));
byte[] bytes = responseBody.getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(responseStatus, bytes.length);
try (var output = exchange.getResponseBody()) { output.write(bytes); }
});
server.start();
endpoint = "http://127.0.0.1:" + server.getAddress().getPort() + "/submit";
}
@AfterEach
void stopServer() { server.stop(0); }
private PreparedGallery prepared(String resolution) {
Gallery gallery = new Gallery();
gallery.setGid(1234567);
var resolutions = new LinkedHashMap<String, String>();
resolutions.put(resolution, "12.5MiB");
gallery.setAvailableResolution(resolutions);
return new PreparedGallery(gallery, endpoint,
GalleryUtil.verifyLink("https://exhentai.org/g/1234567/0123456789/"));
}
@Test
void originalSubmissionMatchesBrowserFormAndHeaders() throws Exception {
Gallery gallery = GalleryUtil.submit(prepared("Original"), "Original");
assertEquals("已提交", gallery.getStatus());
assertEquals(1, requestCount.get());
Received request = received.get();
assertEquals("POST", request.method());
assertEquals("https://exhentai.org", request.origin());
assertEquals(endpoint, request.referer());
assertTrue(request.contentType().startsWith("application/x-www-form-urlencoded"));
assertEquals("hathdl_xres=org", request.body());
}
@Test
void resampledSubmissionSendsNumericWidth() throws Exception {
GalleryUtil.submit(prepared("1280x"), "1280x");
assertEquals("hathdl_xres=1280", received.get().body());
}
@ParameterizedTest
@ValueSource(ints = {403, 429, 503})
void rejectedSubmissionReportsStatusWithoutSecretsOrAutomaticReplay(int status) {
responseStatus = status;
responseBody = "private upstream response";
PreparedGallery initial = prepared("Original");
PreparedGallery withToken = new PreparedGallery(initial.gallery(),
endpoint + "?token=private-gallery-token", initial.origin());
IOException error = assertThrows(IOException.class,
() -> GalleryUtil.submit(withToken, "Original"));
assertTrue(error.getMessage().contains("HTTP " + status));
assertFalse(error.getMessage().contains("private-gallery-token"));
assertFalse(error.getMessage().contains(responseBody));
assertEquals(1, requestCount.get(), "A rejected download POST must never be replayed");
}
@Test
void jsonImageApiRetainsItsEncodingAndDoesNotMutateCaller() throws Exception {
HashMap<String, String> body = new HashMap<>();
body.put("payload", "json");
body.put("method", "imagedispatch");
GalleryUtil.requests(endpoint, "post", null, body);
assertTrue(received.get().contentType().startsWith("application/json"));
assertEquals("imagedispatch", CustomUtil.objectMapper.readTree(received.get().body()).get("method").asText());
assertFalse(received.get().body().contains("payload"));
assertEquals("json", body.get("payload"));
}
}
@@ -41,7 +41,15 @@ class GalleryUtilTest {
void verifyLinkAcceptsWellFormedGalleryUrl() { void verifyLinkAcceptsWellFormedGalleryUrl() {
String url = "https://exhentai.org/g/1234567/0123456789ab/"; String url = "https://exhentai.org/g/1234567/0123456789ab/";
assertNotNull(GalleryUtil.verifyLink(url)); assertNotNull(GalleryUtil.verifyLink(url));
assertEquals("", GalleryUtil.verifyLink(url), "通过校验时返回空串作为 origin"); assertEquals("https://exhentai.org", GalleryUtil.verifyLink(url), "返回真实 origin,不能发送空 Origin");
}
@Test
void verifyLinkRejectsLookalikeHostsAndMalformedUrls() {
assertNull(GalleryUtil.verifyLink("https://exhentai.org.example.com/g/1234567/0123456789ab/"));
assertNull(GalleryUtil.verifyLink("https://example.com/g/1234567/exhentai0123456789/"));
assertNull(GalleryUtil.verifyLink("https://user@exhentai.org/g/1234567/0123456789ab/"));
assertNull(GalleryUtil.verifyLink("https://exhentai.org/g/1234567/invalid key/"));
} }
// ---------- parseGid ---------- // ---------- parseGid ----------
@@ -0,0 +1,296 @@
package com.lion.lionwebsite.Util;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import static org.junit.jupiter.api.Assertions.*;
/**
* 订阅缓存分代存储的核心契约:两份文件作为一个版本发布,失败时保留上次完整版本。
*/
class SubscriptionCacheStoreTest {
private static byte[] bytes(String value) {
return value.getBytes(StandardCharsets.UTF_8);
}
/** 发布成功后,读取到的两份内容应来自同一分代。 */
@Test
void publishExposesBothFilesFromOneGeneration(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(1, bytes("v2-new"), bytes("clash-new"));
SubscriptionCacheStore.Pair pair = store.read(1).orElseThrow();
assertEquals("v2-new", new String(pair.v2(), StandardCharsets.UTF_8));
assertEquals("clash-new", new String(pair.clash(), StandardCharsets.UTF_8));
assertEquals(pair.generation(), store.read(1).orElseThrow().generation());
assertTrue(store.hasComplete(1));
assertEquals(store.resolve(1, "v2").orElseThrow().getParent(),
store.resolve(1, "cat").orElseThrow().getParent(), "两份文件必须同目录");
}
/** 第二份文件写入失败时,指针不得前进,上一次的完整版本必须原样可读。 */
@Test
void secondFileFailureKeepsPreviousPairIntact(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root) {
boolean firstPublished;
@Override
protected void writeFile(Path target, byte[] data) throws java.io.IOException {
if (firstPublished && target.getFileName().toString().equals("clash.yaml"))
throw new java.io.IOException("disk full");
super.writeFile(target, data);
}
@Override
protected void publishManifest(Path accountDir, String generation, String v2Sha256, String clashSha256)
throws java.io.IOException {
super.publishManifest(accountDir, generation, v2Sha256, clashSha256);
firstPublished = true;
}
};
store.publish(7, bytes("v2-old"), bytes("clash-old"));
assertThrows(java.io.IOException.class, () -> store.publish(7, bytes("v2-new"), bytes("clash-new")));
SubscriptionCacheStore.Pair pair = store.read(7).orElseThrow();
assertEquals("v2-old", new String(pair.v2(), StandardCharsets.UTF_8),
"第二份写入失败时不应出现新旧混用");
assertEquals("clash-old", new String(pair.clash(), StandardCharsets.UTF_8));
// 未发布的分代目录不应残留。
try (var generations = Files.list(root.resolve("7/generations"))) {
assertEquals(1, generations.count(), "失败的分代应被清理");
}
}
/** 指针发布失败时同样不得改变当前版本。 */
@Test
void manifestPublishFailureKeepsPreviousPairIntact(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(8, bytes("v2-old"), bytes("clash-old"));
// 用同一目录上的第二个实例覆写发布接缝,模拟指针写入失败。
SubscriptionCacheStore failing = new SubscriptionCacheStore(root) {
@Override
protected void publishManifest(Path accountDir, String generation, String v2Sha256, String clashSha256)
throws java.io.IOException {
throw new java.io.IOException("manifest write failed");
}
};
assertThrows(java.io.IOException.class, () -> failing.publish(8, bytes("v2-new"), bytes("clash-new")));
SubscriptionCacheStore.Pair pair = store.read(8).orElseThrow();
assertEquals("v2-old", new String(pair.v2(), StandardCharsets.UTF_8),
"指针发布失败时不应切换版本");
assertEquals("clash-old", new String(pair.clash(), StandardCharsets.UTF_8));
}
/** 旧结构:账号根目录下的两份文件都要在,才算可读的 legacy 版本。 */
@Test
void legacyRootIsReadOnlyWhenBothFilesExist(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
Path account = root.resolve("9");
Files.createDirectories(account);
Files.writeString(account.resolve("v2ray.txt"), "legacy-v2");
assertTrue(store.read(9).isEmpty(), "只有一份旧文件时不应视为可用");
assertFalse(store.hasComplete(9));
Files.writeString(account.resolve("clash.yaml"), "legacy-clash");
SubscriptionCacheStore.Pair pair = store.read(9).orElseThrow();
assertEquals("legacy-v2", new String(pair.v2(), StandardCharsets.UTF_8));
assertEquals("legacy-clash", new String(pair.clash(), StandardCharsets.UTF_8));
assertEquals("legacy", pair.generation());
assertEquals(account.resolve("v2ray.txt"), store.resolve(9, "v2").orElseThrow());
}
/** 半份旧缓存不得被解析为可分发路径,否则分发端会把残缺内容当订阅发出。 */
@Test
void incompleteLegacyIsNotResolvable(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
Path account = root.resolve("20");
Files.createDirectories(account);
Files.writeString(account.resolve("v2ray.txt"), "only-v2");
assertTrue(store.resolve(20, "v2").isEmpty(), "只有一份旧文件时不得解析出 v2 路径");
assertTrue(store.resolve(20, "cat").isEmpty(), "缺 clash 时不得解析出 cat 路径");
assertFalse(store.hasComplete(20));
}
/** 分代里缺一份文件时,两份都必须判定为不可用。 */
@Test
void generationMissingOneFileIsNotResolvable(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(21, bytes("v2"), bytes("clash"));
Files.delete(store.resolve(21, "cat").orElseThrow());
assertTrue(store.resolve(21, "v2").isEmpty());
assertTrue(store.resolve(21, "cat").isEmpty());
assertFalse(store.hasComplete(21));
assertTrue(store.read(21).isEmpty());
}
/** 内容与指针哈希不符时,resolve 与 hasComplete 必须一起拒绝。 */
@Test
void tamperedGenerationIsNotResolvable(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(22, bytes("v2"), bytes("clash"));
Files.writeString(store.resolve(22, "v2").orElseThrow(), "tampered");
assertTrue(store.resolve(22, "v2").isEmpty());
assertTrue(store.resolve(22, "cat").isEmpty(), "同一版本里另一份也应一并拒绝");
assertFalse(store.hasComplete(22));
}
/** 指针被写坏(缺哈希字段)时不得抛异常,也不得回退到半份旧缓存。 */
@Test
void corruptedManifestNeverFallsBackToLegacy(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
Path account = root.resolve("23");
Files.createDirectories(account);
// 故意只放一份旧文件,模拟「指针损坏 + 半份 legacy」的最坏组合。
Files.writeString(account.resolve("v2ray.txt"), "legacy-only-v2");
Files.writeString(account.resolve("current"), "generation=g1\n");
assertDoesNotThrow(() -> store.read(23));
assertTrue(store.read(23).isEmpty(), "坏指针不应回退半份 legacy");
assertTrue(store.resolve(23, "v2").isEmpty());
assertFalse(store.hasComplete(23));
}
/** 指针指向的分代不存在时同样不得回退,且不应抛异常。 */
@Test
void manifestPointingAtMissingGenerationIsNotResolvable(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
Path account = root.resolve("24");
Files.createDirectories(account);
Files.writeString(account.resolve("v2ray.txt"), "legacy-v2");
Files.writeString(account.resolve("clash.yaml"), "legacy-clash");
Files.writeString(account.resolve("current"),
"generation=0000000000000-000001\nv2-sha256=deadbeef\nclash-sha256=deadbeef\n");
assertTrue(store.read(24).isEmpty(), "指针存在时不得回退 legacy");
assertTrue(store.resolve(24, "v2").isEmpty());
assertFalse(store.hasComplete(24));
}
/**
* 每个 API 调用只读一次指针:整份版本由同一次快照校验,不会出现
* 「用新分代名配旧哈希」的撕裂窗口。
*/
@Test
void everyLookupReadsTheManifestExactlyOnce(@TempDir Path root) throws Exception {
var reads = new int[1];
SubscriptionCacheStore counted = new SubscriptionCacheStore(root) {
@Override
Properties readManifest(Path account) {
reads[0]++;
return super.readManifest(account);
}
};
counted.publish(25, bytes("v2"), bytes("clash"));
reads[0] = 0;
counted.read(25);
assertEquals(1, reads[0], "read 应只读一次指针");
reads[0] = 0;
counted.resolve(25, "v2");
assertEquals(1, reads[0], "resolve 应只读一次指针");
reads[0] = 0;
counted.hasComplete(25);
assertEquals(1, reads[0], "hasComplete 应只读一次指针");
}
/** 哨兵路径必须保证不存在,且与真实的 legacy 路径不同。 */
@Test
void unavailablePathIsNeverAnExistingFile(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
Path account = root.resolve("26");
Files.createDirectories(account);
Files.writeString(account.resolve("v2ray.txt"), "legacy-v2");
Path sentinel = store.unavailablePath(26, "v2");
assertFalse(Files.exists(sentinel));
assertNotEquals(store.legacyPath(26, "v2"), sentinel);
assertTrue(store.resolve(26, "v2").isEmpty(), "半缓存不得解析成真实路径");
}
/** 新分代发布后取代旧结构,读取应切到分代内容。 */
@Test
void publishedGenerationSupersedesLegacyFiles(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
Path account = root.resolve("10");
Files.createDirectories(account);
Files.writeString(account.resolve("v2ray.txt"), "legacy-v2");
Files.writeString(account.resolve("clash.yaml"), "legacy-clash");
store.publish(10, bytes("new-v2"), bytes("new-clash"));
SubscriptionCacheStore.Pair pair = store.read(10).orElseThrow();
assertEquals("new-v2", new String(pair.v2(), StandardCharsets.UTF_8));
assertNotEquals("legacy", pair.generation());
assertFalse(Files.exists(account.resolve("v2ray.txt")), "发布新分代后旧结构文件应被移除");
}
/** 失效后不应再读到任何版本,并允许后续重新发布。 */
@Test
void invalidateRemovesAllVersionsAndAllowsRepublish(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(11, bytes("v2"), bytes("clash"));
store.invalidate(11);
assertTrue(store.read(11).isEmpty());
assertFalse(store.hasComplete(11));
assertTrue(store.resolve(11, "v2").isEmpty());
store.publish(11, bytes("v2-again"), bytes("clash-again"));
assertEquals("v2-again", new String(store.read(11).orElseThrow().v2(), StandardCharsets.UTF_8));
}
/** 分代目录只保留有限数量,避免长期运行撑爆磁盘。 */
@Test
void oldGenerationsArePruned(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
for (int i = 0; i < 6; i++)
store.publish(12, bytes("v2-" + i), bytes("clash-" + i));
try (var generations = Files.list(root.resolve("12/generations"))) {
List<Path> retained = generations.toList();
assertTrue(retained.size() <= 2, "分代数应被裁剪,实际 " + retained.size());
assertTrue(retained.contains(store.resolve(12, "v2").orElseThrow().getParent()),
"当前分代必须保留");
}
assertEquals("v2-5", new String(store.read(12).orElseThrow().v2(), StandardCharsets.UTF_8));
}
/** 指针缺失但分代目录存在时不应误读为当前版本。 */
@Test
void missingManifestIsNotTreatedAsPublished(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(13, bytes("v2"), bytes("clash"));
Files.delete(root.resolve("13/current"));
assertTrue(store.read(13).isEmpty(), "没有指针就没有已发布版本");
assertFalse(store.hasComplete(13));
}
/** 内容被篡改(与指针记录的哈希不符)时应拒绝读取,避免把坏内容推给备机。 */
@Test
void tamperedGenerationContentIsRejected(@TempDir Path root) throws Exception {
SubscriptionCacheStore store = new SubscriptionCacheStore(root);
store.publish(14, bytes("v2"), bytes("clash"));
Path published = store.resolve(14, "v2").orElseThrow();
Files.writeString(published, "tampered");
Optional<SubscriptionCacheStore.Pair> pair = store.read(14);
assertTrue(pair.isEmpty(), "校验失败的分代不应被读取");
}
}
@@ -0,0 +1,154 @@
package com.lion.lionwebsite.Util;
import org.jsoup.Jsoup;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import static org.junit.jupiter.api.Assertions.*;
class UpstreamParserVariantsTest {
@Test void metadataUsesLabelsInsteadOfRowPositions() throws Exception {
var document = Jsoup.parse("<h1 id=gj>Alternate title</h1><div id=gdd><table>"
+ "<tr><td>Length:</td><td>12 pages</td></tr><tr><td>Extra:</td><td>ignored</td></tr>"
+ "<tr><td>File Size:</td><td>2 MiB</td></tr><tr><td>Language:</td><td>Chinese</td></tr>"
+ "</table></div><div id=gd1><img src='/cover.jpg'></div>");
var gallery = GalleryPageParser.metadata(document, "https://exhentai.org/g/12/key/", 12);
assertEquals(12, gallery.getPages());
assertEquals("Chinese", gallery.getLanguage());
assertEquals("Alternate title [12]", gallery.getName());
assertEquals("/cover.jpg", gallery.getThumb_link());
assertThrows(IOException.class, () -> GalleryPageParser.metadata(Jsoup.parse("<h1 id=gn>Error</h1>"), "x", 1));
}
@Test void archiveEntrySupportsRelativeHrefAndMovedAnchor() throws Exception {
var document = Jsoup.parse("<div id=gd5><a href='/unrelated'>Other</a><p><a href='/archive/12'>Archive Download</a></p></div>", "https://exhentai.org/g/12/key/");
assertEquals("https://exhentai.org/archive/12", GalleryPageParser.archiveLink(document));
assertThrows(IOException.class, () -> GalleryPageParser.archiveLink(Jsoup.parse("<html/>")));
}
@Test void arrayAssignmentHandlesMultilineAndClosingBracketsInsideStrings() throws Exception {
String script = "let unrelated=[];\nconst imagelist = [\n{\"k\":\"one]two\"},\n{\"k\":\"three\"}\n]; var after=1;";
var array = CustomUtil.objectMapper.readTree(JavaScriptAssignments.array(script, "imagelist"));
assertEquals(2, array.size());
assertEquals("one]two", array.get(0).get("k").asText());
assertEquals("value", JavaScriptAssignments.string("let mpvkey = 'value';", "mpvkey"));
assertThrows(IOException.class, () -> JavaScriptAssignments.array("imagelist = [{\"k\":\"x\"}", "imagelist"));
}
/** 线上真实结构:每个单元格拆成三个 <p>(分辨率/体积/费用)。旧实现逐段匹配导致全部漏掉。 */
@Test void resolutionsReadSplitParagraphCellsFromLiveLayout() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody><tr>"
+ "<td><p>Original</p><p>36.31 MiB</p><p>Free</p></td>"
+ "<td><p>800x</p><p>5.65 MiB</p><p>Free</p></td>"
+ "<td><p>1280x</p><p>12.5 MiB</p><p>Free</p></td>"
+ "<td><p>1920x</p><p>30.2 MiB</p><p>Free</p></td>"
+ "<td><p>2560x</p><p>N/A</p><p>N/A</p></td>"
+ "</tr></tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(List.of("800x", "1280x", "1920x", "Original"), new ArrayList<>(resolutions.keySet()));
assertEquals("5.65MiB", resolutions.get("800x"));
assertEquals("12.5MiB", resolutions.get("1280x"));
assertEquals("30.2MiB", resolutions.get("1920x"));
assertEquals("36.31MiB", resolutions.get("Original"), "Original 应排在最后");
assertFalse(resolutions.containsKey("2560x"), "N/A 的体积不得进结果");
}
/** 历史/单段布局:分辨率与体积在同一段 <p> 也必须继续兼容。 */
@Test void resolutionsKeepMergedParagraphLayoutCompatible() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody>"
+ "<tr><td><p>1280x 12.5 MiB</p></td></tr>"
+ "<tr><td><p>Original 30.2 MiB</p></td></tr>"
+ "</tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(List.of("1280x", "Original"), new ArrayList<>(resolutions.keySet()));
assertEquals("12.5MiB", resolutions.get("1280x"));
assertEquals("30.2MiB", resolutions.get("Original"));
}
/** N/A 与体积分处两格时不得跨 td 拼接。 */
@Test void resolutionsNeverStitchAcrossCells() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody><tr>"
+ "<td><p>2560x</p><p>N/A</p><p>N/A</p></td>"
+ "<td><p>1920x</p><p>30.2 MiB</p></td>"
+ "<td><p>800x</p><p>N/A</p></td>"
+ "</tr></tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(Set.of("1920x"), resolutions.keySet());
assertEquals("30.2MiB", resolutions.get("1920x"));
assertTrue(resolutions.values().stream().noneMatch(size -> size.toUpperCase().contains("N/A")));
}
/** 单元格开头不是“分辨率标签 + 体积”时不得凭空造出条目。 */
@Test void resolutionsRequireResolutionLabelWithinSameCell() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody><tr>"
+ "<td><p>800x</p><p>5.65 MiB</p></td>"
+ "<td><p>12.5 MiB</p></td>"
+ "<td><p>1920x</p><p>Costs 50 GP</p></td>"
+ "</tr></tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(List.of("800x"), new ArrayList<>(resolutions.keySet()));
assertEquals("5.65MiB", resolutions.get("800x"));
}
/** 嵌套标签(strong/span/div)与 &nbsp; 由 text() 归并后仍应正确取到分辨率与体积。 */
@Test void resolutionsReadNestedTagsInsideCells() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody><tr>"
+ "<td><div><p><strong>1280x</strong></p><div><p><span>12.5</span> <em>MiB</em></p></div></div></td>"
+ "<td><p>Original</p><p>30.2&nbsp;MiB</p></td>"
+ "</tr></tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(List.of("1280x", "Original"), new ArrayList<>(resolutions.keySet()));
assertEquals("12.5MiB", resolutions.get("1280x"));
assertEquals("30.2MiB", resolutions.get("Original"), "&nbsp; 分隔的体积也要认");
}
/** 未知单位、坏数字和费用文本都不得当作体积,只保留可解析的项。 */
@Test void resolutionsRejectUnknownUnitAndCostText() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody><tr>"
+ "<td><p>1280x</p><p>12.5 ZiB</p></td>"
+ "<td><p>1920x</p><p>Costs 50 GP</p></td>"
+ "<td><p>2560x</p><p>.5 MiB</p></td>"
+ "<td><p>800x</p><p>5.65 MiB</p></td>"
+ "</tr></tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(List.of("800x"), new ArrayList<>(resolutions.keySet()));
assertEquals("5.65MiB", resolutions.get("800x"));
}
/** 完全取不到可用分辨率(空文档或只有 N/A)必须抛 IOException,而不是返回空表。 */
@Test void resolutionsThrowWhenNothingUsable() throws Exception {
assertThrows(IOException.class, () -> GalleryPageParser.resolutions(Jsoup.parse("<html/>")));
assertThrows(IOException.class, () -> GalleryPageParser.resolutions(Jsoup.parse(
"<div id=db><table><tbody><tr><td><p>2560x</p><p>N/A</p><p>N/A</p></td>"
+ "<td><p>N/A</p></td></tr></tbody></table></div>")));
}
/** 宽度为 0 或溢出、以及单位后粘连字符的项都要跳过,不能崩到调用方。 */
@Test void resolutionsSkipInvalidWidthAndGluedUnit() throws Exception {
var document = Jsoup.parse("<div id=db><table><tbody><tr>"
+ "<td><p>0x</p><p>5.65 MiB</p><p>Free</p></td>"
+ "<td><p>99999999999x</p><p>12.5 MiB</p><p>Free</p></td>"
+ "<td><p>800x</p><p>5.65 MiB2</p><p>Free</p></td>"
+ "<td><p>1280x</p><p>12.5 MiB</p><p>Free</p></td>"
+ "</tr></tbody></table></div>");
var resolutions = GalleryPageParser.resolutions(document);
assertEquals(List.of("1280x"), new ArrayList<>(resolutions.keySet()));
assertEquals("12.5MiB", resolutions.get("1280x"));
}
}