Author SHA1 Message Date
root b9358d9817 Fix upstream archive submissions to match browser form encoding 2026-10-22 22:27:59 +08:00
root a72d99ff0a Upgrade Tomcat and Jackson to patched dependency versions 2026-10-22 00:10:26 +08:00
root 788fd1e7e2 Reserve SQLite writer before reading subscription event receipts 2026-10-21 10:11:57 +08:00
root d6eb25a0f6 Record standby subscription events with signed durable receipts 2026-10-21 09:57:46 +08:00
root 9faae09851 fix: reconcile v2 retries and parse all gallery search layouts 2026-10-07 23:50:52 +08:00
root 4c1a2d22e2 Fix gallery resolution parsing for split archive paragraphs 2026-10-07 23:09:47 +08:00
root e6c7dab12e Rewrite durable task lifecycle, gallery indexing and subscription publication 2026-10-07 17:46:32 +08:00
root 776eb8cf53 修复 GraalVM 25 原生构建中的 commons-logging 初始化 2026-09-26 12:45:30 +08:00
root e03a2e007b netty 按需声明,并记录 jar 运行期的低内存调优
netty-all 是聚合 pom,会拖进 5 个平台的 native-quic、各架构的 epoll/kqueue/
io_uring 传输以及 htt3/mqtt/redis 等一堆用不到的 codec。主站实际只用
Bootstrap/NioEventLoopGroup/NioSocketChannel/ByteBuf/ByteToMessageCodec/
LengthFieldBasedFrameDecoder/LoggingHandler/Promise,因此收敛为
transport + codec-base + handler(含 buffer/common/resolver 传递依赖)。

效果:jar 68.1MB -> 52.2MB,netty 模块 50 -> 7,进程 RSS 291MB -> 约 227MB。
433 个测试全绿。hutool 未动。

同时把 jar 运行期的 systemd JVM 参数与前后实测数据记入
docs/MEMORY_TUNING_2026-09-21.md,避免这些只存在于服务器 drop-in 的参数
在下次重新部署时丢失。
2026-09-21 17:27:17 +08:00
root 5f1066cdab 去掉订阅快照的周期性重发
节点侧已改为按「收到主站的任何消息」判定新鲜度(过期 ⇔ 主站失联),
主站每 30 分钟的可用性检查即可持续刷新,因此主站不需要任何为续期的周期性推送。

- 删除上一版加入的 scheduledSubscriptionKeepalive(6 小时一次)。
- 保留「内容变化即推」(各处 requestSubscriptionSync)与「节点上线即推」
  (initChannel),两者覆盖变更下发与节点冷启动两种场景。
- 移除对应的 standby 保活配置项。

最终推送时机:只在订阅内容变化时推送,另在节点上线时补推一次;
空闲时零快照传输(此前 60 秒一次时为约 486 MiB/天)。

测试:433 项全过。新增 SubscriptionSnapshotPushTest 锁住两条约定——
RemoteService 不得再有定时任务(续期改由节点存活信号承担),以及订阅刷新成功后
仍必须触发一次推送(用真实 refreshDueAccounts 调用验证)。
2026-09-21 16:49:55 +08:00
root e3d4220997 订阅快照改为变更即推,并加低频保活
原先主站每 60 秒无条件重建并推送整份快照:实测 12 账号时传输约 346 KiB/次、
约 486 MiB/天,节点每次都要 base64 解码、SHA-256、gunzip、JSON 校验后判
APPLY_OLD 丢弃。而 revision 是内容寻址的,订阅不常变动时这些推送纯属浪费。

改为:
- 内容变化即推:各处已有的 requestSubscriptionSync()(刷新成功、绑定/改绑、
  重置 Key、增删停用账号、过滤开关变更)保持原样,延迟仍在数秒内。
- 低频保活:新增 scheduledSubscriptionKeepalive(),默认 6 小时一次,仅用于刷新
  节点「最近收到快照」的时刻。节点的过期判定依据该时刻(见 storageNode 侧修复),
  因此保活周期必须远小于节点 SubscriptionMaxStaleSeconds(默认 7 天)。
- 节点上线即推:initChannel 保持推送一次,覆盖节点刚重启、本地尚无快照的情况。

空闲流量约 486 MiB/天 -> 约 1.4 MiB/天(约三个数量级)。

测试:432 项全过。新增 SubscriptionKeepaliveTest 锁定不变量——保活入口必须存在,
且周期小于节点有效期的 1/4、初始延迟小于一个周期;否则节点可能在两次保活之间判过期。
2026-09-21 16:36:13 +08:00
root 42921ba97b 修复多处隐患并做低风险性能优化
缺陷修复:
- GalleryUtil.parseGid 只捕 IndexOutOfBoundsException:非数字段抛 NumberFormatException、
  link 为 null 抛 NPE,都会穿透成 500(按链接查询直接把用户输入喂进来)。现统一返回 null,
  由调用方转成业务失败。
- 图片索引缓存不是整体生效:先写 gidToKey 再逐页写 ImageKeyCache,中途失败会留下
  「gidToKey 命中但页 key 缺失」的半截缓存,后续请求直接返回已缓存而永远取不到图。
  现任何异常都回滚已写入部分,并识别历史半截缓存自动重建。
- 全站请求日志把 AuthCode 明文记入 INFO,改为只记 present/absent。
- selectEnableAuthCode 的 SQL 与全量查询完全相同(都无 isEnable 条件),停用用户的
  授权码刷新后仍放行,isEnable 形同虚设;补齐条件并允许 null 按默认 true 处理。
- /validate 手工 String.format 拼 JSON,用户名含引号或反斜杠会产出非法 JSON,
  前端 JSON.parse 失败;改用 ObjectMapper 组装。

性能与并发:
- Netty IO 线程不再直接做 JDBC:节点上报的任务状态改由单线程顺序执行器落库与推送,
  既不打乱「按上报顺序覆盖状态」的语义,也不阻塞心跳与响应。
- 取订阅不再等待第三方归属地查询:先以 unknown 落库并立即分发,归属地由后台线程补齐。
- 搜索关键词与分页参数按 UTF-8 编码,&、#、中文不再破坏上游查询串。
- selectAllGallery 的收藏标记由 O(n*m) 嵌套循环改为集合查找。
- 复用 CustomUtil.objectMapper(8 处 new ObjectMapper);gid2MpvKey 改为有界 LRU(2048);
  缩略图清理排序不再每次比较都重读文件属性;FileDownload 关闭响应流,避免大文件下载的
  临时文件堆积到 Full GC;移除从未使用的 downloadThread 线程池。

测试:431 项全过(新增索引回滚、半截缓存重建、用户名特殊字符、畸形链接查询、
搜索编码、分发不等待定位、批量收藏标记等用例;异步状态处理用例加 awaitStatusApplied
屏障保持断言确定性)。
2026-09-21 16:17:17 +08:00
root 584db277a6 登录方案文档更新为全部完成并记录独立复验结论 2026-09-21 11:25:49 +08:00
root 628f3ad0a2 登录方案:确认复用 /login 并一次返回两条链接,密钥已入库 2026-09-21 10:33:00 +08:00
root dfb647bcd3 登录方案文档更新为已实施状态,补机器人侧票据规格 2026-09-21 10:23:26 +08:00
root 00e27152fe 新增一键登录:共享密钥自签票据换会话
- 新增 PanelLoginTicket:v1.<时间戳>.<HMAC-SHA256> 形状,只验签与时间窗,
  不发一次性状态。机器人用同一把密钥本地签发,因此不需要机器人到主站的网络调用,
  一端临时不可达也不影响生成链接。窗口内可重放(用户明确接受),过期即失效。
- PersonalController 新增 GET /login?t= 换 HttpSession(先作废旧会话防固定攻击)、
  /login/logout 退出、/personal/denied 提示页。
- PersonalInterceptor 放行「有效会话或 AuthCode=alone」,拒绝时回 401 供前端区分;
  InterceptorConfiguration 排除登录端点与 /personal/。
- server.servlet.session.cookie.path 固定为 /:nginx 会把 /user 改写成 /personal/user,
  沿用容器推导的 /personal 会让浏览器判定路径不匹配而丢会话。
- 密钥经 PERSONAL_LOGIN_SECRET 环境变量注入,为空时一律拒绝而非放行。
- 新增 PanelLoginTicketTest(7 项)与登录/退出/会话相关用例,共 423 项通过。
2026-09-21 10:20:51 +08:00
root 654181d2ea 登录方案对齐 PersonalHub 既有实现 2026-09-21 10:03:35 +08:00
root bee2f11fd6 移除文件管理与文件分享,接口范围收敛到订阅与用户管理
- 删除 /personal/files、uploadFile、private/**、compress、delete 与对应服务实现,
  连同压缩线程池、TAR 打包与 PersonalArchiveTest
- 删除 /personal/share、extendShareTime、cancelShare 与 /GetFile/{path},
  以及 ShareFile 实体、ShareFileMapper、PublicService.GetFile、每天四点的
  checkShareCode 定时任务和分享码失败黑名单
- CustomBean 去掉 ShareFile 的 Native 反射登记
- 同步收敛 PersonalServiceTest、PublicServiceTest、PersonalControllerTest、
  PublicControllerTest、LocalServiceTest 与三个画廊测试
- 文档同步,并附上 /login 方案 docs/login-command-plan.md

测试:mvn -o test 410 项通过
2026-09-21 10:02:07 +08:00
root 305036ddfe 新增下载人查询接口,并把管理员判定收敛到服务端
- Gallery 增加 downloaderName,仅在管理员查询任务列表时填充;普通用户
  既不获取也不下发他人身份
- 新增 GET /GalleryManage/downloader?gid=&AuthCode=,按 gid 返回实际下载人
  昵称,非管理员一律拒绝
- /validate 下发 isAdmin,前端不再各自硬编码 userId === 3
- 管理员判定统一走 UserService.ADMIN_USER_ID
- 补充列表填充、按 gid 查询的权限与缺任务分支测试
2026-09-20 22:54:23 +08:00
root d949415e4d 分散刷新后补上备机同步
原实现中「全量刷新 → 通知备机同步」是同一步的两个动作:refreshAll()
由调用方跟一次 requestSubscriptionSync()。改为按账号分散刷新后,调度器
自己没有触发同步,刷新出的新缓存要等到下一次周期同步才可能送到备机。

生产上每 60 秒有一次周期同步兜底,所以影响被限制在最长约一分钟的滞后;
但这属于重构时漏掉的语义,且一旦把周期同步间隔调大就会变成真实缺口,
因此仍按原语义在刷新到内容后立即触发同步。没有账号到期时不打扰备机。

测试 441 通过。
2026-09-15 16:27:29 +08:00
root 3adb42c538 订阅刷新改为每账号错峰,保证 24 小时内各刷一次
原先每 24 小时一次性刷新全部子账号:12 个账号在 16 秒内背靠背发完
24 个请求,且 @Scheduled(fixedRate) 未设 initialDelay,首次触发即立刻
执行,导致每次重启都重刷一遍全部账号——上游看到的请求密度实际取决于
部署频率。请求头也只带 HttpClient 默认值,上游收到的是
"Apache-HttpClient/5.6.4 (Java/25.0.4)"。

改为高频轻量 tick(5 分钟,延迟 2 分钟启动),只刷新已到期账号,单轮
最多 2 个。24 小时保证由三层构成:成功后按「成功时刻 + 窗口 − 一个 tick」
推进相位,使分散效果自我维持;到期时刻持久化在数据库,停机期间错过的
账号恢复后立即到期;兜底把被写到超过一个窗口之后的计划判为损坏并立即
刷新。失败用固定间隔重试(数据库没有连续失败次数字段,做不出真正的指数
退避),重试排在将来时被尊重,避免故障账号每个 tick 重打上游。从未成功过
的账号按 tick 间隔尽快刷,已有缓存的按窗口分散。超过两倍窗口仍未成功的
账号推送告警。

请求头按账号与订阅格式确定性地伪装成 mihomo/clash-verge/v2rayN 等真实
客户端身份;同一账号固定用同一身份,逐请求更换反而更像机器。实现前已
实测确认显式设置 Accept-Encoding 不会破坏 HttpClient5 的透明解压。注意
这只能伪装到 HTTP 头,TLS 指纹仍与真实客户端不同。

新增 next_refresh_at、last_success_epoch 两列,均为 Epoch 毫秒整数:SQLite
的 CURRENT_TIMESTAMP 写 UTC 文本而被 JDBC 按本地时区解释,实测偏差 8 小时,
不能用于时间比较。迁移为纯新增列,旧二进制忽略它们,故回滚程序无需回滚
数据库;回填 last_success_epoch 必须执行,否则既有账号会被当成从未成功过
而集中补刷,正好复现本次要消除的爆发。

同时移除 LocalService 的 24 小时全量入口与 refreshAll()(它们正是爆发式
写法),并将调度线程池由默认 1 调到 2,避免刷新阻塞连接自检。

测试 439 通过,指令覆盖率 82.1%。
2026-09-15 16:15:41 +08:00
root a0959618ca 修复 Jackson 3 迁移引入的看图 500,并大幅补充单元测试
生产缺陷(2026-09-14 Spring Boot 4.1.1 / Jackson 2→3 迁移引入):
- GalleryUtil.parseImageKeys 解析 mpv 页 imagelist 时,行尾分号属 JS 语法,
  Jackson 3 默认 FAIL_ON_TRAILING_TOKENS=true 会抛 StreamReadException。
  改为先由 parseImagelist 剥掉 JS 语句外壳(前缀 + 行尾分号)再交给 JSON 解析。
  已用线上真实页面验证:修复前必抛异常,修复后正确解析出 154 页 imgkey。
- GalleryManageService.cacheImagesKey 只 catch IOException,而 Jackson 3 的
  JacksonException 继承 RuntimeException,异常穿透 Controller(无 @ControllerAdvice)
  表现为 HTTP 500。改为同时兜 JacksonException 并按业务失败返回。
  同时用 parseGidFromLink 拦住段数不足的畸形链接(原先会抛
  ArrayIndexOutOfBoundsException 变成 500)。

既有行为修正:
- PersonalService.deleteFile 原先依据 hutool FileUtil.del 的返回值报成功,
  而该函数对不存在的目标返回 true,导致路径写错也显示「删除成功」;
  改为先校验存在性,不存在则回「文件不存在」。
- PersonalService.getFiles 的 listFiles()==null 分支原先报「文件夹为空」,
  但该返回值只在 I/O 出错时出现,会掩盖权限/磁盘问题,改为「读取文件夹失败」。

测试:143 → 379 用例,指令覆盖率 36.2% → 80.2%(分支 34.6% → 69.8%)。
新增控制器(MockMvc standaloneSetup)、服务层、拦截器/过滤器、WebSocket、
配置装配与 MyBatis AOT hints 的测试;全部离线可跑,不加载 Spring 上下文、不连库、不占端口。
2026-09-15 13:42:22 +08:00
root a288846552 收紧 GalleryManage 子路径鉴权并更新项目说明 2026-09-14 19:17:23 +08:00
root 850819ed86 新增 .gitignore:忽略构建产物、本地日志与 IDE 文件
规则来自 storageNode 现有模板(39 行,含 maven-wrapper.jar 的 ! 例外),
并追加 run.out / *.log(后端与 storageNode 本地都会产生运行时日志)。
application.yaml 与 .mvn/wrapper 均被跟踪,不受影响。
2026-09-14 15:38:48 +08:00
root 576092578d 修复越权删除、畸形链接 500 及两处 NPE
修复上一提交的测试所发现的缺陷。

1) 越权删除(安全,最严重):deleteGalleryByGid 的授权条件是
   `!(collector.isEmpty() || collector.size()==1 && ...)`。当画廊无任何收藏时
   collector.isEmpty() 使整个条件短路放行,下载者身份完全未校验,
   任意有效授权码用户都能删除他人任务记录(无收藏正是最常见情形)。
   改为与提示文案一致的判定:被别人收藏 或 请求者不是下载人 即拒绝。

2) 被拒请求仍删除节点文件且对外显示成功:remoteService.deleteGallery 原先位于
   授权判断之外,即使拒绝也会向节点下发删除指令;且 switch 中 case 0 的
   response.success() 会覆盖先前的 failure。现改为授权失败即提前返回。

3) 畸形链接导致 500:link.split("/")[4] 段数不足抛 ArrayIndexOutOfBoundsException,
   而只捕获 NumberFormatException;项目无 @ControllerAdvice,异常穿透为 500。
   抽出 parseGidFromLink() 统一把 null/段数不足/非数字转成业务失败。

4) Response.isSuccess()/getData()/get() 在键缺失时抛 NPE,改为安全返回
   (isSuccess 视未设置为失败)。deleteGalleryByGid 也显式处理节点无响应(-1),
   原先该分支不设置 result,末尾 response.get("result") 会 NPE。

5) getWeekUsedAmount() 在配置行缺失/值非法时给默认 0,不再 NPE。

验证:新增/改写回归用例后 143 项测试全过;并用真实数据做了新旧对照实测——
取 downloader=4 且无收藏的任务,以 id=25 用户的授权码删除:
  旧 jar:数据库记录被删(count 0),随后 500;
  新 jar:记录保留(count 1),返回「你不是下载人」并被拒。
覆盖率的 GalleryManageService 由 40.1% 升至 45.8%。
2026-09-14 15:23:42 +08:00
root 213ed4e7f6 补充测试用例并接入 JaCoCo 覆盖率统计
原 8 个测试类、16 个用例,指令覆盖率仅 16.9%,协议编解码、
鉴权拦截器、UA 分流过滤器、子账号业务规则等高风险代码完全未覆盖。

接入 JaCoCo(pom 增加 jacoco-maven-plugin 0.8.13,test 阶段出报告)。
新增 9 个测试类、118 个用例,测试总数 16 -> 134,全部通过:

- Message.MessageCodecTest(12):8 种消息的帧格式与往返;
  锁死帧头 [类型 1B][长度 4B][JSON]、@JsonIgnore 的 path 不外泄、
  多字节 UTF-8 按字节计长、未知类型静默丢弃、编解码器间不共享状态。
- Interceptor.TaskHandlerInterceptorTest(11):授权码放行/拒绝两侧,
  含空值、空白、前后缀、大小写、列表含 null 等必须拒绝的形态,
  以及 updateAuthCodes 后被吊销授权码立即失效。
- Filter.AdaptorFilterTest(9):桌面放行 / 移动 UA 跳转 /validate 例外 /
  无 UA 既不跳转也不放行 / AuthCode=alone 透传与不泄漏非 alone 授权码。
- Service.SubServiceTest(24):子账号增改删的校验与落库边界、
  绑定与改绑的前置条件、key 生成冲突重试、公开订阅的 404/400/503 分支。
- Service.GalleryManageServiceTest(19):任务创建校验、节点离线、
  查询/重试幂等、删除的授权与节点回执分支。
- Util.CustomUtilTest(17):体积换算双向互逆与档位边界、时间格式化、
  端口探测与占用回退、404 输出容错。
- Util.GalleryUtilTest(15):链接校验、gid 提取、取图失败的降级返回 null、
  请求体契约字段、mpvKey 缓存命中不刷新。
- Util.ResponseTest(11):result/data 字段契约、结构化数据不被双重转义、
  实例间不共享状态、非 ASCII 输出仍是合法 JSON。

覆盖率:指令 16.9% -> 35.5%,分支 -> 32.6%。
TaskHandlerInterceptor/Response 100%、CustomUtil 99%、MessageCodec 94.6%、
AdaptorFilter 94.4%、SubService 78.1%。

测试过程中发现 3 处既有缺陷,均未擅自修改生产逻辑,改为在测试中
显式断言现状并注明「若断言失败说明已修复」:
1) GalleryManageService.deleteGalleryByGid:当画廊无任何收藏时
   collector.isEmpty() 使授权条件短路放行,**未校验下载者身份**,
   任意有效授权码用户可删除他人任务记录。
2) 同一方法:remoteService.deleteGallery 位于授权判断之外,被拒请求
   仍会向节点下发删除指令;且 switch 中 case 0 会 response.success()
   覆盖先前的 failure,对外表现为成功。
3) createTask:link.split("/")[4] 段数不足抛 ArrayIndexOutOfBoundsException,
   而只捕获 NumberFormatException,且项目无 @ControllerAdvice,会穿透为 500。
另有 Response.isSuccess() 在未设置 result 键时 NPE、
getWeekUsedAmount 在配置行缺失时 NPE,一并记录。
2026-09-14 15:04:52 +08:00
root d5b97b82a0 升级剩余依赖并将 HttpClient 4 迁移到 5
- HttpClient 4.5.14 -> httpclient5 5.6.4(Boot 4.1.1 托管版本):
  HttpClient 4.x 最后一次发布是 2022-11,已 EOL,且 Boot 自 3.1 起不再管理它;
  httpmime 一并移除(httpclient5 已内置 multipart)。
  代码迁移 3 个文件(LocalService、SubscriptionRefreshService、GalleryUtil):
  包名 org.apache.http.* -> org.apache.hc.client5.http.*;HttpClient 5 用
  response.getCode() 取代 response.getStatusLine().getStatusCode();
  RequestConfig 的 socket 读超时改名 setSocketTimeout -> setResponseTimeout,
  超时参数改为 Timeout.ofMilliseconds(...)。
  MultipartEntityBuilder/EntityBuilder 仍在,仅换包名,行为不变。
- java-telegram-bot-api 7.9.1 -> 10.1.0(跨 3 个大版本)。本项目只用到
  new TelegramBot(token)、new SendMessage(chatId, text)、bot.execute(msg),
  已核对 10.1.0 的构造器与 execute 签名均兼容。
- hutool-all 5.8.26 -> 5.8.47;commons-compress 1.26.1 -> 1.28.0。

验证:mvn test 16 项全过(注意单测对 HTTP 是 mock,不能证明迁移后的真实网络路径);
另外做了针对性验证——隔离实例上 POST /personal/subBind/accounts/1/refresh 触发
真实上游下载,经迁移后的 httpclient5 成功取回 127,379 字节订阅并写入隔离缓存目录
(生产目录未被触碰);另用等价配置的 httpclient5 客户端实拉 https://example.com
返回 200 且正文可读。启动 5.09s、各接口正常、日志无 httpclient 相关异常。
2026-09-14 14:23:23 +08:00
root 4623fca8b0 升级到 Spring Boot 4.1.1 并迁移 Jackson 3
- 父 POM 3.3.2 -> 4.1.1;spring-boot-starter-web 改名为 webmvc;
  mybatis-spring-boot-starter(-test) 3.0.3 -> 4.1.0(对标 Boot 4)。
- Jackson 2 -> 3:databind/core 包名改为 tools.jackson.*(注解仍在
  com.fasterxml.jackson.annotation,无需改动)。Boot 4 移除了
  JsonProcessingException,PersonalController.ip() 实际不抛该异常,去掉声明。
- @ServletComponentScan 迁移到 org.springframework.boot.web.server.servlet.context。
- 顺带修复无 Boot 关联的漏洞:jsoup 1.15.3 -> 1.23.2(CVE-2026-71497)、
  commons-io -> 2.22.0(CVE-2024-47554)、lombok 对齐托管版 1.18.46。
- native-maven-plugin 0.10.3 -> 1.1.8(Boot 4.1.1 托管版本)。
  注意:原生构建仍需 GraalVM/JDK 25,本次仅验证 JVM 运行。

验证:mvn test 16 项全过;隔离冒烟(真实库副本、独立端口)通过——
Tomcat 11 启动 4.5s、/ 302、/GalleryManage 200、订阅分发 /sub/v2|cat 字节数与
3.x 一致、WebSocket 升级 101、@ServletComponentScan 过滤器生效、
与 storageNode 的 Jackson 2.15.2 线上格式双向兼容。
2026-09-14 13:44:03 +08:00
root f5ef8ca487 修复测试构造与 commons-io 版本不匹配
- RemoteServiceTest 原先调用 ResponseMessage(int, byte),主源码中不存在该构造器,
  导致测试源码无法编译;改为在测试内用 setter 组装。
- commons-compress 1.26.1 依赖 commons-io 2.15.1 的 IOUtils.skip(InputStream,long,Supplier),
  而 POM 钉在 2.11.0,任何 tar 读取都会抛 NoSuchMethodError;commons-io 升至 2.15.1
  (同时覆盖 CVE-2024-47554)。

后端 mvn test 16 项、storageNode 5 项、桌面前端 9 项全部通过。
2026-09-14 13:36:45 +08:00
root 50e513b8cc 记录审查修复提交与离线编译验证事项 2026-09-08 12:51:59 +08:00
root d3b18f90fa 修复图片Key缓存冷启动并关闭异常HTTP响应 2026-09-08 12:50:44 +08:00
root a9ba631847 补齐连接反复断开和缓存失败重试边界 2026-09-08 12:48:24 +08:00
root 60facae9b5 原子发布图片缓存并关闭下载和打包文件流 2026-09-08 09:30:23 +08:00
root b15eeaf45e 统一节点请求生命周期并释放重连资源 2026-09-08 09:25:45 +08:00
root 1e6e3a1557 缩短订阅状态锁范围并拒绝过期刷新结果 2026-09-08 09:23:14 +08:00
root 7f823b6150 修复断点下载范围解析与读取越界 2026-09-08 09:21:01 +08:00
root f65c5ad860 先保存下载任务再下发节点以保留即时状态 2026-09-08 09:19:47 +08:00
root 3c8be3e7c9 修复订阅访问IP记录缺失问题 2026-08-30 23:07:25 +08:00
150 changed files with 18544 additions and 1465 deletions
+46
View File
@@ -0,0 +1,46 @@
target/
!.mvn/wrapper/maven-wrapper.jar
!**/src/main/**/target/
!**/src/test/**/target/
# 本地运行与测试产生的日志
run.out
*.log
### IntelliJ IDEA ###
.idea/modules.xml
.idea/jarRepositories.xml
.idea/compiler.xml
.idea/libraries/
*.iws
*.iml
*.ipr
### Eclipse ###
.apt_generated
.classpath
.factorypath
.project
.settings
.springBeans
.sts4-cache
### NetBeans ###
/nbproject/private/
/nbbuild/
/dist/
/nbdist/
/.nb-gradle/
build/
!**/src/main/**/build/
!**/src/test/**/build/
### VS Code ###
.vscode/
### Mac OS ###
.DS_Store
/.idea/encodings.xml
# Migration script bytecode
scripts/__pycache__/
+150
View File
@@ -0,0 +1,150 @@
# 图片索引缓存(GalleryIndex)— 重写模块说明
## 问题
在线阅读的索引是「gid → 每一页 imgkey」,存在 `cache.db` 的两张表里:
| 表 | 列 | 含义 |
|---|---|---|
| `gidToKey` | `gid`, `key`, `pages` | 画廊头部:gid、画廊 key、总页数 |
| `ImageKeyCache` | `gid`, `imgkey`, `page` | 每页的 imgkey |
旧实现(`GalleryManageService.cacheImagesKey`)依次做四件事:查缓存、判断是否需要重建、
外呼抓 mpv 页、逐页 `INSERT`。由此产生的实际问题:
1. **没有唯一约束**。实测生产缓存库有 4 组重复 gid、97 组重复 `(gid,page)`。
2. **整组替换不是原子的**。先写头部再逐页写,中途失败会留下「头部命中、页 key 缺失」的
半截索引;后续请求直接判定为已缓存,永远取不到图。
3. **用行数判断完整性**。`count(*) == pages` 无法发现「页码 1、3 而 pages=2」这种跳号。
4. **没有并发合并**。同一本画廊被并发打开时,两个构建者会互相删除对方刚写入的行。
5. **逐页写入放大写事务**,抓取与写入之间没有事务边界。
## 交付物与职责
| 文件 | 职责 |
|---|---|
| `Service/GalleryIndexService.java` | 索引的唯一读写入口:同 gid 合并、抓取在事务外、校验后整组短事务替换 |
| `Exception/GalleryIndexException.java` | 分类失败原因:INVALID_LINK / UNAVAILABLE / MALFORMED / TRANSIENT |
| `Configuration/GalleryIndexConfiguration.java` | 缓存库的事务管理器与读写模板(独立文件,未改 SqlConfiguration) |
| `Dao/cache/ImageCacheMapper.java` | 新增页码查询、脏行计数、批量写入(原有方法全部保留) |
| `Maintenance/GalleryIndexMaintenance.java` | 离线体检与修复工具(只删不健康 gid 的整组索引) |
| `scripts/migrate_image_index_cache.{sh,sql}` | 迁移:默认 dry-run、备份、清理、建唯一索引 |
| `Service/GalleryIndexServiceTest.java` | 服务编排的行为约束(无 Spring、无网络) |
| `Maintenance/GalleryIndexMaintenanceTest.java` | 工具在真实临时 SQLite 上的判定与删除范围 |
| `Dao/cache/ImageCacheMapperSqlTest.java` | 注解 SQL 在真实 MyBatis + SQLite 上的验证 |
## 接口
```java
GidToKey ensureIndex(String url) throws GalleryIndexException;
```
返回该画廊的 `gid` / `key` / `pages`。三条不变量:
- **抓取在事务外**:`upstream.fetchPageKeys(url)` 先于任何写事务执行;写事务只在
`publish(...)` 里开启,删除旧索引与批量写入在同一个事务内整体生效。
- **校验先于写入**:行必须满足 gid 与请求一致、页码恰好为 1..N、不重复、imgkey 非空,
否则抛 MALFORMED,此时尚未开启写事务,库中旧索引原样保留。
- **读在快照内**:`lookup` 在同一个读事务里读头部与页码,避免读到整组替换的中间状态。
失败原因:
| Reason | 触发条件 | isPermanent() | 建议处置 |
|---|---|---|---|
| INVALID_LINK | 链接里取不到 gid 或 key | true | 直接回「链接错误」 |
| UNAVAILABLE | 上游返回空页列表(下架/删除) | true | 回「该图片已下架或已被删除」 |
| MALFORMED | 上游页 key 不自洽 | false | 可重试,回「网络波动或其他异常」 |
| TRANSIENT | 上游异常或写库失败 | false | 可重试,回「网络波动或其他异常」 |
## 接入方式(由主模型完成 GalleryManageService 侧改动)
构造函数注入:
```java
final GalleryIndexService galleryIndexService;
```
`cacheImagesKey` 改为:
```java
public String cacheImagesKey(String url) {
Response response = Response.generateResponse();
try {
GidToKey gidToKey = galleryIndexService.ensureIndex(url);
return response.success(objectMapper.valueToTree(gidToKey)).toJSONString();
} catch (GalleryIndexException e) {
response.failure(e.getReason() == GalleryIndexException.Reason.INVALID_LINK
? "链接错误"
: e.isPermanent() ? "该图片已下架或已被删除" : "网络波动或其他异常");
return response.toJSONString();
}
}
```
Spring 装配无需额外配置:`GalleryIndexConfiguration` 已把 `GalleryIndexService` 注册为 bean,
依赖 `ImageCacheMapper`(cache 数据源)与两个 TransactionTemplate。原来的
`isIndexComplete` / `rollbackImageIndex` 私有方法与 `ImageCacheMapper` 的
`countImageKeyCacheByGid` / `delete*ByGid` 仍保留,供旧调用方与非索引路径使用。
读取路径 `getCachedImage` 不需要改动:它按 `(gid,page)` 查 ImageKeyCache,现在会走新加的
`ux_ImageKeyCache_gid_page` 唯一索引。
## 迁移
```bash
# 1. dry-run(只读,不带 --apply 绝不写入)
scripts/migrate_image_index_cache.sh --db /path/to/copy.db
# 2. 先在 SQLite 备份副本上演练
sqlite3 -readonly /home/lionwebsite/cache.db ".backup '/tmp/cache-copy.db'"
# 3. 确认后执行(自动做 .backup 备份并校验)
scripts/migrate_image_index_cache.sh --db /tmp/cache-copy.db --apply
```
迁移只做两类写入:删除不健康 gid 的整组索引(重复头部、半截、跳号、空 imgkey)、
删除孤儿页 key;随后建立 `ux_gidToKey_gid` 与 `ux_ImageKeyCache_gid_page`。
若清理后仍有重复,建唯一索引会失败并整体回滚。真实的 `/home/lionwebsite/cache.db` 未被本次工作修改。
离线修复工具:
```bash
java -cp sqlite-jdbc.jar Maintenance/GalleryIndexMaintenance.java --db /tmp/cache-copy.db # dry-run
java -cp sqlite-jdbc.jar Maintenance/GalleryIndexMaintenance.java --db /tmp/cache-copy.db --apply # 执行
```
对 `/home/lionwebsite` 下的库运行 `--apply` 会被拒绝,除非显式加 `--allow-production`。
## 测试
在 `/tmp` 的独立源码副本中运行(未复制任何生产 runtime config、未改生产库):
```
mvn -o -Djacoco.skip=true \
-Dtest='GalleryIndexServiceTest,GalleryIndexMaintenanceTest,ImageCacheMapperSqlTest' \
-DfailIfNoSpecifiedTests=false test
Tests run: 9, Failures: 0, Errors: 0 -- GalleryIndexMaintenanceTest
Tests run: 21, Failures: 0, Errors: 0 -- GalleryIndexServiceTest
Tests run: 7, Failures: 0, Errors: 0 -- ImageCacheMapperSqlTest
Tests run: 37, Failures: 0, Errors: 0
BUILD SUCCESS
```
覆盖的关键场景:
- 同 gid 24 线程并发只抓取一次、只发布一次;失败结果也被等待者共享;失败后可重试。
- 抓取发生在事务之外;所有删除与写入都在事务内;整组替换只有一个写事务。
- 行数相同但页码跳号、空 imgkey、空 key、pages<=0 均判定为不完整并重建。
- 写入失败整体回滚,旧索引与失败前逐字段一致。
- 上游畸形数据(gid 不符、页码重复/跳号/越界、imgkey 为空)在开启写事务之前被拒绝。
- 大规模索引按 BATCH_SIZE 分片写入;重复发布替换旧行而非累积。
- 注解 SQL 在真实 SQLite 上验证,含唯一约束确实拒绝重复写入。
## 已知边界
- 迁移后 gidToKey 433 条、ImageKeyCache 43704 条(原 443 / 43918)。
被删除的是 4 个冲突或不完整 gid 的整组索引与 214 行关联页 key,由下一次在线阅读重抓。
- 未做性能压测,不承诺加索引后的具体延迟收益,只确认查询计划已由全表扫描变为索引查找。
- lookup 的读事务只是为了拿到一致快照;实测 sqlite-jdbc 不允许在已建立的连接上设置
readOnly,因此读模板只设超时、不设只读标志。
+78
View File
@@ -0,0 +1,78 @@
# 2026-09-21 低内存调优记录(jar 运行期)
编译机未上线期间,两个后端都以 jar 方式运行。本机为 2 核 / 2.9G,活跃堆很小但 RSS 偏高,
做了一轮只改 JVM 参数与依赖声明的低内存调优。全程未改业务代码。
## 实测前后对照
| 服务 | 项目 | 调优前 | 调优后 |
| --- | --- | --- | --- |
| 主站 lionwebsite | RSS | 291 MB | 约 227 MB |
| 主站 lionwebsite | jar 体积 | 68.1 MB | 52.2 MB |
| 主站 lionwebsite | netty 模块 | 50 个 | 7 个 |
| 主站 lionwebsite | 线程数 | 40 | 37 |
| 存储节点 storageNode | RSS | 171 MB | 约 155-162 MB |
| 存储节点 storageNode | netty 模块 | 35 个 | 5 个 |
| 存储节点 storageNode | 线程数 | 24 | 20-22 |
调优前后堆使用都在 40-52 MB 量级,RSS 的主要成分是元空间、代码缓存与各类映射,
因此收益主要来自收敛预留和减少类加载面,而不是堆。
## 一、JVM 参数(systemd drop-in,不在版本控制内)
主站 `/etc/systemd/system/lionwebsite.service.d/jar-run.conf`:
/usr/local/jdk-25/bin/java -Xmx256m -Xss512k -XX:MaxMetaspaceSize=160m \
-XX:+UseSerialGC -XX:+UseCompactObjectHeaders \
-XX:ReservedCodeCacheSize=96m -XX:+ExitOnOutOfMemoryError \
-jar /home/lionwebsite/lionwebsite.jar
存储节点 `/etc/systemd/system/storageNode.service.d/jar-run.conf`:
/usr/local/jdk-25/bin/java -Xmx192m -Xss512k -XX:MaxMetaspaceSize=96m \
-XX:+UseSerialGC -XX:+UseCompactObjectHeaders \
-XX:ReservedCodeCacheSize=96m -XX:+ExitOnOutOfMemoryError \
-cp /root/gallery/storageNode/storageNode-jar.jar:/root/gallery/storageNode/lib/* lion.Main
取值依据:
- `UseSerialGC`:小堆、低吞吐场景,省掉 G1 的分区元数据与并发标记线程。存储节点原本就是默认 SerialGC,主站原本是默认 G1,本次对齐。
- `UseCompactObjectHeaders`:对象头 12 到 8 字节,堆内以 String 与 map 节点为主时收益直接。
- `ReservedCodeCacheSize=96m`:原先 240m 预留,实测主站只用约 14 MB、节点约 6 MB。
- `-Xmx` 收敛:主站由 384m 到 256m,节点由 256m 到 192m,相对活跃堆仍留 4-6 倍余量。
回滚:恢复同目录 `jar-run.conf.bak.20260921T090605Z`(主站)或
`jar-run.conf.bak.20260921T090746Z`(节点)后 `daemon-reload` 加 `restart`。
## 二、依赖精简(已入库)
`netty-all` 是聚合 pom,会拖进全部模块。两个后端只用到 NIO 传输与基础编解码,
因此改为按需声明:
- 主站(netty 4.2.17):`netty-transport` 加 `netty-codec-base` 加 `netty-handler`。
`handler` 用于 `RemoteService` 的 `LoggingHandler` 协议调试日志。
- 存储节点(netty 4.1.138):`netty-transport` 加 `netty-codec`。
节点侧未使用 `handler` 模块的类,故不声明。4.1 无独立 `codec-base`,编解码基类在 `netty-codec`。
剔除的内容包括 5 个平台的 native-quic、aarch64 与 riscv64 与 osx 的 epoll 与 kqueue 与 io_uring 传输、
以及 codec-http3 与 mqtt 与 redis 与 smtp 与 stomp 与 xml 与 protobuf 等。
`hutool-all` 未调整。
## 三、已评估但未采用:AOT 缓存
JDK 25 的 `-XX:AOTCache` 可免原生编译生成类缓存。实测交替 A/B 各两轮:
| | 启动耗时 | RSS |
| --- | --- | --- |
| 无 AOT | 6.37s / 6.74s | 220.9 MB / 223.1 MB |
| 带 AOT | 6.27s / 6.36s | 221.4 MB / 222.7 MB |
启动约快 0.2s(噪声量级),RSS 与元空间均无改善,代价是 80 MB 缓存文件且每次发版需重训。
按降低内存的目标不值得,故未在生产启用。
## 四、遗留与后续
- 原生编译机恢复后应重新评估原生镜像:内存会显著低于 jar,届时上述 JVM 参数失去意义。
- 存储节点目录内仍保留 `lib.bak.20260921T091617Z`(47 jar)与 `storageNode.jar.bak.*`,为本次回滚路径。
- 两个 `jar-run.conf` 的 JVM 参数只存在于服务器 drop-in,未纳入版本控制;重新部署时需按本文恢复。
+64
View File
@@ -0,0 +1,64 @@
# 2026-09-08 审查修复与编译交接
本批源码已在本机按问题分别提交,尚未推送、后端编译或部署。编译机恢复后,需要先取得下列仓库的提交,再按现有发布手册构建、验证与上线。此次没有调整鉴权或数据库表结构。
## 提交清单
| 仓库 | 提交 | 内容 |
| --- | --- | --- |
| storageNode | cbfd634 | 压缩临时文件、ZIP 完整性校验、失败保留源目录并恢复可重试状态 |
| lionwebsite-backend | f65c5ad | 下发节点前持久化任务,超时保留记录,原子累加用量 |
| lionwebsite-backend | 7f823b6 | 修复 Range 解析、读取长度、空文件和越界处理 |
| lionwebsite-backend | 1e6e3a1 | 上游订阅下载移出全局锁,设置超时,拒绝旧配置/旧请求的刷新结果 |
| lionwebsite-backend | b15eeaf | 复用连接线程组,登记等待对象后再发送,统一超时/失败/关闭清理 |
| lionwebsite-backend | 60facae | 关闭图片和 TAR 文件流,合并同图请求,完成后发布缓存,检查转换结果 |
| lionwebsite-frontend-desktop | 287dc09 | 图片失败继续加载,单图重试,忽略旧页面图片事件 |
| lionwebsite-frontend-desktop | b0f832c | 退避重连、连接状态提示、唤醒后重连、重连与提交后的全量刷新 |
| lionwebsite-frontend-desktop | d0eb2f0 | 链接 GID 类型匹配,补充任务提交回归测试 |
| lionwebsite-backend | a9ba631 | 连接反复断开时重新安排监听,验证缓存失败后可重试 |
| lionwebsite-backend | d3b18f9 | 修复图片 Key 缓存递归更新与 null 写入,异常时关闭 HTTP 响应 |
各仓库分别保留了原有历史。前端原先未提交的 `index.html`、`vite.config.js`、`llm_readme.md` 以及未跟踪的 `dist/`、`node_modules/` 未纳入本批提交。
## 已验证与待验证
已完成:
- 前端 `node --test tests/*.test.mjs`:9 项通过。
- 前端生产构建:成功,产物输出到 `/tmp/lionwebsite-review-build`,没有覆盖现有 `dist/` 或线上资源。构建提示主包体积超过 500 kB,本批未做依赖拆包。
- 各次提交的 `git diff --check` 与最终源码调用链检查。
尚未执行:
- 主站、存储节点的 Java 编译及 JUnit 测试。
- GraalVM 原生构建、真实节点断连、真实上游下载和生产端到端验证。
**以下 Java 测试仅在编译机的独立工作副本执行,不在生产主站或存储机执行。两个仓库依次运行 `mvn test`,通过后再按现有部署手册构建原生程序。** 新增测试均使用临时目录、Mock 或内存通道,不需要生产数据库和真实上游。
主站新增测试:`GallerySubmissionTest`、`FileDownloadTest`、`SubscriptionRefreshServiceTest`、`RemoteServiceTest`、`PersonalArchiveTest`、`ImageFileCacheTest`、`GalleryKeyCacheTest`。
存储节点新增测试:`DownloadCheckServiceTest`。两个仓库已有的订阅快照测试也应一起运行。
## 行为与接口说明
HTTP 路由、参数名和节点消息格式保持兼容,任务状态继续使用 `已提交`、`下载中`、`等待压缩`、`压缩中`、`下载完成`。
- `POST /GalleryManage`:先保存任务再下发。节点未确认时返回业务 failure,但任务记录仍在,前端刷新后可以重试;不会把已经收到的完成状态写回已提交。新增任务时即计入本周用量,重试不重复计量。
- 存储节点压缩失败:保留源文件,回到等待压缩状态,30 秒后可自动重试;手动重试会取消等待。仅完整校验通过的 ZIP 才视为完成。
- 使用 `FileDownload.export` 的文件接口:支持单个普通范围、后缀范围和开放结尾范围;非法或不可满足范围返回 416,`Content-Range: bytes */<size>`;多个范围回退为完整 200 响应;HEAD 不输出正文;普通完整响应不再带 Content-Range。
- 图片缓存:同一图片的并发请求共享下载,临时文件关闭且转换成功后才发布。下载连接超时 5 秒、读取超时 15 秒,转换超时 60 秒。图片缺失或下载失败仍返回 404,中断可返回 503。
- 订阅刷新:连接和连接池等待超时 5 秒,读取超时 15 秒;网络操作不占用订阅状态锁。上游 Key、过滤选项或启用状态改变后会使旧缓存失效,旧刷新结果不能重新发布。仅名称修改且刷新失败时保留原有缓存。
- WebSocket:连接失败按 1/2/4/8/16/30 秒退避,建立连接超时 10 秒;重连后重新读取任务列表和用量。页面恢复可见或网络恢复时重新连接。
## 上线验证重点
按现有发布手册安排存储节点、主站和前端上线,避开正在运行的生产任务。源码修复本身不代表生产已经更新。
1. 新建任务,模拟节点已有归档或立即返回状态,确认数据库与页面均保留完成状态且没有重复行。
2. 使用临时测试目录制造一次压缩失败,确认源文件保留,恢复目录可写后能重试,损坏 ZIP 不会标记完成。
3. 下载完整文件和小范围/尾部范围,确认响应字节数;验证空文件和越界请求。
4. 在测试上游阻塞刷新时修改其他账号/读取快照,确认操作不被网络等待阻塞;旧请求结束后不能覆盖新配置。
5. 阅读器人为令一张图片失败,确认后续图片继续加载、单图重试有效。
6. 浏览器断网、恢复网络、切后台再返回,确认连接提示和进度能够恢复。
根目录 `/home/lionwebsite/API_DOCUMENTATION.md` 已追加待发布说明;本文件是这部分接口行为变更的仓库内交接记录。
+61
View File
@@ -0,0 +1,61 @@
# 订阅缓存分代化 — 重写模块说明
## 问题
一个订阅账号对应两份内容:`v2ray.txt`(Base64 节点)与 `clash.yaml`(Clash 配置)。
旧实现先 `atomicWrite` 写 v2ray,再 `atomicWrite` 写 clash,两次替换彼此独立:
1. **两份文件可以不同版本**。第二次写入失败(磁盘满、进程被杀)时,用户会拿到
「新 v2ray + 旧 clash」的混合版本,且没有任何状态能表明它已经撕裂。
2. **读取方无法保证配对**。分发路径与备机快照各自 `Files.readAllBytes` 两个路径,
在这两次读之间完成一次刷新,就会读到跨版本组合。
3. **过期刷新可能复活**。在途刷新返回时只比较上游 key 与过滤开关;一次
`invalidateCache` 之后,迟到的响应仍会把内容写回被清空的路径。
4. **失败留半成品**。第一次写成功、第二次失败时,磁盘上就是只有一份文件的账号。
## 交付物与职责
| 文件 | 职责 |
|---|---|
| `Util/SubscriptionCacheStore.java` | 分代存储:两份文件同一分代写入 → 校验 → 指针原子发布;读取始终来自指针指向的同一分代 |
| `Service/SubscriptionRefreshService.java` | 抓取与解析在协调器锁外进行,发布短持锁;`invalidateCache` 递增版本使在途刷新失效 |
| `Service/SubscriptionStandbySnapshotService.java` | 快照构建在读锁内通过同一 store 解析成对内容,保证同版本 |
| `Util/ClashSubscriptionParser.java` | 按 YAML 语义过滤高倍节点并清理 `proxy-groups` 引用(由主模型提供) |
## 目录布局
```
<cacheRoot>/<accountId>/current 指针(Properties 文本,原子替换)
<cacheRoot>/<accountId>/generations/<generation>/v2ray.txt
<cacheRoot>/<accountId>/generations/<generation>/clash.yaml
<cacheRoot>/<accountId>/v2ray.txt + clash.yaml 旧结构,仅在两份都在时按 legacy 读
```
`current` 记录分代名与两份文件的 SHA-256。分代名是定宽 `时间戳-序号`,
因此字典序即时间序;默认只保留最近两个分代(当前与上一个)。
## 不变量
- **发布最后一步才切指针**:先写两份文件,读回校验长度与内容,再原子替换 `current`。
抓取失败、解析失败、任一份写入失败、指针写入失败,都会回滚新分代目录并保留上一个完整版本。
- **读取配对**:`read` / `resolve` / `hasComplete` 走同一份 `load` 契约,每个调用只读一次
指针快照,再按该快照校验双文件哈希。只有两份齐全且匹配才算一个有效版本;
指针缺失才回退 legacy(两份齐全时),指针存在但读不出完整版本(半写、篡改、缺字段)
时按无可用缓存处理,不回退到更旧的半份内容。
- **分发不会发出半份缓存**:`cachedPath` 在无有效版本时返回保证不存在的哨兵路径
(`.unavailable/`),使 `SubService` 判定为 503,而不会被单份 legacy 文件骗过。
`SubService.updateSub` 在读锁内完成 resolve 与 `FileDownload.export`,与 publish/invalidate 互斥。
- **失效不可复活**:`invalidateCache` 在写锁内删除指针/分代/旧结构文件,并移除该账号的
刷新版本标记;在途响应返回时因版本标记对不上而被丢弃,不会重新发布。
- **兼容旧结构**:只有当账号根下 `v2ray.txt` 与 `clash.yaml` 都在时才读为 legacy 版本;
新分代发布成功后旧结构文件被移除,避免两份来源并存。
## 测试
| 测试 | 锁定行为 |
|---|---|
| `Util/SubscriptionCacheStoreTest` | 发布后两份同分代;第二份写入失败 / 指针发布失败均保留旧版本;legacy 双文件才有效;半份 legacy、缺一份分代、内容被篡改、坏指针、指针指向缺失分代均判定不可用;每次查询只读一次指针;哨兵路径保证不存在 |
| `Service/SubscriptionCacheGenerationTest` | 在途旧刷新在失效与新版发布后被丢弃;被丢弃的刷新不留任何缓存 |
| `Service/SubscriptionFilteringTest` | 过滤规则;失败保留上次完整版本;`cachedPath` / `hasCompleteCache` 新契约;legacy 路径定位 |
| `Service/SubscriptionStandbySnapshotServiceTest` | 快照从单一分代读成对内容;不完整 legacy 账号不入快照 |
| `Util/ClashSubscriptionParserTest` | YAML 语义过滤(内联节点与引用)与畸形正文拒绝 |
+30
View File
@@ -0,0 +1,30 @@
# 备用订阅更新事件
主站 `/sub/{client}/{key}` 接受网关的签名事件;`GET` 在客户端类型与缓存路径有效后,同一个主库事务写更新记录和长期收据。签名有效的 `HEAD` 不写记录。普通请求以及无效签名仍走原来的更新记录逻辑。
`POST /internal/subscription-events` 有界读取原始请求(最多 256 KiB),批次最多 100 条。签名算法为 HMAC-SHA256,同步密钥来自 `SUBSCRIPTION_SYNC_SECRET`;签名输入是 `v1\n` + epoch 秒时间戳 + `\n` + 原始请求体 SHA256 十六进制。时间偏差最多 300 秒。请求头为 `X-Subscription-Timestamp` 和 `X-Subscription-Signature`。主站 GET 将单个事件的原始 JSON 使用无 padding 的 base64url 编码放入 `X-Subscription-Event`,同样对原始事件 JSON 签名。
事件字段:`eventId`(规范小写 UUID)、`keyHash`(公开 key 的小写 SHA256)、`client`(v2 或 cat)、`clientIp`(IPv4/IPv6 字面值)、`userAgent`(最多 2048 字符)、`occurredAt`(epoch 毫秒)。允许历史事件,事件时间不可超过未来 5 分钟。签名中的 IP 为权威来源,不能由未经签名的 XFF 覆盖。
响应 `acknowledged` 只在事务已提交后返回,包括已存在的相同事件。冲突事件 ID 返回 400;签名无效返回 403;数据库异常返回 5xx,无确认列表。未知 hash 不确认;历史映射中用户已删除则写终态收据并确认,不猜测用户归属。可见记录按事件时间和 ID 倒序保留每用户最近 10 条,收据独立长期保留。批次任意数据库写入失败会全部回滚。
## 显式迁移
运行前先执行稳定 user_id 关联迁移。新表不会在启动时创建。
1. 先在数据库副本上演练:`python3 scripts/migrate_subscription_events.py /path/to/copy.db`。
2. 停止生产主站服务,确认没有其他写入者。
3. 执行 `python3 scripts/migrate_subscription_events.py /path/to/main.db --apply --service-stopped`。
4. 检查输出 integrity 为 ok,保留输出中的 SQLite 备份,然后启动新程序。
脚本在写入前使用 SQLite backup API 生成权限 0600 的备份,在单个事务中建表并初始化历史 hash,重复执行保留既有收据和历史。不将公开 key 写入历史表或输出;原数据库备份含既有敏感数据,须保留原有保护措施。
快照构建记住所有现有绑定;重置或删除绑定前也保存历史 hash。用户改名时回传事件按 stable user_id 读取当前名字。合法旧订阅标签的 user_id 可以为 NULL,历史映射保留标签并按原标签写回,不创建用户,也不猜测同名新用户归属。可见日志删除不删除历史映射或收据。
Java 编译与测试必须在部署手册指定的 192.168.5.30 执行:
```sh
mvn -Dtest=SubscriptionEventIntegrationTest,SubServiceTest,SubscriptionStandbySnapshotServiceTest,MyBatisNativeConfigurationTest test
```
Python 迁移验证可本机执行:`python3 -m unittest discover -s scripts -p 'test_subscription_events.py' -v`。
+80
View File
@@ -0,0 +1,80 @@
# 任务生命周期与协议 v2
范围:主站下载任务的持久化生命周期、与存储节点的可靠上报协议。对应评估报告第 2、6 节。
## 为什么重写
原实现里创建先`POST`上游、后落库,删除先删库、后等节点,节点上报异步写完后立刻清队列:上下游任一侧失败都会留下「主站与节点说法不一致」的状态,且没有任何东西能把它们对回来。
复现过的具体后果:模拟状态消息发送失败后,完成任务已从节点队列移除、消息却未发出,于是「归档已存在、主站仍显示未完成」。
## 不变量
1. `gid` 只有一个有效生命周期意图。创建先落唯一 `CREATE` 意图(并写一条可见的占位任务行),靠 `gallery(gid)` 主键唯一把并发创建收敛成一个赢家;输家直接得到业务失败。
2. 上游副作用只在意图持久化之后发生。顺序固定为:登记意图 → 只读解析(`prepare`,无 POST)→ 持久化 `SUBMITTING` → 真正 `submit`。
3. `SUBMITTING` 之后绝不自动重发。上游不支持幂等,因此超时/断连/落库失败/重启中断一律落 `UNKNOWN` 等待人工对账处理,而不是再 `POST` 一次。
4. 任务行与周用量在同一短事务里写入;重复落库不重复计量。数据库失败时任务行与意图保持一致(要么都在,要么明确为 `UNKNOWN`),不允许出现「有任务没计量」。
5. 删除失败保留记录。先落 `DELETE_PENDING` 与稳定 `deleteOperationId`,只有节点确认(或文件已不存在)后才在同一事务里清收藏、删任务行并留 `DELETED` 墓碑。
6. 上报按 `revision` 单调推进,按 `operationId` 归属。迟到的旧世代上报被丢弃且不改变当前任务;墓碑(`DELETED`)永久忽略所有上报,已删除任务只能由显式 `CREATE` 重新建立新世代。
7. 回执只在提交之后发送。数据库失败不回 ACK,节点保留快照重投;重复上报也回 ACK 以停止重放,但不会重复通知。
8. 通知与 WebSocket 推送在事务提交之后、由独立执行器发送;慢通知不阻塞状态提交。
9. 只有确认已提交(`SUBMITTED`)或迁移前确实提交过的旧任务才允许下发给节点。`UNKNOWN`、`RESERVED`、`SUBMITTING`、`DELETE_PENDING`、`DELETED` 一律不下发:上游不支持幂等,把未确认/未计量的任务重投会在节点侧变成「下载完成却 0 额度」。
10. 启动时先绑定上报处理器与判定解析器、执行恢复(`SUBMITTING → UNKNOWN`、释放 `RESERVED`),完成之后才允许连接与自动重投;恢复失败时只保持连接,禁止自动派发。
## 状态表
| 层 | 取值 |
| --- | --- |
| `task_operation.state` | `RESERVED` `SUBMITTING` `SUBMITTED` `UNKNOWN` `DELETE_PENDING` `DELETED` |
| 任务展示状态 | `提交中` `已提交` `下载中` `等待压缩` `压缩中` `下载完成` `提交未知` `待删除` `删除失败` |
| 节点上报状态码 | `1` 下载中 `2` 等待压缩 `3` 压缩中 `4` 下载完成(沿用旧码) |
状态码到中文文案只在 `TaskStatus` 一处映射。
## 线协议
帧格式不变:`[messageType(1B)][length(4B, 大端)][JSON]`。旧类型 `0..9` 的字段一个都不动(旧节点 Jackson 会拒绝未知字段)。
| 类型 | 消息 | 关键字段 |
| --- | --- | --- |
| 10 | `ProtocolHelloMessage` | `protocolVersion=2`, `features=["reliable-task-v2"]` |
| 11 | `TaskCommandMessage` | `operationId`, `action=DOWNLOAD\|DELETE`, `gid`, `name` |
| 12 | `TaskReportMessage` | `reportId`, `revision`, `galleryTasks[]`, `operationIds{gid→op}` |
| 13 | `TaskReportAckMessage` | `reportId`, `result` |
流程:主站先发旧 `Identity`,再发类型 10;节点回类型 10 声明 v2 后,才允许使用 `11..13`。未协商成功时自动退化为旧的 `DownloadPostMessage`/`DeleteGalleryMessage`(此时不承诺可靠 ACK,属已知兼容边界)。
## 部署与迁移
1. 用 `scripts/migrate_task_operation_v2.sql` 建表。脚本只有独立语句、不含 `BEGIN/COMMIT`,由外层 wrapper 包事务并在执行前备份。
2. 该脚本还会在 `gallery(gid)` 上建唯一索引。历史库若已有重复 gid,这条语句会失败:先人工确认重复来源再处理,不要直接删行。
3. 主库连接需要 busy 等待(`SqlConfiguration` 已为主数据源设置 `PRAGMA busy_timeout=5000`)。单文件 SQLite 并发开短事务时,没有等待窗口会直接抛出 `SQLITE_BUSY`。生产库建议同时启用 WAL。
4. 旧任务行没有操作记录,首次收到上报时会按显式策略补建一次「世代」(优先采纳上报里的 `operationId`),不会改动其已有状态。
## 验证
`mvn -o test`:549 项,0 失败/错误/跳过。
新增关键用例(`TaskLifecycleIntegrationTest`,真实 SQLite 文件 + 真实事务/触发器/唯一约束,落在 `/tmp` 临时目录):
- 意图与占位行先于上游 `POST`;`SUBMITTING` 先于 `submit`。
- 同 `gid` 6 线程并发创建:上游只被提交一次、任务行不超过一条。
- 上游超时后落 `UNKNOWN`;再次点击创建也不会第二次 `POST`。
- 重启恢复把中断的 `SUBMITTING` 判为 `UNKNOWN`(任务行同步显示「提交未知」),安全释放 `RESERVED` 占位,且不改写 `operationId`、不重发。
- 额度写入失败时落库事务回滚,任务保持可对账的 `提交未知` + `UNKNOWN`。
- 重投/重试复用同一 `operationId`。
- 判定为拒绝的任务(`UNKNOWN` 等)绝不下发,也不退化为无 `operationId` 的旧命令。
- 删除失败保留任务行与意图,重试复用同一删除 ID;成功后清收藏、删行、留墓碑。删除意图 ID 不匹配时不得完成删除。
- 正在提交(`RESERVED`/`SUBMITTING`)的任务拒绝删除;释放预占只清匹配世代的记录与占位行。
- 落库严格匹配本次创建世代(expectedOperationId),被删除/换代后不得复活已删 gid。
- 墓碑永久忽略所有上报(含同一 `operationId`),只允许显式 `CREATE` 用新 `operationId` 重建。
- 旧节点不带 `operationId` 的连续上报保持可接受;新 v2 已知世代缺归属的上报仍被拒。
- 过期 `revision`、缺 `operationId`、待删除任务的上报都被丢弃;未知 gid 忽略但回执。
- 重复上报不重复通知;回执写入失败时 `apply` 返回不可回执结果。
`TaskProtocolV2CodecTest` 锁定类型 10..13 的帧头与字段名,并断言旧 `DownloadPostMessage` 没有混入新字段。
## 已知限制
未做双机真实故障演练。`UNKNOWN` 目前只提供状态与持久化记录:任务行会显示「提交未知」并拒绝自动重投/重试,对账与人工解决的接口由后续批次补齐。
+137
View File
@@ -0,0 +1,137 @@
# LionWebsite 一键登录(/login):实现说明
状态:**已全部完成并上线**。主站侧提交 `00e2715`(文档 `dfb647b`、`628f3ad`),
桌面端 `9b5bc94`;机器人侧由 `debian-qq-chatgpt` 于 host-vm103-debian-qq 完成
(见共享盘 `tasks/2026-09-21-lionwebsite-login-bot-command/`)。
关联待办:`td-20260921-a1b2c3`(已归档为
`todos/archive/done-td-20260921-a1b2c3-lionwebsite-login-tg.md`)。
用户已明确选择**接法 B:共享密钥自签票据**,理由是机器人到主站的网络调用不可靠。
因此主站**不提供**任何票据签发端点,也不需要机器人侧发起 HTTP 请求——机器人本地
用共享密钥算出票据即可。
## 1. 已经上线的东西(主站侧)
### 1.1 票据格式
```
v1.<签发时间的 Unix 秒级时间戳>.<HMAC-SHA256 十六进制小写>
```
签名原文为 `v1.<时间戳>`(固定前缀 `v1.` 加时间戳本身),密钥为共享密钥的 UTF-8 字节。
等价于 `hex(HMAC_SHA256(key, "v1." + timestamp))`。
### 1.2 校验规则
- 必须恰好三段,第一段必须是 `v1`;
- 签名比较用常量时间(`MessageDigest.isEqual`);
- 时间戳只接受 `[now - 300s, now]`,**未来时间戳一律拒绝**(防伪造者用远期时间换长期有效);
- 窗口内**允许重放**(用户明确接受),过期即失效;
- 默认 300 秒有效期,可用 `personal.login.ticket-ttl-seconds` 调整,下限 30 秒。
### 1.3 端点
| 端点 | 作用 |
| --- | --- |
| `GET /login?t=<票据>` | 校验票据 → 作废旧会话(防会话固定)→ 建新会话 → 302 跳 `/index`;失败 302 跳 `/denied` |
| `GET /login/logout` | 销毁会话 → 302 跳 `/denied` |
| `GET /denied` | 静态提示页:「请在机器人里发送 `/login`」 |
nginx 无需改动:`location /` 会把 `/login` 改写成后端 `/personal/login`。
### 1.4 会话与鉴权
- 登录成功下发 `JSESSIONID`,参数为 `Path=/`、`HttpOnly`、`SameSite=Lax`,14 天滑动过期。
**`Path=/` 是必须的**:nginx 会把 `/user` 改写成后端 `/personal/user`,若沿用容器按
请求路径推导的 `/personal`,浏览器判定 `/user` 不匹配就不会带会话,面板会一直 401。
- `PersonalInterceptor` 放行「有效会话 **或** `AuthCode=alone`」,两者都拒绝时返回 **401**。
`alone` 是留给下载器前端与存储节点 `/message2me` 推送的,本次**没有**退役它。
### 1.5 密钥
- 主站从环境变量 `PERSONAL_LOGIN_SECRET` 读取(`personal.login.secret`),
由 systemd drop-in `/etc/systemd/system/lionwebsite.service.d/login-secret.conf`
加载 `/etc/lionwebsite/login-secret.env`(`600` 权限,不在仓库里)。
- 密钥为空时**一律拒绝**票据,不会退化成放行。
- 轮换方式:改两侧配置并重启。轮换后旧票据在 300 秒内自然失效。
## 2. 机器人侧待实施(host-vm103-debian-qq)
### 2.1 生成票据
密钥必须与主站一致。**已由主人批准写入 PersonalHub 保险库**,引用为
`secret://services/service-lionwebsite/panel-login-secret`;机器人侧按既有秘密读取流程
申请取值,不要写进仓库或聊天记录。Python 侧计算方式:
```python
import hashlib, hmac, time
def lionwebsite_login_link(secret: str, base: str = "https://personal.lionwebsite.xyz") -> str:
stamp = str(int(time.time()))
signed = f"v1.{stamp}"
digest = hmac.new(secret.encode(), signed.encode(), hashlib.sha256).hexdigest()
return f"{base}/login?t=v1.{stamp}.{digest}"
```
Shell 等价写法(`openssl`)供人工验证用:
```sh
SIG=$(printf 'v1.%s' "$STAMP" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $NF}')
```
### 2.2 命令接入(已实施:复用既有 `/login`)
**复用既有 `/login`,不新建命令名;一条回复里同时给出两个登录链接。**
实现方式(vm103 实际做法,比原计划更稳):**机器人进程不持有密钥**。
签发走保险库固定动作 `CollabSecretService.use_secret`,值只交给进程内签名函数,
不返回、不落盘、不进日志;审计只记引用与动作名 `lionwebsite-login-link`。
QQ 侧 `plugins/login.py` 只经内部令牌 POST `/internal/panel-login`,
由 PersonalService 统一产出文案,因此 QQ 与 Telegram 不会漂移。
- Telegram:`telegram_panel_login()` 的返回文案改为同时给出 PersonalHub 面板链接与
LionWebsite 面板链接,`TELEGRAM_COMMANDS` 里 `login` 的描述同步更新。
- QQ(LionQQBot):既有 `/login`(别名 `/登录`)返回同样两条链接,共用同一套生成逻辑。
- PersonalHub 未完整配置(缺 password / session_secret / TOTP)时,现有逻辑会回一句
配置提示;此时**仍要发出 LionWebsite 链接**,不要让一条链接的失败带掉另一条。
- 帮助菜单(Telegram `/help` 与 QQ 菜单)中 `/login` 的描述同步更新。
- 文案不要写成「两条都只能用一次」:PersonalHub 链接是一次性的,
LionWebsite 票据是 300 秒窗口内可重放。
示例文案:
```
已生成登录链接:
· Personal Hub:[打开面板](< PersonalHub 链接 >)
· LionWebsite:[打开面板](< LionWebsite 链接 >)
两条链接 5 分钟内有效,请勿转发。
```
### 2.3 与 PersonalHub 现有 /login 的关系
PersonalHub 的 `/login` 原本只签发**它自己面板**的票据
(`OneTimePanelLoginTickets` + `/panel/telegram-login`)。改造时保留这条,
再在同一回复里附加 LionWebsite 链接;两者的票据机制与有效期互不影响。
## 3. 交付与复验
- 机器人侧提交状态:**两个机器人仓库的改动尚未提交 Git**(`v103` 侧待主人确认后
按项目约定提交,注意 PersonalHub 工作区另有前一轮未提交改动,勿混提)。
- `us9929main-chatgpt` 的独立复验(2026-09-21):
- 票据边界实测:`now` 与 `now-290s` 放行,`now-301s`、`now-400s`、`now+60s` 均落 `/denied`;
篡改签名与畸形串同样落 `/denied`。
- 密钥明文扫描:抓取两个任务目录全部正文与线程消息共 17 KiB,未出现密钥明文,
只有 4 处 `secret://` 引用。
- 内部接口鉴权:`POST /internal/panel-login` 无令牌与错令牌均返回 403。
- PersonalHub `/health` 返回 200;共享盘 waiting 指针已消费,队列已清空。
## 4. 可选收尾(未做,需要时另开任务)
- 移动端 `PrivateMainForMobile` 仍写死 `authCode: "alone"`。后端两种方式都接受,
所以它能用;要一并切到会话,照搬 `PrivateMain/src/store/index.js` 的改法即可。
- `sourcecode/storageNode` 有两处硬编码 `alone`(`CustomUtil.java` 的 `/message2me`
与 `MultiThreadedHTTPServer.java` 的本机鉴权),属另一仓库,需单独发布后才能退役字面量。
- 概览页的「本机订阅」链接 `https://personal.lionwebsite.xyz/sub/self` 目前是 404,
后端没有对应映射,属历史遗留,与本次登录改造无关。
+15 -21
View File
@@ -5,12 +5,12 @@
一、项目概况 一、项目概况
────────────────────────────────────────────────────────────────────────────── ──────────────────────────────────────────────────────────────────────────────
名称: LionWebsite 名称: LionWebsite
技术栈: Spring Boot 3.3.2, Java 21, Maven, SQLite, MyBatis, Netty 技术栈: Spring Boot 4.1.1, Java 21 字节码(生产 JDK 25), Maven, SQLite, MyBatis 4, Netty 4.2
定位: 个人/私有 Web 应用,兼具 E-Hentai 画廊下载管理、个人文件服务、 定位: 个人/私有 Web 应用,兼具 E-Hentai 画廊下载管理、代理订阅
代理订阅管理等功能。 管理等功能。
运行端口: 8888 运行端口: 8888
数据库: 双 SQLite 数据库 — LionWebsite.db (主库) + cache.db (缓存库) 数据库: 双 SQLite 数据库 — LionWebsite.db (主库) + cache.db (缓存库)
构建目标: 支持 GraalVM Native Image (AOT 编译) 构建目标: 当前以 JVM/JAR 运行;保留 GraalVM Native Image 配置但尚未在 JDK 25 完成原生验证
二、项目结构 二、项目结构
────────────────────────────────────────────────────────────────────────────── ──────────────────────────────────────────────────────────────────────────────
@@ -29,8 +29,8 @@ src/main/java/com/lion/lionwebsite/
├── Controller/ ├── Controller/
│ ├── GalleryManageController.java 画廊任务 CRUD、收藏、图片在线缓存 /GalleryManage │ ├── GalleryManageController.java 画廊任务 CRUD、收藏、图片在线缓存 /GalleryManage
│ ├── QueryController.java E-Hentai 搜索代理 /query │ ├── QueryController.java E-Hentai 搜索代理 /query
│ ├── PublicController.java 根路由、IP、订阅、文件分享、验证 /、/ip、/sub/、/GetFile/、/validate │ ├── PublicController.java 根路由、IP、订阅、验证 /、/ip、/sub/、/validate
│ ├── PersonalController.java 个人文件管理 /personal/ │ ├── PersonalController.java 个人管理 (订阅更新/最后更新时间/IP/留言) /personal/
│ ├── SubController.java 订阅绑定管理 /personal/subBind/ │ ├── SubController.java 订阅绑定管理 /personal/subBind/
│ └── UserController.java 用户管理 /personal/user │ └── UserController.java 用户管理 /personal/user
│ │
@@ -41,8 +41,8 @@ src/main/java/com/lion/lionwebsite/
│ ├── PushService.java Telegram Bot 通知 (admin 告警) │ ├── PushService.java Telegram Bot 通知 (admin 告警)
│ ├── QueryService.java E-Hentai 搜索 + 缩略图代理缓存 (转 AVIF) │ ├── QueryService.java E-Hentai 搜索 + 缩略图代理缓存 (转 AVIF)
│ ├── LocalServiceImpl.java 定时任务 (连接检测/额度重置/Cookie验证/订阅更新/缩略图清理) │ ├── LocalServiceImpl.java 定时任务 (连接检测/额度重置/Cookie验证/订阅更新/缩略图清理)
│ ├── PublicServiceImpl.java IP 记录、分享码文件获取、授权码修改 │ ├── PublicServiceImpl.java IP 记录、授权码修改、身份查询
│ ├── PersonalServiceImpl.java 文件管理 (浏览/上传/下载/分享/压缩/删除/TAR打包) │ ├── PersonalServiceImpl.java 订阅最后更新时间、家里 IP、留言转发
│ ├── SubService.java 代理订阅绑定/重置/查询/更新记录 │ ├── SubService.java 代理订阅绑定/重置/查询/更新记录
│ ├── CollectService.java 画廊收藏/取消收藏 │ ├── CollectService.java 画廊收藏/取消收藏
│ └── UserServiceImpl.java 用户 CRUD + 授权码管理 │ └── UserServiceImpl.java 用户 CRUD + 授权码管理
@@ -52,7 +52,6 @@ src/main/java/com/lion/lionwebsite/
│ │ ├── GalleryMapper.java gallery 表 CRUD │ │ ├── GalleryMapper.java gallery 表 CRUD
│ │ ├── UserMapper.java user 表 CRUD │ │ ├── UserMapper.java user 表 CRUD
│ │ ├── CollectMapper.java collect 表 CRUD │ │ ├── CollectMapper.java collect 表 CRUD
│ │ ├── ShareFileMapper.java ShareFile 表 CRUD
│ │ ├── CustomConfigurationMapper.java 配置键值对读写 │ │ ├── CustomConfigurationMapper.java 配置键值对读写
│ │ └── SubMapper.java 订阅绑定 & 更新记录 │ │ └── SubMapper.java 订阅绑定 & 更新记录
│ └── cache/ 缓存库 Mapper │ └── cache/ 缓存库 Mapper
@@ -66,7 +65,6 @@ src/main/java/com/lion/lionwebsite/
│ ├── GidToKey.java 画廊 GID → MPV Key 映射 │ ├── GidToKey.java 画廊 GID → MPV Key 映射
│ ├── ImageKeyCache.java 图片 key 缓存 (gid, page, imgkey) │ ├── ImageKeyCache.java 图片 key 缓存 (gid, page, imgkey)
│ ├── CustomConfiguration.java 配置键常量定义 │ ├── CustomConfiguration.java 配置键常量定义
│ ├── ShareFile.java 文件分享 (ShareCode, FilePath, ExpireTime)
│ ├── SubBind.java 订阅绑定 (key, user) │ ├── SubBind.java 订阅绑定 (key, user)
│ ├── SubUpdateRecord.java 订阅更新记录 (ip, UA, time, location) │ ├── SubUpdateRecord.java 订阅更新记录 (ip, UA, time, location)
│ └── PageNameCache.java 页面名缓存 (gid, page, pageName) │ └── PageNameCache.java 页面名缓存 (gid, page, pageName)
@@ -115,13 +113,11 @@ src/main/java/com/lion/lionwebsite/
- 支持图片在线预览: 缓存 MPV key → 按需下载单页 → 转为 AVIF 格式 - 支持图片在线预览: 缓存 MPV key → 按需下载单页 → 转为 AVIF 格式
- 画廊收藏/取消收藏 - 画廊收藏/取消收藏
2. 个人文件服务 2. 个人管理服务
- 文件浏览器: 按路径浏览文件/文件夹,显示大小、分享状态 - 手动触发订阅更新,返回最近更新时间与已记录的家庭 IP
- 上传: MultipartFile 上传到指定路径 - 管理员留言经 PushService 转发到 Telegram
- 下载: 支持 HTTP Range 断点续传
- 分享: 生成 8 位随机分享码,设置过期时间,可延长/取消 (文件浏览/上传/下载/分享/压缩/删除已在 td-20260921-a1b2c3 中整体下线)
- 压缩: 异步 TAR 打包目录
- 删除文件/文件夹
3. E-Hentai 搜索代理 3. E-Hentai 搜索代理
- 代理搜索 exhentai.org,返回格式化结果 (含缩略图 URL) - 代理搜索 exhentai.org,返回格式化结果 (含缩略图 URL)
@@ -141,7 +137,6 @@ src/main/java/com/lion/lionwebsite/
- 每 30 分钟: 检测存储节点连接,断开则自动重连 - 每 30 分钟: 检测存储节点连接,断开则自动重连
- 每周一 4:00: 重置每周下载额度 - 每周一 4:00: 重置每周下载额度
- 每天 0:00: 验证 E-Hentai Cookie 有效性 - 每天 0:00: 验证 E-Hentai Cookie 有效性
- 每天 4:00: 清理过期分享码
- 每 24 小时: 更新代理订阅配置文件 - 每 24 小时: 更新代理订阅配置文件
- 每周一 4:00: 清理缩略图缓存 (保留最近 10000 个) - 每周一 4:00: 清理缩略图缓存 (保留最近 10000 个)
@@ -151,13 +146,12 @@ src/main/java/com/lion/lionwebsite/
- /personal 和 /remote 路径限 AuthCode="alone" 用户 - /personal 和 /remote 路径限 AuthCode="alone" 用户
- HumanInterceptor 拒绝无 User-Agent 请求 - HumanInterceptor 拒绝无 User-Agent 请求
- AdaptorFilter 记录所有请求日志 (IP/路径/UA/时间) - AdaptorFilter 记录所有请求日志 (IP/路径/UA/时间)
- 分享码失败黑名单 (连续失败则屏蔽)
六、依赖 六、依赖
────────────────────────────────────────────────────────────────────────────── ──────────────────────────────────────────────────────────────────────────────
spring-boot-starter-web, spring-boot-starter-websocket, mybatis-spring-boot-starter spring-boot-starter-webmvc, spring-boot-starter-websocket, mybatis-spring-boot-starter 4.1
jsoup (HTML 解析), hutool-all (工具集), sqlite-jdbc (数据库) jsoup (HTML 解析), hutool-all (工具集), sqlite-jdbc (数据库)
httpclient + httpmime (HTTP 请求), commons-compress (TAR 打包) httpclient5 (HTTP 请求)
commons-io, commons-lang3, netty-all (TCP 通信) commons-io, commons-lang3, netty-all (TCP 通信)
java-telegram-bot-api (Telegram Bot), graalvm native-maven-plugin (AOT) java-telegram-bot-api (Telegram Bot), graalvm native-maven-plugin (AOT)
+64 -23
View File
@@ -5,7 +5,7 @@
<parent> <parent>
<groupId>org.springframework.boot</groupId> <groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId> <artifactId>spring-boot-starter-parent</artifactId>
<version>3.3.2</version> <version>4.1.1</version>
<relativePath/> <!-- lookup parent from repository --> <relativePath/> <!-- lookup parent from repository -->
</parent> </parent>
<groupId>com.lion</groupId> <groupId>com.lion</groupId>
@@ -15,22 +15,29 @@
<description>LionWebsite</description> <description>LionWebsite</description>
<properties> <properties>
<java.version>21</java.version> <java.version>21</java.version>
<tomcat.version>11.0.27</tomcat.version>
<jackson-bom.version>3.1.7</jackson-bom.version>
</properties> </properties>
<dependencies> <dependencies>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>2.6</version>
</dependency>
<dependency> <dependency>
<groupId>org.springframework.boot</groupId> <groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId> <artifactId>spring-boot-starter-webmvc</artifactId>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.mybatis.spring.boot</groupId> <groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId> <artifactId>mybatis-spring-boot-starter</artifactId>
<version>3.0.3</version> <version>4.1.0</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.projectlombok</groupId> <groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId> <artifactId>lombok</artifactId>
<version>1.18.40</version> <version>1.18.46</version>
<optional>true</optional> <optional>true</optional>
</dependency> </dependency>
<dependency> <dependency>
@@ -41,20 +48,20 @@
<dependency> <dependency>
<groupId>org.mybatis.spring.boot</groupId> <groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter-test</artifactId> <artifactId>mybatis-spring-boot-starter-test</artifactId>
<version>3.0.3</version> <version>4.1.0</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.jsoup</groupId> <groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId> <artifactId>jsoup</artifactId>
<version>1.15.3</version> <version>1.23.2</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>cn.hutool</groupId> <groupId>cn.hutool</groupId>
<artifactId>hutool-all</artifactId> <artifactId>hutool-all</artifactId>
<version>5.8.26</version> <version>5.8.47</version>
</dependency> </dependency>
<dependency> <dependency>
@@ -63,15 +70,8 @@
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.httpcomponents</groupId> <groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient</artifactId> <artifactId>httpclient5</artifactId>
<version>4.5.14</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.14</version>
</dependency> </dependency>
<dependency> <dependency>
@@ -82,24 +82,45 @@
<dependency> <dependency>
<groupId>org.apache.commons</groupId> <groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId> <artifactId>commons-compress</artifactId>
<version>1.26.1</version> <version>1.28.0</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>commons-io</groupId> <groupId>commons-io</groupId>
<artifactId>commons-io</artifactId> <artifactId>commons-io</artifactId>
<version>2.11.0</version> <version>2.22.0</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>io.netty</groupId> <groupId>io.netty</groupId>
<artifactId>netty-all</artifactId> <!--
只保留实际用到的模块。原先的 netty-all 是聚合 pom,会拖进 50 个模块
(含 5 个平台的 native-quic、aarch64/riscv64/osx 的 epoll/kqueue/io_uring、
codec-http3/mqtt/redis/smtp/stomp/xml/protobuf 等),
而本项目只用 Bootstrap/NioEventLoopGroup/NioSocketChannel/ByteBuf/
ByteToMessageCodec/LengthFieldBasedFrameDecoder/LoggingHandler/Promise。
transport 提供 Bootstrap/EventLoop/Channel,codec-base 提供
ByteToMessageCodec 与 LengthFieldBasedFrameDecoder,handler 提供
LoggingHandler(RemoteService 的协议调试日志用)。
buffer/common/resolver 由它们传递引入。
-->
<artifactId>netty-transport</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-base</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
</dependency> </dependency>
<dependency> <dependency>
<groupId>com.github.pengrad</groupId> <groupId>com.github.pengrad</groupId>
<artifactId>java-telegram-bot-api</artifactId> <artifactId>java-telegram-bot-api</artifactId>
<version>7.9.1</version> <version>10.1.0</version>
</dependency> </dependency>
<dependency> <dependency>
@@ -118,20 +139,40 @@
<path> <path>
<groupId>org.projectlombok</groupId> <groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId> <artifactId>lombok</artifactId>
<version>1.18.40</version> <version>1.18.46</version>
</path> </path>
</annotationProcessorPaths> </annotationProcessorPaths>
</configuration> </configuration>
</plugin> </plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.13</version>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin> <plugin>
<groupId>org.graalvm.buildtools</groupId> <groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId> <artifactId>native-maven-plugin</artifactId>
<version>0.10.3</version> <version>1.1.8</version>
<configuration> <configuration>
<imageName>lionwebsite</imageName> <imageName>lionwebsite</imageName>
<buildArgs> <buildArgs>
<arg>-H:+ReportExceptionStackTraces</arg> <arg>-H:+ReportExceptionStackTraces</arg>
<arg>--initialize-at-build-time=org.apache.commons.logging.LogFactory,org.apache.commons.logging.LogFactoryService,org.sqlite.util.ProcessRunner</arg> <arg>--initialize-at-build-time=org.apache.commons.logging.LogFactory,org.apache.commons.logging.LogFactoryService,org.apache.commons.logging.impl.WeakHashtable,org.sqlite.util.ProcessRunner</arg>
</buildArgs> </buildArgs>
<metadataRepository> <metadataRepository>
<enabled>true</enabled> <enabled>true</enabled>
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env bash
# 图片索引(cache.db)迁移:为缓存表补唯一约束与查询索引。
#
# 设计约束:
# * 默认 dry-run,只读检查;必须显式 --apply 才执行写入。
# * 必须显式 --db 指定数据库文件,不读取任何运行时配置,也不推断生产路径。
# * 写库用 sqlite3 -cmd '.bail on',任一语句出错立即以非零码退出,事务不提交。
# * 只以读写模式打开目标库(文件必须已存在,不会新建);dry-run 全用 -readonly。
# * 备份用 SQLite 自身的 .backup(online backup API),而不是 cp:
# cp 会漏掉 WAL 里尚未 checkpoint 的已提交内容。
# * 备份后立即校验备份可读且含两张缓存表,失败则中止并删除无效备份。
# * 不健康(不完整/冲突)的 gid 整组索引会被删除,由下一次在线阅读重新抓取。
# * 唯一约束建在唯一索引上:若清理后仍有重复,建索引会失败并整体回滚。
#
# 用法:
# scripts/migrate_image_index_cache.sh --db /tmp/cache-copy.db # dry-run(只读)
# scripts/migrate_image_index_cache.sh --db /tmp/cache-copy.db --apply # 执行迁移
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SQL_FILE="$SCRIPT_DIR/migrate_image_index_cache.sql"
db_path=""
apply="no"
usage() {
sed -n '2,30p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
}
while [[ $# -gt 0 ]]; do
case "$1" in
--db) db_path="${2:-}"; shift 2 ;;
--apply) apply="yes"; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "未知参数: $1" >&2
exit 2 ;;
esac
done
if [[ -z "$db_path" ]]; then
echo "必须显式指定缓存库: --db <cache.db>;本脚本不会自动推断数据库位置。" >&2
exit 2
fi
if [[ ! -f "$db_path" ]]; then
echo "数据库不存在: $db_path" >&2
exit 2
fi
if [[ ! -f "$SQL_FILE" ]]; then
echo "缺少 SQL: $SQL_FILE" >&2
exit 2
fi
if ! command -v sqlite3 >/dev/null 2>&1; then
echo "缺少 sqlite3 命令行工具" >&2
exit 2
fi
if [[ "$(sqlite3 -readonly "$db_path" \
"select count(*) from sqlite_master where type='table' and name in ('gidToKey','ImageKeyCache');")" != "2" ]]; then
echo "缓存表缺失:$db_path 中找不到 gidToKey 与 ImageKeyCache。" >&2
exit 2
fi
# 体检统计,与 SQL 里实际清理的范围一致:
# bad_gids 将被整组删除的 gid 个数(去重),覆盖:
# 重复头部 / 行数不符 / 页码去重后不足 / 页码区间异常 /
# 空 imgkey / 空 key / 非法 pages
# bad_gid_rows 这些 gid 占用的头部行数(重复头部的 gid 会多于 1 行)
# duplicate_pages 重复的 (gid,page) 组数,仅用于展示,会随所属 gid 一并清理
# orphan_gids 有页 key 但无头部记录的 gid 个数
read_stats() {
sqlite3 -readonly "$db_path" "
select 'header_rows', count(*) from gidToKey;
select 'page_rows', count(*) from ImageKeyCache;
select 'duplicate_gids', count(*) from (select gid from gidToKey group by gid having count(*) > 1);
select 'duplicate_pages', count(*) from (select gid,page from ImageKeyCache group by gid,page having count(*) > 1);
select 'bad_gids', count(distinct g.gid) from gidToKey g
where g.pages is null or g.pages <= 0
or g.key is null or trim(g.key) = ''
or (select count(*) from ImageKeyCache i where i.gid = g.gid) <> g.pages
or (select count(distinct i.page) from ImageKeyCache i where i.gid = g.gid) <> g.pages
or (select min(i.page) from ImageKeyCache i where i.gid = g.gid) <> 1
or (select max(i.page) from ImageKeyCache i where i.gid = g.gid) <> g.pages
or exists (select 1 from ImageKeyCache i
where i.gid = g.gid and (i.imgkey is null or trim(i.imgkey) = ''));
select 'bad_gid_rows', coalesce(sum(c), 0) from (
select count(*) as c from gidToKey g where g.gid in (
select g2.gid from gidToKey g2 where g2.pages is null or g2.pages <= 0
or g2.key is null or trim(g2.key) = ''
or (select count(*) from ImageKeyCache i where i.gid = g2.gid) <> g2.pages
or (select count(distinct i.page) from ImageKeyCache i where i.gid = g2.gid) <> g2.pages
or (select min(i.page) from ImageKeyCache i where i.gid = g2.gid) <> 1
or (select max(i.page) from ImageKeyCache i where i.gid = g2.gid) <> g2.pages
or exists (select 1 from ImageKeyCache i
where i.gid = g2.gid and (i.imgkey is null or trim(i.imgkey) = '')))
group by g.gid);
select 'orphan_gids', count(distinct gid) from ImageKeyCache
where gid not in (select gid from gidToKey);
select 'strict_index', count(*) from sqlite_master
where type='index' and name='ux_ImageKeyCache_gid_page';
"
}
mode_label="dry-run(只读)"
if [[ "$apply" == "yes" ]]; then mode_label="apply(读写)"; fi
echo "=== 缓存库图片索引迁移($mode_label) ==="
echo "数据库: $db_path"
echo "--- 迁移前体检 ---"
read_stats | while IFS='|' read -r name value; do
printf ' %-16s %s\n' "$name" "$value"
done
if [[ "$apply" != "yes" ]]; then
echo
echo "dry-run:未做任何写入。确认无误后加 --apply 执行迁移。"
echo "提示:请先在 SQLite 副本上演练,副本应这样生成(含 WAL 内容):"
echo " sqlite3 -readonly <生产库> \".backup '<副本路径>'\""
exit 0
fi
# ---- 备份:SQLite online backup,包含 WAL 中已提交但未 checkpoint 的内容 ----
backup="$db_path.before-image-index.$(date -u +%Y%m%dT%H%M%SZ)"
if ! sqlite3 "$db_path" ".backup '$backup'"; then
echo "备份失败,未做任何修改: $backup" >&2
exit 3
fi
if [[ ! -s "$backup" ]]; then
echo "备份文件为空,未做任何修改: $backup" >&2
exit 3
fi
if [[ "$(sqlite3 -readonly "$backup" \
"select count(*) from sqlite_master where type='table' and name in ('gidToKey','ImageKeyCache');")" != "2" ]]; then
echo "备份校验失败(缺少缓存表),未做任何修改,已删除无效备份。" >&2
rm -f -- "$backup"
exit 3
fi
echo "已备份(SQLite .backup): $backup"
# ---- 执行迁移:.bail on 保证任一语句失败即退出,事务不提交 ----
sqlite3 -cmd '.bail on' "$db_path" < "$SQL_FILE"
echo "--- 迁移后体检 ---"
read_stats | while IFS='|' read -r name value; do
printf ' %-16s %s\n' "$name" "$value"
done
echo "图片索引迁移完成: $db_path"
+67
View File
@@ -0,0 +1,67 @@
-- 图片索引缓存迁移:为 cache.db 的两张索引表补唯一约束与查询索引。
--
-- 目标(对应本次重写的三条不变量):
-- 1. gid 唯一 —— 同一本画廊只能有一条 gidToKey 记录;
-- 2. (gid,page) 唯一 —— 同一页只能有一个 imgkey,同时天然满足按页查询的索引需求;
-- 3. 只保留「完整」的索引 —— 半截或冲突的 gid 整组删除,让下一次在线阅读重新抓取。
--
-- 为什么用唯一索引而不是重建表:
-- * 唯一索引对 SQLite 与唯一约束等价,都会拒绝重复写入;
-- * 不需要搬表、改列,旧版本二进制仍可读写同一份数据;
-- * 回滚只需 drop index,最坏情况也只需回到脚本自动生成的备份文件。
--
-- 幂等:所有重建索引都带 IF NOT EXISTS;清理语句只按集合删除,重复执行结果相同。
-- 安全:本脚本只做 DELETE 与 CREATE INDEX,不建表、不改列、不碰 gidToKey/ImageKeyCache
-- 之外的任何对象。必须在事务中整体提交,调用方使用 sqlite3 -bail 保证任一语句
-- 失败即终止、事务回滚。
--
-- 执行前请务必先备份数据库文件(迁移脚本 migrate_image_index_cache.sh --apply 会自动备份)。
BEGIN IMMEDIATE;
-- ---------- 1. 冲突 gid:同一 gid 有多条头部记录 ----------
-- 无法判断哪一条正确,整组移除,由下次访问重抓。
CREATE TEMP TABLE __conflict_gid AS
SELECT gid FROM gidToKey GROUP BY gid HAVING COUNT(*) > 1;
DELETE FROM ImageKeyCache WHERE gid IN (SELECT gid FROM __conflict_gid);
DELETE FROM gidToKey WHERE gid IN (SELECT gid FROM __conflict_gid);
-- ---------- 2. 半截或非法 gid:头部与页 key 不一致 ----------
-- 判定同时覆盖:页码数量不符、页码不连续(跳号/重复)、区间不是 1..pages、
-- 头部字段为空、以及存在空的 imgkey 行。
CREATE TEMP TABLE __broken_gid AS
SELECT g.gid
FROM gidToKey g
WHERE g.pages IS NULL OR g.pages <= 0
OR g.key IS NULL OR trim(g.key) = ''
OR (SELECT COUNT(*) FROM ImageKeyCache i WHERE i.gid = g.gid) <> g.pages
OR (SELECT COUNT(DISTINCT i.page) FROM ImageKeyCache i WHERE i.gid = g.gid) <> g.pages
OR (SELECT MIN(i.page) FROM ImageKeyCache i WHERE i.gid = g.gid) <> 1
OR (SELECT MAX(i.page) FROM ImageKeyCache i WHERE i.gid = g.gid) <> g.pages
OR EXISTS (SELECT 1 FROM ImageKeyCache i
WHERE i.gid = g.gid AND (i.imgkey IS NULL OR trim(i.imgkey) = ''));
DELETE FROM ImageKeyCache WHERE gid IN (SELECT gid FROM __broken_gid);
DELETE FROM gidToKey WHERE gid IN (SELECT gid FROM __broken_gid);
-- ---------- 3. 孤儿页 key:没有对应头部记录 ----------
DELETE FROM ImageKeyCache WHERE gid NOT IN (SELECT gid FROM gidToKey);
-- ---------- 4. 唯一约束 ----------
-- 到这里若仍有重复,说明上面的判定有遗漏;CREATE UNIQUE INDEX 会失败,
-- 配合 -bail 让整个事务回滚,不会留下「一半约束」的库。
CREATE UNIQUE INDEX IF NOT EXISTS ux_gidToKey_gid
ON gidToKey (gid);
CREATE UNIQUE INDEX IF NOT EXISTS ux_ImageKeyCache_gid_page
ON ImageKeyCache (gid, page);
-- 按 gid 统计页数会用到 (gid,page) 前缀,按页取 key 也走同一索引,
-- 因此不再单独建 page 索引,避免重复占用写放大。
DROP TABLE __conflict_gid;
DROP TABLE __broken_gid;
COMMIT;
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Create durable subscription event receipts/history, with a SQLite backup.
Stop the production backend before --apply and pass --service-stopped to confirm.
No automatic startup DDL exists. Dry run is read-only; only hashes enter history.
Receipt retention is independent of the most recent ten visible update records.
"""
import argparse
import datetime
import hashlib
import json
from pathlib import Path
import sqlite3
DDL = (
"""CREATE TABLE IF NOT EXISTS subscription_event_receipt (
event_id TEXT PRIMARY KEY NOT NULL, key_hash TEXT NOT NULL,
client TEXT NOT NULL, occurred_at INTEGER NOT NULL, source TEXT NOT NULL,
ingested_at INTEGER NOT NULL, fingerprint TEXT NOT NULL, outcome TEXT NOT NULL
)""",
"""CREATE TABLE IF NOT EXISTS subscription_event_binding (
key_hash TEXT PRIMARY KEY NOT NULL, user_id INTEGER,
user_name TEXT NOT NULL, first_seen_at INTEGER NOT NULL, last_seen_at INTEGER NOT NULL
)""",
"CREATE INDEX IF NOT EXISTS idx_subscription_event_binding_user ON subscription_event_binding(user_id)",
"CREATE INDEX IF NOT EXISTS idx_sub_record_user_time ON sub_update_record(user_id,time DESC,id DESC)",
)
def migrate(database: Path, apply=False, service_stopped=False):
if not database.is_file():
raise ValueError("database must already exist")
if apply and not service_stopped:
raise ValueError("stop backend and supply --service-stopped before applying")
uri = database.resolve().as_uri() + ("?mode=rw" if apply else "?mode=ro")
with sqlite3.connect(uri, uri=True) as db:
for table in ("User", "sub_bind", "sub_update_record"):
columns = {row[1] for row in db.execute(f'PRAGMA table_info("{table}")')}
if not columns or (table != "User" and "user_id" not in columns):
raise ValueError("stable user links migration is required first")
unresolved = db.execute("SELECT count(*) FROM sub_bind sb LEFT JOIN User u ON u.id=sb.user_id WHERE sb.user_id IS NOT NULL AND u.id IS NULL").fetchone()[0]
if unresolved:
raise ValueError("resolve non-null dangling subscription user IDs before migration")
tables = {row[0] for row in db.execute("SELECT name FROM sqlite_master WHERE type='table'")}
result = {"apply": apply, "tables_to_create": sorted({"subscription_event_receipt", "subscription_event_binding"} - tables)}
if not apply:
return result
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
backup = database.with_name(database.name + ".before-subscription-events." + stamp)
# Create mode 600 before any sensitive database bytes are written.
backup.touch(mode=0o600, exist_ok=False)
with sqlite3.connect(backup) as snapshot:
db.backup(snapshot)
try:
db.execute("BEGIN IMMEDIATE")
for statement in DDL:
db.execute(statement)
now = int(datetime.datetime.now(datetime.timezone.utc).timestamp() * 1000)
for key, user_id, name in db.execute("SELECT sb.key,sb.user_id,coalesce(u.username,sb.user) FROM sub_bind sb LEFT JOIN User u ON u.id=sb.user_id"):
key_hash = hashlib.sha256(key.encode("utf-8")).hexdigest()
db.execute("""INSERT INTO subscription_event_binding VALUES(?,?,?,?,?)
ON CONFLICT(key_hash) DO UPDATE SET user_name=excluded.user_name,last_seen_at=excluded.last_seen_at
WHERE subscription_event_binding.user_id IS excluded.user_id""", (key_hash, user_id, name, now, now))
db.commit()
except BaseException:
db.rollback()
raise
result["backup"] = str(backup)
result["history_bindings"] = db.execute("SELECT count(*) FROM subscription_event_binding").fetchone()[0]
result["integrity"] = db.execute("PRAGMA quick_check").fetchone()[0]
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("database", type=Path)
parser.add_argument("--apply", action="store_true")
parser.add_argument("--service-stopped", action="store_true")
args = parser.parse_args()
print(json.dumps(migrate(args.database, args.apply, args.service_stopped), ensure_ascii=False))
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -euo pipefail
db_path="${1:-LionWebsite.db}"
if [[ ! -f "$db_path" ]]; then
echo "数据库不存在: $db_path" >&2
exit 2
fi
if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet lionwebsite 2>/dev/null; then
echo "检测到 lionwebsite.service 仍在运行;请先停止服务再迁移,避免写冲突。" >&2
exit 3
fi
if pgrep -f 'java .*lionwebsite\.jar' >/dev/null 2>&1; then
echo "检测到主站进程仍在运行;请先停止服务再迁移,避免写冲突。" >&2
exit 3
fi
backup="${db_path}.before-refresh-schedule.$(date -u +%Y%m%dT%H%M%SZ)"
cp "$db_path" "$backup"
echo "已备份: $backup"
sqlite3 "$db_path" < "$(dirname "$0")/migrate_subscription_refresh.sql"
echo "--- 校验 ---"
sqlite3 "$db_path" "select count(*) as accounts, sum(next_refresh_at is null) as missing_schedule from subscription_account;"
echo "订阅刷新计划迁移完成: $db_path"
+26
View File
@@ -0,0 +1,26 @@
-- 为每账号分散刷新增加计划字段。
-- 纯新增列:旧版本二进制忽略这两列,因此回滚程序时不必回滚数据库。
BEGIN;
-- next_refresh_at:该子账号下一次应刷新的时刻,Epoch 毫秒。
-- 用整数存时刻,避免 SQLite 文本时间戳被按本地时区解释(实证存在 8 小时偏差)。
-- last_success_epoch:最近一次成功刷新的真实时刻,Epoch 毫秒。
-- 既有的 last_success_at 保持原样,仅供页面展示,不参与调度判断。
ALTER TABLE subscription_account ADD COLUMN next_refresh_at INTEGER;
ALTER TABLE subscription_account ADD COLUMN last_success_epoch INTEGER;
-- 回填 last_success_epoch:SQLite 的 strftime 把文本时间按 UTC 解释,而
-- CURRENT_TIMESTAMP 写入的正是 UTC,因此这里能得到正确时刻(JDBC 读取则会偏 8 小时)。
-- 必须回填:否则所有既有账号都会被当成「从未成功过」,迁移后立刻集中补刷一遍,
-- 正好复现本次改造要消除的爆发。
UPDATE subscription_account
SET last_success_epoch = CAST(strftime('%s', last_success_at) AS INTEGER) * 1000
WHERE last_success_epoch IS NULL AND last_success_at IS NOT NULL;
-- 已有账号在窗口内按 id 错开,避免迁移后同一 tick 集中开火。
-- 以迁移时刻为基准按 37 分钟步长铺开(12 个账号约 7.4 小时排完)。
UPDATE subscription_account
SET next_refresh_at = (CAST(strftime('%s', 'now') AS INTEGER) * 1000) + (id * 37 * 60 * 1000)
WHERE next_refresh_at IS NULL;
COMMIT;
+31
View File
@@ -0,0 +1,31 @@
-- 任务操作记录表(协议 v2:先持久化意图,再产生外部副作用)
--
-- 这是显式迁移脚本,不会被应用自动执行,也不对生产库运行。
-- 执行前请备份:sqlite3 LionWebsite.db ".backup LionWebsite.db.before-task-operation"
--
-- 幂等:可重复执行。
create table if not exists task_operation
(
gid integer not null
constraint task_operation_pk
primary key,
operationId varchar(64),
deleteOperationId varchar(64),
state varchar(20) not null,
lastRevision integer not null default 0,
lastReportId varchar(64),
updateTime long not null default 0
);
-- 节点上报消费只按 gid 命中主键,这里额外为「按状态找待处理意图」提供查询入口。
create index if not exists idx_task_operation_state on task_operation (state);
-- gallery.gid 目前只有 table 级主键约束(部分历史库甚至是自增主键的衍生约束),
-- 这里显式建立唯一索引,保证同一 gid 不会出现两条任务行:
-- 「同一 gid 只有一个有效意图」是 v2 生命周期成立的前提。
-- 若该语句因历史重复数据失败,请先人工确认重复行来源再处理,不要直接删除。
create unique index if not exists idx_gallery_gid_unique on gallery (gid);
-- 旧任务行没有操作记录:启动/查询时按 gid 惰性补建为 SUBMITTED(已在上游提交过),
-- 不需要在迁移里批量写入,避免把正在进行的任务判成 UNKNOWN。
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env python3
"""Explicit task-v2 schema migration. Default is read-only; --apply backs up first."""
import argparse
import datetime
import json
from pathlib import Path
import sqlite3
def migrate(database: Path, apply: bool = False) -> dict:
if not database.is_file():
raise ValueError("database must already exist")
with sqlite3.connect(database.resolve().as_uri() + ("?mode=rw" if apply else "?mode=ro"), uri=True) as db:
if not db.execute("pragma table_info(gallery)").fetchall():
raise ValueError("required table missing: gallery")
duplicates = db.execute("select count(*) from (select gid from gallery group by gid having count(*)>1)").fetchone()[0]
if duplicates:
raise ValueError("duplicate gallery gid; reconcile before migration")
present = bool(db.execute("pragma table_info(task_operation)").fetchall())
result = {"apply": apply, "create_operation_table": not present}
if not apply:
return result
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
backup = database.with_name(database.name + ".before-task-v2." + stamp)
with sqlite3.connect(backup) as copy:
db.backup(copy)
backup.chmod(0o600)
sql = Path(__file__).with_name("migrate_task_operation_v2.sql").read_text()
try:
db.executescript("BEGIN IMMEDIATE;\n" + sql + "\nCOMMIT;")
except BaseException:
db.rollback()
raise
result["backup"] = str(backup)
result["operation_rows"] = db.execute("select count(*) from task_operation").fetchone()[0]
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("database", type=Path)
parser.add_argument("--apply", action="store_true")
arguments = parser.parse_args()
print(json.dumps(migrate(arguments.database, arguments.apply), ensure_ascii=False))
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""Add stable user identity to subscription bindings and access records.
The default invocation only reports schema changes. --apply creates a consistent
SQLite backup before the transaction. Run against an isolated database first.
"""
import argparse
import datetime
import json
from pathlib import Path
import sqlite3
def migrate(db_path: Path, apply: bool = False) -> dict:
if not db_path.is_file():
raise ValueError("database must already exist")
uri = db_path.resolve().as_uri() + ("?mode=rw" if apply else "?mode=ro")
with sqlite3.connect(uri, uri=True) as db:
changes = []
for table in ("sub_bind", "sub_update_record"):
columns = {row[1] for row in db.execute(f'PRAGMA table_info("{table}")')}
if not columns:
raise ValueError(f"required table missing: {table}")
if "user_id" not in columns:
changes.append(f"add {table}.user_id")
duplicates = db.execute(
"select count(*) from (select username from User group by username having count(*) > 1)"
).fetchone()[0]
if duplicates:
raise ValueError("ambiguous duplicate usernames; resolve identities before migration")
result = {"apply": apply, "schema_changes": changes}
if not apply:
return result
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
backup = db_path.with_name(db_path.name + ".before-user-links." + stamp)
with sqlite3.connect(backup) as snapshot:
db.backup(snapshot)
backup.chmod(0o600)
db.execute("BEGIN IMMEDIATE")
try:
for change in changes:
table = change.split()[1].split(".")[0]
db.execute(f'ALTER TABLE "{table}" ADD COLUMN user_id INTEGER REFERENCES User(id)')
for table in ("sub_bind", "sub_update_record"):
db.execute(f'UPDATE "{table}" SET user_id=(SELECT id FROM User WHERE username="{table}".user) WHERE user_id IS NULL')
db.execute("CREATE UNIQUE INDEX IF NOT EXISTS User_username_uindex ON User(username)")
db.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_sub_bind_user_id ON sub_bind(user_id) WHERE user_id IS NOT NULL")
db.execute("CREATE INDEX IF NOT EXISTS idx_sub_record_user_id ON sub_update_record(user_id, id)")
db.commit()
except BaseException:
db.rollback()
raise
result["backup"] = str(backup)
result["unresolved_bindings"] = db.execute("select count(*) from sub_bind where user_id is null").fetchone()[0]
result["unresolved_records"] = db.execute("select count(*) from sub_update_record where user_id is null").fetchone()[0]
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("database", type=Path)
parser.add_argument("--apply", action="store_true")
args = parser.parse_args()
print(json.dumps(migrate(args.database, args.apply), ensure_ascii=False))
+82
View File
@@ -0,0 +1,82 @@
import importlib.util
from pathlib import Path
import sqlite3
import tempfile
import unittest
import hashlib
spec = importlib.util.spec_from_file_location("migration", Path(__file__).with_name("migrate_subscription_events.py"))
migration = importlib.util.module_from_spec(spec)
spec.loader.exec_module(migration)
class SubscriptionEventsMigrationTest(unittest.TestCase):
def test_explicit_backup_atomic_and_idempotent(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "main.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY,username TEXT);
CREATE TABLE sub_bind(key TEXT,user TEXT,user_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY,user_id INTEGER,time INTEGER);
INSERT INTO User VALUES(3,'alice');
INSERT INTO sub_bind VALUES('public-test-key','alice',3);
""")
dry = migration.migrate(path)
self.assertEqual(2, len(dry["tables_to_create"]))
with self.assertRaises(ValueError):
migration.migrate(path, True)
first = migration.migrate(path, True, True)
self.assertEqual("ok", first["integrity"])
self.assertEqual(0o600, Path(first["backup"]).stat().st_mode & 0o777)
with sqlite3.connect(first["backup"]) as db:
self.assertEqual(0, db.execute("SELECT count(*) FROM sqlite_master WHERE name='subscription_event_binding'").fetchone()[0])
with sqlite3.connect(path) as db:
row = db.execute("SELECT key_hash,user_id,user_name FROM subscription_event_binding").fetchone()
self.assertEqual((hashlib.sha256(b"public-test-key").hexdigest(), 3, "alice"), row)
db.execute("INSERT INTO subscription_event_receipt VALUES('id','hash','v2',0,'primary',1,'fingerprint','recorded')")
second = migration.migrate(path, True, True)
self.assertEqual([], second["tables_to_create"])
with sqlite3.connect(path) as db:
self.assertEqual(1, db.execute("SELECT count(*) FROM subscription_event_receipt").fetchone()[0])
self.assertEqual(1, db.execute("SELECT count(*) FROM subscription_event_binding").fetchone()[0])
def test_legacy_labels_without_user_ids_remain_null(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "main.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY,username TEXT);
CREATE TABLE sub_bind(key TEXT,user TEXT,user_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY,user_id INTEGER,time INTEGER);
INSERT INTO sub_bind VALUES('legacy-key','legacy-label',NULL);
""")
result = migration.migrate(path, True, True)
self.assertEqual(1, result["history_bindings"])
with sqlite3.connect(path) as db:
row = db.execute("SELECT key_hash,user_id,user_name FROM subscription_event_binding").fetchone()
self.assertEqual((hashlib.sha256(b"legacy-key").hexdigest(), None, "legacy-label"), row)
self.assertIsNone(db.execute("SELECT user_id FROM sub_bind").fetchone()[0])
self.assertEqual(0, db.execute("SELECT count(*) FROM User").fetchone()[0])
migration.migrate(path, True, True)
with sqlite3.connect(path) as db:
self.assertEqual(1, db.execute("SELECT count(*) FROM subscription_event_binding").fetchone()[0])
def test_orphan_binding_aborts_without_ddl(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "main.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY,username TEXT);
CREATE TABLE sub_bind(key TEXT,user TEXT,user_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY,user_id INTEGER,time INTEGER);
INSERT INTO sub_bind VALUES('test','deleted',9);
""")
with self.assertRaises(ValueError):
migration.migrate(path, True, True)
with sqlite3.connect(path) as db:
self.assertEqual(0, db.execute("SELECT count(*) FROM sqlite_master WHERE name LIKE 'subscription_event_%'").fetchone()[0])
if __name__ == "__main__":
unittest.main()
+55
View File
@@ -0,0 +1,55 @@
import importlib.util
from pathlib import Path
import sqlite3
import tempfile
import unittest
spec = importlib.util.spec_from_file_location("migration", Path(__file__).with_name("migrate_user_links.py"))
migration = importlib.util.module_from_spec(spec)
spec.loader.exec_module(migration)
class MigrationTest(unittest.TestCase):
def test_dry_run_idempotent_migration_and_rename(self):
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "test.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY, username TEXT);
CREATE TABLE sub_bind(key TEXT PRIMARY KEY, user TEXT, subscription_account_id INTEGER);
CREATE TABLE sub_update_record(id INTEGER PRIMARY KEY, user TEXT, location TEXT);
INSERT INTO User VALUES(1,'old-name');
INSERT INTO sub_bind VALUES('example-key','old-name',2);
INSERT INTO sub_update_record VALUES(10,'old-name','unknown');
INSERT INTO sub_update_record VALUES(11,'old-name','unknown');
""")
before = path.read_bytes()
self.assertEqual(2, len(migration.migrate(path)["schema_changes"]))
self.assertEqual(before, path.read_bytes())
self.assertEqual(0, migration.migrate(path, True)["unresolved_bindings"])
self.assertEqual([], migration.migrate(path, True)["schema_changes"])
with sqlite3.connect(path) as db:
db.execute("update User set username='new-name' where id=1")
self.assertEqual((1,), db.execute("select user_id from sub_bind").fetchone())
# A slow first lookup must not overwrite the second request's row.
db.execute("update sub_update_record set location='first-location' where id=10")
self.assertEqual(("unknown",), db.execute("select location from sub_update_record where id=11").fetchone())
def test_ambiguous_names_fail_without_modifying_database(self):
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "test.db"
with sqlite3.connect(path) as db:
db.executescript("""
CREATE TABLE User(id INTEGER PRIMARY KEY, username TEXT);
CREATE TABLE sub_bind(user TEXT);
CREATE TABLE sub_update_record(user TEXT);
INSERT INTO User VALUES(1,'same'),(2,'same');
""")
before = path.read_bytes()
with self.assertRaises(ValueError):
migration.migrate(path, True)
self.assertEqual(before, path.read_bytes())
if __name__ == "__main__":
unittest.main()
@@ -2,6 +2,7 @@ package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Domain.*; import com.lion.lionwebsite.Domain.*;
import com.lion.lionwebsite.Message.*; import com.lion.lionwebsite.Message.*;
import com.lion.lionwebsite.Service.SubscriptionRefreshPlanner;
import com.lion.lionwebsite.Util.GalleryUtil; import com.lion.lionwebsite.Util.GalleryUtil;
import com.pengrad.telegrambot.TelegramBot; import com.pengrad.telegrambot.TelegramBot;
import com.pengrad.telegrambot.model.*; import com.pengrad.telegrambot.model.*;
@@ -14,14 +15,21 @@ import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
import java.time.Clock;
import java.time.Duration;
import java.util.Random;
@Configuration @Configuration
@RegisterReflectionForBinding(classes = {CustomConfiguration.class, GidToKey.class, ImageKeyCache.class, @RegisterReflectionForBinding(classes = {CustomConfiguration.class, GidToKey.class, ImageKeyCache.class,
GalleryForQuery.class, Gallery.class, GalleryTask.class, HikariConfig.class, GalleryForQuery.class, Gallery.class, GalleryTask.class, HikariConfig.class,
PageNameCache.class, ShareFile.class, User.class, PageNameCache.class, User.class,
SendResponse.class, Message.class, com.pengrad.telegrambot.model.User.class, SendResponse.class, Message.class, com.pengrad.telegrambot.model.User.class,
Chat.class, MessageEntity.class, Chat.class, MessageEntity.class,
AbstractMethodError.class, DeleteGalleryMessage.class, DownloadPostMessage.class, DownloadStatusMessage.class, AbstractMethodError.class, DeleteGalleryMessage.class, DownloadPostMessage.class, DownloadStatusMessage.class,
IdentityMessage.class, MaintainMessage.class, ResponseMessage.class, AvailableCheckMessage.class, IdentityMessage.class, MaintainMessage.class, ResponseMessage.class, AvailableCheckMessage.class,
// 协议 v2 的消息与任务操作实体:原生镜像下 Jackson/MyBatis 都要反射构造它们。
ProtocolHelloMessage.class, TaskCommandMessage.class, TaskReportMessage.class, TaskReportAckMessage.class,
TaskOperation.class,
SubscriptionSnapshotMessage.class, SubscriptionSnapshotPayload.class, SubscriptionAccountSnapshot.class, SubscriptionSnapshotMessage.class, SubscriptionSnapshotPayload.class, SubscriptionAccountSnapshot.class,
SubscriptionBindingSnapshot.class, LinkPreviewOptions.class}) SubscriptionBindingSnapshot.class, LinkPreviewOptions.class})
public class CustomBean { public class CustomBean {
@@ -41,4 +49,22 @@ public class CustomBean {
public TelegramBot getTelegramBot(){ public TelegramBot getTelegramBot(){
return new TelegramBot(botToken); return new TelegramBot(botToken);
} }
/**
* 订阅分散刷新的排程器。
*
* <p>放在这里是为了让窗口、最小间隔、tick 周期与重试间隔集中由配置注入,
* 同时保持 {@link SubscriptionRefreshPlanner} 本身是可直接构造的纯对象
* (便于用固定时钟与固定随机种子做确定性单元测试)。
*/
@Bean
public SubscriptionRefreshPlanner subscriptionRefreshPlanner(
@Value("${subscription.refresh.window-hours:24}") long windowHours,
@Value("${subscription.refresh.min-gap-minutes:60}") long minGapMinutes,
@Value("${subscription.refresh.tick-interval-ms:300000}") long tickIntervalMs,
@Value("${subscription.refresh.retry-delay-minutes:60}") long retryDelayMinutes) {
return new SubscriptionRefreshPlanner(Clock.systemDefaultZone(), new Random(),
Duration.ofHours(windowHours), Duration.ofMinutes(minGapMinutes),
Duration.ofMillis(tickIntervalMs), Duration.ofMinutes(retryDelayMinutes));
}
} }
@@ -0,0 +1,83 @@
package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Dao.cache.ImageCacheMapper;
import com.lion.lionwebsite.Domain.ImageKeyCache;
import com.lion.lionwebsite.Service.GalleryIndexService;
import com.lion.lionwebsite.Util.GalleryUtil;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import javax.sql.DataSource;
import java.util.List;
/**
* 图片索引服务的事务装配。
*
* <p>单独成文件是为了不触碰既有的 SqlConfiguration:那里的两个 SqlSessionFactory
* 保持原样,这里只额外声明缓存库的事务管理器与模板。
*
* <p>事务管理器必须绑定 datasource-cache —— 索引只写缓存库,
* 若误用主库事务管理器,整组替换就失去原子性(删除与写入落在不同连接上)。
*/
@Configuration
public class GalleryIndexConfiguration {
@Bean
public PlatformTransactionManager cacheTransactionManager(
@Qualifier("datasource-cache") DataSource cacheDataSource) {
return new DataSourceTransactionManager(cacheDataSource);
}
/** 只用于图片索引的短事务发布,超时留给上层请求超时控制,这里显式设为 30 秒。 */
@Bean
public TransactionTemplate cacheTransactionTemplate(
@Qualifier("cacheTransactionManager") PlatformTransactionManager cacheTransactionManager) {
TransactionTemplate template = new TransactionTemplate(cacheTransactionManager);
template.setTimeout(30);
return template;
}
/**
* 只读快照事务,用于一次性读取头部与页码。
*
* <p>读事务不写库,因此只读连接可以与写入并发;它的作用是让「读头部 + 读页码」
* 落在同一个快照上,避免恰好读到整组替换的中间状态。
*/
@Bean
public TransactionTemplate cacheReadTransactionTemplate(
@Qualifier("cacheTransactionManager") PlatformTransactionManager cacheTransactionManager) {
TransactionTemplate template = new TransactionTemplate(cacheTransactionManager);
template.setTimeout(10);
return template;
}
/**
* 这里显式不设置 {@code readOnly}:sqlite-jdbc 不允许在已建立的连接上改只读标志
* (实测抛 SQLException: Cannot change read-only flag after establishing a connection),
* 而事务模板拿到的是连接池里的既有连接,设置只读会让每次读取都直接失败。
* 快照一致性由「同一个事务」保证,不需要额外的只读标志。
*/
@Bean
public GalleryIndexService galleryIndexService(
ImageCacheMapper imageCacheMapper,
@Qualifier("cacheTransactionTemplate") TransactionTemplate cacheTransactionTemplate,
@Qualifier("cacheReadTransactionTemplate") TransactionTemplate cacheReadTransactionTemplate) {
return new GalleryIndexService(imageCacheMapper, cacheTransactionTemplate,
cacheReadTransactionTemplate,
GalleryIndexConfiguration::fetchPageKeys);
}
/**
* 生产环境的上游实现:沿用 GalleryUtil.parseImageKeys(HTTP + JS 解析)。
*
* <p>放在这里而不是服务内部,是为了让服务在测试中可以脱离静态工具类。
*/
static List<ImageKeyCache> fetchPageKeys(String url) throws Exception {
return GalleryUtil.parseImageKeys(url);
}
}
@@ -17,8 +17,13 @@ public class InterceptorConfiguration implements WebMvcConfigurer {
@Override @Override
public void addInterceptors(InterceptorRegistry registry) { public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(getPersonalInterceptor()).addPathPatterns("/personal/**", "/remote/**"); // 登录端点必须排除在闸门之外,否则校验票据的入口会被自己拦住。
registry.addInterceptor(taskHandlerInterceptor).addPathPatterns("/GalleryManage", "/validate"); // /personal/ 只是跳到静态入口 /index,也一并排除:未登录时让前端自己去提示,
// 否则用户直接访问根域名只会看到 401。
registry.addInterceptor(getPersonalInterceptor())
.addPathPatterns("/personal/**", "/remote/**")
.excludePathPatterns("/personal/", "/personal/login", "/personal/login/logout", "/personal/denied");
registry.addInterceptor(taskHandlerInterceptor).addPathPatterns("/GalleryManage", "/GalleryManage/**", "/validate");
registry.addInterceptor(getHumanInterceptor()).addPathPatterns("/", "/mobile"); registry.addInterceptor(getHumanInterceptor()).addPathPatterns("/", "/mobile");
} }
@@ -8,6 +8,11 @@ import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder; import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
import com.zaxxer.hikari.HikariDataSource;
import javax.sql.DataSource; import javax.sql.DataSource;
@@ -16,7 +21,9 @@ public class SqlConfiguration {
@Bean(name = "datasource-main") @Bean(name = "datasource-main")
@ConfigurationProperties(prefix = "spring.datasource-main") @ConfigurationProperties(prefix = "spring.datasource-main")
public DataSource dataSourceMain() { public DataSource dataSourceMain() {
return DataSourceBuilder.create().build(); DataSource dataSource = DataSourceBuilder.create().build();
applyBusyTimeout(dataSource);
return dataSource;
} }
@Bean(name = "datasource-cache") @Bean(name = "datasource-cache")
@@ -25,6 +32,18 @@ public class SqlConfiguration {
return DataSourceBuilder.create().build(); return DataSourceBuilder.create().build();
} }
/**
* 给主库连接打开 busy 等待。
*
* <p>主库是单文件 SQLite,任务生命周期会并发开启短事务(创建、状态回执、删除)。
* 默认没有 busy 超时,第二个写事务会立刻拿到 SQLITE_BUSY 并被当作 500;
* 打开等待后,写冲突会按序排队重试而不是失败。
*/
private void applyBusyTimeout(DataSource dataSource) {
if (dataSource instanceof HikariDataSource hikari)
hikari.setConnectionInitSql("PRAGMA busy_timeout=5000");
}
@Bean(name = "sqlSessionFactory-main") @Bean(name = "sqlSessionFactory-main")
public SqlSessionFactory sqlSessionFactoryMain(@Qualifier("datasource-main") DataSource dataSource) throws Exception { public SqlSessionFactory sqlSessionFactoryMain(@Qualifier("datasource-main") DataSource dataSource) throws Exception {
SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean(); SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean();
@@ -48,4 +67,30 @@ public class SqlConfiguration {
public SqlSessionTemplate sqlSessionTemplateCache(@Qualifier("sqlSessionFactory-cache") SqlSessionFactory sqlSessionFactory) { public SqlSessionTemplate sqlSessionTemplateCache(@Qualifier("sqlSessionFactory-cache") SqlSessionFactory sqlSessionFactory) {
return new SqlSessionTemplate(sqlSessionFactory); return new SqlSessionTemplate(sqlSessionFactory);
} }
/**
* 主库事务管理器。
*
* <p>必须显式绑定到 datasource-main:应用里有两个 DataSource,Spring Boot 的自动配置
* 会退避,若没有这个 bean 就没有任何事务管理器,@Transactional 会静默失效。
* @Primary 同时让「不指定名字就注入 TransactionManager」的旧代码继续可用。
*/
@Bean(name = "mainTransactionManager")
@Primary
public DataSourceTransactionManager mainTransactionManager(@Qualifier("datasource-main") DataSource dataSource) {
return new DataSourceTransactionManager(dataSource);
}
/**
* 主库编程式事务模板。
*
* <p>任务生命周期需要把「短事务」放在网络调用之外,并且要求「回调里捕获异常后不再抛出」
* 不能错误提交,因此调用方必须用 {@code execute} 的返回状态或再次抛出决定回滚,
* 而不是在回调里吞掉异常(见 GalleryTaskLifecycleService)。
*/
@Bean(name = "mainTransactionTemplate")
@Primary
public TransactionTemplate mainTransactionTemplate(@Qualifier("mainTransactionManager") DataSourceTransactionManager transactionManager) {
return new TransactionTemplate(transactionManager);
}
} }
@@ -0,0 +1,57 @@
package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Domain.TaskOperation;
import com.lion.lionwebsite.Service.GalleryTaskLifecycleService;
import com.lion.lionwebsite.Service.RemoteService;
import com.lion.lionwebsite.Service.TaskReportHandler;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
/**
* 装配协议 v2 的双向依赖,并做启动恢复。
*
* <p>RemoteService 消费上报时需要生命周期服务,而生命周期服务下发命令时需要 RemoteService。
* 直接互相注入构造参数会形成环,所以这里在两端都构造完成之后用 setter 接上;
* 同时也把「启动时把中断的 SUBMITTING 判为 UNKNOWN、释放 RESERVED 占位」这一步接到应用就绪事件上。
*
* <p>顺序是不可交换的:
* <ol>
* <li>先绑定上报处理器、下发判定解析器与 operationId 解析器;</li>
* <li>再执行启动恢复(SUBMITTING → UNKNOWN、释放 RESERVED);</li>
* <li>最后才允许建立连接与自动重投。连接由本类显式触发,不再由 RemoteService 的
* PostConstruct 在装配前发起,否则会把状态不明的任务盲目重投给节点。</li>
* </ol>
* 恢复失败时仍然允许连接(保持节点在线/心跳),但明确禁止自动派发。
*/
@Component
@Slf4j
@RequiredArgsConstructor
public class TaskLifecycleWiring {
final RemoteService remoteService;
final GalleryTaskLifecycleService lifecycleService;
@EventListener(ApplicationReadyEvent.class)
public void wire() {
TaskReportHandler handler = lifecycleService;
remoteService.setTaskReportHandler(handler);
remoteService.setDownloadOperationIdResolver(lifecycleService::currentOperationId);
remoteService.setDispatchResolver(lifecycleService::resolveDispatch);
boolean recoverySafe = false;
try {
int recovered = lifecycleService.recoverInterruptedSubmissions();
recoverySafe = true;
if (recovered > 0)
log.warn("启动恢复:{} 个中断的创建/预占任务已处理(提交中的标为提交未知)", recovered);
} catch (RuntimeException e) {
// 恢复失败说明库中任务真实状态未知:可以保持连接,但绝不能自动重投/派发。
log.error("启动恢复失败,本次启动将拒绝自动重投未完成任务", e);
}
// 绑定与恢复完成之后才连接:连接成功后的自动重投受 lifecycleReady/recoverySafe 双重约束。
remoteService.startAfterWiring(recoverySafe);
}
}
@@ -67,6 +67,14 @@ public class GalleryManageController {
return galleryManageService.deleteGalleryByGid(gid, AuthCode); return galleryManageService.deleteGalleryByGid(gid, AuthCode);
} }
/** 按 gid 查实际下载人昵称,仅管理员可用。 */
@GetMapping("/downloader")
public String selectDownloader(Integer gid, String AuthCode){
if(gid == null)
return Response._failure("参数不全");
return galleryManageService.selectDownloaderByGid(gid, AuthCode);
}
@PostMapping("/collect") @PostMapping("/collect")
public String collectGallery(Integer gid, String AuthCode){ public String collectGallery(Integer gid, String AuthCode){
@@ -2,17 +2,19 @@ package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.LocalService; import com.lion.lionwebsite.Service.LocalService;
import com.lion.lionwebsite.Service.PersonalService; import com.lion.lionwebsite.Service.PersonalService;
import com.lion.lionwebsite.Util.PanelLoginTicket;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import com.fasterxml.jackson.core.JsonProcessingException;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
import jakarta.servlet.http.HttpSession;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController; import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import java.io.IOException; import java.io.IOException;
@@ -23,59 +25,99 @@ import java.io.IOException;
@RequestMapping("/personal") @RequestMapping("/personal")
public class PersonalController { public class PersonalController {
/** 会话有效期,与 PersonalHub 的面板会话保持一致:14 天滑动过期。 */
private static final int SESSION_MAX_INACTIVE_SECONDS = 60 * 60 * 24 * 14;
final PersonalService personalService; final PersonalService personalService;
final LocalService localService; final LocalService localService;
final PanelLoginTicket panelLoginTicket;
@GetMapping("/") @GetMapping("/")
public void index(HttpServletResponse resp) throws IOException { public void index(HttpServletResponse resp) throws IOException {
resp.sendRedirect("/index"); resp.sendRedirect("/index");
} }
@GetMapping("/files") /**
public String file(String path){ * 一键登录:校验机器人签发的 HMAC 票据,通过后建立会话并跳到个人面板。
return personalService.getFiles(path); *
* <p>票据一次性之外的防护全靠有效期,因此失败的票据不重定向到登录页,
* 而是直接回到提示页,避免把参数回显到浏览器历史里。
*/
@GetMapping("/login")
public void login(@RequestParam(value = "t", required = false) String ticket,
HttpServletRequest request,
HttpServletResponse response) throws IOException {
if (!panelLoginTicket.verify(ticket)) {
log.warn("一键登录票据无效或已过期,来自 {}", clientIp(request));
// 重定向目标用浏览器可见路径:nginx 的 location / 会把它改写成 /personal/…
response.sendRedirect("/denied");
return;
}
// 先作废旧会话再建新的,避免会话固定攻击。
HttpSession existing = request.getSession(false);
if (existing != null)
existing.invalidate();
HttpSession session = request.getSession(true);
session.setAttribute("personalAuthenticated", Boolean.TRUE);
session.setMaxInactiveInterval(SESSION_MAX_INACTIVE_SECONDS);
log.info("一键登录成功,来自 {}", clientIp(request));
response.sendRedirect("/index");
} }
@PostMapping("/uploadFile") @GetMapping("/login/logout")
public String uploadFile(String path, String fileName, MultipartFile file){ public void logout(HttpServletRequest request, HttpServletResponse response) throws IOException {
return personalService.uploadFile(path, fileName, file); HttpSession session = request.getSession(false);
if (session != null)
session.invalidate();
response.sendRedirect("/denied");
} }
@GetMapping("/private/**") /** 未登录或票据失效时的提示页;纯静态文案,不含任何可推断的信息。 */
public void getFile(HttpServletRequest request, HttpServletResponse response, String path){ @GetMapping(value = "/denied", produces = MediaType.TEXT_HTML_VALUE)
personalService.download(request, response, path); public String denied() {
return """
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>需要登录 · LionWebsite</title>
<style>
body { margin:0; display:flex; min-height:100vh; align-items:center; justify-content:center;
background:#f6f7f9; color:#1f2328;
font-family:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",sans-serif; }
.card { max-width:26rem; padding:2rem; background:#fff; border:1px solid #e2e5e9;
border-radius:10px; box-shadow:0 1px 2px rgba(0,0,0,.04); }
h1 { margin:0 0 .75rem; font-size:1.15rem; }
p { margin:0; line-height:1.7; color:#4a5259; }
code { background:#f0f2f5; padding:.1rem .35rem; border-radius:4px; }
</style>
</head>
<body>
<div class="card">
<h1>需要登录</h1>
<p>请在机器人里发送 <code>/login</code>,用返回的一次性链接打开个人面板。</p>
</div>
</body>
</html>
""";
} }
@PostMapping("/share") private static String clientIp(HttpServletRequest request) {
public String shareFile(String path, Integer expireHour) { String forwarded = request.getHeader("X-Forwarded-For");
return personalService.shareFile(path, expireHour); if (forwarded != null && !forwarded.isBlank())
} return forwarded.split(",")[0].trim();
return request.getRemoteAddr();
@PostMapping("/compress")
public String compress(String path){
return personalService.compress(path);
}
@PostMapping("/delete")
public String deleteFile(String path){
return personalService.deleteFile(path);
}
@PostMapping("/extendShareTime")
public String extendShareTime(String path, Integer extendHour) {
return personalService.extendShareTime(path, extendHour);
}
@PostMapping("/cancelShare")
public String cancelShare(String path){
return personalService.cancelShare(path);
} }
@PostMapping("/updateSub") @PostMapping("/updateSub")
public String updateSub() throws IOException { public String updateSub() throws IOException {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
if(localService.updateSub(true)) if(localService.updateSub())
response.success(); response.success();
else else
response.failure(); response.failure();
@@ -88,7 +130,7 @@ public class PersonalController {
} }
@GetMapping("/ip") @GetMapping("/ip")
public String ip() throws JsonProcessingException { public String ip() {
return personalService.getIp(); return personalService.getIp();
} }
@@ -6,7 +6,10 @@ import com.lion.lionwebsite.Service.PublicService;
import com.lion.lionwebsite.Service.QueryService; import com.lion.lionwebsite.Service.QueryService;
import com.lion.lionwebsite.Service.RemoteService; import com.lion.lionwebsite.Service.RemoteService;
import com.lion.lionwebsite.Service.SubService; import com.lion.lionwebsite.Service.SubService;
import com.lion.lionwebsite.Service.UserService;
import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import tools.jackson.databind.node.ObjectNode;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
@@ -15,16 +18,12 @@ import org.springframework.web.bind.annotation.*;
import java.io.IOException; import java.io.IOException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.LinkedList;
import java.util.List;
@RestController @RestController
@Slf4j @Slf4j
@RequiredArgsConstructor @RequiredArgsConstructor
public class PublicController { public class PublicController {
final List<String> black_share_codes = new LinkedList<>();
final PublicService publicService; final PublicService publicService;
final RemoteService remoteService; final RemoteService remoteService;
@@ -59,32 +58,18 @@ public class PublicController {
subService.updateSub(response, request, client, key); subService.updateSub(response, request, client, key);
} }
@GetMapping("/GetFile/{path}")
public void getFile(HttpServletRequest request, HttpServletResponse response, String ShareCode, @PathVariable("path") String path) throws IOException {
synchronized (black_share_codes) {
if (black_share_codes.contains(ShareCode))
return;
}
log.info("ShareCode:{}", ShareCode);
log.info("Path:{}", path);
boolean result = publicService.GetFile(request, response, ShareCode);
if(!result)
black_share_codes.add(ShareCode);
if(black_share_codes.size() > 100)
black_share_codes.removeFirst();
}
@PostMapping("/validate") @PostMapping("/validate")
public String validate(String AuthCode){ public String validate(String AuthCode){
Response response = Response.generateResponse(); Response response = Response.generateResponse();
User user = publicService.getUserId(AuthCode); User user = publicService.getUserId(AuthCode);
String isAvailable = remoteService.isDead() ? "false": "true"; // 身份字段直接作为结构化 data 返回;共享前端解码器兼容旧版本字符串。
response.success(String.format("{\"userId\": %d, " + ObjectNode identity = CustomUtil.objectMapper.createObjectNode();
"\"username\": \"%s\", " + identity.put("userId", user.getId());
"\"isAvailable\": %s}", user.getId(), user.getUsername(), isAvailable)); identity.put("username", user.getUsername());
identity.put("isAvailable", !remoteService.isDead());
// 管理员标记随登录一起下发,前端据此决定是否显示下载人信息与筛选。
identity.put("isAdmin", user.getId() == UserService.ADMIN_USER_ID);
response.success(identity);
return response.toJSONString(); return response.toJSONString();
} }
@@ -0,0 +1,33 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.SubscriptionEventService;
import jakarta.servlet.http.HttpServletRequest;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;
import java.io.IOException;
import java.util.Map;
@RestController
public class SubscriptionEventController {
private final SubscriptionEventService events;
public SubscriptionEventController(SubscriptionEventService events) { this.events = events; }
@PostMapping(value = "/internal/subscription-events", produces = "application/json")
public ResponseEntity<Map<String, Object>> ingest(HttpServletRequest request) throws IOException {
// Bound actual bytes even for chunked requests and dishonest Content-Length headers.
byte[] body = request.getInputStream().readNBytes(SubscriptionEventService.MAX_BODY + 1);
if (body.length > SubscriptionEventService.MAX_BODY)
return ResponseEntity.badRequest().body(Map.of("error", "invalid subscription events"));
if (!events.verify(body, request.getHeader("X-Subscription-Timestamp"), request.getHeader("X-Subscription-Signature")))
return ResponseEntity.status(403).body(Map.of("error", "authentication failed"));
try {
return ResponseEntity.ok(events.ingest(events.parseBatch(body)));
} catch (IllegalArgumentException invalid) {
return ResponseEntity.badRequest().body(Map.of("error", "invalid subscription events"));
}
// All infrastructure failures propagate as 5xx; never acknowledge an uncommitted batch.
}
}
@@ -5,8 +5,11 @@ import com.lion.lionwebsite.Domain.ImageKeyCache;
import org.apache.ibatis.annotations.Insert; import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Mapper; import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param; import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Select; import org.apache.ibatis.annotations.Select;
import java.util.List;
@Mapper @Mapper
public interface ImageCacheMapper { public interface ImageCacheMapper {
@Insert("insert into ImageKeyCache values (#{gid}, #{imgkey}, #{page})") @Insert("insert into ImageKeyCache values (#{gid}, #{imgkey}, #{page})")
@@ -21,4 +24,41 @@ public interface ImageCacheMapper {
@Select("select * from gidToKey where gid=#{gid}") @Select("select * from gidToKey where gid=#{gid}")
GidToKey selectKeyByGid(String gid); GidToKey selectKeyByGid(String gid);
/** 已缓存的页 key 行数,用于判断索引是否完整(半截缓存需要重建)。 */
@Select("select count(*) from ImageKeyCache where gid=#{gid}")
int countImageKeyCacheByGid(String gid);
@Delete("delete from ImageKeyCache where gid=#{gid}")
void deleteImageKeyCacheByGid(String gid);
@Delete("delete from gidToKey where gid=#{gid}")
void deleteGidToKey(String gid);
/**
* 某个 gid 已落库的页码列表(升序)。
*
* <p>只按行数判断完整性会把「页码 1 和 3、共 2 行」当成完整索引,
* 因此完整性判定改为按页码集合比对——先取回页码,再校验它是否恰好是 1..pages。
*/
@Select("select page from ImageKeyCache where gid=#{gid} order by page")
List<Integer> selectPageNumbersByGid(String gid);
/** 该 gid 下 imgkey 为空的脏行数;任何一行脏行都说明索引不可用。 */
@Select("select count(*) from ImageKeyCache where gid=#{gid} and (imgkey is null or trim(imgkey)='')")
int countInvalidImageKeysByGid(String gid);
/**
* 批量写入页 key(单条多值 INSERT)。
*
* <p>整本画廊的索引可能上千页,逐行 INSERT 会拉长写事务并放大锁竞争;
* 调用方按固定大小分片后,一个写事务里只需少量语句即可写完。
*/
// MyBatis 只在注解值「以 <script> 开头」时才按 XML 动态 SQL 解析,因此这里不能有前导空白。
@Insert("<script>"
+ "insert into ImageKeyCache (gid, imgkey, page) values "
+ "<foreach collection='pages' item='p' separator=','>"
+ "(#{p.gid}, #{p.imgkey}, #{p.page})"
+ "</foreach>"
+ "</script>")
int insertImageKeyCacheBatch(@Param("pages") List<ImageKeyCache> pages);
} }
@@ -20,4 +20,8 @@ public interface CollectMapper {
@Select("select collector from collect where gid=#{gid}") @Select("select collector from collect where gid=#{gid}")
ArrayList<Integer> selectCollectorByGid(int gid); ArrayList<Integer> selectCollectorByGid(int gid);
/** 任务删除完成后清理该 gid 的全部收藏记录(与删任务行同一事务)。 */
@Delete("delete from collect where gid=#{gid}")
void deleteByGid(@Param("gid") int gid);
} }
@@ -16,6 +16,9 @@ public interface CustomConfigurationMapper {
// @Delete("delete from customConfiguration where parameter=#{parameter}") // @Delete("delete from customConfiguration where parameter=#{parameter}")
// void deleteConfiguration(CustomConfiguration configuration); // void deleteConfiguration(CustomConfiguration configuration);
@Update("update customConfiguration set value=cast(value as integer)+#{amount} where parameter=#{parameter}")
void incrementConfiguration(@Param("parameter") String parameter, @Param("amount") long amount);
@Select("select * from customConfiguration where parameter=#{parameter}") @Select("select * from customConfiguration where parameter=#{parameter}")
CustomConfiguration selectConfiguration(String parameter); CustomConfiguration selectConfiguration(String parameter);
} }
@@ -36,9 +36,10 @@ public interface GalleryMapper {
@Update(""" @Update("""
update gallery set name=#{name}, link=#{link}, language=#{language}, update gallery set name=#{name}, link=#{link}, language=#{language}, downloader=#{downloader},
pages=#{pages}, status=#{status}, fileSize=#{fileSize}, createTime=#{createTime}, pages=#{pages}, status=#{status}, fileSize=#{fileSize}, createTime=#{createTime},
proceeding=#{proceeding}, resolution=#{resolution}, displayFileSize=#{displayFileSize}, is_download=#{is_download} proceeding=#{proceeding}, resolution=#{resolution}, displayFileSize=#{displayFileSize},
thumb_link=#{thumb_link}, is_download=#{is_download}
where gid=#{gid}""") where gid=#{gid}""")
void updateGallery(Gallery gallery); void updateGallery(Gallery gallery);
@@ -47,4 +48,8 @@ public interface GalleryMapper {
@Delete("delete from gallery where gid=#{gid}") @Delete("delete from gallery where gid=#{gid}")
void deleteGalleryByGid(Integer gid); void deleteGalleryByGid(Integer gid);
/** 只改状态列,用于「待删除」这类不牵动其他字段的意图标记。 */
@Update("update gallery set status=#{status} where gid=#{gid}")
int updateGalleryStatus(@Param("gid") int gid, @Param("status") String status);
} }
@@ -1,31 +0,0 @@
package com.lion.lionwebsite.Dao.normal;
import com.lion.lionwebsite.Domain.ShareFile;
import org.apache.ibatis.annotations.*;
import java.util.ArrayList;
import java.util.Date;
@Mapper
public interface ShareFileMapper {
@Insert("insert into ShareFile (ShareCode, FilePath, ExpireTime) values (#{ShareCode}, #{FilePath}, #{ExpireTime})")
void insertShareFile(@Param("ShareCode")String ShareCode, @Param("FilePath")String FilePath, @Param("ExpireTime") Date ExpireTime);
@Select("select * from ShareFile where ShareCode=#{ShareCode}")
ShareFile selectShareFileByShareCode(String ShareCode);
@Select("select * from ShareFile where FilePath=#{FilePath}")
ShareFile selectShareFileByFilePath(String FilePath);
@Select("select * from ShareFile where FilePath like '%' || #{FilePath} || '%'")
ArrayList<ShareFile> selectShareFilesByFilePath(String FilePath);
@Delete("delete from ShareFile where ShareCode=#{ShareCode}")
void deleteShareFile(String ShareCode);
@Select("select * from ShareFile")
ShareFile[] selectAllShareFile();
@Update("update ShareFile set ExpireTime=#{ExpireTime}, ShareCode=#{ShareCode} where FilePath=#{FilePath}")
void updateShareFile(ShareFile ShareFile);
}
@@ -13,10 +13,10 @@ public interface SubMapper {
@Options(useGeneratedKeys = true, keyProperty = "id") @Options(useGeneratedKeys = true, keyProperty = "id")
void insertSubscriptionAccount(SubscriptionAccount account); void insertSubscriptionAccount(SubscriptionAccount account);
@Select("select id, name, upstream_key as upstreamKey, filter_high_multiplier as filterHighMultiplier, enabled, last_success_at as lastSuccessAt, last_error as lastError, created_at as createdAt, updated_at as updatedAt, (select count(*) from sub_bind sb where sb.subscription_account_id = sa.id) as boundUserCount from subscription_account sa order by id") @Select("select id, name, upstream_key as upstreamKey, filter_high_multiplier as filterHighMultiplier, enabled, last_success_at as lastSuccessAt, last_error as lastError, created_at as createdAt, updated_at as updatedAt, next_refresh_at as nextRefreshAt, last_success_epoch as lastSuccessEpoch, (select count(*) from sub_bind sb where sb.subscription_account_id = sa.id) as boundUserCount from subscription_account sa order by id")
ArrayList<SubscriptionAccount> selectAllSubscriptionAccounts(); ArrayList<SubscriptionAccount> selectAllSubscriptionAccounts();
@Select("select id, name, upstream_key as upstreamKey, filter_high_multiplier as filterHighMultiplier, enabled, last_success_at as lastSuccessAt, last_error as lastError, created_at as createdAt, updated_at as updatedAt, (select count(*) from sub_bind sb where sb.subscription_account_id = sa.id) as boundUserCount from subscription_account sa where id=#{id}") @Select("select id, name, upstream_key as upstreamKey, filter_high_multiplier as filterHighMultiplier, enabled, last_success_at as lastSuccessAt, last_error as lastError, created_at as createdAt, updated_at as updatedAt, next_refresh_at as nextRefreshAt, last_success_epoch as lastSuccessEpoch, (select count(*) from sub_bind sb where sb.subscription_account_id = sa.id) as boundUserCount from subscription_account sa where id=#{id}")
SubscriptionAccount selectSubscriptionAccount(Integer id); SubscriptionAccount selectSubscriptionAccount(Integer id);
@Select("select count(*) from subscription_account where name=#{name}") @Select("select count(*) from subscription_account where name=#{name}")
@@ -28,54 +28,72 @@ public interface SubMapper {
@Update("update subscription_account set name=#{name}, upstream_key=#{upstreamKey}, filter_high_multiplier=#{filterHighMultiplier}, enabled=#{enabled}, updated_at=CURRENT_TIMESTAMP, last_error=null where id=#{id}") @Update("update subscription_account set name=#{name}, upstream_key=#{upstreamKey}, filter_high_multiplier=#{filterHighMultiplier}, enabled=#{enabled}, updated_at=CURRENT_TIMESTAMP, last_error=null where id=#{id}")
void updateSubscriptionAccount(SubscriptionAccount account); void updateSubscriptionAccount(SubscriptionAccount account);
@Update("update subscription_account set last_success_at=CURRENT_TIMESTAMP, last_error=null, updated_at=CURRENT_TIMESTAMP where id=#{id}") // last_success_at 仍写 SQLite 的 UTC 文本时间,仅供页面显示;
void markSubscriptionRefreshSuccess(Integer id); // last_success_epoch 是调度判断用的权威时刻,两者口径不同,不要互相推导。
@Update("update subscription_account set last_success_at=CURRENT_TIMESTAMP, last_success_epoch=#{epoch}, last_error=null, updated_at=CURRENT_TIMESTAMP where id=#{id}")
void markSubscriptionRefreshSuccess(@Param("id") Integer id, @Param("epoch") long epoch);
@Update("update subscription_account set last_error=#{error}, updated_at=CURRENT_TIMESTAMP where id=#{id}") @Update("update subscription_account set last_error=#{error}, updated_at=CURRENT_TIMESTAMP where id=#{id}")
void markSubscriptionRefreshFailure(@Param("id") Integer id, @Param("error") String error); void markSubscriptionRefreshFailure(@Param("id") Integer id, @Param("error") String error);
/** 排定下一次刷新时刻;失败重试与次日分槽都走这里。 */
@Update("update subscription_account set next_refresh_at=#{nextRefreshAt}, updated_at=CURRENT_TIMESTAMP where id=#{id}")
void updateNextRefreshAt(@Param("id") Integer id, @Param("nextRefreshAt") long nextRefreshAt);
@Delete("delete from subscription_account where id=#{id}") @Delete("delete from subscription_account where id=#{id}")
void deleteSubscriptionAccount(Integer id); void deleteSubscriptionAccount(Integer id);
@Insert("insert into sub_bind (key, user, subscription_account_id) values (#{key}, #{user}, #{subscriptionAccountId})") @Insert("insert into sub_bind (key, user, subscription_account_id, user_id) values (#{key}, #{user}, #{subscriptionAccountId}, (select id from User where username=#{user}))")
void insertSubBind(SubBind subBind); void insertSubBind(SubBind subBind);
@Select("select sb.key, sb.user, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join subscription_account sa on sa.id=sb.subscription_account_id order by sb.user") @Select("select sb.key, coalesce(u.username, sb.user) as user, sb.user_id as userId, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join User u on u.id=sb.user_id left join subscription_account sa on sa.id=sb.subscription_account_id order by sb.user")
ArrayList<SubBind> selectAllSubBind(); ArrayList<SubBind> selectAllSubBind();
@Select("select sb.key, sb.user, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join subscription_account sa on sa.id=sb.subscription_account_id where sb.key=#{key}") @Select("select sb.key, coalesce(u.username, sb.user) as user, sb.user_id as userId, sb.subscription_account_id as subscriptionAccountId, sa.name as subscriptionAccountName, sa.enabled as subscriptionAccountEnabled, sa.filter_high_multiplier as filterHighMultiplier from sub_bind sb left join User u on u.id=sb.user_id left join subscription_account sa on sa.id=sb.subscription_account_id where sb.key=#{key}")
SubBind selectSubBind(String key); SubBind selectSubBind(String key);
@Select("select count(key) from sub_bind where key=#{key}") @Select("select count(key) from sub_bind where key=#{key}")
boolean selectSubBindExist(String key); boolean selectSubBindExist(String key);
@Select("select count(*) from sub_bind where user=#{user}") @Select("select count(*) from sub_bind where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
int countSubBindByUser(String user); int countSubBindByUser(String user);
@Update("update sub_bind set subscription_account_id=#{accountId} where user=#{user}") @Update("update sub_bind set subscription_account_id=#{accountId} where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
int updateSubBindAccount(@Param("user") String user, @Param("accountId") Integer accountId); int updateSubBindAccount(@Param("user") String user, @Param("accountId") Integer accountId);
@Update("update sub_bind set key=#{key} where user=#{user}") @Update("update sub_bind set key=#{key} where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
int updateSubBindKey(@Param("user") String user, @Param("key") String key); int updateSubBindKey(@Param("user") String user, @Param("key") String key);
@Delete("delete from sub_bind where user=#{user}") @Delete("delete from sub_bind where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
void deleteSubBind(String user); void deleteSubBind(String user);
@Insert("insert into sub_update_record (user, ip, UA, time, location) values (#{user}, #{ip}, #{UA}, #{time}, #{location})") @Insert("insert into sub_update_record (user, ip, UA, time, location, user_id) values (#{user}, #{ip}, #{UA}, #{time}, #{location}, coalesce(#{userId}, (select id from User where username=#{user})))")
@Options(useGeneratedKeys = true, keyProperty = "id")
void insertSubUpdateRecord(SubUpdateRecord subUpdateRecord); void insertSubUpdateRecord(SubUpdateRecord subUpdateRecord);
@Select("select * from sub_update_record order by time desc") @Select("select id,user,ip,UA,time,location,user_id as userId from sub_update_record order by time desc")
ArrayList<SubUpdateRecord> selectAllSubUpdateRecord(); ArrayList<SubUpdateRecord> selectAllSubUpdateRecord();
@Delete("delete from sub_update_record where user=#{user}") @Delete("delete from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
void deleteSubUpdateRecord(String user); void deleteSubUpdateRecord(String user);
@Select("select count(user) from sub_update_record where user=#{user}") @Select("select count(user) from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user})")
Integer selectUpdateRecordCount(String user); Integer selectUpdateRecordCount(String user);
@Select("select min(id) from sub_update_record where user=#{user}") @Select("select id from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user}) order by time asc,id asc limit 1")
Integer selectMinUpdateRecordId(String user); Integer selectMinUpdateRecordId(String user);
/** 后台补齐归属地:只更新该用户最近一条记录,避免覆盖更早的历史记录。 */
@Update("update sub_update_record set location=#{location} where id=(select max(id) from sub_update_record where user_id=(select id from User where username=#{user}) or (user_id is null and user=#{user}))")
void updateLatestSubUpdateRecordLocation(@Param("user") String user, @Param("location") String location);
@Update("update sub_update_record set location=#{location} where id=#{id}")
void updateSubUpdateRecordLocation(@Param("id") int id, @Param("location") String location);
@Update("update sub_bind set user=#{newName} where user_id=#{userId} or (user_id is null and user=#{oldName})")
void renameBoundUser(@Param("userId") int userId, @Param("oldName") String oldName,
@Param("newName") String newName);
@Delete("delete from sub_update_record where id=#{id}") @Delete("delete from sub_update_record where id=#{id}")
void deleteSubUpdateRecordById(int id); void deleteSubUpdateRecordById(int id);
} }
@@ -0,0 +1,35 @@
package com.lion.lionwebsite.Dao.normal;
import com.lion.lionwebsite.Domain.SubscriptionEvent;
import com.lion.lionwebsite.Domain.SubUpdateRecord;
import com.lion.lionwebsite.Domain.SubscriptionEventBinding;
import org.apache.ibatis.annotations.*;
@Mapper
public interface SubscriptionEventMapper {
/** Reserve SQLite's writer before the receipt read; a deferred read-to-write upgrade can fail immediately. */
@Update("update subscription_event_receipt set ingested_at=ingested_at where 0")
void reserveWriter();
@Select("select fingerprint from subscription_event_receipt where event_id=#{eventId}")
String fingerprint(String eventId);
@Insert("insert into subscription_event_receipt(event_id,key_hash,client,occurred_at,source,ingested_at,fingerprint,outcome) values(#{event.eventId},#{event.keyHash},#{event.client},#{event.occurredAt},#{source},#{ingestedAt},#{fingerprint},#{outcome})")
void insertReceipt(@Param("event") SubscriptionEvent event, @Param("source") String source,
@Param("ingestedAt") long ingestedAt, @Param("fingerprint") String fingerprint,
@Param("outcome") String outcome);
@Select("select key_hash as keyHash,user_id as userId,user_name as userName from subscription_event_binding where key_hash=#{hash}")
SubscriptionEventBinding binding(String hash);
@Insert("insert into subscription_event_binding(key_hash,user_id,user_name,first_seen_at,last_seen_at) values(#{hash},#{userId},#{userName},#{now},#{now}) on conflict(key_hash) do update set user_name=excluded.user_name,last_seen_at=excluded.last_seen_at where subscription_event_binding.user_id IS excluded.user_id")
void rememberBinding(@Param("hash") String hash, @Param("userId") Integer userId,
@Param("userName") String userName, @Param("now") long now);
@Insert("insert into sub_update_record(user,ip,UA,time,location,user_id) values(#{user},#{ip},#{UA},#{time},#{location},#{userId})")
@Options(useGeneratedKeys = true, keyProperty = "id")
void insertRecord(SubUpdateRecord record);
@Delete("delete from sub_update_record where (user_id=#{userId} or (user_id is null and user=#{userName})) and id not in (select id from sub_update_record where user_id=#{userId} or (user_id is null and user=#{userName}) order by time desc,id desc limit 10)")
void pruneRecords(@Param("userId") Integer userId, @Param("userName") String userName);
}
@@ -0,0 +1,107 @@
package com.lion.lionwebsite.Dao.normal;
import com.lion.lionwebsite.Domain.TaskOperation;
import org.apache.ibatis.annotations.*;
/**
* 任务操作记录的持久化入口(主库)。
*
* <p>所有写操作都在调用方的事务里执行;这里的注解只描述 SQL,不包含业务判断。
* 主键为 gid,保证「同一 gid 只能有一条有效操作意图」。
*/
@Mapper
public interface TaskOperationRepository {
@Select("select * from task_operation where gid=#{gid}")
TaskOperation selectByGid(int gid);
/** 生成该 gid 的稳定下载 operationId:只有还没有记录时才写入。 */
@Insert("insert or ignore into task_operation (gid, operationId, deleteOperationId, state, lastRevision," +
" lastReportId, updateTime) values (#{gid}, #{operationId}, null, #{state}, 0, null, #{updateTime})")
int insertIfAbsent(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("state") String state, @Param("updateTime") long updateTime);
@Insert("insert into task_operation (gid, operationId, deleteOperationId, state, lastRevision, lastReportId, updateTime)" +
" values (#{gid}, #{operationId}, #{deleteOperationId}, #{state}, #{lastRevision}, #{lastReportId}, #{updateTime})")
void insert(TaskOperation operation);
/** 推进创建意图的状态(RESERVED → SUBMITTING → SUBMITTED/UNKNOWN)。 */
@Update("update task_operation set state=#{state}, updateTime=#{updateTime} where gid=#{gid}")
int updateState(@Param("gid") int gid, @Param("state") String state, @Param("updateTime") long updateTime);
/**
* 带前置条件的创建状态推进:只有当前状态与 operationId 都匹配时才写入。
*
* <p>这是「提交前先确认这一代意图仍然有效」的 CAS:若期间被删除/重建,
* 返回值 0,调用方必须停下,绝不能再产生上游副作用。
*/
@Update("update task_operation set state=#{state}, updateTime=#{updateTime}" +
" where gid=#{gid} and operationId=#{operationId} and state=#{expectedState}")
int compareAndSetCreateState(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("expectedState") String expectedState, @Param("state") String state,
@Param("updateTime") long updateTime);
/**
* 只按当前删除意图 ID 匹配的删除完成:期间若已被删除/重建,则不改动新世代记录。
*/
@Update("update task_operation set deleteOperationId=null, state=#{state}, updateTime=#{updateTime}" +
" where gid=#{gid} and deleteOperationId=#{deleteOperationId}")
int completeDeleteIfSameGeneration(@Param("gid") int gid, @Param("deleteOperationId") String deleteOperationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/**
* 删除「本次创建意图」的预占记录:必须同时匹配 gid 与 operationId,且状态仍是
* RESERVED/SUBMITTING。
*
* <p>三重匹配保证释放流程只能清掉自己那一代:期间被删除/重建(新 operationId 或墓碑)
* 的记录一律不动,避免把新世代抹掉。
*/
@Delete("delete from task_operation where gid=#{gid} and operationId=#{operationId}" +
" and state in ('RESERVED','SUBMITTING')")
int deleteReservationIfSameGeneration(@Param("gid") int gid, @Param("operationId") String operationId);
/** 记录/替换当前下载 operationId 并推进状态。 */
@Update("update task_operation set operationId=#{operationId}, state=#{state}, updateTime=#{updateTime} where gid=#{gid}")
int updateOperationId(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/**
* 给「有记录但缺 operationId」的历史任务补一个稳定 operationId。
* 只在确实缺失时写入,避免覆盖并发创建出来的新世代。
*/
@Update("update task_operation set operationId=#{operationId}, updateTime=#{updateTime}" +
" where gid=#{gid} and operationId is null")
int assignOperationIdIfMissing(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("updateTime") long updateTime);
/**
* 墓碑复活:删除完成后的 gid 允许用全新 operationId 重新创建。
* revision 与回执一并清零,旧操作的迟到上报会因 operationId 不匹配被丢弃。
*/
@Update("update task_operation set operationId=#{operationId}, deleteOperationId=null, state=#{state}," +
" lastRevision=0, lastReportId=null, updateTime=#{updateTime} where gid=#{gid}")
int reactivate(@Param("gid") int gid, @Param("operationId") String operationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/** 记录删除意图与其稳定的删除操作 ID(重试必须复用同一个 ID)。 */
@Update("update task_operation set deleteOperationId=#{deleteOperationId}, state=#{state}, updateTime=#{updateTime} where gid=#{gid}")
int updateDeleteIntent(@Param("gid") int gid, @Param("deleteOperationId") String deleteOperationId,
@Param("state") String state, @Param("updateTime") long updateTime);
/**
* 写入状态上报回执(revision + reportId)。
*
* <p>回执与任务行更新在同一个事务里提交,提交成功之后才回 ACK;
* 因此节点重复投递同一份上报时,只能看到已经提交过的回执,不会重复落库或重复通知。
*/
@Update("update task_operation set lastRevision=#{revision}, lastReportId=#{reportId}, updateTime=#{updateTime}" +
" where gid=#{gid}")
int updateReportReceipt(@Param("gid") int gid, @Param("revision") long revision,
@Param("reportId") String reportId, @Param("updateTime") long updateTime);
@Select("select * from task_operation where state=#{state}")
java.util.List<TaskOperation> selectByState(String state);
@Delete("delete from task_operation where gid=#{gid}")
int deleteByGid(int gid);
}
@@ -14,10 +14,20 @@ public interface UserMapper {
@Select("select * from User where username=#{username}") @Select("select * from User where username=#{username}")
User selectUserByUsername(String username); User selectUserByUsername(String username);
@Select("select * from User where id=#{id}")
User selectUserById(int id);
@Select("select AuthCode from User") @Select("select AuthCode from User")
String[] selectAllAuthCode(); String[] selectAllAuthCode();
@Select("select AuthCode from User") /**
* 仅返回仍启用的授权码。
*
* <p>此前这条 SQL 与 {@link #selectAllAuthCode()} 完全相同(都没有 isEnable 条件),
* 于是「停用用户」后刷新授权码集合依旧把该码放行,isEnable 形同虚设。
* isEnable 为空按 DDL 默认值 true 处理,避免历史行被锁死。
*/
@Select("select AuthCode from User where isEnable is null or isEnable = 1")
String[] selectEnableAuthCode(); String[] selectEnableAuthCode();
@Select("select * from User") @Select("select * from User")
@@ -48,6 +48,11 @@ public class Gallery {
@JsonProperty("downloader") @JsonProperty("downloader")
private int downloader; //下载人 private int downloader; //下载人
/** 下载人昵称。仅管理员查询时填充,普通用户拿不到,避免暴露他人信息。 */
@JsonProperty("downloaderName")
@JsonInclude(JsonInclude.Include.NON_EMPTY)
private String downloaderName;
@JsonProperty("collector") @JsonProperty("collector")
@JsonInclude(JsonInclude.Include.NON_EMPTY) @JsonInclude(JsonInclude.Include.NON_EMPTY)
private String collector; //收藏人 private String collector; //收藏人
@@ -1,12 +0,0 @@
package com.lion.lionwebsite.Domain;
import lombok.Data;
import java.util.Date;
@Data
public class ShareFile {
String ShareCode;
String FilePath;
Date ExpireTime;
}
@@ -14,4 +14,10 @@ public class SubBind {
String subscriptionAccountName; String subscriptionAccountName;
boolean subscriptionAccountEnabled; boolean subscriptionAccountEnabled;
boolean filterHighMultiplier; boolean filterHighMultiplier;
Integer userId;
public SubBind(String key, String user, Integer accountId, String accountName,
boolean accountEnabled, boolean filterHighMultiplier) {
this(key, user, accountId, accountName, accountEnabled, filterHighMultiplier, null);
}
} }
@@ -16,4 +16,9 @@ public class SubUpdateRecord {
String UA; String UA;
Date time; Date time;
String location; String location;
Integer userId;
public SubUpdateRecord(int id, String user, String ip, String UA, Date time, String location) {
this(id, user, ip, UA, time, location, null);
}
} }
@@ -1,6 +1,5 @@
package com.lion.lionwebsite.Domain; package com.lion.lionwebsite.Domain;
import lombok.AllArgsConstructor;
import lombok.Data; import lombok.Data;
import lombok.NoArgsConstructor; import lombok.NoArgsConstructor;
@@ -8,7 +7,6 @@ import java.util.Date;
@Data @Data
@NoArgsConstructor @NoArgsConstructor
@AllArgsConstructor
public class SubscriptionAccount { public class SubscriptionAccount {
private Integer id; private Integer id;
private String name; private String name;
@@ -22,4 +20,12 @@ public class SubscriptionAccount {
private Integer boundUserCount; private Integer boundUserCount;
private String v2Url; private String v2Url;
private String clashUrl; private String clashUrl;
/**
* 下一次应刷新的时刻(Epoch 毫秒),由分散调度器排定。
* 刻意用整数而非 DATETIME:SQLite 文本时间戳会被 JDBC 按本地时区解释,
* 实测偏差 8 小时,不能用于时间比较。
*/
private Long nextRefreshAt;
/** 最近一次成功刷新的真实时刻(Epoch 毫秒),用于「24 小时内必刷一次」的判断与陈旧告警。 */
private Long lastSuccessEpoch;
} }
@@ -0,0 +1,16 @@
package com.lion.lionwebsite.Domain;
import lombok.AllArgsConstructor;
import lombok.Data;
/** Validated, authenticated event. Never contains the public subscription key. */
@Data
@AllArgsConstructor
public class SubscriptionEvent {
private String eventId;
private String keyHash;
private String client;
private String clientIp;
private String userAgent;
private long occurredAt;
}
@@ -0,0 +1,10 @@
package com.lion.lionwebsite.Domain;
import lombok.Data;
@Data
public class SubscriptionEventBinding {
private String keyHash;
private Integer userId;
private String userName;
}
@@ -0,0 +1,53 @@
package com.lion.lionwebsite.Domain;
import lombok.Data;
/**
* 单个 gid 的下载生命周期操作记录(协议 v2 的持久化意图)。
*
* <p>这张表是「先持久化意图、再产生外部副作用」的落点:
* <ul>
* <li>CREATE 意图先于上游 POST 落库,用来保证同一 gid 只有一个未完成意图;</li>
* <li>提交结果不确定时只能停在 {@link #STATE_UNKNOWN},绝不自动重复上游 POST;</li>
* <li>DELETE 意图先于节点删除落库,节点确认后(或文件已不存在)才在同一事务里删任务行并留墓碑。</li>
* </ul>
*
* <p>旧任务行没有对应操作记录,查询时按 gid 惰性补建(见 GalleryTaskLifecycleService)。
*/
@Data
public class TaskOperation {
/** 已登记创建意图,尚未向上游提交。 */
public static final String STATE_RESERVED = "RESERVED";
/** 正在向上游提交;此时不可再发第二次 POST。 */
public static final String STATE_SUBMITTING = "SUBMITTING";
/** 上游确认已接受下载请求。 */
public static final String STATE_SUBMITTED = "SUBMITTED";
/**
* 提交结果未知:超时、断连或在 SUBMITTING 期间重启。
* 上游不支持幂等,因此这里只能等对账(reconcile),不能自动重发。
*/
public static final String STATE_UNKNOWN = "UNKNOWN";
/** 已登记删除意图,等待节点确认。 */
public static final String STATE_DELETE_PENDING = "DELETE_PENDING";
/** 节点已确认删除,任务行也随之删除;本行作为墓碑保留,用于停止迟到状态的重放。 */
public static final String STATE_DELETED = "DELETED";
private int gid;
/** 当前有效的下载操作 ID;同一 gid 的 DOWNLOAD 重试必须复用它。 */
private String operationId;
/** 独立于下载操作 ID 的删除操作 ID,删除重试复用它。 */
private String deleteOperationId;
private String state;
/** 最近一次应用的状态上报 revision;用于丢弃乱序/重放的旧上报。 */
private long lastRevision;
/** 节点已确认收到的最新上报 ID;用于重复 ACK 的幂等判断。 */
private String lastReportId;
private long updateTime;
}
@@ -0,0 +1,44 @@
package com.lion.lionwebsite.Exception;
/**
* 图片索引构建失败。
*
* <p>调用方需要区分三类结果,才能给出正确的用户可见响应:
* 链接本身有问题({@link Reason#INVALID_LINK},属于请求错误)、
* 画廊已不存在({@link Reason#UNAVAILABLE},重试无意义)、
* 上游或数据库暂时不可用({@link Reason#TRANSIENT},可以重试)。
*/
public class GalleryIndexException extends Exception {
public enum Reason {
/** 链接无法解析出 gid 或 key。 */
INVALID_LINK,
/** 上游明确表示画廊不存在(下架/删除/解析不到页 key)。 */
UNAVAILABLE,
/** 上游返回的页 key 无法自洽(gid 不匹配、页码重复/跳号、imgkey 为空)。 */
MALFORMED,
/** 上游波动或索引写入失败,重试可能成功。 */
TRANSIENT
}
private final Reason reason;
public GalleryIndexException(Reason reason, String message) {
super(message);
this.reason = reason;
}
public GalleryIndexException(Reason reason, String message, Throwable cause) {
super(message, cause);
this.reason = reason;
}
public Reason getReason() {
return reason;
}
/** 重试没有意义(链接错误或画廊不可用);上游数据畸形与会话级故障仍可重试。 */
public boolean isPermanent() {
return reason == Reason.INVALID_LINK || reason == Reason.UNAVAILABLE;
}
}
@@ -7,11 +7,15 @@ import jakarta.servlet.http.HttpServletResponse;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import java.io.IOException; import java.io.IOException;
import java.util.Calendar; import java.time.LocalTime;
import java.time.format.DateTimeFormatter;
@WebFilter(filterName = "AdaptorFilter", urlPatterns = {"/", "/personal/"}) @WebFilter(filterName = "AdaptorFilter", urlPatterns = {"/", "/personal/"})
@Slf4j @Slf4j
public class AdaptorFilter implements Filter { public class AdaptorFilter implements Filter {
private static final DateTimeFormatter TIME_FORMATTER = DateTimeFormatter.ofPattern("HH:mm:ss");
@Override @Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException { public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
HttpServletRequest request = (HttpServletRequest) servletRequest; HttpServletRequest request = (HttpServletRequest) servletRequest;
@@ -22,14 +26,15 @@ public class AdaptorFilter implements Filter {
if(UserAgent == null) if(UserAgent == null)
return; return;
String AuthCode = request.getParameter("AuthCode") == null ? "null" : request.getParameter("AuthCode"); // 这里是全站请求日志,AuthCode 是真正的凭据;只记录是否携带,绝不落明文。
boolean hasAuthCode = request.getParameter("AuthCode") != null;
String ServletPath = request.getServletPath(); String ServletPath = request.getServletPath();
String ip = request.getHeader("X-Forwarded-For") == null ? request.getRemoteAddr(): request.getHeader("X-Forwarded-For"); String ip = request.getHeader("X-Forwarded-For") == null ? request.getRemoteAddr(): request.getHeader("X-Forwarded-For");
Calendar calendar = Calendar.getInstance(); String now = LocalTime.now().format(TIME_FORMATTER);
String now = String.format("%s:%s:%s", calendar.get(Calendar.HOUR_OF_DAY), calendar.get(Calendar.MINUTE), calendar.get(Calendar.SECOND));
//日志 //日志
log.info("{} ip:{} \tpath:{} \tAuthCode:{} ua:{}", now, ip, ServletPath, AuthCode, UserAgent.length() > 61 ? UserAgent.substring(0, 60): UserAgent); log.info("{} ip:{} \tpath:{} \tAuthCode:{} ua:{}", now, ip, ServletPath,
hasAuthCode ? "present" : "absent", UserAgent.length() > 61 ? UserAgent.substring(0, 60): UserAgent);
//如果是验证,则直接跳转 //如果是验证,则直接跳转
if(ServletPath.equals("/validate")) if(ServletPath.equals("/validate"))
@@ -37,7 +42,7 @@ public class AdaptorFilter implements Filter {
//如果不是,则根据UA判断是否跳转 //如果不是,则根据UA判断是否跳转
else if ((UserAgent.contains("Android") || UserAgent.contains("iPhone"))) else if ((UserAgent.contains("Android") || UserAgent.contains("iPhone")))
if (ServletPath.equals("/personal/") && AuthCode.equals("alone")) if (ServletPath.equals("/personal/") && "alone".equals(request.getParameter("AuthCode")))
response.sendRedirect("/mobile?AuthCode=alone"); response.sendRedirect("/mobile?AuthCode=alone");
else else
response.sendRedirect("/mobile"); response.sendRedirect("/mobile");
@@ -2,13 +2,40 @@ package com.lion.lionwebsite.Interceptor;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
import jakarta.servlet.http.HttpSession;
import org.springframework.web.servlet.HandlerInterceptor; import org.springframework.web.servlet.HandlerInterceptor;
/**
* 个人管理区的访问闸门:有效会话或合法 AuthCode。
*
* <p>会话来自机器人签发的登录链接(见 {@code PanelLoginTicket} 与
* {@code /personal/login});固定的 {@code alone} 授权码保留给下载器与存储节点推送,
* 因此这里两条路径都放行,等那些调用方切换完成后再退役字面量。
*/
public class PersonalInterceptor implements HandlerInterceptor { public class PersonalInterceptor implements HandlerInterceptor {
/** 会话标记:{@code /personal/login} 校验票据后写入。 */
public static final String SESSION_ATTRIBUTE = "personalAuthenticated";
@Override @Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler){ public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler){
return request.getParameter("AuthCode") != null && request.getParameter("AuthCode").equals("alone"); if (authenticatedSession(request))
return true;
String authCode = request.getParameter("AuthCode");
if (authCode != null && authCode.equals("alone"))
return true;
// 会话缺失或过期时回 401,前端据此提示「去机器人发 /login」而不是静默失败。
response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
return false;
}
/** 只认服务端写入的会话标记,不信任客户端可伪造的 Cookie 内容。 */
public static boolean authenticatedSession(HttpServletRequest request) {
// 已有会话才可能已登录;不调用 getSession(),以免为匿名访问创建空会话。
HttpSession session = request.getSession(false);
return session != null && Boolean.TRUE.equals(session.getAttribute(SESSION_ATTRIBUTE));
} }
} }
@@ -16,7 +16,9 @@ public class TaskHandlerInterceptor implements HandlerInterceptor {
final UserMapper userMapper; final UserMapper userMapper;
String[] AuthCodes; // 由 PostConstruct 加载后仅被读(发布式更新)与被测试线程读取;
// volatile 保证 refresh 后其它线程立即看到新数组,避免停用的授权码短暂仍可用。
volatile String[] AuthCodes;
@PostConstruct @PostConstruct
void init() { void init() {
@@ -5,7 +5,7 @@ import org.mybatis.spring.annotation.MapperScan;
import org.mybatis.spring.annotation.MapperScans; import org.mybatis.spring.annotation.MapperScans;
import org.springframework.boot.SpringApplication; import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletComponentScan; import org.springframework.boot.web.server.servlet.context.ServletComponentScan;
import org.springframework.scheduling.annotation.EnableScheduling; import org.springframework.scheduling.annotation.EnableScheduling;
@SpringBootApplication @SpringBootApplication
@@ -0,0 +1,319 @@
package com.lion.lionwebsite.Maintenance;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* 缓存库图片索引的离线体检与修复工具。
*
* <p>它是独立可执行类(只用 JDBC,不依赖 Spring / MyBatis),因此可以在服务停止后
* 直接对数据库文件运行,也可以先对生产库的副本演练。</p>
*
* <pre>
* java -cp sqlite-jdbc.jar GalleryIndexMaintenance.java --db /path/cache.db
* java -cp sqlite-jdbc.jar GalleryIndexMaintenance.java --db /path/cache.db --apply
* </pre>
*
* <p>它只做两类写入:删除不完整或有冲突的 gid 整组索引(gidToKey 头部加该 gid 的全部
* 页 key),以及删除没有头部记录的孤儿页 key。被删掉的 gid 由下一次在线阅读重新抓取,
* 因此不需要猜测保留哪一组 key。工具不建表、不改列、不碰其它业务表。</p>
*/
public final class GalleryIndexMaintenance {
/** 生产运行时数据库所在目录;默认拒绝直接写入,必须显式加 --allow-production。 */
private static final String PRODUCTION_DIR = "/home/lionwebsite";
private static final String PLAIN_REPAIR_NOTE =
"这些画廊将在下次被访问时重新抓取。";
public static void main(String[] args) {
Map<String, String> options = parseOptions(args);
if (options.containsKey("help") || !options.containsKey("db")) {
printUsage();
System.exit(options.containsKey("help") ? 0 : 2);
return;
}
Path database = Path.of(options.get("db")).toAbsolutePath().normalize();
boolean apply = options.containsKey("apply");
if (!Files.isRegularFile(database)) {
System.err.println("数据库不存在: " + database);
System.exit(2);
return;
}
if (apply && isProductionPath(database) && !options.containsKey("allow-production")) {
System.err.println("拒绝写入生产库路径 " + database
+ "。请先在副本上演练;确需在本机生产库执行时显式加 --allow-production,"
+ "并确保服务已停止。");
System.exit(3);
return;
}
try (Connection connection = open(database)) {
Report before = inspect(connection);
System.out.println(before.render(apply ? "修复前" : "体检结果"));
if (!apply) {
System.out.println();
System.out.println("dry-run:未做任何写入。确认无误后加 --apply 执行修复。");
System.exit(before.isHealthy() ? 0 : 1);
return;
}
if (before.isHealthy()) {
System.out.println();
System.out.println("索引已健康,无需修复。");
return;
}
connection.setAutoCommit(false);
int removed = repair(connection, before.badGids, before.orphanGids);
connection.commit();
System.out.println();
System.out.println("已删除 " + removed + " 个 gid 的索引整组," + PLAIN_REPAIR_NOTE);
Report after = inspect(connection);
System.out.println();
System.out.println(after.render("修复后"));
if (!after.isHealthy()) {
System.err.println("修复后仍未达到健康状态,请人工检查。");
System.exit(4);
}
} catch (SQLException e) {
System.err.println("执行失败: " + e.getMessage());
e.printStackTrace(System.err);
System.exit(5);
}
}
private static Connection open(Path database) throws SQLException {
Connection connection = DriverManager.getConnection("jdbc:sqlite:" + database);
try (Statement statement = connection.createStatement()) {
statement.execute("pragma busy_timeout=5000");
statement.execute("pragma foreign_keys=off");
}
return connection;
}
/** 是否位于生产运行时目录;工具据此拒绝在未显式确认时写入生产库。 */
static boolean isProductionPath(Path database) {
return database.startsWith(Path.of(PRODUCTION_DIR).toAbsolutePath().normalize());
}
// ---------- 体检 ----------
static Report inspect(Connection connection) throws SQLException {
Report report = new Report();
report.hasHeaderTable = tableExists(connection, "gidToKey");
report.hasPageTable = tableExists(connection, "ImageKeyCache");
if (!report.hasHeaderTable || !report.hasPageTable)
return report;
report.headerRows = count(connection, "select count(*) from gidToKey");
report.pageRows = count(connection, "select count(*) from ImageKeyCache");
report.duplicateHeaderGids = count(connection,
"select count(*) from (select gid from gidToKey group by gid having count(*) > 1)");
report.duplicatePageGroups = count(connection,
"select count(*) from (select gid, page from ImageKeyCache group by gid, page having count(*) > 1)");
try (PreparedStatement statement = connection.prepareStatement(BAD_GID_QUERY);
ResultSet rows = statement.executeQuery()) {
while (rows.next()) {
String gid = rows.getString(1);
report.badGids.add(gid);
report.badReasons.put(gid, describe(rows));
}
}
try (PreparedStatement statement = connection.prepareStatement(
"select distinct gid from ImageKeyCache where gid not in (select gid from gidToKey)");
ResultSet rows = statement.executeQuery()) {
while (rows.next())
report.orphanGids.add(rows.getString(1));
}
return report;
}
/**
* 一个 gid 触发的全部不健康条件集中在一句里检查,避免多次全表扫描。
*
* <p>必须按 gid 分组后再判断:重复 gid(同一 gid 有多条头部记录)本身就是要清除的对象,
* 而重复行里的某一条可能恰好与页 key 数一致,逐行判断会把它当成健康记录漏掉。
*/
private static final String BAD_GID_QUERY = """
select g.gid as gid,
count(*) as header_rows,
max(g.pages) as pages,
min(g.key) as key,
(select count(*) from ImageKeyCache i where i.gid = g.gid) as page_rows,
(select count(distinct i.page) from ImageKeyCache i where i.gid = g.gid) as distinct_pages,
(select coalesce(min(i.page), 0) from ImageKeyCache i where i.gid = g.gid) as min_page,
(select coalesce(max(i.page), 0) from ImageKeyCache i where i.gid = g.gid) as max_page,
(select count(*) from ImageKeyCache i where i.gid = g.gid
and (i.imgkey is null or trim(i.imgkey) = '')) as dirty_keys
from gidToKey g
group by g.gid
having count(*) > 1
or max(g.pages) is null or max(g.pages) <= 0
or min(g.key) is null or trim(min(g.key)) = ''
or (select count(*) from ImageKeyCache i where i.gid = g.gid) <> max(g.pages)
or (select count(distinct i.page) from ImageKeyCache i where i.gid = g.gid) <> max(g.pages)
or (select coalesce(min(i.page), 0) from ImageKeyCache i where i.gid = g.gid) <> 1
or (select coalesce(max(i.page), 0) from ImageKeyCache i where i.gid = g.gid) <> max(g.pages)
or (select count(*) from ImageKeyCache i where i.gid = g.gid
and (i.imgkey is null or trim(i.imgkey) = '')) > 0
order by g.gid
""";
private static String describe(ResultSet row) throws SQLException {
List<String> reasons = new ArrayList<>();
int headerRows = row.getInt("header_rows");
int pages = row.getInt("pages");
String key = row.getString("key");
int pageRows = row.getInt("page_rows");
int distinctPages = row.getInt("distinct_pages");
int minPage = row.getInt("min_page");
int maxPage = row.getInt("max_page");
int dirtyKeys = row.getInt("dirty_keys");
if (headerRows > 1) reasons.add("重复头部记录 " + headerRows + " 条");
if (key == null || key.isBlank()) reasons.add("key 为空");
if (pages <= 0) reasons.add("pages 非法(" + pages + ")");
if (pageRows != pages) reasons.add("页 key 行数 " + pageRows + " 与 pages " + pages + " 不符");
if (distinctPages != pages) reasons.add("页码去重后 " + distinctPages + " 个");
if (minPage != 1 || maxPage != pages) reasons.add("页码区间 " + minPage + "-" + maxPage);
if (dirtyKeys > 0) reasons.add("imgkey 为空 " + dirtyKeys + " 行");
return String.join("、", reasons);
}
// ---------- 修复 ----------
/** 删除不健康 gid 的整组索引与孤儿页 key,返回被移除索引的 gid 个数。 */
static int repair(Connection connection, List<String> badGids, List<String> orphanGids)
throws SQLException {
int touched = 0;
try (PreparedStatement pages = connection.prepareStatement("delete from ImageKeyCache where gid = ?");
PreparedStatement header = connection.prepareStatement("delete from gidToKey where gid = ?")) {
for (String gid : badGids) {
pages.setString(1, gid);
pages.executeUpdate();
header.setString(1, gid);
header.executeUpdate();
touched++;
}
for (String gid : orphanGids) {
pages.setString(1, gid);
if (pages.executeUpdate() > 0)
touched++;
}
}
return touched;
}
// ---------- 基础设施 ----------
private static boolean tableExists(Connection connection, String table) throws SQLException {
try (PreparedStatement statement = connection.prepareStatement(
"select count(*) from sqlite_master where type='table' and name=?")) {
statement.setString(1, table);
try (ResultSet rows = statement.executeQuery()) {
return rows.next() && rows.getInt(1) > 0;
}
}
}
private static int count(Connection connection, String sql) throws SQLException {
try (Statement statement = connection.createStatement();
ResultSet rows = statement.executeQuery(sql)) {
return rows.next() ? rows.getInt(1) : 0;
}
}
private static Map<String, String> parseOptions(String[] args) {
Map<String, String> options = new LinkedHashMap<>();
for (int i = 0; i < args.length; i++) {
String arg = args[i];
if (!arg.startsWith("--"))
continue;
String name = arg.substring(2);
if (name.equals("apply") || name.equals("help") || name.equals("allow-production")) {
options.put(name, "true");
continue;
}
if (i + 1 >= args.length) {
System.err.println("缺少参数值: " + arg);
System.exit(2);
}
options.put(name, args[++i]);
}
return options;
}
private static void printUsage() {
System.out.println("""
用法: java -cp <sqlite-jdbc.jar> GalleryIndexMaintenance.java --db <cache.db> [--apply] [--allow-production]
--db <path> 必填,缓存库文件
--apply 执行修复;不加则只做 dry-run 体检
--allow-production 允许写入 /home/lionwebsite 下的数据库(默认拒绝)
退出码: 0 健康或修复成功 / 1 存在待修复项(dry-run) / 2 参数错误 /
3 拒绝写生产路径 / 4 修复后仍不健康 / 5 执行异常
""");
}
private GalleryIndexMaintenance() { }
/** 一次体检结果。 */
static final class Report {
boolean hasHeaderTable;
boolean hasPageTable;
int headerRows;
int pageRows;
int duplicateHeaderGids;
int duplicatePageGroups;
final List<String> badGids = new ArrayList<>();
final List<String> orphanGids = new ArrayList<>();
final Map<String, String> badReasons = new LinkedHashMap<>();
boolean isHealthy() {
return hasHeaderTable && hasPageTable && badGids.isEmpty() && orphanGids.isEmpty();
}
String render(String title) {
StringBuilder out = new StringBuilder();
out.append("=== 图片索引").append(title).append("(").append(Instant.now()).append(")===\n");
if (!hasHeaderTable || !hasPageTable) {
out.append("缺少缓存表: gidToKey=").append(hasHeaderTable)
.append(" ImageKeyCache=").append(hasPageTable).append('\n');
return out.toString();
}
out.append("gidToKey 行数: ").append(headerRows).append('\n');
out.append("ImageKeyCache 行数: ").append(pageRows).append('\n');
out.append("重复 gid 组: ").append(duplicateHeaderGids).append('\n');
out.append("重复 (gid,page) 组: ").append(duplicatePageGroups).append('\n');
out.append("不完整或冲突 gid: ").append(badGids.size()).append('\n');
int limit = 20;
for (int i = 0; i < badGids.size() && i < limit; i++) {
String gid = badGids.get(i);
out.append(" - gid=").append(gid).append(' ')
.append(badReasons.getOrDefault(gid, "")).append('\n');
}
if (badGids.size() > limit)
out.append(" ... 其余 ").append(badGids.size() - limit).append(" 个略\n");
out.append("孤儿页 key 的 gid: ").append(orphanGids.size()).append('\n');
out.append(isHealthy() ? "结论: 健康\n" : "结论: 需要修复\n");
return out.toString();
}
}
}
@@ -21,6 +21,20 @@ public class AbstractMessage {
public static final byte SUBSCRIPTION_SNAPSHOT_MESSAGE = 9; public static final byte SUBSCRIPTION_SNAPSHOT_MESSAGE = 9;
// ---- 协议 v2(可靠任务)。旧类型 0..9 的字段保持不变,不能新增字段。 ----
/** 能力协商:{@code {protocolVersion: 2, features: ["reliable-task-v2"]}}。 */
public static final byte PROTOCOL_HELLO_MESSAGE = 10;
/** 带稳定 operationId 的任务命令(DOWNLOAD/DELETE)。 */
public static final byte TASK_COMMAND_MESSAGE = 11;
/** 带 reportId + revision 的状态上报,替代旧的无确认上报。 */
public static final byte TASK_REPORT_MESSAGE = 12;
/** 主站提交事务成功后的上报回执,节点据此清理快照。 */
public static final byte TASK_REPORT_ACK_MESSAGE = 13;
public byte messageType; public byte messageType;
public int messageId; public int messageId;
@@ -1,7 +1,7 @@
package com.lion.lionwebsite.Message; package com.lion.lionwebsite.Message;
import com.fasterxml.jackson.databind.ObjectMapper; import tools.jackson.databind.ObjectMapper;
import io.netty.buffer.ByteBuf; import io.netty.buffer.ByteBuf;
import io.netty.channel.ChannelHandlerContext; import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.ByteToMessageCodec; import io.netty.handler.codec.ByteToMessageCodec;
@@ -45,6 +45,10 @@ public class MessageCodec extends ByteToMessageCodec<AbstractMessage> {
case AbstractMessage.MAINTAIN_MESSAGE -> objectMapper.readValue(metadata, MaintainMessage.class); case AbstractMessage.MAINTAIN_MESSAGE -> objectMapper.readValue(metadata, MaintainMessage.class);
case AbstractMessage.AVAILABLE_CHECK_MESSAGE -> objectMapper.readValue(metadata, AvailableCheckMessage.class); case AbstractMessage.AVAILABLE_CHECK_MESSAGE -> objectMapper.readValue(metadata, AvailableCheckMessage.class);
case AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE -> objectMapper.readValue(metadata, SubscriptionSnapshotMessage.class); case AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE -> objectMapper.readValue(metadata, SubscriptionSnapshotMessage.class);
case AbstractMessage.PROTOCOL_HELLO_MESSAGE -> objectMapper.readValue(metadata, ProtocolHelloMessage.class);
case AbstractMessage.TASK_COMMAND_MESSAGE -> objectMapper.readValue(metadata, TaskCommandMessage.class);
case AbstractMessage.TASK_REPORT_MESSAGE -> objectMapper.readValue(metadata, TaskReportMessage.class);
case AbstractMessage.TASK_REPORT_ACK_MESSAGE -> objectMapper.readValue(metadata, TaskReportAckMessage.class);
default -> null; default -> null;
}; };
@@ -0,0 +1,30 @@
package com.lion.lionwebsite.Message;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.ArrayList;
import java.util.List;
/** 协议 v2 能力协商(类型 10)。双方都以对方声明了 version>=2 为前提才启用新消息。 */
@Data
@NoArgsConstructor
public class ProtocolHelloMessage extends AbstractMessage {
{
messageType = PROTOCOL_HELLO_MESSAGE;
}
public static final int PROTOCOL_V2 = 2;
public static final String FEATURE_RELIABLE_TASK_V2 = "reliable-task-v2";
private int protocolVersion = PROTOCOL_V2;
private List<String> features = new ArrayList<>(List.of(FEATURE_RELIABLE_TASK_V2));
/** 对端是否声明了 v2 可靠任务能力。 */
public boolean supportsReliableTasks() {
return protocolVersion >= PROTOCOL_V2
&& features != null && features.contains(FEATURE_RELIABLE_TASK_V2);
}
}
@@ -0,0 +1,29 @@
package com.lion.lionwebsite.Message;
import lombok.Data;
/**
* 任务命令(类型 11)。
*
* <p>只有确认对端支持 v2 之后才发送。同一逻辑操作在重试/重连后必须复用同一个
* {@link #operationId},节点据此返回原结果而不重复执行副作用。
*/
@Data
public class TaskCommandMessage extends AbstractMessage {
{
messageType = TASK_COMMAND_MESSAGE;
}
public static final String ACTION_DOWNLOAD = "DOWNLOAD";
public static final String ACTION_DELETE = "DELETE";
/** 稳定操作 ID:DOWNLOAD 与 DELETE 各自独立。 */
private String operationId;
private String action;
private int gid;
private String name;
}
@@ -0,0 +1,29 @@
package com.lion.lionwebsite.Message;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* 上报回执(类型 13)。
*
* <p>只在任务行与回执已经提交之后才发送;重复上报(同一 reportId)也会得到回执,
* 让节点停止重放。{@link #result} 0 表示已落库。
*/
@Data
@NoArgsConstructor
public class TaskReportAckMessage extends AbstractMessage {
{
messageType = TASK_REPORT_ACK_MESSAGE;
}
public static final byte RESULT_SUCCESS = 0;
private String reportId;
private byte result;
public TaskReportAckMessage(String reportId, byte result) {
this.reportId = reportId;
this.result = result;
}
}
@@ -0,0 +1,32 @@
package com.lion.lionwebsite.Message;
import com.lion.lionwebsite.Domain.GalleryTask;
import lombok.Data;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* 节点状态上报(类型 12)。
*
* <p>相比旧的无确认上报,这里带上单调递增的 {@link #revision} 与唯一 {@link #reportId}:
* 主站据此丢弃乱序/重放的旧快照,并在事务提交后回 {@link TaskReportAckMessage},
* 节点只有在收到匹配回执后才清理自己的快照。
*/
@Data
public class TaskReportMessage extends AbstractMessage {
{
messageType = TASK_REPORT_MESSAGE;
}
/** 本次上报的唯一 ID,重复投递同一份快照时保持不变。 */
private String reportId;
/** 节点侧单调递增版本号;主站只接受比已记录更大的 revision。 */
private long revision;
private GalleryTask[] galleryTasks;
/** gid → 产生该状态的下载 operationId,用于丢弃属于旧操作的迟到上报。 */
private Map<Integer, String> operationIds = new LinkedHashMap<>();
}
@@ -0,0 +1,262 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.cache.ImageCacheMapper;
import com.lion.lionwebsite.Domain.GidToKey;
import com.lion.lionwebsite.Domain.ImageKeyCache;
import com.lion.lionwebsite.Exception.GalleryIndexException;
import com.lion.lionwebsite.Util.GalleryUtil;
import com.lion.lionwebsite.Util.SingleFlight;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.transaction.support.TransactionOperations;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
/**
* 图片索引(gid 到每页 imgkey)的唯一读写入口。
*
* <p>这个类替换掉原先「读缓存 / 写缓存 / 回滚」分散在业务服务里、并且逐页 INSERT
* 的写法。三件事在这里被固定下来:
*
* <ol>
* <li><b>同 gid 合并</b>:并发请求同一本画廊时只有一个线程真的去抓上游,
* 其余线程等待同一个结果(含失败),因此不会出现两个构建者互相删除对方写入的索引。</li>
* <li><b>抓取在事务外</b>:上游 HTTP 可能耗时十几秒,绝不能占着 SQLite 写事务,
* 否则整个缓存库的写入都会排队。</li>
* <li><b>整组短事务发布</b>:一个事务里先删该 gid 的旧索引,再批量写入新索引。
* 失败整体回滚,磁盘上仍保留旧的完整索引;只有写入成功这一个状态是对外可见的。</li>
* </ol>
*
* <p>依赖全部由构造函数注入:mapper 负责落库,operations 负责事务边界,
* upstream 负责把 mpv 页解析成页 key 列表。测试可以直接给假的实现,
* 不需要起 Spring 上下文,也不需要网络。
*/
public class GalleryIndexService {
private static final Logger log = LoggerFactory.getLogger(GalleryIndexService.class);
/** 单条 INSERT 携带的最大行数,避免 SQL 过长。 */
static final int BATCH_SIZE = 200;
private final ImageCacheMapper mapper;
private final TransactionOperations operations;
private final TransactionOperations readOperations;
private final GalleryKeyUpstream upstream;
private final SingleFlight<String, GidToKey> builds = new SingleFlight<>();
public GalleryIndexService(ImageCacheMapper mapper, TransactionOperations operations,
GalleryKeyUpstream upstream) {
this(mapper, operations, operations, upstream);
}
/**
* @param operations 发布索引的写事务(删除加批量写入必须整体生效)
* @param readOperations 读取索引的一致性快照事务;传 null 表示不额外开启读事务
*/
public GalleryIndexService(ImageCacheMapper mapper, TransactionOperations operations,
TransactionOperations readOperations, GalleryKeyUpstream upstream) {
if (mapper == null || operations == null || upstream == null)
throw new IllegalArgumentException("mapper/operations/upstream 不能为空");
this.mapper = mapper;
this.operations = operations;
this.readOperations = readOperations;
this.upstream = upstream;
}
/**
* 保证该画廊的索引可用,返回 gid 到 key 的映射。
*
* @param url 画廊链接,形如 https://exhentai.org/g/1596929/f08534d87d/
* @throws GalleryIndexException 链接错误 / 画廊不可用 / 暂时失败,见 GalleryIndexException.Reason
*/
public GidToKey ensureIndex(String url) throws GalleryIndexException {
return ensureIndex(url, this::buildAndPublish);
}
/**
* 同 ensureIndex,但允许调用方提供构建动作。
*
* <p>离线重建需要「先检查再决定是否重建」的不同策略,因此把构建步骤作为参数暴露出来;
* 生产调用使用默认实现即可。
*/
GidToKey ensureIndex(String url, IndexBuilder builder) throws GalleryIndexException {
GalleryLink link = GalleryLink.parse(url);
GidToKey cached = lookup(link);
if (cached != null)
return cached;
try {
return builds.run(link.gid(), () -> {
// 进入临界区后必须再看一眼:等待期间别的线程可能已经把索引建好了。
GidToKey existing = lookup(link);
return existing != null ? existing : builder.build(url);
});
} catch (GalleryIndexException e) {
throw e;
} catch (Exception e) {
throw new GalleryIndexException(GalleryIndexException.Reason.TRANSIENT,
"构建图片索引失败 gid=" + link.gid(), e);
}
}
/**
* 已缓存且完整则返回索引,否则返回 null。
*
* <p>完整性由页码集合判定,而不是行数:历史重复行与半截写入都会让行数与页数不符。
*/
GidToKey lookup(GalleryLink link) {
if (readOperations != null)
return readOperations.execute(status -> lookupInTransaction(link));
return lookupInTransaction(link);
}
/**
* 在同一个读事务里读头部与页码,得到一个一致的快照。
*
* <p>两个独立的 SELECT 之间可能夹着另一个线程的整组替换,于是头部是旧的、页码是新的
* (或相反),完整性判断会得出错误结论。SQLite 的读事务能看到固定快照,避免这种交错。
*/
private GidToKey lookupInTransaction(GalleryLink link) {
GidToKey header = mapper.selectKeyByGid(link.gid());
if (header == null)
return null;
if (header.getPages() <= 0 || header.getKey() == null || header.getKey().isBlank())
return null;
if (!isComplete(header))
return null;
return header;
}
private boolean isComplete(GidToKey header) {
List<Integer> pages = mapper.selectPageNumbersByGid(header.getGid());
if (pages == null || pages.size() != header.getPages())
return false;
if (mapper.countInvalidImageKeysByGid(header.getGid()) != 0)
return false;
// 页码必须是严格递增的 1..pages,重复或跳号都视为不完整。
for (int i = 0; i < pages.size(); i++) {
Integer page = pages.get(i);
if (page == null || page != i + 1)
return false;
}
return true;
}
/** 抓上游(事务外)后短事务整组替换。 */
private GidToKey buildAndPublish(String url) throws GalleryIndexException {
GalleryLink link = GalleryLink.parse(url);
List<ImageKeyCache> pageKeys;
try {
// 抓取必须发生在事务之外:上游可能耗时十几秒,占着 SQLite 写事务会拖垮整个缓存库。
pageKeys = upstream.fetchPageKeys(url);
} catch (Exception e) {
throw new GalleryIndexException(GalleryIndexException.Reason.TRANSIENT,
"抓取图片索引失败 gid=" + link.gid(), e);
}
if (pageKeys == null || pageKeys.isEmpty())
throw new GalleryIndexException(GalleryIndexException.Reason.UNAVAILABLE,
"该图片已下架或已被删除 gid=" + link.gid());
// 落库前先校验上游数据自洽:整组替换一旦开始就没有回头路,
// 与其把畸形页码写进唯一约束再回滚,不如在此之前就判为失败。
validatePageKeys(link, pageKeys);
GidToKey header = new GidToKey();
header.setGid(link.gid());
header.setKey(link.key());
header.setPages(pageKeys.size());
publish(header, pageKeys);
return header;
}
/**
* 校验上游返回的页 key 是否自洽。
*
* <p>四条要求缺一不可,否则索引会以「看起来完整」的样子写进库:
* gid 属于请求的那本画廊、页码恰好是 1..N、页码不重复、imgkey 非空。
* 任一不满足都抛出 {@link GalleryIndexException.Reason#MALFORMED},此时尚未开启写事务,
* 库中旧索引保持原样。
*/
void validatePageKeys(GalleryLink link, List<ImageKeyCache> pageKeys) throws GalleryIndexException {
int expected = pageKeys.size();
Set<Integer> seen = new HashSet<>(expected * 2);
for (ImageKeyCache row : pageKeys) {
if (row == null || row.getGid() == null || !row.getGid().equals(link.gid()))
throw malformed(link, "上游页 key 的 gid 与请求不一致");
if (row.getImgkey() == null || row.getImgkey().isBlank())
throw malformed(link, "上游页 key 的 imgkey 为空");
int page = row.getPage();
if (page < 1 || page > expected)
throw malformed(link, "上游页码越界: " + page + "(共 " + expected + " 页)");
if (!seen.add(page))
throw malformed(link, "上游页码重复: " + page);
}
if (seen.size() != expected)
throw malformed(link, "上游页码不连续,去重后 " + seen.size() + " 个,期望 " + expected + " 个");
}
private GalleryIndexException malformed(GalleryLink link, String detail) {
log.warn("上游图片索引非法 gid={} detail={}", link.gid(), detail);
return new GalleryIndexException(GalleryIndexException.Reason.MALFORMED,
detail + " gid=" + link.gid());
}
/**
* 在单个事务里整组替换索引。
*
* <p>失败时事务整体回滚:要么看到旧的完整索引,要么看到新的完整索引,
* 不存在「头部已更新、页 key 缺失」的中间态。
*/
void publish(GidToKey header, List<ImageKeyCache> pageKeys) {
operations.executeWithoutResult(status -> {
mapper.deleteImageKeyCacheByGid(header.getGid());
mapper.deleteGidToKey(header.getGid());
mapper.insertGidToKey(header);
writeBatches(pageKeys);
});
log.info("图片索引已整体替换 gid={} pages={}", header.getGid(), header.getPages());
}
private void writeBatches(List<ImageKeyCache> pageKeys) {
for (int start = 0; start < pageKeys.size(); start += BATCH_SIZE) {
int end = Math.min(start + BATCH_SIZE, pageKeys.size());
mapper.insertImageKeyCacheBatch(new ArrayList<>(pageKeys.subList(start, end)));
}
}
/** 便于日志与错误信息使用的解析结果。 */
record GalleryLink(String gid, String key) {
static GalleryLink parse(String url) throws GalleryIndexException {
String[] segments = url == null ? null : url.split("/");
Integer gid = GalleryUtil.parseGid(url);
if (gid == null || segments == null || segments.length <= 5 || segments[5].isBlank())
throw new GalleryIndexException(GalleryIndexException.Reason.INVALID_LINK,
"链接错误");
return new GalleryLink(String.valueOf(gid), segments[5].strip());
}
}
/**
* 上游取页 key 的端口。
*
* <p>生产实现只是把 URL 转交给 GalleryUtil.parseImageKeys;
* 单独抽出来是为了让本服务在测试里完全脱离网络与静态工具类。
*/
@FunctionalInterface
public interface GalleryKeyUpstream {
List<ImageKeyCache> fetchPageKeys(String url) throws Exception;
}
/**
* 构建索引的动作。允许抛出业务异常,因此不用 java.util.function.Function
* (后者不能抛受检异常,会让调用方被迫包装成运行时异常)。
*/
@FunctionalInterface
interface IndexBuilder {
GidToKey build(String url) throws GalleryIndexException;
}
}
@@ -7,9 +7,11 @@ import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import com.lion.lionwebsite.Error.ErrorCode; import com.lion.lionwebsite.Error.ErrorCode;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.FileDownload; import com.lion.lionwebsite.Util.FileDownload;
import com.lion.lionwebsite.Util.ImageFileCache;
import java.nio.file.Path;
import com.lion.lionwebsite.Util.GalleryUtil; import com.lion.lionwebsite.Util.GalleryUtil;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import com.fasterxml.jackson.databind.ObjectMapper; import tools.jackson.databind.node.ObjectNode;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
import lombok.Data; import lombok.Data;
@@ -38,14 +40,22 @@ public class GalleryManageService {
final UserMapper userMapper; final UserMapper userMapper;
final ShareFileMapper shareFileMapper;
final ImageCacheMapper imageCacheMapper; final ImageCacheMapper imageCacheMapper;
final RemoteService remoteService; final RemoteService remoteService;
final PushService pushService; final PushService pushService;
/**
* 任务生命周期(协议 v2)。创建/删除都先走它做持久化意图,再产生外部副作用。
* 用字段注入是为了兼容现有构造式 mock 测试与旧调用点。
*/
@org.springframework.beans.factory.annotation.Autowired(required = false)
GalleryTaskLifecycleService lifecycleService;
@org.springframework.beans.factory.annotation.Autowired
GalleryIndexService galleryIndexService;
/** /**
* 创建任务 * 创建任务
* @param link 任务链接 * @param link 任务链接
@@ -58,14 +68,20 @@ public class GalleryManageService {
User user = userMapper.selectUserByAuthCode(AuthCode); User user = userMapper.selectUserByAuthCode(AuthCode);
// return Response._failure("调试中,请勿提交任务"); // return Response._failure("调试中,请勿提交任务");
int gid; if (user == null) {
try { response.failure("授权码无效");
gid = Integer.parseInt(link.split("/")[4]); return response.toJSONString();
} catch (NumberFormatException e) { }
// 段数不足会先抛 ArrayIndexOutOfBoundsException,非数字段抛 NumberFormatException;
// 只捕后者会让畸形链接穿透为 500(本项目无 @ControllerAdvice)。
Integer parsedGid = parseGidFromLink(link);
if (parsedGid == null) {
response.failure("链接错误"); response.failure("链接错误");
pushService.taskCreateReport(user.getUsername(), "未知任务", response); pushService.taskCreateReport(user.getUsername(), "未知任务", response);
return response.toJSONString(); return response.toJSONString();
} }
int gid = parsedGid;
String taskName = "任务 [" + gid + "]"; String taskName = "任务 [" + gid + "]";
if (remoteService.isDead()) { if (remoteService.isDead()) {
@@ -73,6 +89,20 @@ public class GalleryManageService {
pushService.taskCreateReport(user.getUsername(), taskName, response); pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString(); return response.toJSONString();
} }
// 协议 v2 生命周期:gid 唯一意图先落库(占位行「提交中」),提交前持久化 SUBMITTING,
// 上游结果不确定时报 UNKNOWN 而不是自动重发;差额额度与任务行在同一短事务里写入。
if (lifecycleService != null) {
response = lifecycleService.createTask(gid, taskName, link, targetResolution, user.getId());
Gallery current = galleryMapper.selectGalleryByGid(gid);
taskName = current == null || current.getName() == null ? taskName : current.getName();
// 业务结果由生命周期服务给出,data 用结构化对象下发(前端解码器兼容旧字符串)。
if (response.isSuccess() && current != null)
response.success(objectMapper.valueToTree(current));
pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString();
}
Gallery gallery; Gallery gallery;
//判断数据库中是否有这个任务,有则返回状态 //判断数据库中是否有这个任务,有则返回状态
@@ -93,9 +123,14 @@ public class GalleryManageService {
} else { } else {
taskName = gallery.getName(); taskName = gallery.getName();
log.info("创建任务: {} 目标分辨率:{}", link, targetResolution); log.info("创建任务: {} 目标分辨率:{}", link, targetResolution);
// Persist before dispatch: the node sends its current status before its ACK.
gallery.setDownloader(user.getId());
gallery.set_download(true);
galleryMapper.insertGallery(gallery);
configurationMapper.incrementConfiguration(CustomConfiguration.WEEK_USED_AMOUNT, gallery.getFileSize());
if (remoteService.addGalleryToQueue(gallery) != 0) { if (remoteService.addGalleryToQueue(gallery) != 0) {
log.error("传送任务{}失败, 未知原因", gallery.getName()); log.error("传送任务{}失败, 未知原因", gallery.getName());
response.failure("任务传送失败,未知原因,尝试点击重连按钮看看"); response.failure("任务已保存,但节点未确认接收;请刷新任务列表后重试");
pushService.taskCreateReport(user.getUsername(), taskName, response); pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString(); return response.toJSONString();
} }
@@ -111,25 +146,31 @@ public class GalleryManageService {
return response.toJSONString(); return response.toJSONString();
} }
//处理下载结果,将任务插入数据库并且更新每周用量 // Do not overwrite an immediate node status with the original submitted state.
if (gallery.getStatus().equals("已提交")) { Gallery current = galleryMapper.selectGalleryByGid(gallery.getGid());
response.success(gallery.toString()); response.success(objectMapper.valueToTree(current == null ? gallery : current));
gallery.setDownloader(user.getId());
gallery.set_download(true);
galleryMapper.insertGallery(gallery);
long usedAmount = Long.parseLong(configurationMapper.selectConfiguration(CustomConfiguration.WEEK_USED_AMOUNT).getValue());
usedAmount += gallery.getFileSize();
configurationMapper.updateConfiguration(CustomConfiguration.WEEK_USED_AMOUNT, String.valueOf(usedAmount));
} else {
response.failure("提交失败,未知原因");
galleryMapper.deleteGalleryByGid(gallery.getGid());
}
pushService.taskCreateReport(user.getUsername(), taskName, response); pushService.taskCreateReport(user.getUsername(), taskName, response);
return response.toJSONString(); return response.toJSONString();
} }
/**
* 从任务链接中安全提取 gid。任何畸形输入(null、段数不足、非数字)都返回 null,
* 由调用方转成业务失败,避免异常穿透为 500。
*/
static Integer parseGidFromLink(String link) {
if (link == null)
return null;
String[] segments = link.split("/");
if (segments.length <= 4)
return null;
try {
return Integer.parseInt(segments[4]);
} catch (NumberFormatException e) {
return null;
}
}
/** /**
* 尝试重新连接 * 尝试重新连接
* @return 重连结果 * @return 重连结果
@@ -167,7 +208,7 @@ public class GalleryManageService {
try { try {
gallery = GalleryUtil.parse(link, false, null); gallery = GalleryUtil.parse(link, false, null);
if (gallery != null) if (gallery != null)
response.success(new ObjectMapper().valueToTree(gallery).toString()); response.success(objectMapper.valueToTree(gallery));
else else
response.failure("查询失败"); response.failure("查询失败");
} catch (Exception e) { } catch (Exception e) {
@@ -176,7 +217,7 @@ public class GalleryManageService {
} }
else else
response.success(new ObjectMapper().valueToTree(gallery).toString()); response.success(objectMapper.valueToTree(gallery));
return response.toJSONString(); return response.toJSONString();
} }
@@ -195,7 +236,7 @@ public class GalleryManageService {
response.failure("未找到该图片,请使用链接"); response.failure("未找到该图片,请使用链接");
else else
response.success(gallery.toString()); response.success(objectMapper.valueToTree(gallery));
return response.toJSONString(); return response.toJSONString();
} }
@@ -215,26 +256,62 @@ public class GalleryManageService {
} }
ArrayList<Integer> galleryIds = collectMapper.selectGidByCollector(userId); ArrayList<Integer> galleryIds = collectMapper.selectGidByCollector(userId);
Iterator<Integer> idIterator; // 原先对每个画廊线性扫描收藏列表(O(n*m))并就地删除迭代器元素;
// 改成集合查找后同样是「命中即标记」,复杂度降到 O(n+m)。
if (!galleryIds.isEmpty()) {
Set<Integer> collected = new HashSet<>(galleryIds);
for (Gallery gallery : galleries)
if (collected.contains(gallery.getGid()))
gallery.setCollect(true);
}
if (!galleryIds.isEmpty()) //如果该用户收藏了图片 // 下载人昵称只对管理员下发;一次建表避免逐条查询。
galleryLoop:for (Gallery gallery : galleries) { //遍历图片 if (userId == UserService.ADMIN_USER_ID)
idIterator = galleryIds.iterator(); fillDownloaderNames(galleries);
while (idIterator.hasNext()) { //遍历收藏的gid
Integer id = idIterator.next();
if (id.equals(gallery.getGid())) { //如果找到对应的gid,修改对应图片的属性,删除当前gid,判断是否需要跳出或者结束循环
gallery.setCollect(true);
idIterator.remove();
if (galleryIds.isEmpty()) response.success(objectMapper.valueToTree(galleries));
break galleryLoop; return response.toJSONString();
else }
continue galleryLoop;
}
}
}
response.success(new ObjectMapper().valueToTree(galleries).toString()); /** 给任务列表补上下载人昵称,找不到用户时保留空值而不是报错。 */
private void fillDownloaderNames(Gallery[] galleries) {
User[] users = userMapper.selectAllUser();
Map<Integer, String> names = new HashMap<>();
if (users != null)
for (User user : users)
names.put(user.getId(), user.getUsername());
for (Gallery gallery : galleries)
gallery.setDownloaderName(names.get(gallery.getDownloader()));
}
/**
* 查询某个任务的实际下载人昵称。
* 只有管理员可调用,普通用户一律拒绝,避免暴露他人身份。
*
* @param gid 任务 gid
* @param AuthCode 调用方授权码
*/
public String selectDownloaderByGid(int gid, String AuthCode) {
Response response = Response.generateResponse();
User requester = userMapper.selectUserByAuthCode(AuthCode);
if (requester == null || requester.getId() != UserService.ADMIN_USER_ID) {
response.failure("无权查看下载人");
return response.toJSONString();
}
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null) {
response.failure("任务不存在");
return response.toJSONString();
}
User downloader = userMapper.selectUserById(gallery.getDownloader());
ObjectNode node = objectMapper.createObjectNode();
node.put("gid", gid);
node.put("downloader", gallery.getDownloader());
node.put("downloaderName", downloader == null ? "" : downloader.getUsername());
response.success(node);
return response.toJSONString(); return response.toJSONString();
} }
@@ -249,7 +326,7 @@ public class GalleryManageService {
Gallery gallery = galleryMapper.selectGalleryByName("%" + name + "%"); Gallery gallery = galleryMapper.selectGalleryByName("%" + name + "%");
if (gallery != null) if (gallery != null)
response.success(new ObjectMapper().valueToTree(gallery).toString()); response.success(objectMapper.valueToTree(gallery));
else else
response.failure("没有找到该名字的图片"); response.failure("没有找到该名字的图片");
@@ -267,7 +344,7 @@ public class GalleryManageService {
Gallery[] galleries = galleryMapper.selectGalleryByDownloader(userMapper.selectUserByAuthCode(AuthCode).getId()); Gallery[] galleries = galleryMapper.selectGalleryByDownloader(userMapper.selectUserByAuthCode(AuthCode).getId());
if (galleries.length > 0) if (galleries.length > 0)
response.success(new ObjectMapper().valueToTree(galleries).toString()); response.success(objectMapper.valueToTree(galleries));
else else
response.failure("您未下载图片"); response.failure("您未下载图片");
@@ -288,21 +365,73 @@ public class GalleryManageService {
response.failure("删除失败,该图片不存在"); response.failure("删除失败,该图片不存在");
return response.toJSONString(); return response.toJSONString();
} }
if (user == null) {
response.failure("删除失败,授权码无效");
return response.toJSONString();
}
ArrayList<Integer> collector = collectMapper.selectCollectorByGid(gallery.getGid()); ArrayList<Integer> collector = collectMapper.selectCollectorByGid(gallery.getGid());
if (!(collector.isEmpty() || collector.size() == 1 && collector.getFirst().equals(user.getId()) //判断收藏 // 拒绝条件与提示文案一致:被别人收藏,或者请求者不是下载人。
&& gallery.getDownloader() == user.getId())) //判断下载 // 注意不能写成 collector.isEmpty() || ...:那样在「无任何收藏」时会短路放行,
// 从而完全跳过下载者校验,导致任何有效授权码都能删除他人任务。
boolean collectedByOthers = collector.stream().anyMatch(id -> id != user.getId());
boolean isDownloader = gallery.getDownloader() == user.getId();
if (collectedByOthers || !isDownloader) {
response.failure("删除失败,该图片已被别人收藏或你不是下载人"); response.failure("删除失败,该图片已被别人收藏或你不是下载人");
else { log.info("拒绝删除 gid={}:collectedByOthers={} isDownloader={}", gid, collectedByOthers, isDownloader);
log.info("删除图片{}", gallery.getName()); return response.toJSONString();
galleryMapper.deleteGalleryByGid(gallery.getGid()); //删除图片记录
} }
// 通过授权后才产生外部副作用。协议 v2 的删除顺序:
// 先落「待删除」意图 → 节点确认(或文件已不存在)→ 才在同一事务删任务行并留墓碑。
// 失败时任务行与意图都保留,可沿原记录重试并复用同一个删除 operationId。
log.info("删除图片{}", gallery.getName());
if (lifecycleService != null) {
// 被拒绝(例如任务仍在提交中)时必须回业务失败:
// 绝不能退避到裸删除,否则会与并发的创建 POST 同时执行并写回已删/重建的 gid。
GalleryTaskLifecycleService.DeleteIntent intent = lifecycleService.beginDelete(gallery.getGid());
if (!intent.accepted()) {
response.failure(intent.message());
log.info("拒绝删除 gid={}:{}", gid, intent.message());
return response.toJSONString();
}
byte result = remoteService.sendDeleteCommand(intent.deleteOperationId(),
gallery.getGid(), gallery.getName());
switch (result) {
case 0 -> {
lifecycleService.completeDelete(gallery.getGid(), intent.deleteOperationId());
response.success();
}
// 文件已不存在等于目标达成:同样完成本地清理,避免留下永远删不掉的任务行。
case ErrorCode.FILE_NOT_FOUND -> {
lifecycleService.completeDelete(gallery.getGid(), intent.deleteOperationId());
response.success();
}
case ErrorCode.IO_ERROR -> {
lifecycleService.failDelete(gallery.getGid());
response.failure("图片:" + gallery.getName() + "删除失败,IO错误");
}
// 节点无响应/超时:保留记录与待删除意图,可重试。
default -> {
lifecycleService.failDelete(gallery.getGid());
response.failure("图片:" + gallery.getName() + "删除失败,节点无响应");
}
}
if (!response.isSuccess())
log.info(response.getData());
return response.toJSONString();
}
galleryMapper.deleteGalleryByGid(gallery.getGid());
switch (remoteService.deleteGallery(gallery)) { switch (remoteService.deleteGallery(gallery)) {
case ErrorCode.IO_ERROR -> response.failure("图片:" + gallery.getName() + "删除失败,IO错误"); case ErrorCode.IO_ERROR -> response.failure("图片:" + gallery.getName() + "删除失败,IO错误");
case ErrorCode.FILE_NOT_FOUND -> response.failure("图片:" + gallery.getName() + "删除失败,文件不存在"); case ErrorCode.FILE_NOT_FOUND -> response.failure("图片:" + gallery.getName() + "删除失败,文件不存在");
case 0 -> response.success(); case 0 -> response.success();
// 节点无响应/超时会返回 -1 等非枚举值;必须显式判失败,
// 否则 result 保持未设置,末尾的 response.get("result") 会抛 NPE。
default -> response.failure("图片:" + gallery.getName() + "删除失败,节点无响应");
} }
if (response.get("result").equals("failure")) if (!response.isSuccess())
log.info(response.getData()); log.info(response.getData());
return response.toJSONString(); return response.toJSONString();
} }
@@ -317,110 +446,75 @@ public class GalleryManageService {
CustomConfiguration lastResetAmountTime = configurationMapper.selectConfiguration(CustomConfiguration.LAST_RESET_AMOUNT_TIME); CustomConfiguration lastResetAmountTime = configurationMapper.selectConfiguration(CustomConfiguration.LAST_RESET_AMOUNT_TIME);
Map<String, String> data = new HashMap<>(); Map<String, String> data = new HashMap<>();
data.put("weekUsedAmount", CustomUtil.fileSizeToString(Long.parseLong(weekUsedAmount.getValue()))); // 配置行缺失时给出默认值,避免 NPE 让用量接口整体不可用。
data.put("lastResetAmountTime", lastResetAmountTime.getValue()); String usedValue = weekUsedAmount == null || weekUsedAmount.getValue() == null
? "0" : weekUsedAmount.getValue();
long used;
try {
used = Long.parseLong(usedValue);
} catch (NumberFormatException e) {
log.warn("每周用量配置值非法,按 0 处理: {}", usedValue);
used = 0L;
}
data.put("weekUsedAmount", CustomUtil.fileSizeToString(used));
data.put("lastResetAmountTime",
lastResetAmountTime == null || lastResetAmountTime.getValue() == null
? "" : lastResetAmountTime.getValue());
response.success(new ObjectMapper().valueToTree(data).toString()); response.success(objectMapper.valueToTree(data));
return response.toJSONString(); return response.toJSONString();
} }
public String cacheImagesKey(String url) { public String cacheImagesKey(String url) {
Response response = Response.generateResponse();
String gid = String.valueOf(GalleryUtil.parseGid(url));
GidToKey gidToKey = imageCacheMapper.selectKeyByGid(gid);
//已缓存过,直接返回
if(gidToKey != null) {
return response.success(objectMapper.valueToTree(gidToKey)).toJSONString();
}
try { try {
gidToKey = new GidToKey(); GidToKey index = galleryIndexService.ensureIndex(url);
gidToKey.setGid(gid); return Response.generateResponse().success(objectMapper.valueToTree(index)).toJSONString();
gidToKey.setKey(url.split("/")[5].strip()); } catch (com.lion.lionwebsite.Exception.GalleryIndexException error) {
ArrayList<ImageKeyCache> imageKeyCaches = GalleryUtil.parseImageKeys(url); String message = switch (error.getReason()) {
if(imageKeyCaches == null) case INVALID_LINK -> "链接错误";
return response.failure("该图片已下架或已被删除").toJSONString(); case UNAVAILABLE -> "该图片已下架或已被删除";
gidToKey.setPages(imageKeyCaches.size()); case MALFORMED, TRANSIENT -> "网络波动或其他异常";
imageCacheMapper.insertGidToKey(gidToKey); };
for (ImageKeyCache imageKeyCache : imageKeyCaches) log.warn("缓存图片索引失败 reason={}", error.getReason());
imageCacheMapper.insertImageKeyCache(imageKeyCache); return Response._failure(message);
response.success(objectMapper.valueToTree(gidToKey)); } catch (RuntimeException error) {
}catch (IOException e){ log.warn("缓存图片索引失败 errorType={}", error.getClass().getSimpleName());
log.error(e.getMessage()); return Response._failure("网络波动或其他异常");
response.failure("网络波动或其他异常");
} }
return response.toJSONString();
} }
String[] suffixes = {".avif", ".gif"};
public Callable<?> getCachedImage(String gid, Integer page, HttpServletRequest request, HttpServletResponse response) { public Callable<?> getCachedImage(String gid, Integer page, HttpServletRequest request, HttpServletResponse response) {
//检查文件夹是否存在 Path directory = Path.of(cachePath, gid);
File folder = new File(cachePath + gid); String name = String.valueOf(page);
if(!folder.isDirectory()) Path cached = ImageFileCache.find(directory, name);
folder.mkdirs(); if (cached != null) {
FileDownload.export(request, response, cached.toString());
//检查对应图片是否存在,存在则直接返回 return null;
for (String suffix : suffixes) {
if(new File(cachePath + gid + "/" + page + suffix).exists()){
FileDownload.export(request, response, cachePath + gid + "/" + page + suffix);
return null;
}
} }
//检查该图片缓存是否存在
GidToKey gidToKey = imageCacheMapper.selectKeyByGid(gid);
if(gidToKey == null)
try {
log.error("未缓存gid:{}", gid);
response.sendError(404);
return null;
}catch (IOException e){
log.warn("sendError 404 failed", e);
return null;
}
return () -> { return () -> {
if(response.isCommitted()) { if (response.isCommitted()) return null;
log.info("连接已关闭: gid={} page={}", gid, page); try {
return null; Path image = ImageFileCache.get(directory, name, () -> {
} GidToKey gidToKey = imageCacheMapper.selectKeyByGid(gid);
ImageKeyCache imageKey = imageCacheMapper.selectImageKeyCacheByGidAndPage(gid, page);
String imageUrl = null; if (gidToKey == null || imageKey == null)
//获取该图片key throw new IOException("图片索引不存在");
ImageKeyCache imageKeyCache = imageCacheMapper.selectImageKeyCacheByGidAndPage(gid, page); for (int attempt = 0; attempt < 2; attempt++) {
if (imageKeyCache == null) { String url = GalleryUtil.getImageUrl(getMpvKey(gidToKey.toUrl()), imageKey);
CustomUtil.fourZeroFour(response); if (url != null) return url;
return null; GalleryUtil.refreshMpvKey(gidToKey.toUrl());
} }
throw new IOException("无法获取图片地址");
//获取图片地址 });
for (int i = 0; i < 2; i++) { FileDownload.export(request, response, image.toString());
imageUrl = GalleryUtil.getImageUrl(getMpvKey(gidToKey.toUrl()), imageKeyCache); } catch (InterruptedException e) {
if (imageUrl != null) Thread.currentThread().interrupt();
break; if (!response.isCommitted()) response.sendError(503);
GalleryUtil.refreshMpvKey(gidToKey.toUrl()); } catch (Exception e) {
} log.warn("获取在线图片失败: gid={} page={} errorType={}", gid, page, e.getClass().getSimpleName());
if (!response.isCommitted()) response.sendError(404);
if (imageUrl == null) { }
CustomUtil.fourZeroFour(response); return null;
log.error("获取图片url失败:gid={} page={} imageKey={}", gid, page, imageKeyCache.getImgkey());
return null;
}
//下载图片,转格式并返回
String suffix = imageUrl.substring(imageUrl.lastIndexOf("."));
String imagePath = cachePath + gid + "/" + page + suffix;
try {
new URI(imageUrl).toURL().openConnection().getInputStream().transferTo(new FileOutputStream(imagePath));
}catch (Exception e){
log.error("下载图片失败:url{}", imageUrl, e);
CustomUtil.fourZeroFour(response);
return null;
}
if (!suffix.equals(".gif"))
imagePath = GalleryUtil.convertImg(imagePath, suffix);
FileDownload.export(request, response, imagePath);
return null;
}; };
} }
@@ -434,10 +528,24 @@ public class GalleryManageService {
Gallery[] galleries = galleryMapper.selectUnDoneGalleries(); Gallery[] galleries = galleryMapper.selectUnDoneGalleries();
if(galleries != null && galleries.length != 0) { if(galleries != null && galleries.length != 0) {
log.info("发送未下载完成图片至节点,{}本", galleries.length); // 只重投「确实已提交上游」的任务:提交未知/预占中/待删除的任务重投会变成
for (Gallery gallery : galleries) // 未计量副作用(节点侧显示下载完成但没有任何额度记录)。
int sent = 0;
int skipped = 0;
for (Gallery gallery : galleries) {
if (lifecycleService != null
&& !lifecycleService.resolveDispatch(gallery.getGid()).dispatchable()) {
skipped++;
continue;
}
remoteService.addGalleryToQueue(gallery); remoteService.addGalleryToQueue(gallery);
response.success(String.format("发送未下载完成图片至节点,%s本", galleries.length)); sent++;
}
log.info("发送未下载完成图片至节点,{}本(跳过需要对账的{}本)", sent, skipped);
if (sent == 0)
response.failure("没有可安全重投的任务,需先人工对账提交未知的任务");
else
response.success(String.format("发送未下载完成图片至节点,%s本", sent));
}else{ }else{
response.failure("当前没有未下载完成的图片"); response.failure("当前没有未下载完成的图片");
} }
@@ -453,6 +561,13 @@ public class GalleryManageService {
return Response._success("下载完成"); return Response._success("下载完成");
if(remoteService.isDead()) if(remoteService.isDead())
return Response._failure("节点不在线,无法重试"); return Response._failure("节点不在线,无法重试");
// 提交结果未知(UNKNOWN)或正在删除的任务不能重试:上游不支持幂等,重发会让
// 「可能已受理但未计量」的任务在节点侧变成下载完成却 0 额度。这类任务需人工对账。
if (lifecycleService != null) {
var decision = lifecycleService.resolveDispatch(gid);
if (decision.kind() == GalleryTaskLifecycleService.DispatchDecision.Kind.REFUSE)
return Response._failure("该任务提交结果未知或正在删除,请先人工对账后再重试");
}
RemoteService.RetryResult retryResult = remoteService.retryGallery(gallery); RemoteService.RetryResult retryResult = remoteService.retryGallery(gallery);
if(retryResult.success()) if(retryResult.success())
@@ -0,0 +1,783 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.*;
import com.lion.lionwebsite.Domain.*;
import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import com.lion.lionwebsite.Message.TaskCommandMessage;
import com.lion.lionwebsite.Message.TaskReportMessage;
import com.lion.lionwebsite.Util.PreparedGallery;
import com.lion.lionwebsite.Util.Response;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionTemplate;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
/**
* 下载任务的持久化生命周期(协议 v2)。
*
* <p>核心约束:
* <ol>
* <li><b>先保存意图,再产生外部副作用</b>:同一 gid 的唯一 CREATE 意图先落库(占位任务行 + RESERVED 操作记录),
* 通过 gid 的主键唯一约束把并发创建收敛成「一个赢家」;</li>
* <li><b>SUBMITTING 之后绝不自动重发</b>:上游不支持幂等,提交结果不确定时只能落 UNKNOWN,等显式对账/解决;</li>
* <li><b>提交结果与额度用一个短事务写入</b>:Gallery 与 weekUsedAmount 要么一起成功,要么一起回滚,
* 事务提交之后才把 DOWNLOAD 命令下发给节点;</li>
* <li><b>删除失败保留记录</b>:先落待删除意图,节点确认(或文件已不存在)后才在同一事务里删任务行并留墓碑;
* 失败则保留 Gallery 与 DELETE_PENDING 意图,可沿原记录重试且复用同一删除 operationId。</li>
* </ol>
*
* <p>所有网络调用都发生在事务之外;事务回调内部不做 I/O,也不吞异常(吞掉会错误提交)。
*/
@Service
@Slf4j
public class GalleryTaskLifecycleService implements TaskReportHandler {
final GalleryMapper galleryMapper;
final CollectMapper collectMapper;
final CustomConfigurationMapper configurationMapper;
final UserMapper userMapper;
final TaskOperationRepository taskOperationRepository;
final UpstreamGateway upstreamGateway;
final RemoteService remoteService;
/**
* 短全局创建预占锁。
*
* <p>只有「登记唯一 CREATE 意图」这一小段数据库写入在锁内,网络调用一律在锁外。
* 单进程内它把并发同 gid 创建直接收敛成一个赢家;跨进程的正确性仍由
* {@code task_operation}/{@code gallery} 的主键唯一约束兜底。
*/
private final Object reservationLock = new Object();
/**
* 测试钩子:进入上游只读解析之前执行。
* 用来让并发删除发生在预占之后、SUBMITTING 之前。生产环境是空实现。
*/
volatile Runnable beforePrepare = () -> { };
/**
* 测试钩子:SUBMITTING 已持久化之后、真正 POST 上游之前执行。
* 用来制造「提交瞬间任务被删除」的 late generation 场景。生产环境是空实现。
*/
volatile Runnable afterMarkSubmitting = () -> { };
/** 主库事务模板;测试可注入内存 sqlite 的事务模板。 */
TransactionTemplate transactions;
public GalleryTaskLifecycleService(GalleryMapper galleryMapper,
CollectMapper collectMapper,
CustomConfigurationMapper configurationMapper,
UserMapper userMapper,
TaskOperationRepository taskOperationRepository,
UpstreamGateway upstreamGateway,
RemoteService remoteService,
@Qualifier("mainTransactionTemplate") TransactionTemplate transactions) {
this.galleryMapper = galleryMapper;
this.collectMapper = collectMapper;
this.configurationMapper = configurationMapper;
this.userMapper = userMapper;
this.taskOperationRepository = taskOperationRepository;
this.upstreamGateway = upstreamGateway;
this.remoteService = remoteService;
this.transactions = transactions;
}
/** 测试用:替换事务模板或在没有 Spring 事务时退化为直连执行。 */
void setTransactions(TransactionTemplate transactions) {
this.transactions = transactions;
}
/**
* 一次「登记创建意图」的结果。
*
* <p>只有赢家能继续做外部副作用;输家(已存在有效意图 / 数据库竞争失败)
* 一律转成业务失败,绝不再 POST 上游。
*/
private enum ReserveOutcome {
/** 已成功登记唯一 CREATE 意图,可以做只读解析与提交。 */
RESERVED,
/** 同一 gid 已存在有效意图(或任务行)。 */
DUPLICATE,
/** 登记时的数据库竞争/失败:没有产生任何副作用,必须转业务失败。 */
CONTENDED
}
/** 一次「登记创建意图」的返回:结果 + 赢家持有的操作记录。 */
private record Reservation(ReserveOutcome outcome, TaskOperation operation) {
}
/**
* 单次下发决策。
*
* <p>把「能不能把这个任务交给节点」从「下载是否已提交」里拆出来:
* 未确认上游提交结果(UNKNOWN / 预占 / 待删除 / 删除墓碑)绝不能下发,
* 否则未计量的人工对账任务可能在节点侧变成「下载完成却 0 额度」。
*/
public static final class DispatchDecision {
public enum Kind {
/** 当前 operation 已是 SUBMITTED:可以按稳定 operationId 下发。 */
READY,
/** 迁移前的旧任务行:没有操作记录但历史上确实提交过下载。 */
LEGACY,
/** 不可下发:状态不明或被删除,下发会产生未计量副作用。 */
REFUSE
}
private final Kind kind;
private final String operationId;
private DispatchDecision(Kind kind, String operationId) {
this.kind = kind;
this.operationId = operationId;
}
public Kind kind() {
return kind;
}
public String operationId() {
return operationId;
}
/** 只有 READY/LEGACY 允许真正下发到节点。 */
public boolean dispatchable() {
return kind != Kind.REFUSE && operationId != null;
}
/** 拒绝下发(供调用方与测试构造)。 */
public static DispatchDecision refuse() {
return REFUSE;
}
/** 可直接下发(供测试构造)。 */
public static DispatchDecision ready(String operationId) {
return new DispatchDecision(Kind.READY, operationId);
}
}
private static final DispatchDecision REFUSE = new DispatchDecision(DispatchDecision.Kind.REFUSE, null);
/**
* 一次删除登记的返回。
*
* <p>用显式结果替代「null 即退避到裸删除」的旧语义:
* 被拒绝时调用方必须回业务失败,不能绕过待删除意图直接删节点文件。
*/
public record DeleteIntent(boolean accepted, String deleteOperationId, String message) {
static DeleteIntent refused(String message) {
return new DeleteIntent(false, null, message);
}
static DeleteIntent accepted(String deleteOperationId) {
return new DeleteIntent(true, deleteOperationId, null);
}
}
/** 在事务里执行;没有配置事务模板时候直接执行(仅便于构造 mock 测试)。 */
private <T> T inTransaction(java.util.function.Supplier<T> body) {
TransactionTemplate template = transactions;
if (template == null)
return body.get();
return template.execute(status -> {
try {
return body.get();
} catch (RuntimeException e) {
// 回调里必须把异常继续抛出,否则 Spring 会把只读了一半的事务当成正常提交。
status.setRollbackOnly();
throw e;
}
});
}
/**
* 创建任务的持久化前置:在同一个事务里登记唯一 CREATE 意图。
*
* <p>同时写入可见的占位任务行(状态「提交中」),这样提交期间任务列表就能解析该 gid,
* 不会出现「上游已受理但本地查不到」的空窗。
*
* @return 已登记意图(赢家)返回它;同一 gid 已有有效意图返回 {@code null}(输家)。
*/
private Reservation reserveCreateIntent(int gid, String placeholderName, int downloaderId) {
// 锁内只做数据库写入:把并发同 gid 创建串行化,网络调用留在锁外。
synchronized (reservationLock) {
try {
TaskOperation operation = inTransaction(() -> {
TaskOperation existingOperation = taskOperationRepository.selectByGid(gid);
Gallery existingGallery = galleryMapper.selectGalleryByGid(gid);
if (existingOperation != null) {
// 只有「删除已完成」的墓碑才允许用全新 operationId 重新创建(显式重建)。
if (!TaskOperation.STATE_DELETED.equals(existingOperation.getState()))
return null;
if (existingGallery != null)
return null;
String operationId = TaskProtocol.newDownloadOperationId();
taskOperationRepository.reactivate(gid, operationId, TaskOperation.STATE_RESERVED, now());
insertPlaceholder(gid, placeholderName, downloaderId);
return taskOperationRepository.selectByGid(gid);
}
if (existingGallery != null)
return null;
TaskOperation created = new TaskOperation();
created.setGid(gid);
created.setOperationId(TaskProtocol.newDownloadOperationId());
created.setState(TaskOperation.STATE_RESERVED);
created.setUpdateTime(now());
taskOperationRepository.insert(created);
insertPlaceholder(gid, placeholderName, downloaderId);
return created;
});
return operation == null
? new Reservation(ReserveOutcome.DUPLICATE, null)
: new Reservation(ReserveOutcome.RESERVED, operation);
} catch (RuntimeException e) {
// 数据库竞争/失败发生在任何外部副作用之前,绝不能穿透成 500,也不能继续提交。
log.warn("登记创建意图失败,按业务失败处理 gid={} error={}", gid, e.getClass().getSimpleName());
return new Reservation(ReserveOutcome.CONTENDED, null);
}
}
}
/** 写入占位任务行:让提交期间的任务在列表里可见。 */
private void insertPlaceholder(int gid, String name, int downloaderId) {
Gallery placeholder = new Gallery();
placeholder.setGid(gid);
placeholder.setName(name);
placeholder.setStatus(TaskStatus.PENDING_SUBMIT);
placeholder.setDownloader(downloaderId);
placeholder.setCreateTime(System.currentTimeMillis() / 1000);
placeholder.setProceeding(0);
galleryMapper.insertGallery(placeholder);
}
/**
* 标记为正在提交,并用 CAS 确认这一代意图仍然有效。
*
* <p>只有当前状态与 operationId 都匹配(RESERVED → SUBMITTING)才返回 true;
* 期间被删除或重建返回 false,调用方必须停下,绝不能再 POST 上游。
*
* @return true 表示已持久化 SUBMITTING,可以安全产生上游副作用
*/
boolean markSubmitting(String operationId, int gid) {
return inTransaction(() -> taskOperationRepository.compareAndSetCreateState(gid, operationId,
TaskOperation.STATE_RESERVED, TaskOperation.STATE_SUBMITTING, now()) == 1);
}
/**
* 提交结果未知(超时/断连/中断)。保留意图,等待人工对账,不自动重发。
*
* <p>TaskOperation 与 Gallery 状态在同一个短事务里更新:
* 操作落 UNKNOWN 而任务行仍显示「提交中」会让运营侧误以为还能自动重试,
* 因此两边必须一致地表达「不确定,需要人工对账」。
*/
void markUnknown(String operationId, int gid) {
inTransaction(() -> {
int advanced = taskOperationRepository.compareAndSetCreateState(gid, operationId,
TaskOperation.STATE_SUBMITTING, TaskOperation.STATE_UNKNOWN, now());
// 只有本次这一代确实从 SUBMITTING 推进到 UNKNOWN 时才同步任务行:
// 若期间已被删除/重建(CAS 返回 0),绝不能再改那条记录。
if (advanced != 1)
return null;
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery != null && !TaskStatus.UNKNOWN.equals(gallery.getStatus()))
galleryMapper.updateGalleryStatus(gid, TaskStatus.UNKNOWN);
return null;
});
}
/**
* 提交成功后的「一次性短事务」:写入 Gallery 行、累计周用量、把操作推进为 SUBMITTED。
* 三者同一个事务,任何一个失败都一起回滚,避免出现「任务存在但用量未计」这类不一致。
*/
/**
* 提交成功后的落库(带世代校验)。
*
* @param expectedOperationId 本次提交所归属的 operationId;只有它仍是当前有效意图时才能落库,
* 否则(被删除/重建)必须抛错,避免复活已删 gid 或污染新世代
*/
int persistSubmitted(Gallery gallery, int downloaderId, String expectedOperationId) {
return inTransaction(() -> {
TaskOperation operation = taskOperationRepository.selectByGid(gallery.getGid());
// 严格匹配本次世代:任务被删除、被重建或换了 operationId 时都不能落库。
if (operation == null || TaskOperation.STATE_DELETED.equals(operation.getState())
|| TaskOperation.STATE_DELETE_PENDING.equals(operation.getState()))
throw new IllegalStateException("任务已删除或没有有效创建意图,禁止落库复活 gid=" + gallery.getGid());
if (expectedOperationId == null || !expectedOperationId.equals(operation.getOperationId()))
throw new IllegalStateException("创建意图世代不匹配,禁止落库 gid=" + gallery.getGid());
// 幂等:同一意图重复落库(例如上游已提交后重跑)不得把用量累计两次。
boolean alreadyCounted = TaskOperation.STATE_SUBMITTED.equals(operation.getState());
gallery.setDownloader(downloaderId);
gallery.set_download(true);
if (!TaskStatus.SUBMITTED.equals(gallery.getStatus()))
gallery.setStatus(TaskStatus.SUBMITTED);
Gallery existing = galleryMapper.selectGalleryByGid(gallery.getGid());
if (existing == null)
galleryMapper.insertGallery(gallery);
else
galleryMapper.updateGallery(gallery);
if (!alreadyCounted)
configurationMapper.incrementConfiguration(CustomConfiguration.WEEK_USED_AMOUNT, gallery.getFileSize());
taskOperationRepository.updateState(gallery.getGid(), TaskOperation.STATE_SUBMITTED, now());
return 1;
});
}
/**
* 预占式创建:登记 gid 唯一意图(提交任务行占位)→ 只读解析上游 → 标 SUBMITTING → 真正提交。
*
* <p>上游提交发生在事务之外,且只在 SUBMITTING 已持久化之后执行一次。网络异常一律落 UNKNOWN。
*/
public Response createTask(int gid, String placeholderName, String link, String targetResolution, int downloaderId) {
Response response = Response.generateResponse();
Reservation reservation = reserveCreateIntent(gid, placeholderName, downloaderId);
if (reservation.outcome() == ReserveOutcome.DUPLICATE) {
response.failure("任务队列已有此任务,请点击查找任务");
return response;
}
if (reservation.outcome() == ReserveOutcome.CONTENDED) {
// 预占阶段的数据库竞争不产生任何副作用;转业务失败,绝不下发上游、绝不穿透成 500。
response.failure("任务提交失败,请稍后重试");
return response;
}
// 解析上游是只读步骤;真正的 POST 只在 SUBMITTING 持久化之后发生。
// 生命周期约束:RESERVED 阶段被中断是安全的(可释放重来),SUBMITTING 之后一律按 UNKNOWN 处理。
return submitPreparedIntent(gid, reservation.operation(), link, targetResolution, downloaderId);
}
private Response submitPreparedIntent(int gid, TaskOperation operation, String link, String targetResolution,
int downloaderId) {
Response response = Response.generateResponse();
beforePrepare.run();
PreparedGallery prepared;
try {
prepared = upstreamGateway.prepare(link);
} catch (IOException e) {
log.warn("解析任务链接失败 gid={}", gid, e);
releaseReservation(gid, operation.getOperationId());
response.failure("IO错误,可能是网络波动");
return response;
} catch (RuntimeException e) {
log.warn("解析任务链接异常 gid={}", gid, e);
releaseReservation(gid, operation.getOperationId());
response.failure("提交任务失败,未知原因,请检查链接是否正常");
return response;
}
if (prepared == null || prepared.gallery() == null) {
releaseReservation(gid, operation.getOperationId());
response.failure("提交任务失败,未知原因,请检查链接是否正常");
return response;
}
Gallery gallery = prepared.gallery();
gallery.setGid(gid);
// CAS 确认这一代意图仍然有效(未被并发删除/重建),并把状态持久化为 SUBMITTING。
// 只有成功之后才允许真正 POST;失败说明任务已不在当前生命周期里,直接放弃。
if (!markSubmitting(operation.getOperationId(), gid)) {
log.warn("创建意图已失效(可能被并发删除),放弃提交 gid={} operationId={}",
gid, operation.getOperationId());
response.failure("任务已被删除或状态已变化,请刷新任务列表");
return response;
}
afterMarkSubmitting.run();
Gallery submitted;
try {
submitted = upstreamGateway.submit(prepared, targetResolution);
} catch (ResolutionNotMatchException e) {
releaseReservation(gid, operation.getOperationId());
response.failure("提交失败,分辨率不存在");
return response;
} catch (IOException | RuntimeException e) {
// 超时或断连时上游可能已经受理,无法确认:只能记 UNKNOWN,绝不自动重发。
log.warn("上游提交结果未知 gid={} operationId={}", gid, operation.getOperationId(), e);
markUnknown(operation.getOperationId(), gid);
response.failure("上游提交结果未知,请稍后在任务列表中确认,不要重复提交");
return response;
}
if (submitted == null || !TaskStatus.SUBMITTED.equals(submitted.getStatus())) {
markUnknown(operation.getOperationId(), gid);
response.failure("上游提交结果未知,请稍后在任务列表中确认,不要重复提交");
return response;
}
submitted.setGid(gid);
try {
persistSubmitted(submitted, downloaderId, operation.getOperationId());
} catch (RuntimeException e) {
// 上游已受理但本地落库失败:保留 SUBMITTING 意图并标 UNKNOWN,任务行仍在列表里可对账。
log.error("提交成功后落库失败 gid={} operationId={}", gid, operation.getOperationId(), e);
markUnknown(operation.getOperationId(), gid);
response.failure("任务已提交,但本地保存失败,请在任务列表中确认");
return response;
}
// 只有事务提交之后才向节点下发命令;重复下发由节点按 operationId 去重。
byte dispatchResult = dispatchDownload(gid, submitted.getName());
Gallery current = galleryMapper.selectGalleryByGid(gid);
Gallery payload = current == null ? submitted : current;
if (dispatchResult == 0)
response.success(objectMapper.valueToTree(payload));
else
response.failure("任务已保存,但节点未确认接收;请刷新任务列表后重试");
return response;
}
/**
* 提交成功后的落库 + 下发。落库(含用量)在一个短事务里完成,事务提交后才向节点发命令。
*
* <p>先解析当前世代再落库:世代不存在(未提交过 / 已删除)时不落库也不下发。
*
* @return 已持久化的 Gallery;数据库失败或世代无效时抛出异常,调用方回业务失败且不下发节点。
*/
public Gallery persistAndDispatch(Gallery gallery, int downloaderId) {
DispatchDecision decision = resolveDispatch(gallery.getGid());
if (!decision.dispatchable())
throw new IllegalStateException("任务没有可下发的有效世代 gid=" + gallery.getGid());
persistSubmitted(gallery, downloaderId, decision.operationId());
dispatchDownload(gallery.getGid(), gallery.getName());
return gallery;
}
/** 向节点下发 DOWNLOAD 命令(复用当前有效 operationId),返回节点结果码。 */
public byte dispatchDownload(int gid, String name) {
DispatchDecision decision = resolveDispatch(gid);
if (!decision.dispatchable())
return -1;
return remoteService.sendTaskCommand(TaskCommandMessage.ACTION_DOWNLOAD, decision.operationId(), gid, name);
}
/**
* 判定一个 gid 能否下发给节点,并为迁移前的旧任务补齐稳定 operationId。
*
* <p>规则:
* <ul>
* <li>v2 任务只有 {@code SUBMITTED} 可下发;{@code UNKNOWN} 必须人工对账,
* 否则未计量的未知任务会在节点侧变成「下载完成却 0 额度」;</li>
* <li>预占中的 {@code RESERVED}/{@code SUBMITTING}、待删除与删除墓碑都不下发;</li>
* <li>没有任何操作记录、但任务行确实存在且状态已是「已提交」及之后的历史任务,
* 补建一个稳定 operationId 后再下发(这是唯一的 legacy 兼容路径)。</li>
* </ul>
*/
public DispatchDecision resolveDispatch(int gid) {
TaskOperation operation = taskOperationRepository.selectByGid(gid);
if (operation != null && TaskOperation.STATE_SUBMITTED.equals(operation.getState())
&& operation.getOperationId() != null)
return new DispatchDecision(DispatchDecision.Kind.READY, operation.getOperationId());
// 没有操作记录,或只有「旧世代」记录(缺 operationId):必须先确认任务行确实已提交过,
// 再补一个稳定 operationId;UNKNOWN/预占/待删除一律拒绝。
if (operation != null && !TaskOperation.STATE_SUBMITTED.equals(operation.getState()))
return REFUSE;
if (!isLegacySubmitted(gid))
return REFUSE;
String freshId = ensureStableOperationId(gid);
return freshId == null ? REFUSE : new DispatchDecision(DispatchDecision.Kind.LEGACY, freshId);
}
/** 迁移前的旧任务行:没有操作记录,但状态表明上游确实已受理过下载,可安全重投。 */
private boolean isLegacySubmitted(int gid) {
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null || gallery.getStatus() == null)
return false;
return switch (gallery.getStatus()) {
case TaskStatus.SUBMITTED, TaskStatus.DOWNLOADING, TaskStatus.WAITING_COMPRESS,
TaskStatus.COMPRESSING -> true;
// 提交未知 / 提交中 / 待删除 / 删除失败 一律不算已提交,不能下发。
default -> false;
};
}
/**
* 为一个已经真正提交过的旧任务持久化稳定 operationId。
*
* <p>必须是短事务写入、先落库后返回,之后重投/重连才复用同一个 ID;
* 用 {@code insert or ignore} + 回读,避免与并发创建互相覆盖。
*
* @return 稳定 operationId;无法安全建立时返回 null(调用方不下发)
*/
String ensureStableOperationId(int gid) {
// 与创建预占共用同一把短锁(锁内只有数据库写),并发为同一 gid 补 ID 时只能有一个赢家。
synchronized (reservationLock) {
String candidate = TaskProtocol.newDownloadOperationId();
return inTransaction(() -> {
if (taskOperationRepository.selectByGid(gid) == null)
taskOperationRepository.insertIfAbsent(gid, candidate, TaskOperation.STATE_SUBMITTED, now());
else
taskOperationRepository.assignOperationIdIfMissing(gid, candidate, now());
TaskOperation current = taskOperationRepository.selectByGid(gid);
if (current == null || current.getOperationId() == null)
return null;
// 只有确实处于 SUBMITTED 的世代才可下发;并发写入的其他状态一律拒绝。
return TaskOperation.STATE_SUBMITTED.equals(current.getState()) ? current.getOperationId() : null;
});
}
}
/**
* 取消预占(解析失败、分辨率不存在等:还没有产生任何上游副作用)。
*
* <p>只删除与本次创建 intent 匹配的 RESERVED 记录:若期间该 gid 已被删除并重建
* (新的 operationId / 墓碑),这里绝不能把新世代记录抹掉。占位行同理只删「提交中」。
*/
private void releaseReservation(int gid, String operationId) {
inTransaction(() -> {
// 只删与本代 operationId 匹配且仍停在 RESERVED/SUBMITTING 的记录。
int removed = taskOperationRepository.deleteReservationIfSameGeneration(gid, operationId);
// 只有确实删掉了本代预占记录,才能删对应的占位行:
// 若期间已出现新的 RESERVED 代,删除返回 0,此时绝不能误删新占位。
if (removed == 1) {
Gallery existing = galleryMapper.selectGalleryByGid(gid);
if (existing != null && TaskStatus.PENDING_SUBMIT.equals(existing.getStatus()))
galleryMapper.deleteGalleryByGid(gid);
}
return null;
});
}
/**
* 删除的持久化部分。
*
* <p>正在创建意图(RESERVED/SUBMITTING)的任务不能删除:此刻 DELETE 与创建 POST 会同时发生,
* 随后创建落库还会把已删/重建的 gid 写回。这里直接拒绝,由用户在提交结果明确后再操作。
*
* @return 显式结果:被拒时调用方必须回业务失败,不能退避到裸删除
*/
public DeleteIntent beginDelete(int gid) {
return inTransaction(() -> {
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null)
return DeleteIntent.refused("删除失败,该图片不存在");
TaskOperation operation = taskOperationRepository.selectByGid(gid);
if (operation != null && (TaskOperation.STATE_RESERVED.equals(operation.getState())
|| TaskOperation.STATE_SUBMITTING.equals(operation.getState())))
return DeleteIntent.refused("任务正在提交中,请等待提交结果明确后再删除");
if (operation == null) {
// 历史任务行没有操作记录:直接落一条 DELETE_PENDING 意图,避免中间态缺字段。
TaskOperation fresh = new TaskOperation();
fresh.setGid(gid);
fresh.setOperationId(TaskProtocol.newDownloadOperationId());
fresh.setDeleteOperationId(TaskProtocol.newDeleteOperationId());
fresh.setState(TaskOperation.STATE_DELETE_PENDING);
fresh.setUpdateTime(now());
taskOperationRepository.insert(fresh);
galleryMapper.updateGalleryStatus(gid, TaskStatus.DELETE_PENDING);
return DeleteIntent.accepted(fresh.getDeleteOperationId());
}
if (operation.getDeleteOperationId() == null) {
operation.setDeleteOperationId(TaskProtocol.newDeleteOperationId());
operation.setUpdateTime(now());
}
taskOperationRepository.updateDeleteIntent(gid, operation.getDeleteOperationId(),
TaskOperation.STATE_DELETE_PENDING, now());
galleryMapper.updateGalleryStatus(gid, TaskStatus.DELETE_PENDING);
return DeleteIntent.accepted(operation.getDeleteOperationId());
});
}
/**
* 删除的完成部分:节点已确认(或文件已不存在)后,在一个短事务里删任务行、收藏并留墓碑。
*
* <p>事务提交前若失败,任务行与 DELETE_PENDING 意图都保留,可以继续重试。
* 删除意图 ID 必须匹配(防止并发删除/重建时抹掉别的一代),随后只把 DELETE_PENDING
* 世代推进为 DELETED;墓碑(DELETED 且无删除意图)不允许在这里被重新激活。
*
* @return true 表示本次删除意图已完成并留下墓碑
*/
public boolean completeDelete(int gid, String deleteOperationId) {
return inTransaction(() -> {
TaskOperation operation = taskOperationRepository.selectByGid(gid);
if (operation == null || operation.getDeleteOperationId() == null
|| !operation.getDeleteOperationId().equals(deleteOperationId))
return false;
// 先清收藏再删任务行:收藏一侧在部分历史库上带有指向 gallery 的外键。
collectMapper.deleteByGid(gid);
galleryMapper.deleteGalleryByGid(gid);
int updated = taskOperationRepository.completeDeleteIfSameGeneration(gid, deleteOperationId,
TaskOperation.STATE_DELETED, now());
return updated == 1;
});
}
/** 删除失败:保留任务行并记一条可重试的失败状态。 */
public void failDelete(int gid) {
inTransaction(() -> {
Gallery gallery = galleryMapper.selectGalleryByGid(gid);
if (gallery == null)
return null;
gallery.setStatus(TaskStatus.DELETE_FAILED);
galleryMapper.updateGallery(gallery);
return null;
});
}
/**
* 启动恢复:把上次中断在 SUBMITTING 的意图判为 UNKNOWN,并安全释放还停在 RESERVED 的预占。
*
* <p>这段恢复必须发生在任何自动重投之前:
* <ul>
* <li>{@code SUBMITTING}:上游可能已经受理但本地无法确认,只能落 UNKNOWN(操作与任务行同时
* 更新为「提交未知」),交人工对账,绝不自动再 POST;</li>
* <li>{@code RESERVED}:还没有产生任何上游副作用,重启后可以直接释放预占
* (删掉匹配的 RESERVED 记录与「提交中」占位行),让用户重新提交。</li>
* </ul>
*
* @return 本次恢复(判未知 + 释放预占)处理的任务数
*/
public int recoverInterruptedSubmissions() {
return inTransaction(() -> {
int recovered = 0;
for (TaskOperation operation : taskOperationRepository.selectByState(TaskOperation.STATE_SUBMITTING)) {
int advanced = taskOperationRepository.compareAndSetCreateState(operation.getGid(), operation.getOperationId(),
TaskOperation.STATE_SUBMITTING, TaskOperation.STATE_UNKNOWN, now());
if (advanced != 1)
continue;
Gallery gallery = galleryMapper.selectGalleryByGid(operation.getGid());
if (gallery != null && !TaskStatus.UNKNOWN.equals(gallery.getStatus()))
galleryMapper.updateGalleryStatus(operation.getGid(), TaskStatus.UNKNOWN);
recovered++;
}
for (TaskOperation operation : taskOperationRepository.selectByState(TaskOperation.STATE_RESERVED)) {
// 还没 POST:释放匹配的预占记录与占位行,允许重启后重新提交。
// 只有确实删掉了这条本代 RESERVED 记录,才删其占位行;否则不动(可能已换代)。
int removed = taskOperationRepository.deleteReservationIfSameGeneration(operation.getGid(),
operation.getOperationId());
if (removed == 1) {
Gallery gallery = galleryMapper.selectGalleryByGid(operation.getGid());
if (gallery != null && TaskStatus.PENDING_SUBMIT.equals(gallery.getStatus()))
galleryMapper.deleteGalleryByGid(operation.getGid());
}
recovered++;
}
return recovered;
});
}
/** 已完成状态上报的处理:丢弃旧 revision、旧 operation、待删除任务与重复上报。 */
@Override
public CommittedReport apply(TaskReportMessage report) {
if (report == null || report.getReportId() == null || report.getGalleryTasks() == null)
return null;
Applied applied;
try {
applied = inTransaction(() -> applyInTransaction(report));
} catch (RuntimeException e) {
// 数据库失败:不返回可回执结果,节点会保留快照并稍后重投;失败写入绝不清任务。
log.warn("应用节点上报失败,未回执 reportId={}", report.getReportId(), e);
return null;
}
if (applied == null)
return null;
return new CommittedReport(report.getReportId(), applied.newlyCompleted(), applied.acceptedTasks());
}
/** 事务内应用上报的结果:新完成的任务与真正被接受的进度更新。 */
private record Applied(List<Gallery> newlyCompleted, List<GalleryTask> acceptedTasks) {
}
/**
* 在一个事务里应用上报。
*
* @return 非 null 表示已经提交(可以回 ACK,重复上报也会返回结果以停止重放)。
*/
private Applied applyInTransaction(TaskReportMessage report) {
long revision = report.getRevision();
List<Gallery> newlyCompleted = new ArrayList<>();
List<GalleryTask> acceptedTasks = new ArrayList<>();
for (GalleryTask task : report.getGalleryTasks()) {
if (task == null)
continue;
TaskOperation operation = taskOperationRepository.selectByGid(task.getGid());
Gallery gallery = galleryMapper.selectGalleryByGid(task.getGid());
if (gallery == null) {
// 未知 gid(任务已删除):忽略但 ACK,让节点停止重放。
log.debug("忽略节点上报的未知任务 gid={}", task.getGid());
continue;
}
String expectedOperationId = report.getOperationIds() == null
? null : report.getOperationIds().get(task.getGid());
if (operation == null) {
// 旧任务行(迁移前创建)没有操作记录:按显式生成策略补建并采纳上报里的 operationId。
operation = adoptGeneration(task.getGid(), expectedOperationId);
} else if (TaskOperation.STATE_DELETED.equals(operation.getState())) {
// 墓碑必须永久忽略旧上报:删除已完成的任务只能由显式 CREATE 重新建立新世代。
// (gallery 已被删除时上面的 null 分支已经提前忽略,这里防御性兜底。)
log.debug("忽略已删除 gid 的迟到上报 gid={}", task.getGid());
continue;
} else if (TaskOperation.STATE_DELETE_PENDING.equals(operation.getState())) {
log.info("忽略待删除任务的迟到上报 gid={}", task.getGid());
continue;
} else if (operation.getLastRevision() > 0 && revision <= operation.getLastRevision()) {
log.debug("丢弃过期上报 gid={} revision={} lastRevision={}",
task.getGid(), revision, operation.getLastRevision());
continue;
} else if (operation.getOperationId() != null) {
// v2 任务必须有归属:缺 operationId 或与当前世代不符都不能改当前任务,
// 否则「删除后重建」的新世代会被旧上报污染。
if (expectedOperationId == null || !operation.getOperationId().equals(expectedOperationId)) {
log.info("丢弃无法归属/旧世代的迟到上报 gid={} 上报={} 当前={}",
task.getGid(), expectedOperationId, operation.getOperationId());
continue;
}
}
String status = TaskStatus.fromReport(task.getStatus());
if (status == null) {
log.warn("未知上报状态码 gid={} status={}", task.getGid(), task.getStatus());
continue;
}
gallery.setProceeding(task.getProceeding());
if (task.getName() != null)
gallery.setName(task.getName());
boolean justCompleted = TaskStatus.COMPLETE.equals(status)
&& !TaskStatus.COMPLETE.equals(gallery.getStatus());
gallery.setStatus(status);
galleryMapper.updateGallery(gallery);
taskOperationRepository.updateReportReceipt(task.getGid(), revision, report.getReportId(), now());
acceptedTasks.add(task);
if (justCompleted)
newlyCompleted.add(gallery);
}
return new Applied(newlyCompleted, acceptedTasks);
}
/**
* 为迁移前创建、尚无操作记录的历史任务建立一次「世代」:优先采纳上报里的 operationId;
* 旧节点不带 operationId 时保留 null,让后续同世代的进度上报仍能被接受。
*
* <p>不在这里凭空生成随机 ID:那会让第一份进度上报之后的所有旧上报都因「无法归属」被丢弃。
* 真正的稳定 operationId 会在首次自动下发时用 {@code ensureStableOperationId} 落库,
* 之后该世代即可被严格匹配。不会覆盖任何已有任务状态。
*/
private TaskOperation adoptGeneration(int gid, String reportedOperationId) {
TaskOperation operation = new TaskOperation();
operation.setGid(gid);
operation.setOperationId(reportedOperationId);
operation.setState(TaskOperation.STATE_SUBMITTED);
operation.setUpdateTime(now());
taskOperationRepository.insert(operation);
return operation;
}
private static long now() {
return System.currentTimeMillis() / 1000;
}
/** 供装配层解析当前有效 operationId(避免装配代码访问包私有字段)。 */
public String currentOperationId(int gid) {
TaskOperation operation = taskOperationRepository.selectByGid(gid);
return operation == null ? null : operation.getOperationId();
}
}
@@ -2,18 +2,16 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.CustomConfigurationMapper; import com.lion.lionwebsite.Dao.normal.CustomConfigurationMapper;
import com.lion.lionwebsite.Dao.normal.GalleryMapper; import com.lion.lionwebsite.Dao.normal.GalleryMapper;
import com.lion.lionwebsite.Dao.normal.ShareFileMapper;
import com.lion.lionwebsite.Domain.CustomConfiguration; import com.lion.lionwebsite.Domain.CustomConfiguration;
import com.lion.lionwebsite.Domain.ShareFile;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.GalleryUtil; import com.lion.lionwebsite.Util.GalleryUtil;
import lombok.Data; import lombok.Data;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpEntity; import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.http.client.methods.HttpGet; import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.http.impl.client.CloseableHttpClient; import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.http.impl.client.HttpClients; import org.apache.hc.core5.http.HttpEntity;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled; import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
@@ -23,6 +21,7 @@ import java.net.URLDecoder;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.*; import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes; import java.nio.file.attribute.BasicFileAttributes;
import java.nio.file.attribute.FileTime;
import java.time.LocalDateTime; import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter; import java.time.format.DateTimeFormatter;
import java.util.*; import java.util.*;
@@ -45,15 +44,13 @@ public class LocalService{
final CustomConfigurationMapper configurationMapper; final CustomConfigurationMapper configurationMapper;
final ShareFileMapper shareFileMapper;
final GalleryMapper galleryMapper; final GalleryMapper galleryMapper;
final PushService pushService; final PushService pushService;
final RemoteService remoteService; final RemoteService remoteService;
final SubscriptionRefreshService subscriptionRefreshService; final SubscriptionRefreshScheduler subscriptionRefreshScheduler;
/** /**
* 检查连接是否有效,如果无效自动重连 * 检查连接是否有效,如果无效自动重连
@@ -104,115 +101,23 @@ public class LocalService{
} }
/** /**
* 定时更新订阅 * 手动「更新订阅」:立即刷新全部启用账号,供管理页按钮调用。
* @throws IOException 下载以及保存异常 *
* <p>定时刷新已移交 {@link SubscriptionRefreshScheduler}。原先这里还有一个
* {@code @Scheduled(fixedRate = 86400000)} 的定时入口,它会一次性刷新全部账号
* (十几个账号在十几秒内打满),且因为没有 initialDelay,每次重启都会立刻重刷一遍,
* 使上游看到的请求密度取决于部署频率。该入口已移除。
*
* <p>不再保留 {@code isManual} 参数:定时路径已不存在,留着它只会让人以为
* 「传 false 就是不刷新」,而实际语义是「什么都不做却报成功」。
*/ */
@Scheduled(fixedRate = 86400000) public boolean updateSub() {
public void updateSubScheduler() throws IOException { boolean success = subscriptionRefreshScheduler.refreshAllNow();
updateSub(false);
}
/**
* 更新订阅链接的实际方法
*/
public boolean updateSub(boolean isManual) throws IOException {
// 手动和定时入口均刷新全部启用子账号;全部成功后更新原有“上次更新时间”。
boolean success = subscriptionRefreshService.refreshAll();
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
if (success) if (success)
configurationMapper.updateConfiguration(CustomConfiguration.LAST_UPDATE_SUB_TIME, configurationMapper.updateConfiguration(CustomConfiguration.LAST_UPDATE_SUB_TIME,
CustomUtil.dateTimeFormatter().format(LocalDateTime.now())); CustomUtil.dateTimeFormatter().format(LocalDateTime.now()));
return success; return success;
/*
DateTimeFormatter dateTimeFormatter = CustomUtil.dateTimeFormatter();
CustomConfiguration customConfiguration = configurationMapper.selectConfiguration(CustomConfiguration.LAST_UPDATE_SUB_TIME);
//如果不是手动,则判断更新间隔是否满足,不满足则取消更新
if (!isManual) {
LocalDateTime lastUpdate = LocalDateTime.parse(customConfiguration.getValue(), CustomUtil.dateTimeFormatter());
LocalDateTime now = LocalDateTime.now();
now = now.plusHours(-3);
if(now.isBefore(lastUpdate))
return false;
}
File DouNaiClashFile = new File("sub/DouNaiClash.txt");
File DouNaiV2rayFile = new File("sub/DouNaiV2ray.txt");
File directory = new File("sub");
if(!directory.isDirectory())
Files.createDirectory(Paths.get("sub"));
List<String> DouNaiClash_profile;
//下载豆奶v2ray订阅
try(FileWriter writer = new FileWriter(DouNaiV2rayFile)) {
String DouNaiV2rayRaw = Get(DouNaiV2ray).getFirst();
String[] v2rayPlain = new String(Base64.getDecoder().decode(DouNaiV2rayRaw)).split("\n");
StringBuilder stringBuilder = new StringBuilder();
Pattern pattern = Pattern.compile("-?\\d+(\\.\\d+)?");
//过滤高倍率节点
for(String node: v2rayPlain){
String name = URLDecoder.decode(node.split("#")[1], StandardCharsets.UTF_8);
if(name.contains("流量")){
Matcher matcher = pattern.matcher(name.substring(name.indexOf("(") + 1, name.indexOf(")")));
if (matcher.find()) {
// 将匹配到的数字添加到列表中
float ratio = Float.parseFloat(matcher.group());
if(ratio <= 2) {
stringBuilder.append(node).append("\n");
continue;
}
}
stringBuilder.append(node).append("\n");
}
else{
stringBuilder.append(node).append("\n");
}
}
writer.write(new String(Base64.getEncoder().encode(stringBuilder.toString().getBytes(StandardCharsets.UTF_8))));
log.info("load DouNai v2ray complete");
}catch (IOException e){
log.error("load DouNai v2ray failure", e);
}
//下载豆奶clash订阅
try(FileWriter writer = new FileWriter(DouNaiClashFile)) {
DouNaiClash_profile = Get(DouNaiClash);
//过滤高倍率节点
ArrayList<String> clashProcessed = new ArrayList<>();
boolean isProxies = false;
boolean skip = false;
for(String line: DouNaiClash_profile){
if(line.equals("proxies:"))
isProxies = true;
else if(line.equals("proxy-groups:") && isProxies)
isProxies = false;
if(isProxies) {
if (line.contains("name"))
skip = line.contains("流量");
if (!skip)
clashProcessed.add(line);
}
else
if (!line.contains("流量"))
clashProcessed.add(line);
}
for(String line: clashProcessed)
writer.write(line + "\n");
log.info("load DouNai clash complete");
}catch (IOException e){
log.error("load DouNai clash failure", e);
}
configurationMapper.updateConfiguration(CustomConfiguration.LAST_UPDATE_SUB_TIME, dateTimeFormatter.format(LocalDateTime.now()));
return true;
*/
} }
/** /**
@@ -228,7 +133,7 @@ public class LocalService{
httpResponse = httpClient.execute(httpGet); httpResponse = httpClient.execute(httpGet);
HttpEntity responseEntity = httpResponse.getEntity(); HttpEntity responseEntity = httpResponse.getEntity();
int statusCode = httpResponse.getStatusLine().getStatusCode(); int statusCode = httpResponse.getCode();
ArrayList<String> temp = new ArrayList<>(); ArrayList<String> temp = new ArrayList<>();
if (statusCode == 200) { if (statusCode == 200) {
@@ -242,23 +147,6 @@ public class LocalService{
return temp; return temp;
} }
/**
* 每天四点,清理过期分享码
*/
@Scheduled(cron = "0 0 4 * * *")
public void checkShareCode(){
ShareFile[] shareFiles = shareFileMapper.selectAllShareFile();
Calendar now;
Calendar expireTime;
for(ShareFile shareFile: shareFiles){
now = Calendar.getInstance();
expireTime = Calendar.getInstance();
expireTime.setTime(shareFile.getExpireTime());
if(now.after(expireTime))
shareFileMapper.deleteShareFile(shareFile.getShareCode());
}
}
/** /**
* 每周,查看缩略图数量,按照访问时间排序,删除超出1w的部分 * 每周,查看缩略图数量,按照访问时间排序,删除超出1w的部分
*/ */
@@ -266,13 +154,15 @@ public class LocalService{
public void clearThumbnailCache(){ public void clearThumbnailCache(){
String cachePath = "/storage/hentaiCache/"; String cachePath = "/storage/hentaiCache/";
File directory = new File(cachePath); File directory = new File(cachePath);
List<Path> files = new ArrayList<>(); // 排序需要文件的最后访问时间。原先在比较器里每次重新读属性(n log n 次系统调用),
// 且属性读取失败会让整个排序抛 RuntimeException;这里在遍历时一次性带出属性。
List<Map.Entry<Path, FileTime>> files = new ArrayList<>();
try { try {
Files.walkFileTree(directory.toPath(), new SimpleFileVisitor<>() { Files.walkFileTree(directory.toPath(), new SimpleFileVisitor<>() {
@Override @Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
if (attrs.isRegularFile()) { if (attrs.isRegularFile()) {
files.add(file); files.add(Map.entry(file, attrs.lastAccessTime()));
} }
return CONTINUE; return CONTINUE;
} }
@@ -281,22 +171,15 @@ public class LocalService{
throw new RuntimeException(e); throw new RuntimeException(e);
} }
files.sort((f1, f2) -> { files.sort(Map.Entry.comparingByValue());
try {
return Files.readAttributes(f1, BasicFileAttributes.class).lastAccessTime()
.compareTo(Files.readAttributes(f2, BasicFileAttributes.class).lastAccessTime());
} catch (IOException e) {
throw new RuntimeException(e);
}
});
if (files.size() > 10000) { if (files.size() > 10000) {
List<Path> toDelete = files.subList(0, files.size() - 10000); List<Map.Entry<Path, FileTime>> toDelete = files.subList(0, files.size() - 10000);
for (Path file : toDelete) { for (Map.Entry<Path, FileTime> entry : toDelete) {
try { try {
Files.delete(file); Files.delete(entry.getKey());
}catch (IOException e){ }catch (IOException e){
log.warn("删除缩略图缓存文件失败: {}", file, e); log.warn("删除缩略图缓存文件失败: {}", entry.getKey(), e);
} }
} }
log.info("Deleted {} files", toDelete.size()); log.info("Deleted {} files", toDelete.size());
@@ -1,42 +1,18 @@
package com.lion.lionwebsite.Service; package com.lion.lionwebsite.Service;
import cn.hutool.core.io.FileUtil;
import cn.hutool.core.util.RandomUtil;
import com.lion.lionwebsite.Dao.normal.CustomConfigurationMapper; import com.lion.lionwebsite.Dao.normal.CustomConfigurationMapper;
import com.lion.lionwebsite.Dao.normal.ShareFileMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.CustomConfiguration; import com.lion.lionwebsite.Domain.CustomConfiguration;
import com.lion.lionwebsite.Domain.ShareFile;
import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor;
import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.FileDownload;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import com.fasterxml.jackson.databind.ObjectMapper;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import lombok.Data; import lombok.Data;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import java.io.BufferedOutputStream; import java.util.HashMap;
import java.io.File; import java.util.Map;
import java.io.IOException;
import java.io.OutputStream; import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
@Service @Service
@@ -45,219 +21,8 @@ import java.util.concurrent.Executors;
public class PersonalService{ public class PersonalService{
final CustomConfigurationMapper configurationMapper; final CustomConfigurationMapper configurationMapper;
final UserMapper userMapper;
final ShareFileMapper shareFileMapper;
final TaskHandlerInterceptor taskHandlerInterceptor;
String StoragePath = "/storage/";
DateTimeFormatter dateTimeFormatter = CustomUtil.dateTimeFormatter();
ExecutorService compressThreadPool = Executors.newFixedThreadPool(1);
final PushService pushService; final PushService pushService;
/**
* 获取文件列表,同时带上分享码以及过期时间
* @param path 路径
* @return 文件列表
*/
public String getFiles(String path) {
Response response = Response.generateResponse();
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
File root_file = new File(StoragePath + path);
Calendar now = Calendar.getInstance();
Calendar expireTime = Calendar.getInstance();
//如果目标路径是文件夹
if(root_file.isDirectory()) {
ArrayList<Map<String, String>> fileMaps = new ArrayList<>();
File[] originalFiles = root_file.listFiles();
//如果文件夹里的文件不为空
if (originalFiles != null) {
originalFiles = Arrays.stream(originalFiles).sorted(Comparator.comparing(File::getName)).toArray(File[]::new);
ArrayList<ShareFile> shareFiles = shareFileMapper.selectShareFilesByFilePath(root_file.getAbsolutePath());
//遍历文件,放入文件信息以及查询对应的分享码
for (File file : originalFiles) {
Map<String, String> fileMap = new LinkedHashMap<>();
fileMap.put("name", file.getName());
fileMap.put("path", file.getAbsolutePath());
if (file.isDirectory()) {
fileMap.put("type", "FOLDER");
} else if (file.isFile()) {
fileMap.put("type", "FILE"); //处理文件大小单位
String fileSize = CustomUtil.fileSizeToString(file.length());
fileMap.put("size", fileSize);
Iterator<ShareFile> iterator = shareFiles.iterator();
while(iterator.hasNext()){
ShareFile shareFile = iterator.next();
if(shareFile.getFilePath().equals(file.getAbsolutePath())){
expireTime.setTime(shareFile.getExpireTime());
if(now.after(expireTime)){
shareFileMapper.deleteShareFile(shareFile.getShareCode());
}
else {
fileMap.put("shareCode", shareFile.getShareCode());
fileMap.put("expireTime", dateTimeFormatter.format(shareFile.getExpireTime().toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime()));
}
iterator.remove();
break;
}
}
}
fileMaps.add(fileMap);
}
response.success(new ObjectMapper().valueToTree(fileMaps).toString());
}
else
response.failure("文件夹为空");
}
return response.toJSONString();
}
/**
* 下载文件
* @param request 请求对象
* @param response 响应对象
* @param path 目标路径
*/
public void download(HttpServletRequest request, HttpServletResponse response, String path){
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
File file = new File(path);
if(file.exists())
FileDownload.export(request, response, path);
else
try{
response.getWriter().print("404 NOT FOUND");
}catch (IOException e){
log.warn("输出404失败", e);
}
}
/**
* 上传文件
* @param path 目标路径
* @param fileName 文件名称
* @param file 文件对象
* @return 上传结果
*/
public String uploadFile(String path, String fileName, MultipartFile file) {
Response response = Response.generateResponse();
log.info("上传文件:{}, 目标路径:{}", fileName, path);
if(path == null || fileName == null || file == null){
response.failure("参数不完整");
return response.toJSONString();
}
File directory = new File(StoragePath + path);
if(directory.isDirectory()){
File targetFile = new File(StoragePath + path, fileName);
if(targetFile.exists())
response.failure("目标文件已存在");
else
try {
file.transferTo(Path.of(StoragePath + path, fileName));
response.success("上传成功");
} catch (IOException e) {
response.failure("上传失败");
log.error("上传失败: {}", fileName, e);
}
}
else
response.failure("该路径不存在或者不是文件夹");
return response.toJSONString();
}
/**
* 创建分享码
* @param path 目标路径
* @param expireHour 过期时间
* @return 如果成功则是分享码以及过期时间,失败则是失败原因
*/
public String shareFile(String path, Integer expireHour) {
Response response = Response.generateResponse();
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
Map<String, String> jsonObject = new HashMap<>();
File file = new File(path);
if(file.isFile()){ //如果是文件,则生成分享码,调用此接口时不需要考虑分享已分享文件以及分享已过期文件
String ShareCode;
ShareCode = RandomUtil.randomString(8);
Calendar expireTime = Calendar.getInstance();
expireTime.add(Calendar.HOUR, expireHour);
shareFileMapper.insertShareFile(ShareCode, path, expireTime.getTime());
jsonObject.put("shareCode", ShareCode);
jsonObject.put("expireTime", dateTimeFormatter.format(expireTime.getTime().toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime()));
response.success(new ObjectMapper().valueToTree(jsonObject).toString());
}
else
response.failure("此路径为文件夹或不存在");
return response.toJSONString();
}
/**
* 延长分享时间
* @param path 目标文件路径
* @param extendHour 延长小时数
* @return 延长结果
*/
public String extendShareTime(String path, Integer extendHour) {
Response response = Response.generateResponse();
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
Map<String, String> data = new LinkedHashMap<>();
ShareFile shareFile = shareFileMapper.selectShareFileByFilePath(path);
if(shareFile != null){
Calendar calendar = Calendar.getInstance();
calendar.setTime(shareFile.getExpireTime());
calendar.add(Calendar.HOUR, extendHour);
shareFile.setExpireTime(calendar.getTime());
shareFileMapper.updateShareFile(shareFile);
data.put("expireTime", dateTimeFormatter.format(shareFile.getExpireTime().toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime()));
data.put("path", shareFile.getFilePath());
response.success(new ObjectMapper().valueToTree(data).toString());
}
else{
response.failure("该文件未被分享");
}
return response.toJSONString();
}
/**
* 取消分享
* @param path 目标路径
* @return 取消结果
*/
public String cancelShare(String path) {
Response response = Response.generateResponse();
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
ShareFile shareFile = shareFileMapper.selectShareFileByFilePath(path);
if(shareFile != null){
shareFileMapper.deleteShareFile(shareFile.getShareCode());
response.success("取消分享成功");
}
else{
response.failure("该文件未被分享");
}
return response.toJSONString();
}
/** /**
* 获取订阅文件上次更新时间 * 获取订阅文件上次更新时间
* @return 订阅文件上次更新时间 * @return 订阅文件上次更新时间
@@ -281,75 +46,7 @@ public class PersonalService{
jsonObject.put("ip", ip); jsonObject.put("ip", ip);
jsonObject.put("lastUpdateTime", updateTime); jsonObject.put("lastUpdateTime", updateTime);
response.success(new ObjectMapper().valueToTree(jsonObject).toString().replace("\"", " ").replace("\\", " ")); response.success(objectMapper.valueToTree(jsonObject).toString().replace("\"", " ").replace("\\", " "));
return response.toJSONString();
}
/**
* 打包文件
* @param path 目标路径
* @return 响应提交成功,因为该方法为异步执行,未完成时后辍为undone
*/
public String compress(String path) {
Response response = Response.generateResponse();
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
File file = new File(path);
String finalPath = path;
if(!file.isDirectory() || !file.exists()){
response.failure("选中的路径不是文件夹");
return response.toJSONString();
}
compressThreadPool.submit(() -> {
try(OutputStream bos = new BufferedOutputStream(Files.newOutputStream(Paths.get(finalPath + ".tar***undone")));
TarArchiveOutputStream aos = new TarArchiveOutputStream(bos)) {
aos.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU); //解除文件名长度限制
Path dirPath = Paths.get(finalPath);
Files.walkFileTree(dirPath, new SimpleFileVisitor<>() {
@Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
TarArchiveEntry entry = new TarArchiveEntry(dir.toFile(), dirPath.relativize(dir).toString());
aos.putArchiveEntry(entry);
aos.closeArchiveEntry();
return super.preVisitDirectory(dir, attrs);
}
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
TarArchiveEntry entry = new TarArchiveEntry(file.toFile(), dirPath.relativize(file).toString());
aos.putArchiveEntry(entry);
IOUtils.copy(Files.newInputStream(file.toFile().toPath()), aos);
aos.closeArchiveEntry();
return super.visitFile(file, attrs);
}
});
File targetFile = new File(finalPath + ".tar***undone");
log.info("打包成功,重命名:" + targetFile.renameTo(new File(finalPath + ".tar")));
}catch (IOException e){
log.error("打包失败", e);
log.info("打包失败,删除文件结果:" + new File(finalPath + ".tar***undone").delete());
}
});
response.success("加入队列成功");
return response.toJSONString();
}
/**
* 删除文件
* @param path 目标路径
* @return 删除结果
*/
public String deleteFile(String path) {
Response response = Response.generateResponse();
path = URLDecoder.decode(path, StandardCharsets.UTF_8);
File file = new File(path);
if(FileUtil.del(file))
response.success("删除成功");
else
response.failure("删除失败");
return response.toJSONString(); return response.toJSONString();
} }
@@ -2,35 +2,23 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.CustomConfigurationMapper; import com.lion.lionwebsite.Dao.normal.CustomConfigurationMapper;
import com.lion.lionwebsite.Dao.normal.ShareFileMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper; import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.CustomConfiguration; import com.lion.lionwebsite.Domain.CustomConfiguration;
import com.lion.lionwebsite.Domain.ShareFile;
import com.lion.lionwebsite.Domain.User; import com.lion.lionwebsite.Domain.User;
import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor; import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.FileDownload;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Calendar;
@Service @Service
@RequiredArgsConstructor @RequiredArgsConstructor
public class PublicService { public class PublicService {
final CustomConfigurationMapper configurationMapper; final CustomConfigurationMapper configurationMapper;
final ShareFileMapper shareFileMapper;
final UserMapper userMapper; final UserMapper userMapper;
final TaskHandlerInterceptor taskHandlerInterceptor; final TaskHandlerInterceptor taskHandlerInterceptor;
@@ -44,41 +32,6 @@ public class PublicService {
configurationMapper.updateConfiguration(CustomConfiguration.LAST_UPDATE_IP_ADDRESS_TIME, CustomUtil.now()); configurationMapper.updateConfiguration(CustomConfiguration.LAST_UPDATE_IP_ADDRESS_TIME, CustomUtil.now());
} }
/**
* 通过分享码获取文件
* @param httpRequest 请求对象
* @param httpResponse 响应对象
* @param ShareCode 分享码
* @throws IOException 响应时的异常
*/
public boolean GetFile(HttpServletRequest httpRequest, HttpServletResponse httpResponse, String ShareCode) throws IOException {
Response response = Response.generateResponse();
//参数为空的情况
if(ShareCode == null) {
response.failure("ShareCode invalid");
} else {
ShareFile shareFile = shareFileMapper.selectShareFileByShareCode(ShareCode);
Calendar ExpireTime = Calendar.getInstance();
Calendar now = Calendar.getInstance();
if (shareFile != null) {
ExpireTime.setTime(shareFile.getExpireTime());
if (ExpireTime.after(now) && new File(shareFile.getFilePath()).isFile()) {
FileDownload.export(httpRequest, httpResponse, shareFile.getFilePath());
return true;
} else {
shareFileMapper.deleteShareFile(shareFile.getShareCode());
response.failure("ShareCode is expired or File is not exist");
}
} else
response.failure("ShareCode is not exist or expired");
}
httpResponse.getOutputStream().write(response.toJSONString().getBytes(StandardCharsets.UTF_8));
return false;
}
/** /**
* 修改授权码,如果能够执行此方法,则授权码一定存在 * 修改授权码,如果能够执行此方法,则授权码一定存在
* @param AuthCode 原来的授权码 * @param AuthCode 原来的授权码
@@ -3,8 +3,9 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.GalleryForQuery; import com.lion.lionwebsite.Domain.GalleryForQuery;
import com.lion.lionwebsite.Util.FileDownload; import com.lion.lionwebsite.Util.FileDownload;
import com.lion.lionwebsite.Util.GalleryUtil; import com.lion.lionwebsite.Util.GalleryUtil;
import com.lion.lionwebsite.Util.ImageFileCache;
import java.nio.file.Path;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import com.fasterxml.jackson.databind.ObjectMapper;
import jakarta.servlet.ServletOutputStream; import jakarta.servlet.ServletOutputStream;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
@@ -18,9 +19,12 @@ import org.springframework.stereotype.Service;
import java.io.*; import java.io.*;
import java.net.URI; import java.net.URI;
import java.net.URISyntaxException; import java.net.URISyntaxException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.*; import java.util.*;
import static com.lion.lionwebsite.Util.CustomUtil.fourZeroFour; import static com.lion.lionwebsite.Util.CustomUtil.fourZeroFour;
import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
@Service @Service
@Slf4j @Slf4j
@@ -31,12 +35,15 @@ public class QueryService {
public String query(String keyword, String prev, String next) { public String query(String keyword, String prev, String next) {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
String result; String result;
String param = "?f_search=" + keyword.replace(" ", "+") + "&f_sft=on&f_sfu=on&f_sfl=on"; // 关键词来自用户输入,必须编码后再拼进查询串:未编码的 &、#、中文、空格
// 会被上游当成额外的查询参数或截断,搜索行为不可预期。
String param = "?f_search=" + URLEncoder.encode(keyword == null ? "" : keyword, StandardCharsets.UTF_8)
+ "&f_sft=on&f_sfu=on&f_sfl=on";
if(prev != null) if(prev != null)
param += "&prev=" + prev; param += "&prev=" + URLEncoder.encode(prev, StandardCharsets.UTF_8);
else if(next != null) else if(next != null)
param += "&next=" + next; param += "&next=" + URLEncoder.encode(next, StandardCharsets.UTF_8);
try{ try{
result = GalleryUtil.requests("https://exhentai.org/" + param, "get", null, null); result = GalleryUtil.requests("https://exhentai.org/" + param, "get", null, null);
@@ -47,35 +54,16 @@ public class QueryService {
} }
Document parse = Jsoup.parse(result); Document parse = Jsoup.parse(result);
Elements elements = parse.select("body > div.ido > div:nth-child(2) > table > tbody > tr"); java.util.List<GalleryForQuery> galleries;
ArrayList<GalleryForQuery> galleries = new ArrayList<>();
ListIterator<Element> elementListIterator = elements.listIterator();
try { try {
elementListIterator.next(); //skip firstlink galleries = com.lion.lionwebsite.Util.GallerySearchParser.parse(parse);
}catch (NullPointerException | NoSuchElementException e){ } catch (IOException | RuntimeException error) {
response.failure("没有搜索到结果"); log.warn("上游搜索页面格式不支持: {}", error.getClass().getSimpleName());
return response.toJSONString(); return response.failure("上游搜索页面格式不支持").toJSONString();
} }
if (galleries.isEmpty()) return response.failure("没有搜索到结果").toJSONString();
while (elementListIterator.hasNext()){ response.success(objectMapper.valueToTree(galleries));
Element element = elementListIterator.next();
GalleryForQuery gallery = new GalleryForQuery();
gallery.setType(element.child(0).text()); //type
String src = element.child(1).select("img").attr("src");
if (src.startsWith("data"))
src = element.child(1).select("img").attr("data-src");
src = src.replace("https://s.exhentai.org", "");
gallery.setThumbnailUrl(src); //thumbnailSrc
gallery.setUploadTime(element.child(1).select("div > div [onclick]").get(1).text()); //uploadTime
gallery.setLink(element.child(2).child(0).attr("href")); //link
gallery.setGid(GalleryUtil.parseGid(gallery.getLink()) + ""); //gid
gallery.setName(element.child(2).child(0).child(0).text()); //name
gallery.setPage(Integer.parseInt(element.child(3).child(1).text().split(" ")[0])); //page
galleries.add(gallery);
}
response.success(new ObjectMapper().valueToTree(galleries).toString());
Elements nextLink = parse.select("#unext"); Elements nextLink = parse.select("#unext");
if(nextLink.hasAttr("href")) if(nextLink.hasAttr("href"))
response.set("next", nextLink.attr("href")); response.set("next", nextLink.attr("href"));
@@ -103,22 +91,16 @@ public class QueryService {
String fileName = path.substring(path.lastIndexOf("/") + 1); String fileName = path.substring(path.lastIndexOf("/") + 1);
String suffix = fileName.substring(fileName.lastIndexOf(".")); String suffix = fileName.substring(fileName.lastIndexOf("."));
fileName = fileName.substring(0, fileName.lastIndexOf(".")); fileName = fileName.substring(0, fileName.lastIndexOf("."));
File image = new File(CachePath, fileName + ".avif"); String sourceUrl = "https://ehgt.org/" + path;
try {
if(image.isFile()){ Path image = ImageFileCache.get(Path.of(CachePath), fileName, () -> sourceUrl);
FileDownload.export(request, response, image.getAbsolutePath()); FileDownload.export(request, response, image.toString());
return; } catch (InterruptedException e) {
} Thread.currentThread().interrupt();
response.setStatus(503);
path = "https://ehgt.org/" + path; } catch (Exception e) {
try(ServletOutputStream outputStream = response.getOutputStream()){ log.warn("获取缩略图失败: errorType={}", e.getClass().getSimpleName());
new URI(path).toURL().openConnection().getInputStream().transferTo(new FileOutputStream(CachePath + fileName + suffix)); if (!response.isCommitted()) response.setStatus(404);
GalleryUtil.convertImg(CachePath + fileName + suffix, suffix);
FileInputStream inputStream = new FileInputStream(image.getAbsoluteFile()); //如果放到括号里,会导致图片未创建时创建文件流失败报错
outputStream.write(inputStream.readAllBytes());
inputStream.close();
}catch (IOException | URISyntaxException e){
log.error("获取缩略图失败", e);
} }
} }
} }
@@ -18,7 +18,6 @@ import io.netty.util.concurrent.Promise;
import lombok.Data; import lombok.Data;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct; import jakarta.annotation.PostConstruct;
@@ -38,15 +37,16 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException; import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicBoolean;
import java.util.List;
@Service @Service
@Data @Data
@Slf4j @Slf4j
public class RemoteService { public class RemoteService {
ChannelFuture channelFuture; volatile ChannelFuture channelFuture;
Channel channel; volatile Channel channel;
@Value("${remote.ip:5.255.110.45}") @Value("${remote.ip:5.255.110.45}")
String ip; String ip;
@@ -62,9 +62,12 @@ public class RemoteService {
ConcurrentHashMap<Integer, CopyOnWriteArrayList<CompletableFuture<String>>> retryStatusWaiters = ConcurrentHashMap<Integer, CopyOnWriteArrayList<CompletableFuture<String>>> retryStatusWaiters =
new ConcurrentHashMap<>(); new ConcurrentHashMap<>();
EventLoop eventLoopGroup = new DefaultEventLoop(); final EventLoop eventLoopGroup = new DefaultEventLoop();
final EventLoopGroup networkGroup = new NioEventLoopGroup(2);
ExecutorService downloadThread = Executors.newCachedThreadPool(); final AtomicBoolean connecting = new AtomicBoolean();
final AtomicBoolean monitoring = new AtomicBoolean();
volatile boolean stopping;
volatile ServerSocket monitorSocket;
Thread monitor; Thread monitor;
@@ -74,12 +77,162 @@ public class RemoteService {
final SubscriptionStandbySnapshotService subscriptionStandbySnapshotService; final SubscriptionStandbySnapshotService subscriptionStandbySnapshotService;
/**
* 协议 v2 上报处理器。用 setter 注入而不是构造参数,避免 RemoteService 与
* GalleryTaskLifecycleService 之间的依赖环(后者需要 RemoteService 下发命令)。
*/
volatile TaskReportHandler taskReportHandler;
/**
* gid → 当前有效的下载 operationId。由生命周期服务在装配期注册(保持依赖方向单向)。
* 未注册或返回 null 时,重投/重试退化为旧的无 operationId 命令。
*/
volatile java.util.function.IntFunction<String> downloadOperationIdResolver;
/**
* gid → 下发判定(是否允许下发 + 用哪个稳定 operationId)。
*
* <p>替代「能解析出 operationId 就下发」的旧语义:UNKNOWN / 预占中 / 待删除的任务
* 必须被明确拒绝,否则未计量的人工对账任务可能在节点侧变成「下载完成却 0 额度」。
*/
volatile java.util.function.IntFunction<GalleryTaskLifecycleService.DispatchDecision> dispatchResolver;
/**
* 生命周期装配是否已经完成(handler/resolver 已绑定 + 启动恢复已执行)。
*
* <p>在它为 true 之前,绝不自动重投未完成任务:启动早期还不知道哪些任务是
* SUBMITTED、哪些是 UNKNOWN,盲目下发会制造未计量副作用。
*/
volatile boolean lifecycleReady;
/**
* 启动恢复是否安全完成。恢复失败时拒绝自动重投,避免在状态不明的库上继续派发。
*/
volatile boolean recoverySafe;
/**
* 节点是否声明了 v2 可靠任务能力。只有协商成功之后才发送类型 11..13,
* 否则继续走旧的无确认上报,保持向后兼容。
*/
volatile boolean v2Negotiated;
/** 本次连接的 Hello 协商结果;收到节点 Hello(或超时)后完成。 */
volatile CompletableFuture<Boolean> helloNegotiation;
/** 明确「未下发」的结果码:与 -1(发送失败/超时)区分开。 */
public static final byte REFUSED = -3;
/** 事务提交之后的通知/推送执行器:外部延迟(Telegram、慢浏览器)不得阻塞状态提交。 */
final ExecutorService notificationExecutor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "gallery-notify");
thread.setDaemon(true);
return thread;
});
/**
* 协议握手的专用线程。hello 等待响应是阻塞调用,绝不能放在 Netty 事件循环上,
* 否则响应本身要由同一个事件循环投递,会直接死锁。
*/
final ExecutorService handshakeExecutor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "node-protocol-handshake");
thread.setDaemon(true);
return thread;
});
final ExecutorService subscriptionSyncExecutor = Executors.newSingleThreadExecutor(r -> { final ExecutorService subscriptionSyncExecutor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "subscription-standby-sync"); Thread thread = new Thread(r, "subscription-standby-sync");
thread.setDaemon(true); thread.setDaemon(true);
return thread; return thread;
}); });
/** 测试/装配用:绑定上报处理器。 */
public void setTaskReportHandler(TaskReportHandler taskReportHandler) {
this.taskReportHandler = taskReportHandler;
}
public void setDownloadOperationIdResolver(java.util.function.IntFunction<String> resolver) {
this.downloadOperationIdResolver = resolver;
}
public void setDispatchResolver(java.util.function.IntFunction<GalleryTaskLifecycleService.DispatchDecision> resolver) {
this.dispatchResolver = resolver;
}
/**
* 由装配层在「绑定 handler/resolver + 完成启动恢复」之后调用:标记可以安全派发并建立连接。
*
* <p>初始连接不再放在 {@code @PostConstruct}:那时 handler/resolver 都还没绑定,
* 连接成功会把尚未确认的 UNKNOWN/预占任务当作普通任务重投。
*
* @param recoverySafe 启动恢复是否成功;false 时只允许连接,不允许自动重投
*/
public void startAfterWiring(boolean recoverySafe) {
this.recoverySafe = recoverySafe;
this.lifecycleReady = true;
initChannel();
}
/** 把 gid 解析成下发判定;未注册解析器或解析失败时按「拒绝下发」处理。 */
private GalleryTaskLifecycleService.DispatchDecision resolveDispatchDecision(int gid) {
java.util.function.IntFunction<GalleryTaskLifecycleService.DispatchDecision> resolver = dispatchResolver;
if (resolver == null)
return null;
try {
return resolver.apply(gid);
} catch (Exception e) {
log.warn("解析任务下发判定失败 gid={}", gid, e);
return null;
}
}
private String resolveDownloadOperationId(int gid) {
java.util.function.IntFunction<String> resolver = downloadOperationIdResolver;
if (resolver == null)
return null;
try {
return resolver.apply(gid);
} catch (Exception e) {
log.warn("解析任务 operationId 失败 gid={}", gid, e);
return null;
}
}
/** 节点是否已完成 v2 能力协商(供健康检查/测试观察)。 */
public boolean isReliableProtocolReady() {
return v2Negotiated;
}
/**
* 处理节点上报的任务状态。原先这些 JDBC 读写直接跑在 Netty 的 IO 线程上,
* N 条上报会阻塞该 IO 线程,连带拖慢心跳与请求响应;改为单线程顺序执行,
* 既不打乱「按上报顺序覆盖状态」的语义,也避免并发写库。
*/
ExecutorService statusApplyExecutor = newStatusApplyExecutor();
private static ExecutorService newStatusApplyExecutor() {
return Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r, "gallery-status-apply");
thread.setDaemon(true);
return thread;
});
}
/**
* 测试钩子:等待已提交的状态处理任务全部执行完毕。
*
* <p>单线程执行器天然有序,投递一个空任务并等它跑完,即说明此前提交的
* 任务都已处理完,从而让「上报后落库」的断言保持确定性。
*/
void awaitStatusApplied() {
try {
statusApplyExecutor.submit(() -> { }).get(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} catch (Exception e) {
log.debug("等待状态处理完成失败", e);
}
}
final AtomicBoolean subscriptionSyncQueued = new AtomicBoolean(); final AtomicBoolean subscriptionSyncQueued = new AtomicBoolean();
final AtomicBoolean subscriptionSyncRunning = new AtomicBoolean(); final AtomicBoolean subscriptionSyncRunning = new AtomicBoolean();
@@ -87,22 +240,28 @@ public class RemoteService {
@Value("${subscription.standby.sync-enabled:false}") @Value("${subscription.standby.sync-enabled:false}")
boolean subscriptionSyncEnabled; boolean subscriptionSyncEnabled;
@PostConstruct /**
void init() { * 这里刻意不再自动连接。
if(!initChannel()){ //如果远程服务器连接失败,则开启本地监听 *
monitor = new Thread(this::monitorFunc); * <p>原实现在 PostConstruct 里立即 initChannel(),而那一刻 TaskLifecycleWiring
monitor.start(); * 还没绑定上报处理器、operationId 解析器,也没执行启动恢复:连接成功后会马上把
} * 未完成任务(含 UNKNOWN/预占)当作普通任务重投给节点。
} * 现在改为由装配层在完成绑定与恢复后调用 startAfterWiring(boolean)。
*/
public boolean initChannel(){ public boolean initChannel(){
if (stopping || !connecting.compareAndSet(false, true))
return !isDead();
try { try {
if (!isDead())
return true;
int i; int i;
for(i=0; i<20; i++) { for(i=0; i<20; i++) {
try { try {
channelFuture = new Bootstrap() channelFuture = new Bootstrap()
.channel(NioSocketChannel.class) .channel(NioSocketChannel.class)
.group(new NioEventLoopGroup()) .group(networkGroup)
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 3_000)
.handler(new ChannelInitializer<NioSocketChannel>() { .handler(new ChannelInitializer<NioSocketChannel>() {
@Override @Override
protected void initChannel(NioSocketChannel channel) { protected void initChannel(NioSocketChannel channel) {
@@ -114,9 +273,14 @@ public class RemoteService {
} }
}).connect(new InetSocketAddress(ip, port + i)).sync(); }).connect(new InetSocketAddress(ip, port + i)).sync();
break; break;
}catch (Exception e){ } catch (InterruptedException e) {
Thread.currentThread().interrupt();
return false;
} catch (Exception e) {
log.error("连接storageNode失败,端口偏移量(重试次数):{}", i); log.error("连接storageNode失败,端口偏移量(重试次数):{}", i);
} }
if (stopping)
return false;
} }
//超过二十次连不上,主动抛出错误,由下方catch //超过二十次连不上,主动抛出错误,由下方catch
@@ -124,20 +288,51 @@ public class RemoteService {
throw new Exception(); throw new Exception();
} }
if (stopping) {
channelFuture.channel().close();
return false;
}
log.info("connect success"); log.info("connect success");
if(pushService != null) if(pushService != null)
pushService.storageNodeOnline(); pushService.storageNodeOnline();
channel = channelFuture.channel(); channel = channelFuture.channel();
closeMonitorSocket();
channel.writeAndFlush(new IdentityMessage("lionwebsite")); channel.writeAndFlush(new IdentityMessage("lionwebsite"));
// 协议 v2:先发旧的 Identity,再发 hello 做能力协商。
//子节点上线时,发送未完成的任务 // 协商完成之前不允许下发任何任务,也不允许自动重投未完成任务。
resetUndone(); CompletableFuture<Boolean> negotiation = queueHelloHandshake();
requestSubscriptionSync(); // 在协商线程上等待结果,再决定重投;绝不阻塞 Netty 事件循环。
handshakeExecutor.execute(() -> {
boolean reliable = awaitNegotiation(negotiation);
// 订阅快照与任务重投都要求生命周期装配已完成;装配前只保持连接。
requestSubscriptionSync();
if (!reliable) {
log.info("节点未确认 v2 能力,跳过自动重投未完成任务");
return;
}
if (!lifecycleReady) {
log.info("生命周期装配未就绪,跳过自动重投未完成任务");
return;
}
if (!recoverySafe) {
log.error("启动恢复未安全完成,拒绝自动重投未完成任务");
return;
}
try {
resetUndone();
} catch (RuntimeException e) {
log.warn("重投未完成任务失败", e);
}
});
return true; return true;
}catch (Exception e){ }catch (Exception e){
log.error("connect node failed, wait for node back online", e); log.error("connect node failed, wait for node back online", e);
return false; return false;
} finally {
connecting.set(false);
if (isDead())
startMonitor();
} }
} }
@@ -147,7 +342,7 @@ public class RemoteService {
return -2; return -2;
} }
channelFuture.channel().close(); channelFuture.channel().close().awaitUninterruptibly();
if(initChannel()){ if(initChannel()){
return 0; return 0;
@@ -157,40 +352,55 @@ public class RemoteService {
} }
public byte checkAvailability(){ public byte checkAvailability(){
AvailableCheckMessage acm = new AvailableCheckMessage(); return sendRequest(new AvailableCheckMessage(), 10, TimeUnit.SECONDS);
acm.setMessageId(atomicInteger.getAndIncrement()); }
channel.writeAndFlush(acm); byte sendRequest(AbstractMessage message, long timeout, TimeUnit unit) {
Channel target = channel;
if (stopping || target == null || !target.isActive())
return -1;
message.setMessageId(atomicInteger.getAndIncrement());
DefaultPromise<AbstractMessage> promise = new DefaultPromise<>(eventLoopGroup); DefaultPromise<AbstractMessage> promise = new DefaultPromise<>(eventLoopGroup);
promiseHashMap.put(acm.messageId, promise); promiseHashMap.put(message.messageId, promise);
try { try {
boolean result = promise.await(10, TimeUnit.SECONDS); target.writeAndFlush(message).addListener(future -> {
if(result){ if (!future.isSuccess())
ResponseMessage rsm = (ResponseMessage)promise.getNow(); promise.tryFailure(future.cause() == null ? new IOException("节点发送失败") : future.cause());
return rsm.getResult(); });
} if (promise.await(timeout, unit) && promise.isSuccess()
else return -1; && promise.getNow() instanceof ResponseMessage response)
}catch (InterruptedException e){ return response.getResult();
log.warn("checkAvailability interrupted", e); return -1;
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); Thread.currentThread().interrupt();
return -1; return -1;
} catch (Exception e) {
log.warn("节点请求失败: messageId={}", message.messageId, e);
return -1;
} finally {
promiseHashMap.remove(message.messageId, promise);
} }
} }
/** 请求将当前全部订阅状态异步同步到存储节点,短时间内的多次请求会合并。 */ /** 请求将当前全部订阅状态异步同步到存储节点,短时间内的多次请求会合并。 */
public void requestSubscriptionSync() { public void requestSubscriptionSync() {
if (!subscriptionSyncEnabled) if (stopping || !subscriptionSyncEnabled)
return; return;
subscriptionSyncQueued.set(true); subscriptionSyncQueued.set(true);
if (!subscriptionSyncRunning.compareAndSet(false, true)) if (!subscriptionSyncRunning.compareAndSet(false, true))
return; return;
subscriptionSyncExecutor.execute(this::drainSubscriptionSyncQueue); try {
subscriptionSyncExecutor.execute(this::drainSubscriptionSyncQueue);
} catch (java.util.concurrent.RejectedExecutionException e) {
subscriptionSyncRunning.set(false);
if (!stopping) throw e;
}
} }
@Scheduled(fixedDelayString = "${subscription.standby.retry-interval-ms:60000}") // 这里刻意没有「定期重发整份快照」的定时任务:
void scheduledSubscriptionSync() { // 快照只在订阅内容变化时推送(各处 requestSubscriptionSync),节点上线时补推一次。
requestSubscriptionSync(); // 备机的过期语义是「主站失联」,由节点按「收到主站的任何消息」判定新鲜度——
} // 主站每 30 分钟的可用性检查即可持续刷新它,因此不需要为续期而周期性传输快照。
private void drainSubscriptionSyncQueue() { private void drainSubscriptionSyncQueue() {
try { try {
@@ -207,32 +417,15 @@ public class RemoteService {
} }
private void syncSubscriptionSnapshotOnce() { private void syncSubscriptionSnapshotOnce() {
SubscriptionSnapshotMessage message = null;
DefaultPromise<AbstractMessage> promise = null;
try { try {
message = subscriptionStandbySnapshotService.build(); SubscriptionSnapshotMessage message = subscriptionStandbySnapshotService.build();
message.setMessageId(atomicInteger.getAndIncrement()); byte result = sendRequest(message, 30, TimeUnit.SECONDS);
promise = new DefaultPromise<>(eventLoopGroup); if (result == 0 || result == 3)
promiseHashMap.put(message.messageId, promise); log.info("订阅快照同步完成 revision={} result={}", shortRevision(message.getRevision()), result);
channel.writeAndFlush(message); else
if (promise.await(30, TimeUnit.SECONDS)) { log.warn("订阅快照同步失败或超时 revision={} result={}", shortRevision(message.getRevision()), result);
AbstractMessage reply = promise.getNow();
if (reply instanceof ResponseMessage response && (response.getResult() == 0 || response.getResult() == 3))
log.info("订阅快照同步完成 revision={} result={}", shortRevision(message.getRevision()), response.getResult());
else
log.warn("订阅快照同步失败 revision={} result={}", shortRevision(message.getRevision()),
reply instanceof ResponseMessage response ? response.getResult() : "invalid-response");
} else {
log.warn("订阅快照同步超时 revision={}", shortRevision(message.getRevision()));
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
log.warn("订阅快照同步线程被中断");
} catch (Exception e) { } catch (Exception e) {
log.warn("生成或发送订阅快照失败: {}", e.getMessage()); log.warn("生成或发送订阅快照失败: {}", e.getMessage());
} finally {
if (message != null && promise != null)
promiseHashMap.remove(message.messageId, promise);
} }
} }
@@ -241,8 +434,26 @@ public class RemoteService {
} }
@PreDestroy @PreDestroy
void shutdownSubscriptionSync() { void shutdownResources() {
stopping = true;
closeMonitorSocket();
if (monitor != null) monitor.interrupt();
failPendingRequests();
if (channel != null) channel.close();
statusApplyExecutor.shutdownNow();
notificationExecutor.shutdownNow();
handshakeExecutor.shutdownNow();
subscriptionSyncExecutor.shutdownNow(); subscriptionSyncExecutor.shutdownNow();
networkGroup.shutdownGracefully();
eventLoopGroup.shutdownGracefully();
}
private void failPendingRequests() {
promiseHashMap.forEach((id, promise) -> promise.tryFailure(new IOException("节点连接已关闭")));
promiseHashMap.clear();
retryStatusWaiters.forEach((gid, waiters) ->
waiters.forEach(waiter -> waiter.completeExceptionally(new IOException("节点连接已关闭"))));
retryStatusWaiters.clear();
} }
public boolean isDead(){ public boolean isDead(){
@@ -250,7 +461,11 @@ public class RemoteService {
} }
public void resetUndone(){ public void resetUndone(){
if (channelFuture.channel() == null || !channelFuture.channel().isActive()) // 只有「连接在线 + v2 已协商 + 生命周期装配完成 + 启动恢复安全完成」才允许批量重投;
// 任何一项不满足都说明还不知道任务真实状态,盲目下发会制造未计量副作用。
if (channelFuture == null || channelFuture.channel() == null || !channelFuture.channel().isActive())
return;
if (!v2Negotiated || !lifecycleReady || !recoverySafe)
return; return;
Gallery[] galleries = galleryMapper.selectUnDoneGalleries(); Gallery[] galleries = galleryMapper.selectUnDoneGalleries();
@@ -263,34 +478,135 @@ public class RemoteService {
} }
public byte addGalleryToQueue(Gallery gallery){ public byte addGalleryToQueue(Gallery gallery){
GalleryTaskLifecycleService.DispatchDecision decision = resolveDispatchDecision(gallery.getGid());
if (decision != null) {
// 明确拒绝(UNKNOWN/预占/待删除等):绝不能退化为无 operationId 的旧命令。
if (!decision.dispatchable())
return REFUSED;
return sendTaskCommand(TaskCommandMessage.ACTION_DOWNLOAD, decision.operationId(),
gallery.getGid(), gallery.getName());
}
// 兼容:装配层未注册判定解析器时(如单元测试),退回只按 operationId 解析的旧行为。
String operationId = resolveDownloadOperationId(gallery.getGid());
if (operationId != null)
return sendTaskCommand(TaskCommandMessage.ACTION_DOWNLOAD, operationId, gallery.getGid(), gallery.getName());
return legacyDownloadCommand(gallery.getGid(), gallery.getName());
}
/** 旧的无 operationId 下载命令,供未启用 v2 或无法解析 operationId 时退避使用。 */
private byte legacyDownloadCommand(int gid, String name) {
GalleryTask galleryTask = new GalleryTask(); GalleryTask galleryTask = new GalleryTask();
galleryTask.setGid(gallery.getGid()); galleryTask.setGid(gid);
galleryTask.setName(gallery.getName()); galleryTask.setName(name);
DownloadPostMessage message = new DownloadPostMessage();
message.setGalleryTask(galleryTask);
return sendRequest(message, 10, TimeUnit.SECONDS);
}
DownloadPostMessage dpm = new DownloadPostMessage(); /**
dpm.messageId = atomicInteger.getAndIncrement(); * 在专用线程上执行 hello 协商,并返回一个在「收到节点 Hello」或「等待超时」后完成的 future。
dpm.setGalleryTask(galleryTask); *
* <p>节点的应答是 Hello 而不是 ResponseMessage,因此绝不能走 sendRequest 的响应等待器
DefaultPromise<AbstractMessage> promise = new DefaultPromise<>(eventLoopGroup); * (那会白等 10 秒超时,排在后面的自动重投也只能 10 秒后才开始)。
promiseHashMap.put(dpm.messageId, promise); * 这里直接 writeAndFlush,并让 channelRead 的 Hello 分支完成 future;
channel.writeAndFlush(dpm); * 超时用 completeOnTimeout 处理,写失败立刻完成。
*/
private CompletableFuture<Boolean> queueHelloHandshake() {
// 10 秒内没等到节点 Hello 就按「未协商」完成,避免后续重投被无限期挂起。
CompletableFuture<Boolean> result = new CompletableFuture<Boolean>()
.completeOnTimeout(false, 10, TimeUnit.SECONDS);
helloNegotiation = result;
if (stopping) {
result.complete(false);
return result;
}
try { try {
boolean result = promise.await(10, TimeUnit.SECONDS); handshakeExecutor.execute(() -> {
if(result){ v2Negotiated = false;
ResponseMessage rsm = (ResponseMessage)promise.getNow(); try {
return rsm.getResult(); Channel target = channel;
} if (target == null || !target.isActive()) {
else return -1; result.complete(false);
}catch (InterruptedException e){ return;
log.warn("addGalleryToQueue interrupted", e); }
// Hello 不写入 promiseHashMap:节点回的是 Hello,没有 ResponseMessage 等待者。
target.writeAndFlush(new ProtocolHelloMessage()).addListener(future -> {
if (!future.isSuccess()) {
if (!stopping)
log.warn("协议协商发送失败: {}", future.cause() == null
? "unknown" : future.cause().getClass().getSimpleName());
result.complete(false);
}
});
} catch (RuntimeException e) {
log.warn("协议协商失败", e);
result.complete(false);
}
});
} catch (java.util.concurrent.RejectedExecutionException e) {
if (!stopping) log.warn("安排协议协商失败", e);
result.complete(false);
}
return result;
}
/** 等待协商结果;超时或中断都按「未协商」处理(宁可不下发,也不盲目重投)。 */
boolean awaitNegotiation(CompletableFuture<Boolean> negotiation) {
if (negotiation == null)
return v2Negotiated;
try {
// future 自身已在 10 秒时按「未协商」完成,这里留少量余量即可。
return negotiation.get(12, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); Thread.currentThread().interrupt();
return -1; return false;
}finally { } catch (Exception e) {
promiseHashMap.remove(dpm.messageId, promise); return false;
} }
} }
/**
* 下发一条带稳定 operationId 的任务命令;未协商 v2 时退化为旧的 DownloadPostMessage,
* 保证新旧节点组合仍可用(旧节点上无法保证可靠 ACK,这是已知的兼容边界)。
*/
public byte sendTaskCommand(String action, String operationId, int gid, String name) {
if (!v2Negotiated)
return TaskCommandMessage.ACTION_DELETE.equals(action)
? legacyDeleteCommand(gid, name)
: legacyDownloadCommand(gid, name);
TaskCommandMessage command = new TaskCommandMessage();
command.setOperationId(operationId);
command.setAction(action);
command.setGid(gid);
command.setName(name);
return sendRequest(command, 10, TimeUnit.SECONDS);
}
/** 删除命令:带独立稳定 operationId,重试复用同一个 ID。 */
public byte sendDeleteCommand(String operationId, int gid, String name) {
if (!v2Negotiated)
return legacyDeleteCommand(gid, name);
TaskCommandMessage command = new TaskCommandMessage();
command.setOperationId(operationId);
command.setAction(TaskCommandMessage.ACTION_DELETE);
command.setGid(gid);
command.setName(name);
return sendRequest(command, 10, TimeUnit.SECONDS);
}
/** 旧的无 operationId 删除命令,供未协商 v2 时退避使用。 */
private byte legacyDeleteCommand(int gid, String name) {
Gallery gallery = new Gallery();
gallery.setGid(gid);
gallery.setName(name);
return deleteGallery(gallery);
}
public RetryResult retryGallery(Gallery gallery){ public RetryResult retryGallery(Gallery gallery){
// 提交结果未知或被删除的任务不能重试:节点可能已经把未计量任务当成新下载。
GalleryTaskLifecycleService.DispatchDecision decision = resolveDispatchDecision(gallery.getGid());
if (decision != null && !decision.dispatchable())
return new RetryResult(false, "该任务提交结果未知或正在删除,需要人工对账后再重试");
CompletableFuture<String> statusFuture = new CompletableFuture<>(); CompletableFuture<String> statusFuture = new CompletableFuture<>();
retryStatusWaiters.computeIfAbsent(gallery.getGid(), ignored -> new CopyOnWriteArrayList<>()) retryStatusWaiters.computeIfAbsent(gallery.getGid(), ignored -> new CopyOnWriteArrayList<>())
.add(statusFuture); .add(statusFuture);
@@ -298,6 +614,13 @@ public class RemoteService {
byte submitResult = addGalleryToQueue(gallery); byte submitResult = addGalleryToQueue(gallery);
if(submitResult != 0 && !statusFuture.isDone()) if(submitResult != 0 && !statusFuture.isDone())
return new RetryResult(false, "节点未接受重试请求"); return new RetryResult(false, "节点未接受重试请求");
// 竞态兜底:本方法读到的 Gallery 是调用方传入的旧快照,注册等待者与节点被通知之间
// 节点可能刚完成清理(例如 ACK 之后已清队列、重复 DOWNLOAD 只回放 receipt 不再产生新上报)。
// 此时按「当前库里的真实状态」收尾:只认重新读到的完成状态,绝不把传入的旧进行中状态
// 当作节点的新确认返回。
String completedStatus = currentCompletedStatus(gallery.getGid());
if (completedStatus != null && !statusFuture.isDone())
statusFuture.complete(completedStatus);
return new RetryResult(true, statusFuture.get(10, TimeUnit.SECONDS)); return new RetryResult(true, statusFuture.get(10, TimeUnit.SECONDS));
}catch (TimeoutException e){ }catch (TimeoutException e){
return new RetryResult(false, "节点已收到重试请求,但未及时返回任务状态"); return new RetryResult(false, "节点已收到重试请求,但未及时返回任务状态");
@@ -322,52 +645,77 @@ public class RemoteService {
waiters.forEach(waiter -> waiter.complete(status)); waiters.forEach(waiter -> waiter.complete(status));
} }
/**
* 重新读库确认任务是否已真正完成;只用于「节点已接受重试」之后的竞态兜底。
*
* <p>读库失败一律返回 null:宁可继续等上报,也不能凭一次失败的读把未知状态说成完成。
*/
private String currentCompletedStatus(int gid) {
try {
Gallery current = galleryMapper.selectGalleryByGid(gid);
if (current != null && TaskStatus.COMPLETE.equals(current.getStatus()))
return current.getStatus();
} catch (RuntimeException e) {
log.warn("重试时读取任务当前状态失败 gid={}", gid, e);
}
return null;
}
public record RetryResult(boolean success, String message) {} public record RetryResult(boolean success, String message) {}
public byte deleteGallery(Gallery gallery){ public byte deleteGallery(Gallery gallery){
DeleteGalleryMessage dgm = new DeleteGalleryMessage(); DeleteGalleryMessage message = new DeleteGalleryMessage();
dgm.setGalleryName(gallery.getName()); message.setGalleryName(gallery.getName());
dgm.messageId = atomicInteger.getAndIncrement(); return sendRequest(message, 10, TimeUnit.SECONDS);
}
channel.writeAndFlush(dgm); private void startMonitor() {
DefaultPromise<AbstractMessage> promise = new DefaultPromise<>(eventLoopGroup); if (stopping || !monitoring.compareAndSet(false, true))
promiseHashMap.put(dgm.messageId, promise); return;
try{ monitor = new Thread(this::monitorFunc, "storage-node-monitor");
boolean result = promise.await(10, TimeUnit.SECONDS); monitor.setDaemon(true);
if(result){ monitor.start();
ResponseMessage rsm = (ResponseMessage) promise.getNow(); }
return rsm.getResult();
}else return -1; private void closeMonitorSocket() {
}catch (InterruptedException e){ ServerSocket socket = monitorSocket;
log.warn("deleteGallery interrupted", e); if (socket != null) {
Thread.currentThread().interrupt(); try { socket.close(); }
return -1; catch (IOException e) { log.debug("关闭节点监听失败", e); }
} }
} }
public void monitorFunc(){ public void monitorFunc(){
int real_port = CustomUtil._findIdlePort(port + 1); try (ServerSocket socket = new ServerSocket(CustomUtil._findIdlePort(port + 1))) {
log.info("监听端口: {}等待节点上线", real_port); monitorSocket = socket;
try(ServerSocket socket = new ServerSocket(real_port)) { if (stopping || !isDead())
Socket client; return;
while(true){ log.info("监听端口: {}等待节点上线", socket.getLocalPort());
client = socket.accept(); while (!stopping) {
try (Socket client = socket.accept()) {
if(client.getInetAddress().getHostAddress().equals(ip)){ if (!client.getInetAddress().getHostAddress().equals(ip))
//连接之后发送lionwebsite,否则存储节点不能确认这个端口是否有效 continue;
OutputStream outputStream = client.getOutputStream(); OutputStream output = client.getOutputStream();
outputStream.write("lionwebsite".getBytes()); output.write("lionwebsite".getBytes(java.nio.charset.StandardCharsets.UTF_8));
outputStream.flush(); output.flush();
outputStream.close(); client.shutdownOutput();
log.info("尝试连接"); if (initChannel())
initChannel(); break;
client.close();
socket.close();
break;
} }
} }
} catch (IOException e) { } catch (IOException e) {
throw new RuntimeException(e); if (!stopping && isDead())
log.warn("等待节点上线失败", e);
} finally {
monitorSocket = null;
monitoring.set(false);
// A connection may flap while the previous monitor is still exiting.
if (!stopping && isDead()) {
try { networkGroup.next().schedule(this::startMonitor, 5, TimeUnit.SECONDS); }
catch (java.util.concurrent.RejectedExecutionException e) {
if (!stopping) log.warn("安排节点监听重试失败", e);
}
}
} }
} }
@@ -382,55 +730,144 @@ public class RemoteService {
//下载状态 //下载状态
if(msg instanceof DownloadStatusMessage dsm){ if(msg instanceof DownloadStatusMessage dsm){
GalleryTask[] galleryTasks = dsm.getGalleryTasks(); GalleryTask[] galleryTasks = dsm.getGalleryTasks();
for (GalleryTask galleryTask : galleryTasks) { // 落库与通知可能较慢(含 Telegram 推送),交给单线程顺序执行,
Gallery gallery = galleryMapper.selectGalleryByGid(galleryTask.getGid()); // 不再占用 Netty IO 线程;异常在此兜住,避免打进 IO 线程。
if (gallery == null) { statusApplyExecutor.execute(() -> {
log.warn("收到节点上报的未知任务状态,已忽略: gid={}, name={}", try {
galleryTask.getGid(), galleryTask.getName()); applyReportedStatus(galleryTasks);
continue; } catch (Exception e) {
log.warn("处理节点上报的任务状态失败", e);
} }
gallery.setProceeding(galleryTask.getProceeding()); });
}
if(!gallery.getName().equals(galleryTask.getName()) && galleryTask.getName() != null) else if(msg instanceof ProtocolHelloMessage hello) {
gallery.setName(galleryTask.getName()); // 节点声明了 v2 可靠任务能力;此后才允许使用类型 11..13。
v2Negotiated = hello.supportsReliableTasks();
if(galleryTask.getStatus() == GalleryTask.COMPRESS_COMPLETE) { log.info("节点协议协商完成 v2={} version={}", v2Negotiated, hello.getProtocolVersion());
boolean justCompleted = !"下载完成".equals(gallery.getStatus()); CompletableFuture<Boolean> negotiation = helloNegotiation;
gallery.setStatus("下载完成"); if (negotiation != null)
if (justCompleted) negotiation.complete(v2Negotiated);
pushService.downloadComplete(gallery); }
} else if(msg instanceof TaskReportMessage report) {
else if(galleryTask.getStatus() == GalleryTask.COMPRESSING) // 事务提交与 ACK 都放在单线程消费队列里顺序执行,避免并发写库。
gallery.setStatus("压缩中"); statusApplyExecutor.execute(() -> consumeV2Report(report));
else if(galleryTask.getStatus() == GalleryTask.DOWNLOAD_COMPLETE)
gallery.setStatus("等待压缩");
else if(galleryTask.getStatus() == GalleryTask.DOWNLOADING)
gallery.setStatus("下载中");
log.info(gallery.getName() + "下载进度:" + gallery.getProceeding() + "/" + gallery.getPages());
galleryMapper.updateGallery(gallery);
completeRetryStatusWaiters(gallery.getGid(), gallery.getStatus());
}
webSocketService.updateTaskProcessing(galleryTasks);
} }
else if(msg instanceof ResponseMessage rsm) { else if(msg instanceof ResponseMessage rsm) {
Promise<AbstractMessage> promise = promiseHashMap.remove(rsm.messageId); Promise<AbstractMessage> promise = promiseHashMap.remove(rsm.messageId);
if(promise != null) if(promise != null)
promise.setSuccess(rsm); promise.trySuccess(rsm);
else else
log.warn("收到无等待者的响应消息: messageId={}", rsm.messageId); log.warn("收到无等待者的响应消息: messageId={}", rsm.messageId);
} }
} }
/** 消费协议 v2 上报:事务提交 → 唤醒重试等待者 → 回 ACK → 独立执行器做通知/推送。 */
private void consumeV2Report(TaskReportMessage report) {
try {
TaskReportHandler handler = taskReportHandler;
if (handler == null) {
log.warn("收到 v2 上报但未装配处理器,暂不回执 reportId={}", report.getReportId());
return;
}
TaskReportHandler.CommittedReport committed = handler.apply(report);
if (committed == null) {
// 提交失败:不回 ACK,节点保留快照后重投。
log.warn("v2 上报未提交,等待节点重投 reportId={}", report.getReportId());
return;
}
// 先完成等待者:apply 已确认提交,这一步绝不能因随后的 ACK 写入抛异常而丢失。
completeRetryStatusWaitersFor(committed.acceptedTasks());
// 提交成功之后才回执,让节点清理快照。
Channel target = channel;
if (target != null && target.isActive())
target.writeAndFlush(new TaskReportAckMessage(committed.reportId(), TaskReportAckMessage.RESULT_SUCCESS));
dispatchNotifications(report, committed);
} catch (Exception e) {
log.warn("处理 v2 上报失败 reportId={}", report.getReportId(), e);
}
}
/** 事务提交之后、ACK 与通知之前,按「真正被接受的更新」唤醒等待重试结果的调用方。
* 被过滤的过期/旧 operation/待删除任务不在 acceptedTasks 里,不得完成等待者。 */
private void completeRetryStatusWaitersFor(List<GalleryTask> acceptedTasks) {
if (acceptedTasks == null || acceptedTasks.isEmpty())
return;
for (GalleryTask task : acceptedTasks) {
if (task == null)
continue;
String status = TaskStatus.fromReport(task.getStatus());
if (status != null)
completeRetryStatusWaiters(task.getGid(), status);
}
}
/** 通知/推送在独立执行器里执行,外部延迟不会阻塞状态提交。 */
private void dispatchNotifications(TaskReportMessage report, TaskReportHandler.CommittedReport committed) {
// 只推送事务里真正被接受的更新(含下载中/压缩中等中间状态);
// 过期、旧世代、待删除任务已被过滤掉,不会把它们带到前端。
GalleryTask[] accepted = committed.acceptedTasks() == null
? new GalleryTask[0]
: committed.acceptedTasks().toArray(new GalleryTask[0]);
List<Gallery> newlyCompleted = committed.newlyCompleted();
try {
notificationExecutor.execute(() -> {
try {
for (Gallery gallery : newlyCompleted)
pushService.downloadComplete(gallery);
if (accepted.length > 0)
webSocketService.updateTaskProcessing(accepted);
} catch (Exception e) {
log.warn("任务完成通知/推送失败", e);
}
});
} catch (java.util.concurrent.RejectedExecutionException e) {
// 通知队列已关闭只是「推不出去」,既不能影响已提交的 ACK,也不能影响重试等待者。
log.warn("通知任务被拒绝,状态仍已提交 reportId={}", report.getReportId());
}
}
/** 把节点上报的任务状态写回库里并推送给前端,按上报顺序逐个处理。 */
private void applyReportedStatus(GalleryTask[] galleryTasks) {
for (GalleryTask galleryTask : galleryTasks) {
Gallery gallery = galleryMapper.selectGalleryByGid(galleryTask.getGid());
if (gallery == null) {
log.warn("收到节点上报的未知任务状态,已忽略: gid={}, name={}",
galleryTask.getGid(), galleryTask.getName());
continue;
}
gallery.setProceeding(galleryTask.getProceeding());
if(!gallery.getName().equals(galleryTask.getName()) && galleryTask.getName() != null)
gallery.setName(galleryTask.getName());
if(galleryTask.getStatus() == GalleryTask.COMPRESS_COMPLETE) {
boolean justCompleted = !"下载完成".equals(gallery.getStatus());
gallery.setStatus("下载完成");
if (justCompleted)
pushService.downloadComplete(gallery);
}
else if(galleryTask.getStatus() == GalleryTask.COMPRESSING)
gallery.setStatus("压缩中");
else if(galleryTask.getStatus() == GalleryTask.DOWNLOAD_COMPLETE)
gallery.setStatus("等待压缩");
else if(galleryTask.getStatus() == GalleryTask.DOWNLOADING)
gallery.setStatus("下载中");
log.info(gallery.getName() + "下载进度:" + gallery.getProceeding() + "/" + gallery.getPages());
galleryMapper.updateGallery(gallery);
completeRetryStatusWaiters(gallery.getGid(), gallery.getStatus());
}
webSocketService.updateTaskProcessing(galleryTasks);
}
@Override @Override
public void channelUnregistered(ChannelHandlerContext ctx) { public void channelUnregistered(ChannelHandlerContext ctx) {
log.info("{}", ctx.channel()); if (ctx.channel() == channel) {
log.info("{}", channel); failPendingRequests();
if(ctx.channel() != null && ctx.channel().remoteAddress().toString().equals(channel.remoteAddress().toString())){ if (!stopping) {
log.info("activate monitor thread, waiting for node back online"); pushService.storageNodeOffline();
pushService.storageNodeOffline(); startMonitor();
monitor = new Thread(RemoteService.this::monitorFunc); }
monitor.start();
} }
} }
} }
@@ -5,6 +5,7 @@ import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper; import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.SubBind; import com.lion.lionwebsite.Domain.SubBind;
import com.lion.lionwebsite.Domain.SubUpdateRecord; import com.lion.lionwebsite.Domain.SubUpdateRecord;
import com.lion.lionwebsite.Domain.SubscriptionEvent;
import com.lion.lionwebsite.Domain.SubscriptionAccount; import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.FileDownload; import com.lion.lionwebsite.Util.FileDownload;
@@ -17,37 +18,75 @@ import lombok.extern.slf4j.Slf4j;
import org.jsoup.Jsoup; import org.jsoup.Jsoup;
import org.jsoup.select.Elements; import org.jsoup.select.Elements;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.transaction.support.TransactionTemplate;
import jakarta.annotation.PreDestroy;
import java.io.IOException; import java.io.IOException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.Files; import java.nio.file.Files;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Date; import java.util.Date;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.Lock;
import java.util.function.Supplier; import java.util.function.Supplier;
@Service @Service
@Slf4j @Slf4j
@RequiredArgsConstructor @RequiredArgsConstructor
public class SubService { public class SubService {
final SubMapper subMapper; final SubMapper subMapper;
final UserMapper userMapper; final UserMapper userMapper;
final SubscriptionRefreshService refreshService; final SubscriptionRefreshService refreshService;
final RemoteService remoteService; final RemoteService remoteService;
final SubscriptionStateCoordinator stateCoordinator; final SubscriptionStateCoordinator stateCoordinator;
@Autowired
@Qualifier("mainTransactionTemplate")
TransactionTemplate transactions;
@Autowired(required = false)
SubscriptionEventService subscriptionEvents;
/**
* 归属地查询是同步外呼(最长十几秒),且失败只影响记录里的一列展示。
* 若放在订阅分发路径上,用户取一次订阅就得先等它完成,因此改为后台补齐。
*
* <p>非 final:测试可替换为受控执行器,避免在单测里触发真实外呼。
*/
Executor locationExecutor = newFixedLocationExecutor();
private static ExecutorService newFixedLocationExecutor() {
return new java.util.concurrent.ThreadPoolExecutor(2, 2, 0, java.util.concurrent.TimeUnit.MILLISECONDS,
new java.util.concurrent.ArrayBlockingQueue<>(128), r -> {
Thread thread = new Thread(r, "sub-location-resolver");
thread.setDaemon(true);
return thread;
}, new java.util.concurrent.ThreadPoolExecutor.AbortPolicy());
}
public String insertSubscriptionAccount(String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) { public String insertSubscriptionAccount(String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) {
return withWriteLock(() -> insertSubscriptionAccountUnlocked(name, upstreamKey, filterHighMultiplier, enabled));
}
private String insertSubscriptionAccountUnlocked(String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
if (name == null || name.isBlank() || upstreamKey == null || upstreamKey.isBlank()) SubscriptionAccount account;
return response.failure("名称和上游 key 不能为空").toJSONString(); Lock lock = stateCoordinator.writeLock();
if (subMapper.countSubscriptionAccountName(name.trim()) > 0 || subMapper.countSubscriptionAccountKey(upstreamKey.trim()) > 0) lock.lock();
return response.failure("名称或上游 key 已存在").toJSONString(); try {
SubscriptionAccount account = new SubscriptionAccount(null, name.trim(), upstreamKey.trim(), filterHighMultiplier, enabled, null, null, null, null, 0, null, null); if (name == null || name.isBlank() || upstreamKey == null || upstreamKey.isBlank())
subMapper.insertSubscriptionAccount(account); return response.failure("名称和上游 key 不能为空").toJSONString();
if (subMapper.countSubscriptionAccountName(name.trim()) > 0 || subMapper.countSubscriptionAccountKey(upstreamKey.trim()) > 0)
return response.failure("名称或上游 key 已存在").toJSONString();
account = new SubscriptionAccount();
account.setName(name.trim());
account.setUpstreamKey(upstreamKey.trim());
account.setFilterHighMultiplier(filterHighMultiplier);
account.setEnabled(enabled);
subMapper.insertSubscriptionAccount(account);
} finally {
lock.unlock();
}
if (enabled) if (enabled)
refreshService.refresh(account.getId()); refreshService.refresh(account.getId());
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
@@ -64,27 +103,34 @@ public class SubService {
} }
public String updateSubscriptionAccount(Integer id, String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) { public String updateSubscriptionAccount(Integer id, String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) {
return withWriteLock(() -> updateSubscriptionAccountUnlocked(id, name, upstreamKey, filterHighMultiplier, enabled));
}
private String updateSubscriptionAccountUnlocked(Integer id, String name, String upstreamKey, boolean filterHighMultiplier, boolean enabled) {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
SubscriptionAccount account = subMapper.selectSubscriptionAccount(id); SubscriptionAccount account;
if (account == null) Lock lock = stateCoordinator.writeLock();
return response.failure("子账号不存在").toJSONString(); lock.lock();
if (name == null || name.isBlank() || upstreamKey == null || upstreamKey.isBlank()) try {
return response.failure("名称和上游 key 不能为空").toJSONString(); account = subMapper.selectSubscriptionAccount(id);
for (SubscriptionAccount existing : subMapper.selectAllSubscriptionAccounts()) { if (account == null)
if (!existing.getId().equals(id) && (existing.getName().equals(name.trim()) || existing.getUpstreamKey().equals(upstreamKey.trim()))) return response.failure("子账号不存在").toJSONString();
return response.failure("名称或上游 key 已存在").toJSONString(); if (name == null || name.isBlank() || upstreamKey == null || upstreamKey.isBlank())
return response.failure("名称和上游 key 不能为空").toJSONString();
for (SubscriptionAccount existing : subMapper.selectAllSubscriptionAccounts()) {
if (!existing.getId().equals(id) && (existing.getName().equals(name.trim()) || existing.getUpstreamKey().equals(upstreamKey.trim())))
return response.failure("名称或上游 key 已存在").toJSONString();
}
boolean changed = !account.getUpstreamKey().equals(upstreamKey.trim())
|| account.isFilterHighMultiplier() != filterHighMultiplier || account.isEnabled() != enabled;
account.setName(name.trim());
account.setUpstreamKey(upstreamKey.trim());
account.setFilterHighMultiplier(filterHighMultiplier);
account.setEnabled(enabled);
subMapper.updateSubscriptionAccount(account);
if (changed)
refreshService.invalidateCache(id);
} finally {
lock.unlock();
} }
account.setName(name.trim()); if (enabled)
account.setUpstreamKey(upstreamKey.trim()); refreshService.refresh(id);
account.setFilterHighMultiplier(filterHighMultiplier);
account.setEnabled(enabled);
subMapper.updateSubscriptionAccount(account);
if (enabled && !refreshService.refresh(id))
refreshService.invalidateCache(id);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
return response.success(accountJson(account)).toJSONString(); return response.success(accountJson(account)).toJSONString();
} }
@@ -101,15 +147,12 @@ public class SubService {
if (account.getBoundUserCount() != null && account.getBoundUserCount() > 0) if (account.getBoundUserCount() != null && account.getBoundUserCount() > 0)
return response.failure("子账号仍绑定用户,请先改绑").toJSONString(); return response.failure("子账号仍绑定用户,请先改绑").toJSONString();
subMapper.deleteSubscriptionAccount(id); subMapper.deleteSubscriptionAccount(id);
refreshService.invalidateCache(id);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
return response.success("删除成功").toJSONString(); return response.success("删除成功").toJSONString();
} }
public String refreshSubscriptionAccount(Integer id) { public String refreshSubscriptionAccount(Integer id) {
return withWriteLock(() -> refreshSubscriptionAccountUnlocked(id));
}
private String refreshSubscriptionAccountUnlocked(Integer id) {
boolean success = refreshService.refresh(id); boolean success = refreshService.refresh(id);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
return success ? Response._success("刷新成功") : Response._failure("刷新失败,请查看子账号错误状态"); return success ? Response._success("刷新成功") : Response._failure("刷新失败,请查看子账号错误状态");
@@ -149,6 +192,7 @@ public class SubService {
String key = RandomUtil.randomString(8); String key = RandomUtil.randomString(8);
while (subMapper.selectSubBindExist(key)) while (subMapper.selectSubBindExist(key))
key = RandomUtil.randomString(8); key = RandomUtil.randomString(8);
rememberEventBindings();
subMapper.updateSubBindKey(user, key); subMapper.updateSubBindKey(user, key);
subMapper.deleteSubUpdateRecord(user); subMapper.deleteSubUpdateRecord(user);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
@@ -181,6 +225,16 @@ public class SubService {
} }
public void updateSub(HttpServletResponse response, HttpServletRequest request, String client, String key) { public void updateSub(HttpServletResponse response, HttpServletRequest request, String client, String key) {
Lock lock = stateCoordinator.readLock();
lock.lock();
try {
distributeSubscription(response, request, client, key);
} finally {
lock.unlock();
}
}
private void distributeSubscription(HttpServletResponse response, HttpServletRequest request, String client, String key) {
if (key == null || client == null) if (key == null || client == null)
return; return;
SubBind subBind = subMapper.selectSubBind(key); SubBind subBind = subMapper.selectSubBind(key);
@@ -188,15 +242,11 @@ public class SubService {
sendStatus(response, HttpServletResponse.SC_NOT_FOUND, "subscription not found"); sendStatus(response, HttpServletResponse.SC_NOT_FOUND, "subscription not found");
return; return;
} }
String ip = request.getRemoteAddr(); String ip = resolveClientIp(request);
if ("127.0.0.1".equals(ip)) {
ip = request.getHeader("X-Forwarded-For");
if (ip != null && ip.contains(",")) ip = ip.split(",")[0].trim();
if (ip != null && ip.contains(":")) ip = ip.split(":")[0].trim();
}
String ua = request.getHeader("User-Agent"); String ua = request.getHeader("User-Agent");
if (ua == null) return; SubscriptionEvent event = subscriptionEvents == null ? null : subscriptionEvents.requestEvent(request, key, client);
recordUpdate(subBind.getUser(), ip, ua); if (ua == null && event == null) return;
if (event == null) recordUpdate(subBind, ip, ua);
if (!"v2".equals(client) && !"cat".equals(client)) { if (!"v2".equals(client) && !"cat".equals(client)) {
sendStatus(response, HttpServletResponse.SC_BAD_REQUEST, "client error"); sendStatus(response, HttpServletResponse.SC_BAD_REQUEST, "client error");
return; return;
@@ -206,21 +256,80 @@ public class SubService {
sendStatus(response, HttpServletResponse.SC_SERVICE_UNAVAILABLE, "subscription unavailable"); sendStatus(response, HttpServletResponse.SC_SERVICE_UNAVAILABLE, "subscription unavailable");
return; return;
} }
if (event != null && !"HEAD".equalsIgnoreCase(request.getMethod()))
subscriptionEvents.recordPrimary(event, subBind);
FileDownload.export(request, response, path.toString()); FileDownload.export(request, response, path.toString());
} }
private void recordUpdate(String user, String ip, String ua) { private static String resolveClientIp(HttpServletRequest request) {
String location; String remoteAddr = headerAddress(request.getRemoteAddr());
if (isLoopback(remoteAddr)) {
String forwardedFor = headerAddress(request.getHeader("X-Forwarded-For"));
if (forwardedFor != null)
return forwardedFor;
String realIp = headerAddress(request.getHeader("X-Real-IP"));
if (realIp != null)
return realIp;
}
return remoteAddr == null ? "unknown" : remoteAddr;
}
private static String headerAddress(String value) {
if (value == null || value.isBlank())
return null;
String address = value.split(",", 2)[0].trim();
return address.isBlank() || "unknown".equalsIgnoreCase(address) ? null : address;
}
private static boolean isLoopback(String address) {
return "127.0.0.1".equals(address) || "::1".equals(address)
|| "0:0:0:0:0:0:0:1".equals(address);
}
private void recordUpdate(SubBind binding, String ip, String ua) {
String user = binding.getUser();
// 先落一条 unknown 记录并立即返回,归属地随后在后台补齐;
// 这样取订阅的延迟不再取决于第三方定位站点的响应时间。
SubUpdateRecord record = new SubUpdateRecord(0, user, ip, ua, new Date(), "unknown");
record.setUserId(binding.getUserId());
subMapper.insertSubUpdateRecord(record);
if (subMapper.selectUpdateRecordCount(user) > 10)
subMapper.deleteSubUpdateRecordById(subMapper.selectMinUpdateRecordId(user));
// Capture the generated row ID: another request may arrive before the lookup finishes.
if (record.getId() > 0) {
try {
locationExecutor.execute(() -> enrichLocation(record.getId(), ip));
} catch (java.util.concurrent.RejectedExecutionException busy) {
log.debug("位置查询队列已满,保留 unknown recordId={}", record.getId());
}
}
}
/** 后台按记录 ID 补齐归属地;任何失败都只记日志,不影响已落库的记录。 */
private void enrichLocation(int recordId, String ip) {
try {
String location = resolveLocation(ip);
if (location == null || location.isBlank() || "unknown".equals(location))
return;
subMapper.updateSubUpdateRecordLocation(recordId, location);
} catch (Exception e) {
log.debug("补齐订阅更新记录归属地失败 recordId={}", recordId, e);
}
}
/**
* 查询 IP 归属地。第三方站点结构变化或网络异常都只降级为 unknown,
* 绝不让定位失败影响订阅分发本身。
*/
static String resolveLocation(String ip) {
try { try {
String page = GalleryUtil.requests("https://www.ip138.com/iplookup.php?ip=" + ip, "get", null, null); String page = GalleryUtil.requests("https://www.ip138.com/iplookup.php?ip=" + ip, "get", null, null);
Elements tds = Jsoup.parse(page).select("body > div > div.container > div.content > div > div:nth-child(2) > div.group-left > div > div.bd > div.table-outer > div.table-box > table > tbody > tr > td"); Elements tds = Jsoup.parse(page).select("body > div > div.container > div.content > div > div:nth-child(2) > div.group-left > div > div.bd > div.table-outer > div.table-box > table > tbody > tr > td");
location = tds.size() > 3 ? tds.get(1).text().replace("中国", "") + " " + tds.get(3).text().trim() : (tds.isEmpty() ? "unknown" : tds.get(1).text()); return tds.size() > 3 ? tds.get(1).text().replace("中国", "") + " " + tds.get(3).text().trim()
: (tds.isEmpty() ? "unknown" : tds.get(1).text());
} catch (Exception e) { } catch (Exception e) {
location = "unknown"; return "unknown";
} }
subMapper.insertSubUpdateRecord(new SubUpdateRecord(0, user, ip, ua, new Date(), location));
if (subMapper.selectUpdateRecordCount(user) > 10)
subMapper.deleteSubUpdateRecordById(subMapper.selectMinUpdateRecordId(user));
} }
public String deleteSubBind(String user) { public String deleteSubBind(String user) {
@@ -228,30 +337,41 @@ public class SubService {
} }
private String deleteSubBindUnlocked(String user) { private String deleteSubBindUnlocked(String user) {
rememberEventBindings();
subMapper.deleteSubBind(user); subMapper.deleteSubBind(user);
subMapper.deleteSubUpdateRecord(user); subMapper.deleteSubUpdateRecord(user);
remoteService.requestSubscriptionSync(); remoteService.requestSubscriptionSync();
return Response._success("删除成功"); return Response._success("删除成功");
} }
private void rememberEventBindings() {
if (subscriptionEvents != null) subscriptionEvents.rememberBindings(subMapper.selectAllSubBind());
}
private String withWriteLock(Supplier<String> action) { private String withWriteLock(Supplier<String> action) {
Lock stateLock = stateCoordinator.writeLock(); Lock stateLock = stateCoordinator.writeLock();
stateLock.lock(); stateLock.lock();
try { try {
return action.get(); return transactions == null ? action.get() : transactions.execute(ignored -> action.get());
} finally { } finally {
stateLock.unlock(); stateLock.unlock();
} }
} }
@PreDestroy
void shutdownLocationResolver() {
if (locationExecutor instanceof ExecutorService service)
service.shutdownNow();
}
private SubscriptionAccount firstEnabledAccount() { private SubscriptionAccount firstEnabledAccount() {
return subMapper.selectAllSubscriptionAccounts().stream().filter(SubscriptionAccount::isEnabled).findFirst().orElse(null); return subMapper.selectAllSubscriptionAccounts().stream().filter(SubscriptionAccount::isEnabled).findFirst().orElse(null);
} }
private String accountJson(SubscriptionAccount account) { private tools.jackson.databind.JsonNode accountJson(SubscriptionAccount account) {
account.setV2Url(refreshService.v2Url(account)); account.setV2Url(refreshService.v2Url(account));
account.setClashUrl(refreshService.clashUrl(account)); account.setClashUrl(refreshService.clashUrl(account));
return CustomUtil.objectMapper.valueToTree(account).toString(); return CustomUtil.objectMapper.valueToTree(account);
} }
private static void sendStatus(HttpServletResponse response, int status, String message) { private static void sendStatus(HttpServletResponse response, int status, String message) {
@@ -0,0 +1,224 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Domain.*;
import com.lion.lionwebsite.Util.CustomUtil;
import jakarta.annotation.PreDestroy;
import jakarta.servlet.http.HttpServletRequest;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionTemplate;
import tools.jackson.databind.JsonNode;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Clock;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.Lock;
@Service
public class SubscriptionEventService {
public static final int MAX_BODY = 256 * 1024;
private final SubscriptionEventMapper events;
private final SubMapper subscriptions;
private final UserMapper users;
private final TransactionTemplate transactions;
private final SubscriptionStateCoordinator coordinator;
@Value("${subscription.standby.sync-secret:}") String syncSecret;
Clock clock = Clock.systemUTC();
private final ExecutorService locations = new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS,
new ArrayBlockingQueue<>(128), runnable -> {
Thread thread = new Thread(runnable, "subscription-event-location");
thread.setDaemon(true);
return thread;
}, new ThreadPoolExecutor.AbortPolicy());
public SubscriptionEventService(SubscriptionEventMapper events, SubMapper subscriptions, UserMapper users,
@Qualifier("mainTransactionTemplate") TransactionTemplate transactions,
SubscriptionStateCoordinator coordinator) {
this.events = events;
this.subscriptions = subscriptions;
this.users = users;
this.transactions = transactions;
this.coordinator = coordinator;
}
/** Auth failure deliberately discloses neither signature nor payload. */
public boolean verify(byte[] body, String timestamp, String signature) {
if (body == null || body.length > MAX_BODY || syncSecret == null || syncSecret.isBlank()
|| timestamp == null || !timestamp.matches("[0-9]{1,12}")
|| signature == null || !signature.matches("[0-9a-fA-F]{64}")) return false;
try {
long seconds = Long.parseLong(timestamp);
if (Math.abs(clock.instant().getEpochSecond() - seconds) > 300) return false;
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(syncSecret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
byte[] expected = mac.doFinal(("v1\n" + timestamp + "\n" + sha256(body)).getBytes(StandardCharsets.UTF_8));
return MessageDigest.isEqual(expected, HexFormat.of().parseHex(signature));
} catch (Exception invalid) { return false; }
}
/** Invalid or absent signed headers fall back to the ordinary logging path. */
public SubscriptionEvent requestEvent(HttpServletRequest request, String key, String client) {
String encoded = request.getHeader("X-Subscription-Event");
if (encoded == null || encoded.length() > 8192 || !encoded.matches("[A-Za-z0-9_-]+")) return null;
try {
byte[] body = Base64.getUrlDecoder().decode(encoded);
if (!verify(body, request.getHeader("X-Subscription-Timestamp"), request.getHeader("X-Subscription-Signature"))) return null;
SubscriptionEvent event = parse(CustomUtil.objectMapper.readTree(body));
String ua = Objects.requireNonNullElse(request.getHeader("User-Agent"), "");
if (!event.getKeyHash().equals(sha256(key.getBytes(StandardCharsets.UTF_8)))
|| !event.getClient().equals(client) || !event.getUserAgent().equals(ua)) return null;
return event;
} catch (Exception invalid) { return null; }
}
public List<SubscriptionEvent> parseBatch(byte[] body) {
try {
if (body == null || body.length > MAX_BODY) throw new IllegalArgumentException();
JsonNode root = CustomUtil.objectMapper.readTree(body);
JsonNode list = root.get("events");
if (!root.isObject() || list == null || !list.isArray() || list.size() > 100) throw new IllegalArgumentException();
List<SubscriptionEvent> result = new ArrayList<>();
for (JsonNode item : list) result.add(parse(item));
return result;
} catch (Exception invalid) { throw new IllegalArgumentException("invalid subscription events"); }
}
SubscriptionEvent parse(JsonNode node) {
if (node == null || !node.isObject()) throw new IllegalArgumentException();
String id = text(node, "eventId", 36);
if (!UUID.fromString(id).toString().equals(id)) throw new IllegalArgumentException();
String hash = text(node, "keyHash", 64);
if (!hash.matches("[0-9a-f]{64}")) throw new IllegalArgumentException();
String client = text(node, "client", 3);
if (!client.equals("v2") && !client.equals("cat")) throw new IllegalArgumentException();
String ip = text(node, "clientIp", 64);
// Literal IPv4/IPv6 only: no host names, URL fragments or query injection into geolocation.
if (ip.isEmpty() || !ip.matches("[0-9A-Fa-f:.]+")) throw new IllegalArgumentException();
String ua = text(node, "userAgent", 2048);
if (ua.chars().anyMatch(c -> c < 32 || c == 127)) throw new IllegalArgumentException();
JsonNode time = node.get("occurredAt");
if (time == null || !time.isIntegralNumber() || !time.canConvertToLong()) throw new IllegalArgumentException();
long occurred = time.longValue();
if (occurred < 0 || occurred > clock.millis() + 300_000) throw new IllegalArgumentException();
return new SubscriptionEvent(id, hash, client, ip, ua, occurred);
}
private static String text(JsonNode node, String field, int max) {
JsonNode value = node.get(field);
if (value == null || !value.isString() || value.stringValue().length() > max) throw new IllegalArgumentException();
return value.stringValue();
}
public Map<String, Object> ingest(List<SubscriptionEvent> batch) {
Lock lock = coordinator.readLock();
lock.lock();
try {
// Return acknowledgements only after execute has returned and the entire transaction committed.
List<String> acknowledged = transactions.execute(status -> {
events.reserveWriter();
List<String> ids = new ArrayList<>();
for (SubscriptionEvent event : batch) if (persist(event, "standby", null)) ids.add(event.getEventId());
return ids;
});
return Map.of("acknowledged", Objects.requireNonNull(acknowledged));
} finally { lock.unlock(); }
}
public void recordPrimary(SubscriptionEvent event, SubBind binding) {
transactions.executeWithoutResult(status -> {
events.reserveWriter();
if (!persist(event, "primary", binding)) throw new IllegalStateException("subscription user unavailable");
});
}
private boolean persist(SubscriptionEvent event, String source, SubBind current) {
String fingerprint = sha256((event.getEventId() + "\n" + event.getKeyHash() + "\n" + event.getClient()
+ "\n" + event.getOccurredAt() + "\n" + event.getClientIp() + "\n" + event.getUserAgent()).getBytes(StandardCharsets.UTF_8));
String old = events.fingerprint(event.getEventId());
if (old != null) {
if (!old.equals(fingerprint)) throw new IllegalArgumentException("event ID conflict");
return true;
}
Integer userId = current == null ? null : current.getUserId();
String userName = current == null ? null : current.getUser();
if (current == null) {
boolean found = false;
for (SubBind binding : subscriptions.selectAllSubBind()) {
if (sha256(binding.getKey().getBytes(StandardCharsets.UTF_8)).equals(event.getKeyHash())) {
userId = binding.getUserId();
userName = binding.getUser();
found = true;
break;
}
}
if (!found) {
SubscriptionEventBinding binding = events.binding(event.getKeyHash());
if (binding == null) return false;
userId = binding.getUserId();
userName = binding.getUserName();
}
}
if (userId != null) {
User user = users.selectUserById(userId);
if (user == null) {
events.insertReceipt(event, source, clock.millis(), fingerprint, "user_deleted");
return true;
}
userName = user.getUsername();
}
// NULL user_id is a valid legacy subscription label, not a missing/deleted User.
// Keep it NULL even if someone later creates a User with the same label.
if (userName == null || userName.isBlank()) return false;
events.rememberBinding(event.getKeyHash(), userId, userName, clock.millis());
SubUpdateRecord record = new SubUpdateRecord(0, userName, event.getClientIp(), event.getUserAgent(),
new Date(event.getOccurredAt()), "unknown", userId);
events.insertRecord(record);
events.pruneRecords(userId, userName);
events.insertReceipt(event, source, clock.millis(), fingerprint, "recorded");
// Never perform network calls before commit. A dropped/rolled-back record has no enrichment task.
org.springframework.transaction.support.TransactionSynchronizationManager.registerSynchronization(
new org.springframework.transaction.support.TransactionSynchronization() {
@Override public void afterCommit() { enrich(record.getId(), event.getClientIp()); }
});
return true;
}
public void rememberBindings(Collection<SubBind> bindings) {
for (SubBind binding : bindings) {
Integer userId = binding.getUserId();
String name = binding.getUser();
if (userId != null) {
User user = users.selectUserById(userId);
if (user != null) name = user.getUsername();
}
if (name != null && !name.isBlank()) events.rememberBinding(
sha256(binding.getKey().getBytes(StandardCharsets.UTF_8)), userId, name, clock.millis());
}
}
private void enrich(int recordId, String ip) {
if (recordId <= 0) return;
try { locations.execute(() -> {
String location = SubService.resolveLocation(ip);
if (location != null && !location.isBlank() && !location.equals("unknown")) {
try { subscriptions.updateSubUpdateRecordLocation(recordId, location); }
catch (Exception ignored) { /* Optional display enrichment; no request failure or sensitive log. */ }
}
}); } catch (RejectedExecutionException ignored) { }
}
public static String sha256(byte[] body) {
try { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(body)); }
catch (Exception failure) { throw new IllegalStateException("SHA-256 unavailable"); }
}
@PreDestroy void close() { locations.shutdownNow(); }
}
@@ -0,0 +1,225 @@
package com.lion.lionwebsite.Service;
import java.time.Clock;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Random;
/**
* 为一批子账号排定刷新时刻。
*
* <p>目标:每个账号每 24 小时内至少刷新一次,且各账号的时刻互相错开,
* 不出现「一次全量、十几个账号在十几秒内打完」的爆发式请求。
*
* <p>排程模型是「错峰 + 固定相位」:
* <ol>
* <li><b>错峰</b>:首次排程把账号随机铺满一个窗口,两两之间至少隔开 {@code minGap};</li>
* <li><b>固定相位</b>:账号每次成功后按窗口长度(略减一个 tick)推进,
* 因此各自的相位保持不变,分散效果自我维持,不需要每天重新洗牌;</li>
* <li><b>兜底</b>:任何情况下下一次刷新都不会晚于「上次成功 + 窗口」,
* 把「24 小时内必然刷新一次」从概率保证变成确定保证。</li>
* </ol>
*
* <p>本类是纯函数式设计(注入 {@link Clock} 与带种子的 {@link Random}),
* 不触碰数据库与网络,因此可以直接做确定性单元测试。
*/
public class SubscriptionRefreshPlanner {
private final Clock clock;
private final Random random;
private final Duration window;
private final Duration minGap;
/** 一个调度周期的最大延迟;用它给窗口留余量,使实际间隔严格落在窗口内。 */
private final Duration tickInterval;
/** 刷新失败后的重试间隔。 */
private final Duration retryDelay;
/**
* @param clock 时间源,测试可注入固定时钟
* @param random 随机源,测试可注入固定种子以复现排程
* @param window 刷新窗口,即「24 小时内至少一次」中的 24 小时
* @param minGap 相邻两个账号之间的最小间隔
* @param tickInterval 调度周期,用于给窗口留出余量
* @param retryDelay 失败后的重试间隔
*/
public SubscriptionRefreshPlanner(Clock clock, Random random, Duration window, Duration minGap,
Duration tickInterval, Duration retryDelay) {
if (window.isZero() || window.isNegative())
throw new IllegalArgumentException("刷新窗口必须为正数");
if (minGap.isNegative())
throw new IllegalArgumentException("最小间隔不能为负数");
if (tickInterval.isNegative())
throw new IllegalArgumentException("调度周期不能为负数");
if (retryDelay.isZero() || retryDelay.isNegative())
throw new IllegalArgumentException("重试间隔必须为正数");
if (retryDelay.compareTo(window) > 0)
throw new IllegalArgumentException("重试间隔不能大于刷新窗口,否则无法保证窗口内重试");
this.clock = clock;
this.random = random;
this.window = window;
this.minGap = minGap;
this.tickInterval = tickInterval;
this.retryDelay = retryDelay;
}
/** 便捷构造:重试间隔默认取 1 小时。 */
public SubscriptionRefreshPlanner(Clock clock, Random random, Duration window, Duration minGap,
Duration tickInterval) {
this(clock, random, window, minGap, tickInterval, Duration.ofHours(1));
}
/**
* 账号数与最小间隔是否放得下。
*
* <p>注意这里刻意只用于「提示」而不是阻断启动:账号数是由用户在管理页决定的,
* 若因为它超限就让整个应用起不来,等于把配置细节变成一次线上故障。
* 放不下时 {@link #initialSchedule} 仍会把账号铺满整个窗口(只是间隔小于期望值),
* 结果是错峰效果变弱,而不是退回串行爆发。
*/
public boolean isCapacitySufficient(int accountCount) {
if (accountCount <= 1)
return true;
return minGap.toMillis() * (accountCount - 1) <= window.toMillis();
}
/** 容量不足时的告警文案,供调用方记录日志。 */
public String capacityMessage(int accountCount) {
long required = minGap.toMillis() * (accountCount - 1);
return String.format(
"刷新窗口放不下最小间隔:账号 %d 个、最小间隔 %d 分钟,需要 %d 分钟,但窗口只有 %d 分钟;"
+ "实际仍会铺满窗口,只是账号间间隔会小于期望值。可调小 subscription.refresh.min-gap-minutes。",
accountCount, minGap.toMinutes(),
Duration.ofMillis(required).toMinutes(), window.toMinutes());
}
/**
* 为一批「已成功刷新过」的账号铺开计划,返回顺序与 accountIds 一致。
*
* <p>已成功过的账号内容仍然可用,因此可以安心分散到整个窗口,
* 不会因为排在 20 小时后而让用户拿不到订阅。
*/
public List<Long> initialSchedule(List<Integer> accountIds) {
List<Long> result = new ArrayList<>(accountIds.size());
if (accountIds.isEmpty())
return result;
long now = clock.millis();
long windowMillis = window.toMillis();
int count = accountIds.size();
long slotWidth = windowMillis / count;
long jitterCap = Math.max(1, Math.min(slotWidth, Math.max(1, slotWidth - minGap.toMillis())));
List<Integer> slots = new ArrayList<>(count);
for (int i = 0; i < count; i++)
slots.add(i);
Collections.shuffle(slots, random);
long[] planned = new long[count];
for (int accountIndex = 0; accountIndex < count; accountIndex++) {
long offset = slots.get(accountIndex) * slotWidth + (long) (random.nextDouble() * jitterCap);
planned[accountIndex] = now + Math.min(offset, windowMillis - 1);
}
for (int i = 0; i < count; i++)
result.add(planned[i]);
return result;
}
/**
* 为尚未成功过的账号排程:尽快刷,但互相错开。
*
* <p>这类账号(新建、或上游 key 刚改过)可能根本没有可用缓存,用户立刻请求订阅
* 就会拿不到内容,因此不能像常规账号那样分散到整个窗口。这里把它们按
* {@code spacing} 依次排开,配合调度器的单轮上限逐批刷新,
* 既尽早补齐,又避免一批新账号同时打上游。
*
* @param accountIds 账号 ID,顺序即刷新优先级
* @param spacing 相邻两个账号的间隔(通常取一个 tick)
*/
public List<Long> coldStartSchedule(List<Integer> accountIds, Duration spacing) {
List<Long> result = new ArrayList<>(accountIds.size());
long now = clock.millis();
long step = Math.max(1, spacing.toMillis());
for (int i = 0; i < accountIds.size(); i++)
result.add(now + i * step);
return result;
}
/**
* 已排定的时刻是否已到期。
*
* <p>直接以排定时刻为准,这样分散效果与失败退避都得以保持。
* 只额外兜住一种异常:计划被写到超过一个窗口之后。这不可能是本调度器产生的
* 排程(成功时排在窗口减一个 tick 之后,失败时排在 4 小时以内),
* 按「计划损坏」处理并立即到期,从而保证 24 小时内必定刷新。
*
* <p>尚未排程({@code scheduled == null})视为立即到期,由调用方先行铺开计划。
*/
public boolean isDue(Long scheduled, long now) {
if (scheduled == null)
return true;
if (scheduled - now > window.toMillis())
return true;
return scheduled <= now;
}
/**
* 成功之后的下一次刷新时刻。
*
* <p>只推进「窗口 - 一个调度周期」,使 tick 即使晚触发也不会突破 24 小时;
* 从上次成功的时刻起算,保证相邻两次的间隔严格小于窗口。
*/
public long nextAfterSuccess(long successMillis) {
long advance = Math.max(1, window.minus(tickInterval).toMillis());
return successMillis + advance;
}
/**
* 失败之后的重试时刻。
*
* <p>用固定的重试间隔而不是「失败次数 × 指数退避」:刷新成功会清空 {@code last_error},
* 数据库里没有「连续失败次数」这一栏,退避指数实际上只能取到 0 或 1,
* 写出来会让人误以为有指数退避而其实没有。固定的重试间隔行为可预期,
* 且远小于窗口,不会威胁「24 小时内尝试刷新一次」。
*
* <p>唯一需要裁剪的情况:若「上次成功 + 窗口」还没到,则重试不得晚于它;
* 已经越过该点时保留重试时刻本身——重复请求并不会提高成功率,
* 反过来在每个 tick 重打上游才是真正的问题,这种情况交由陈旧告警处理。
*
* @param now 当前时刻
* @param lastSuccess 上次成功时刻;未知时传 null,不做裁剪
*/
public long nextAfterFailure(long now, Long lastSuccess) {
long candidate = now + retryDelay.toMillis();
if (lastSuccess == null)
return candidate;
long deadline = lastSuccess + window.toMillis();
return deadline > now ? Math.min(candidate, deadline) : candidate;
}
/** 是否已越过兜底线(距上次成功已达一个窗口),用于诊断与告警判定。 */
public boolean isOverdue(Long lastSuccess, long now) {
return lastSuccess != null && now - lastSuccess >= window.toMillis();
}
private long now() {
return clock.millis();
}
public Clock clock() {
return clock;
}
public Duration window() {
return window;
}
public Duration minGap() {
return minGap;
}
public Duration tickInterval() {
return tickInterval;
}
}
@@ -0,0 +1,247 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.SubscriptionAccount;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
/**
* 分散刷新各子账号的订阅。
*
* <p>取代原先「每 24 小时一次全量」的调度。旧写法有两个问题:所有账号在十几秒内
* 背靠背打完,且 {@code @Scheduled(fixedRate = 86400000)} 没有 initialDelay,
* 首次触发即立刻执行——于是每次重启都会重刷一遍全部账号,上游看到的请求密度取决于
* 部署频率,而不是每天一次。
*
* <p>现在改为高频轻量 tick,每轮只刷新已经到期的账号。「24 小时内每个账号至少
* 更新一次」由三层保证:
* <ol>
* <li>成功后排定「成功时刻 + 窗口 - 一个 tick」,相位自我维持,分散效果不会因
* 重启或重排而退化;</li>
* <li>到期时刻存在数据库里,停机期间错过计划的账号在恢复后立即到期;</li>
* <li>兜底闸:任何排定时刻都被裁剪到「上次成功 + 窗口」以内,越线即立刻刷新。</li>
* </ol>
*/
@Service
@Slf4j
public class SubscriptionRefreshScheduler {
private final SubMapper subMapper;
private final SubscriptionRefreshService refreshService;
private final SubscriptionRefreshPlanner planner;
private final PushService pushService;
private final RemoteService remoteService;
/** 单个 tick 最多刷新几个账号,避免重启后一批到期账号同时开火。 */
@Value("${subscription.refresh.max-per-tick:2}")
int maxPerTick;
/** 超过窗口的这个倍数仍未成功即告警,避免静默失败。 */
@Value("${subscription.refresh.stale-alert-multiplier:2.0}")
double staleAlertMultiplier;
/** 已告警过的账号,避免每轮重复推送;刷新成功后移除。 */
private final Set<Integer> alertedAccounts = new HashSet<>();
public SubscriptionRefreshScheduler(SubMapper subMapper,
SubscriptionRefreshService refreshService,
SubscriptionRefreshPlanner planner,
PushService pushService,
RemoteService remoteService) {
this.subMapper = subMapper;
this.refreshService = refreshService;
this.planner = planner;
this.pushService = pushService;
this.remoteService = remoteService;
}
/**
* 轻量 tick:只处理已到期的账号,且单轮有数量上限。
*
* <p>用 fixedDelay 而非 fixedRate,一轮结束再计下一轮,避免上一轮没跑完就叠加触发;
* initialDelay 让重启后的第一轮稍晚开始,先让应用完成启动。
*/
@Scheduled(fixedDelayString = "${subscription.refresh.tick-interval-ms:300000}",
initialDelayString = "${subscription.refresh.initial-delay-ms:120000}")
public void tick() {
try {
refreshDueAccounts();
} catch (Exception e) {
// 调度方法抛出异常会导致后续触发被取消,这里必须兜住。
log.error("订阅分散刷新 tick 失败", e);
}
}
/**
* 刷新当前已到期的账号。
*
* @return 本轮实际刷新的账号数
*/
public int refreshDueAccounts() {
List<SubscriptionAccount> enabled = enabledAccounts();
ensureScheduled(enabled);
long now = planner.clock().millis();
int refreshed = 0;
for (SubscriptionAccount account : enabled) {
if (refreshed >= maxPerTick)
break;
if (!isDue(account, now))
continue;
refreshOne(account, now);
refreshed++;
}
reportStale(enabled, now);
// 缓存内容变了就要让备机同步。原先全量刷新后由调用方跟一次同步,
// 改为分散刷新后必须在这里补上,否则备机要等到下一次周期同步才拿到新内容。
if (refreshed > 0)
remoteService.requestSubscriptionSync();
return refreshed;
}
/**
* 手动刷新入口:立即刷新全部启用账号。
*
* <p>刻意不受 {@code maxPerTick} 限制——管理页「更新订阅」的语义就是立刻全部刷新。
* 但成功后同样要重排各自的下一次时刻,否则刚手动刷完会让当天的计划作废,
* 该账号反而可能超过 24 小时没有下次更新。
*/
public boolean refreshAllNow() {
boolean success = true;
long now = planner.clock().millis();
for (SubscriptionAccount account : enabledAccounts()) {
if (!refreshOne(account, now))
success = false;
}
return success;
}
private List<SubscriptionAccount> enabledAccounts() {
return subMapper.selectAllSubscriptionAccounts().stream()
.filter(SubscriptionAccount::isEnabled)
.sorted(Comparator.comparing(SubscriptionAccount::getId))
.toList();
}
/** 刷新单个账号并按结果重排下一次时刻。 */
private boolean refreshOne(SubscriptionAccount account, long now) {
boolean ok = refreshService.refresh(account.getId());
if (ok) {
long next = planner.nextAfterSuccess(planner.clock().millis());
subMapper.updateNextRefreshAt(account.getId(), next);
alertedAccounts.remove(account.getId());
log.info("订阅刷新成功 accountId={} 下次刷新={}", account.getId(), next);
} else {
long next = planner.nextAfterFailure(now, account.getLastSuccessEpoch());
subMapper.updateNextRefreshAt(account.getId(), next);
log.warn("订阅刷新失败 accountId={} 下次重试={}", account.getId(), next);
}
return ok;
}
/**
* 账号是否到期。
*
* <p>以排定时刻为准;尚未排程或计划明显损坏(超出窗口)时视为立即到期,
* 由 {@link #ensureScheduled} 先铺开或纠正。
*/
private boolean isDue(SubscriptionAccount account, long now) {
return planner.isDue(account.getNextRefreshAt(), now);
}
/**
* 为尚无计划的账号补齐计划。
*
* <p>按紧迫性分三类,因为它们对「排到多晚」的容忍度不同:
* <ul>
* <li><b>从未成功过</b>(新建账号、刚改过上游 key):可能还没有可用缓存,
* 用户此时请求订阅会拿不到内容,因此尽快刷,只按 tick 间隔互相错开;</li>
* <li><b>已成功过但已超期</b>(距上次成功已达一个窗口):保证即将失效,
* 必须立即刷新,不能排到窗口内更晚的位置;</li>
* <li><b>已成功过且未超期</b>:缓存仍可分发,按窗口分散即可,不必挤在一起。</li>
* </ul>
*/
private void ensureScheduled(List<SubscriptionAccount> enabled) {
long now = planner.clock().millis();
List<SubscriptionAccount> cold = new ArrayList<>();
List<SubscriptionAccount> overdue = new ArrayList<>();
List<SubscriptionAccount> warm = new ArrayList<>();
for (SubscriptionAccount account : enabled) {
if (account.getNextRefreshAt() != null)
continue;
if (account.getLastSuccessEpoch() == null)
cold.add(account);
else if (planner.isOverdue(account.getLastSuccessEpoch(), now))
overdue.add(account);
else
warm.add(account);
}
if (cold.isEmpty() && overdue.isEmpty() && warm.isEmpty())
return;
if (!planner.isCapacitySufficient(enabled.size()))
log.warn("{}", planner.capacityMessage(enabled.size()));
// 超期账号立即到期(计划设为当前时刻),本轮就会被刷新。
for (SubscriptionAccount account : overdue)
schedule(account, now);
int index = 0;
List<Long> coldPlan = planner.coldStartSchedule(
cold.stream().map(SubscriptionAccount::getId).toList(), planner.tickInterval());
for (SubscriptionAccount account : cold)
schedule(account, coldPlan.get(index++));
List<Integer> warmIds = warm.stream().map(SubscriptionAccount::getId).toList();
List<Long> warmPlan = planner.initialSchedule(warmIds);
for (int i = 0; i < warm.size(); i++)
schedule(warm.get(i), warmPlan.get(i));
}
/**
* 写入新计划,并同步更新内存中的对象。
*
* <p>必须同时更新内存副本:否则刚被排到几小时后的账号在本轮仍带着
* {@code nextRefreshAt == null},会被 {@link #isDue} 判为立即到期并当场刷新,
* 于是「铺开」在第一个 tick 就失效了(迁移后首批账号会全部集中刷新)。
*/
private void schedule(SubscriptionAccount account, long nextRefreshAt) {
subMapper.updateNextRefreshAt(account.getId(), nextRefreshAt);
account.setNextRefreshAt(nextRefreshAt);
}
/** 陈旧账号告警:同一账号在连续陈旧期间只推一次,刷新成功后重置。 */
private void reportStale(List<SubscriptionAccount> enabled, long now) {
long threshold = (long) (planner.window().toMillis() * staleAlertMultiplier);
for (SubscriptionAccount account : enabled) {
Long lastSuccess = account.getLastSuccessEpoch();
if (lastSuccess == null || now - lastSuccess < threshold)
continue;
if (!alertedAccounts.add(account.getId()))
continue;
long hours = (now - lastSuccess) / 3_600_000L;
String message = String.format("订阅账号 %d(%s)已 %d 小时未成功刷新,最近错误:%s",
account.getId(), account.getName(), hours,
account.getLastError() == null ? "无" : account.getLastError());
log.error("{}", message);
pushService.sendToMe(message);
}
// 已删除或停用的账号不再保留告警状态,避免集合无限增长。
Set<Integer> alive = new HashSet<>();
for (SubscriptionAccount account : enabled)
alive.add(account.getId());
alertedAccounts.retainAll(alive);
}
SubscriptionRefreshPlanner planner() {
return planner;
}
}
@@ -2,13 +2,18 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.SubMapper; import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.SubscriptionAccount; import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Util.ClashSubscriptionParser;
import com.lion.lionwebsite.Util.SubscriptionClientProfile;
import com.lion.lionwebsite.Util.SubscriptionCacheStore;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpEntity; import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.http.client.methods.HttpGet; import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.http.impl.client.CloseableHttpClient; import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.http.impl.client.HttpClients; import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.util.Timeout;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
@@ -16,6 +21,7 @@ import java.io.*;
import java.net.URLDecoder; import java.net.URLDecoder;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.*; import java.nio.file.*;
import java.time.Clock;
import java.util.*; import java.util.*;
import java.util.regex.Matcher; import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.util.regex.Pattern;
@@ -25,7 +31,13 @@ import java.util.concurrent.locks.Lock;
@Slf4j @Slf4j
@RequiredArgsConstructor @RequiredArgsConstructor
public class SubscriptionRefreshService { public class SubscriptionRefreshService {
private static final CloseableHttpClient HTTP_CLIENT = HttpClients.createDefault(); private static final CloseableHttpClient HTTP_CLIENT = HttpClients.custom()
.setDefaultRequestConfig(RequestConfig.custom()
.setConnectTimeout(Timeout.ofMilliseconds(5_000))
.setConnectionRequestTimeout(Timeout.ofMilliseconds(5_000))
// HttpClient 5 将 socket 读超时改名为 responseTimeout。
.setResponseTimeout(Timeout.ofMilliseconds(15_000)).build())
.build();
private static final Pattern MULTIPLIER = Pattern.compile("(\\d+(?:\\.\\d+)?)x\\s*$", Pattern.CASE_INSENSITIVE); private static final Pattern MULTIPLIER = Pattern.compile("(\\d+(?:\\.\\d+)?)x\\s*$", Pattern.CASE_INSENSITIVE);
final SubMapper subMapper; final SubMapper subMapper;
@@ -44,40 +56,75 @@ public class SubscriptionRefreshService {
@Value("${subscription.cache-root:sub/accounts}") @Value("${subscription.cache-root:sub/accounts}")
String cacheRoot; String cacheRoot;
public boolean refreshAll() { /** 时间源,测试可替换以固定成功时刻。 */
boolean success = true; Clock clock = Clock.systemDefaultZone();
for (SubscriptionAccount account : subMapper.selectAllSubscriptionAccounts()) {
if (account.isEnabled() && !refresh(account.getId())) /**
success = false; * 分代缓存存储。默认按 {@link #cacheRoot} 懒构造;测试可覆写为受控子类来模拟写入失败。
} * 保持为普通字段而不是构造参数,是为了不改动既有两参构造器(Spring 与老测试都在用)。
return success; */
} SubscriptionCacheStore cacheStore;
private long refreshSequence;
// Accessed only while holding the coordinator write lock.
private final Map<Integer, Long> latestRefresh = new HashMap<>();
public boolean refresh(Integer accountId) { public boolean refresh(Integer accountId) {
Lock stateLock = stateCoordinator.writeLock(); Lock stateLock = stateCoordinator.writeLock();
SubscriptionAccount account;
long version;
stateLock.lock(); stateLock.lock();
try { try {
SubscriptionAccount account = subMapper.selectSubscriptionAccount(accountId); account = subMapper.selectSubscriptionAccount(accountId);
if (account == null || !account.isEnabled()) if (account == null || !account.isEnabled())
return false; return false;
try { version = ++refreshSequence;
String v2 = processV2(firstLine(download(v2Url(account))), account.isFilterHighMultiplier(), highMultiplierThreshold); latestRefresh.put(accountId, version);
List<String> clash = processClash(download(clashUrl(account)), account.isFilterHighMultiplier(), highMultiplierThreshold);
Path dir = Paths.get(cacheRoot, String.valueOf(accountId));
Files.createDirectories(dir);
atomicWrite(dir.resolve("v2ray.txt"), v2.getBytes(StandardCharsets.UTF_8));
atomicWrite(dir.resolve("clash.yaml"), String.join("\n", clash).concat("\n").getBytes(StandardCharsets.UTF_8));
subMapper.markSubscriptionRefreshSuccess(accountId);
return true;
} catch (Exception e) {
String message = e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage();
subMapper.markSubscriptionRefreshFailure(accountId, message.length() > 500 ? message.substring(0, 500) : message);
log.error("刷新子账号订阅失败 accountId={}: {}", accountId, message);
return false;
}
} finally { } finally {
stateLock.unlock(); stateLock.unlock();
} }
// Network access and parsing never hold the shared subscription lock.
try {
boolean filter = account.isFilterHighMultiplier();
String v2 = processV2(firstLine(download(v2Url(account), SubscriptionClientProfile.forAccount(accountId, false))),
filter, highMultiplierThreshold);
// 按 YAML 语义过滤高倍节点并同步清理 proxy-groups 引用,不再靠行缩进猜测。
String clash = ClashSubscriptionParser.transform(
download(clashUrl(account), SubscriptionClientProfile.forAccount(accountId, true)),
filter, highMultiplierThreshold);
stateLock.lock();
try {
if (!isCurrent(account, version))
return false;
// 两份内容写进同一分代,校验通过后指针最后发布:任一步失败都保留上次完整版本。
store().publish(accountId, v2.getBytes(StandardCharsets.UTF_8), clash.getBytes(StandardCharsets.UTF_8));
subMapper.markSubscriptionRefreshSuccess(accountId, clock.millis());
return true;
} finally {
stateLock.unlock();
}
} catch (Exception e) {
stateLock.lock();
try {
if (isCurrent(account, version)) {
String message = e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage();
subMapper.markSubscriptionRefreshFailure(accountId, message.length() > 500 ? message.substring(0, 500) : message);
}
} finally {
stateLock.unlock();
}
log.warn("刷新子账号订阅失败 accountId={} errorType={}", accountId, e.getClass().getSimpleName());
return false;
}
}
private boolean isCurrent(SubscriptionAccount expected, long version) {
SubscriptionAccount current = subMapper.selectSubscriptionAccount(expected.getId());
return Objects.equals(latestRefresh.get(expected.getId()), version)
&& current != null && current.isEnabled()
&& Objects.equals(current.getUpstreamKey(), expected.getUpstreamKey())
&& current.isFilterHighMultiplier() == expected.isFilterHighMultiplier();
} }
public String v2Url(SubscriptionAccount account) { public String v2Url(SubscriptionAccount account) {
@@ -124,66 +171,32 @@ public class SubscriptionRefreshService {
return URLDecoder.decode(node.substring(hash + 1), StandardCharsets.UTF_8); return URLDecoder.decode(node.substring(hash + 1), StandardCharsets.UTF_8);
} }
private List<String> processClash(List<String> source, boolean filter, double threshold) {
Set<String> removed = new HashSet<>();
List<String> result = new ArrayList<>();
boolean inProxies = false;
boolean skipNode = false;
for (String line : source) {
if (line.equals("proxies:")) {
inProxies = true;
skipNode = false;
result.add(line);
continue;
}
if (line.equals("proxy-groups:")) {
inProxies = false;
skipNode = false;
result.add(line);
continue;
}
if (inProxies && line.matches("^\\s{2}-\\s+name:.*")) {
String name = clashName(line);
skipNode = filter && isHigh(name, threshold);
if (skipNode)
removed.add(name);
else
result.add(line);
continue;
}
if (skipNode)
continue;
if (!inProxies && !removed.isEmpty() && line.trim().startsWith("- ")) {
String ref = line.trim().substring(2).trim();
if (removed.contains(unquote(ref)))
continue;
}
result.add(line);
}
return result;
}
private static String clashName(String line) {
int index = line.indexOf("name:");
return unquote(line.substring(index + 5).trim());
}
private static String unquote(String value) {
if (value.length() >= 2 && ((value.startsWith("\"") && value.endsWith("\"")) || (value.startsWith("'") && value.endsWith("'"))))
return value.substring(1, value.length() - 1);
return value;
}
private static boolean isHigh(String name, double threshold) { private static boolean isHigh(String name, double threshold) {
Matcher matcher = MULTIPLIER.matcher(name); Matcher matcher = MULTIPLIER.matcher(name);
return matcher.find() && Double.parseDouble(matcher.group(1)) > threshold; return matcher.find() && Double.parseDouble(matcher.group(1)) > threshold;
} }
private static List<String> download(String url) throws IOException { /**
* 下载上游订阅正文。
*
* @param profile 要伪装的客户端身份;为 null 时保留 HttpClient 默认头(仅测试使用)。
* 生产路径必须传入,否则上游会看到 {@code Apache-HttpClient/... (Java/...)}。
*/
List<String> download(String url, SubscriptionClientProfile profile) throws IOException {
HttpGet get = new HttpGet(url); HttpGet get = new HttpGet(url);
if (profile != null) {
get.addHeader("User-Agent", profile.userAgent());
get.addHeader("Accept", profile.accept());
get.addHeader("Accept-Language", profile.acceptLanguage());
// 实测:显式设置 Accept-Encoding 不会破坏 HttpClient5 的透明解压,
// ContentCompressionExec 仍按 Content-Encoding 正确解码。
get.addHeader("Accept-Encoding", "gzip, deflate");
// 订阅客户端是长连接复用,且不会发送 Referer。
get.addHeader("Connection", "keep-alive");
}
try (CloseableHttpResponse response = HTTP_CLIENT.execute(get)) { try (CloseableHttpResponse response = HTTP_CLIENT.execute(get)) {
if (response.getStatusLine().getStatusCode() != 200) if (response.getCode() != 200)
throw new IOException("上游 HTTP 状态码 " + response.getStatusLine().getStatusCode()); throw new IOException("上游 HTTP 状态码 " + response.getCode());
HttpEntity entity = response.getEntity(); HttpEntity entity = response.getEntity();
if (entity == null) if (entity == null)
throw new IOException("上游返回为空"); throw new IOException("上游返回为空");
@@ -197,34 +210,33 @@ public class SubscriptionRefreshService {
} }
} }
private static void atomicWrite(Path target, byte[] data) throws IOException { /**
Path temp = target.resolveSibling(target.getFileName() + ".tmp"); * 当前应分发的缓存文件。只在一整个版本有效时返回真实路径;否则返回一个保证不存在的
Files.write(temp, data); * 哨兵路径,使分发端按「订阅暂不可用」处理,而不是发出半份旧缓存。
try { */
Files.move(temp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(temp, target, StandardCopyOption.REPLACE_EXISTING);
}
}
public Path cachedPath(Integer accountId, String client) { public Path cachedPath(Integer accountId, String client) {
return Paths.get(cacheRoot, String.valueOf(accountId), client.equals("v2") ? "v2ray.txt" : "clash.yaml"); SubscriptionCacheStore store = store();
return store.resolve(accountId, client).orElseGet(() -> store.unavailablePath(accountId, client));
} }
public boolean hasCompleteCache(Integer accountId) { public boolean hasCompleteCache(Integer accountId) {
return Files.isRegularFile(cachedPath(accountId, "v2")) && Files.isRegularFile(cachedPath(accountId, "cat")); return store().hasComplete(accountId);
} }
public void invalidateCache(Integer accountId) { public void invalidateCache(Integer accountId) {
Lock stateLock = stateCoordinator.writeLock(); Lock stateLock = stateCoordinator.writeLock();
stateLock.lock(); stateLock.lock();
try { try {
Files.deleteIfExists(cachedPath(accountId, "v2")); latestRefresh.remove(accountId); // In-flight responses must not restore invalidated content.
Files.deleteIfExists(cachedPath(accountId, "cat")); store().invalidate(accountId);
} catch (IOException e) {
log.warn("清理失效订阅缓存失败 accountId={}", accountId, e);
} finally { } finally {
stateLock.unlock(); stateLock.unlock();
} }
} }
private synchronized SubscriptionCacheStore store() {
if (cacheStore == null)
cacheStore = new SubscriptionCacheStore(cacheRoot);
return cacheStore;
}
} }
@@ -1,13 +1,15 @@
package com.lion.lionwebsite.Service; package com.lion.lionwebsite.Service;
import com.fasterxml.jackson.databind.ObjectMapper; import tools.jackson.databind.ObjectMapper;
import com.lion.lionwebsite.Dao.normal.SubMapper; import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.SubBind; import com.lion.lionwebsite.Domain.SubBind;
import com.lion.lionwebsite.Domain.SubscriptionAccount; import com.lion.lionwebsite.Domain.SubscriptionAccount;
import com.lion.lionwebsite.Message.*; import com.lion.lionwebsite.Message.*;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.SubscriptionCacheStore;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct; import jakarta.annotation.PostConstruct;
@@ -16,8 +18,6 @@ import javax.crypto.spec.SecretKeySpec;
import java.io.ByteArrayOutputStream; import java.io.ByteArrayOutputStream;
import java.io.IOException; import java.io.IOException;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest; import java.security.MessageDigest;
import java.util.*; import java.util.*;
import java.util.zip.GZIPOutputStream; import java.util.zip.GZIPOutputStream;
@@ -30,6 +30,9 @@ public class SubscriptionStandbySnapshotService {
private final SubscriptionStateCoordinator stateCoordinator; private final SubscriptionStateCoordinator stateCoordinator;
private final ObjectMapper objectMapper = CustomUtil.objectMapper; private final ObjectMapper objectMapper = CustomUtil.objectMapper;
@Autowired(required = false)
SubscriptionEventService subscriptionEvents;
@Value("${subscription.standby.sync-secret:}") @Value("${subscription.standby.sync-secret:}")
String syncSecret; String syncSecret;
@@ -39,6 +42,12 @@ public class SubscriptionStandbySnapshotService {
@Value("${subscription.cache-root:sub/accounts}") @Value("${subscription.cache-root:sub/accounts}")
String cacheRoot; String cacheRoot;
/**
* 与 {@link SubscriptionRefreshService} 共用的分代缓存解析器。
* 测试可直接覆写;未设置时按 {@link #cacheRoot} 懒构造。
*/
SubscriptionCacheStore cacheStore;
@PostConstruct @PostConstruct
void validateConfiguration() { void validateConfiguration() {
if (syncEnabled && (syncSecret == null || syncSecret.isBlank())) if (syncEnabled && (syncSecret == null || syncSecret.isBlank()))
@@ -51,12 +60,14 @@ public class SubscriptionStandbySnapshotService {
try { try {
Map<Integer, SubscriptionAccountSnapshot> accountMap = new HashMap<>(); Map<Integer, SubscriptionAccountSnapshot> accountMap = new HashMap<>();
for (SubscriptionAccount account : subMapper.selectAllSubscriptionAccounts()) { for (SubscriptionAccount account : subMapper.selectAllSubscriptionAccounts()) {
Path v2Path = cachedPath(account.getId(), "v2"); if (!account.isEnabled())
Path clashPath = cachedPath(account.getId(), "cat");
if (!account.isEnabled() || !Files.isRegularFile(v2Path) || !Files.isRegularFile(clashPath))
continue; continue;
byte[] v2 = Files.readAllBytes(v2Path); // 从同一分代读出成对内容,保证快照里两份文件同版本;无完整版本则跳过该账号。
byte[] clash = Files.readAllBytes(clashPath); Optional<SubscriptionCacheStore.Pair> pair = store().read(account.getId());
if (pair.isEmpty())
continue;
byte[] v2 = pair.get().v2();
byte[] clash = pair.get().clash();
SubscriptionAccountSnapshot snapshot = new SubscriptionAccountSnapshot(); SubscriptionAccountSnapshot snapshot = new SubscriptionAccountSnapshot();
snapshot.setAccountId(account.getId()); snapshot.setAccountId(account.getId());
snapshot.setEnabled(true); snapshot.setEnabled(true);
@@ -71,7 +82,9 @@ public class SubscriptionStandbySnapshotService {
List<SubscriptionAccountSnapshot> accounts = new ArrayList<>(accountMap.values()); List<SubscriptionAccountSnapshot> accounts = new ArrayList<>(accountMap.values());
accounts.sort(Comparator.comparing(SubscriptionAccountSnapshot::getAccountId)); accounts.sort(Comparator.comparing(SubscriptionAccountSnapshot::getAccountId));
List<SubscriptionBindingSnapshot> bindings = new ArrayList<>(); List<SubscriptionBindingSnapshot> bindings = new ArrayList<>();
for (SubBind bind : subMapper.selectAllSubBind()) { List<SubBind> currentBindings = subMapper.selectAllSubBind();
if (subscriptionEvents != null) subscriptionEvents.rememberBindings(currentBindings);
for (SubBind bind : currentBindings) {
if (bind.getSubscriptionAccountId() == null || !accountMap.containsKey(bind.getSubscriptionAccountId())) if (bind.getSubscriptionAccountId() == null || !accountMap.containsKey(bind.getSubscriptionAccountId()))
continue; continue;
SubscriptionBindingSnapshot snapshot = new SubscriptionBindingSnapshot(); SubscriptionBindingSnapshot snapshot = new SubscriptionBindingSnapshot();
@@ -132,7 +145,9 @@ public class SubscriptionStandbySnapshotService {
private static String hex(byte[] input) { return HexFormat.of().formatHex(input); } private static String hex(byte[] input) { return HexFormat.of().formatHex(input); }
private Path cachedPath(Integer accountId, String client) { private synchronized SubscriptionCacheStore store() {
return Path.of(cacheRoot, String.valueOf(accountId), "v2".equals(client) ? "v2ray.txt" : "clash.yaml"); if (cacheStore == null)
cacheStore = new SubscriptionCacheStore(cacheRoot);
return cacheStore;
} }
} }
@@ -0,0 +1,23 @@
package com.lion.lionwebsite.Service;
import java.util.UUID;
/** 协议 v2 的操作 ID 生成与解析。稳定 ID 是「节点重发/重试不重复执行副作用」的依据。 */
public final class TaskProtocol {
private TaskProtocol() {
}
/** 生成一个新的稳定操作 ID(下载或删除),形如 {@code op-<uuid>}。 */
public static String newOperationId(String action) {
return action + "-" + UUID.randomUUID();
}
public static String newDownloadOperationId() {
return newOperationId("dl");
}
public static String newDeleteOperationId() {
return newOperationId("del");
}
}
@@ -0,0 +1,37 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.Gallery;
import com.lion.lionwebsite.Domain.GalleryTask;
import com.lion.lionwebsite.Message.TaskReportMessage;
import java.util.List;
/**
* 协议 v2 状态上报的消费入口。
*
* <p>定义成接口是为了断开依赖环:RemoteService 需要把上报交给生命周期服务,
* 而生命周期服务需要有 RemoteService 才能下发命令。RemoteService 通过可选的 setter
* 注入本接口(见 {@code RemoteService#setTaskReportHandler}),因此构造期没有环。
*/
public interface TaskReportHandler {
/**
* 提交一份上报。
*
* @return 已成功提交(可回 ACK)时返回提交结果;数据库失败等「不可回 ACK」的情况返回 null。
* 返回 null 时节点会保留快照并稍后重投,这正是「失败写入不得清任务」的要求。
*/
CommittedReport apply(TaskReportMessage report);
/**
* 已提交的结果。
*
* @param reportId 本次上报 ID(回执用)
* @param newlyCompleted 本次「首次」完成的任务,通知与 WebSocket 由独立执行器在事务之后处理,
* 因此慢通知不会阻塞数据库消费线程;重复上报不会重复出现在这里
* @param acceptedTasks 事务内实际被接受的进度更新(含下载中/压缩中等中间状态,已排除过期、
* 旧世代与待删除任务),WebSocket 只推这一批,避免前端看到被丢弃的迟到状态
*/
record CommittedReport(String reportId, List<Gallery> newlyCompleted, List<GalleryTask> acceptedTasks) {
}
}
@@ -0,0 +1,45 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.GalleryTask;
/**
* 任务业务状态码到展示文案的唯一映射点。
*
* <p>状态以稳定代码表达,中文文案只出现在展示层(WebSocket/列表),
* 避免把「失败/待重试/结果未知」塞进成功失败提示字符串。
*/
public final class TaskStatus {
public static final String SUBMITTED = "已提交";
/** 已登记创建意图、尚未确认上游提交;用于让任务列表在提交期间也能看到占位任务。 */
public static final String PENDING_SUBMIT = "提交中";
public static final String DOWNLOADING = "下载中";
public static final String WAITING_COMPRESS = "等待压缩";
public static final String COMPRESSING = "压缩中";
public static final String COMPLETE = "下载完成";
public static final String UNKNOWN = "提交未知";
public static final String DELETE_PENDING = "待删除";
public static final String DELETE_FAILED = "删除失败";
private TaskStatus() {
}
/**
* 节点上报的进度状态映射为业务文案;未知状态码返回 null,由调用方忽略。
*
* <p>GalleryTask 里的状态码是可变 static byte,不是编译期常量,因此不能用 switch case;
* 这里用整数比较,避免为改常量引入风险。
*/
public static String fromReport(byte reportStatus) {
int status = reportStatus;
if (status == GalleryTask.DOWNLOADING)
return DOWNLOADING;
if (status == GalleryTask.DOWNLOAD_COMPLETE)
return WAITING_COMPRESS;
if (status == GalleryTask.COMPRESSING)
return COMPRESSING;
if (status == GalleryTask.COMPRESS_COMPLETE)
return COMPLETE;
return null;
}
}
@@ -0,0 +1,47 @@
package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Domain.Gallery;
import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import com.lion.lionwebsite.Util.GalleryUtil;
import com.lion.lionwebsite.Util.PreparedGallery;
import org.springframework.stereotype.Component;
import java.io.IOException;
/**
* 上游图站的读写边界。
*
* <p>分成两步是协议 v2 的核心约束:
* <ol>
* <li>{@link #prepare} 只做读取(页面、分辨率),绝不产生下载副作用;</li>
* <li>调用方必须在持久化 SUBMITTING 之后才调用 {@link #submit},因为那一步才真正 POST。</li>
* </ol>
*
* <p>拆成接口是为了让生命周期服务可以在测试里注入假实现,也方便后续把抓取/解析/提交
* 进一步分开;默认实现 {@link GalleryUtilAdapter} 直接转发到现有 GalleryUtil。
*/
public interface UpstreamGateway {
/** 只读解析:失败返回 null 表示链接无效或页面不可用,不产生任何副作用。 */
PreparedGallery prepare(String link) throws IOException;
/** 真正提交下载请求。抛异常表示结果未知,由调用方落 UNKNOWN,不做自动重发。 */
Gallery submit(PreparedGallery prepared, String targetResolution)
throws IOException, ResolutionNotMatchException;
/** 默认实现:转发现有 GalleryUtil 的拆分方法。 */
@Component
class GalleryUtilAdapter implements UpstreamGateway {
@Override
public PreparedGallery prepare(String link) throws IOException {
return GalleryUtil.prepare(link);
}
@Override
public Gallery submit(PreparedGallery prepared, String targetResolution)
throws IOException, ResolutionNotMatchException {
return GalleryUtil.submit(prepared, targetResolution);
}
}
}
@@ -3,16 +3,23 @@ package com.lion.lionwebsite.Service;
import com.lion.lionwebsite.Dao.normal.CollectMapper; import com.lion.lionwebsite.Dao.normal.CollectMapper;
import com.lion.lionwebsite.Dao.normal.GalleryMapper; import com.lion.lionwebsite.Dao.normal.GalleryMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper; import com.lion.lionwebsite.Dao.normal.UserMapper;
import com.lion.lionwebsite.Dao.normal.SubMapper;
import com.lion.lionwebsite.Domain.User; import com.lion.lionwebsite.Domain.User;
import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor; import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import com.lion.lionwebsite.Util.Response; import com.lion.lionwebsite.Util.Response;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.transaction.support.TransactionTemplate;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.concurrent.locks.Lock;
import java.util.function.Supplier;
import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
@Service @Service
@Slf4j @Slf4j
@@ -27,6 +34,11 @@ public class UserService{
final TaskHandlerInterceptor taskHandlerInterceptor; final TaskHandlerInterceptor taskHandlerInterceptor;
@Autowired SubMapper subMapper;
@Autowired SubscriptionStateCoordinator subscriptionState;
@Autowired RemoteService remoteService;
@Autowired @Qualifier("mainTransactionTemplate") TransactionTemplate transactions;
public String addAuthCode(String targetAuthCode, String people) { public String addAuthCode(String targetAuthCode, String people) {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
@@ -65,12 +77,22 @@ public class UserService{
Response response = Response.generateResponse(); Response response = Response.generateResponse();
try{ try{
if(userMapper.isExist(targetAuthCode) != 0){ return coordinated(() -> {
userMapper.updateUsername(targetAuthCode, newUsername); if (userMapper.isExist(targetAuthCode) == 0)
response.success("修改成功"); return response.failure("授权码不存在").toJSONString();
}else { if (newUsername == null || newUsername.isBlank())
response.failure("授权码不存在"); return response.failure("用户名不能为空").toJSONString();
} String name = newUsername.trim();
User current = userMapper.selectUserByAuthCode(targetAuthCode);
User existing = userMapper.selectUserByUsername(name);
if (existing != null && (current == null || existing.getId() != current.getId()))
return response.failure("用户名已存在").toJSONString();
userMapper.updateUsername(targetAuthCode, name);
if (subMapper != null && current != null)
subMapper.renameBoundUser(current.getId(), current.getUsername(), name);
if (remoteService != null) remoteService.requestSubscriptionSync();
return response.success("修改成功").toJSONString();
});
}catch (Exception e){ }catch (Exception e){
log.error("修改用户名失败", e); log.error("修改用户名失败", e);
response.failure("修改失败"); response.failure("修改失败");
@@ -84,17 +106,22 @@ public class UserService{
if(userMapper.isExist(targetAuthCode) != 0) if(userMapper.isExist(targetAuthCode) != 0)
try{ try{
User user = userMapper.selectUserByAuthCode(targetAuthCode); coordinated(() -> {
User user = userMapper.selectUserByAuthCode(targetAuthCode);
//修改下载人以及取消收藏 ArrayList<Integer> galleries = collectMapper.selectGidByCollector(user.getId());
ArrayList<Integer> galleries = collectMapper.selectGidByCollector(user.getId()); for(Integer gallery: galleries)
for(Integer gallery: galleries) collectMapper.disCollect(gallery, user.getId());
collectMapper.disCollect(gallery, user.getId()); galleryMapper.updateGalleryDownloader(ADMIN_USER_ID, user.getId());
galleryMapper.updateGalleryDownloader(3, user.getId()); if (subMapper != null) {
subMapper.deleteSubUpdateRecord(user.getUsername());
userMapper.deleteUserByAuthCode(targetAuthCode); subMapper.deleteSubBind(user.getUsername());
}
userMapper.deleteUserByAuthCode(targetAuthCode);
return "deleted";
});
response.success("删除成功"); response.success("删除成功");
taskHandlerInterceptor.updateAuthCodes(); taskHandlerInterceptor.updateAuthCodes();
if (remoteService != null) remoteService.requestSubscriptionSync();
}catch (Exception e){ }catch (Exception e){
log.error("删除授权码失败", e); log.error("删除授权码失败", e);
response.failure("删除失败"); response.failure("删除失败");
@@ -107,27 +134,29 @@ public class UserService{
} }
public String alterStatus(String AuthCode, boolean isEnable){ public String alterStatus(String AuthCode, boolean isEnable){
Response response = Response.generateResponse(); try {
String result = coordinated(() -> {
if(userMapper.isExist(AuthCode) == 1){ if (userMapper.isExist(AuthCode) != 1)
int id = userMapper.selectUserByAuthCode(AuthCode).getId(); return Response._failure("该用户不存在");
userMapper.updateIsEnableById(id, isEnable); int id = userMapper.selectUserByAuthCode(AuthCode).getId();
galleryMapper.updateGalleryDownloader(3, id); userMapper.updateIsEnableById(id, isEnable);
response.success("修改用户状态成功"); if (!isEnable) galleryMapper.updateGalleryDownloader(ADMIN_USER_ID, id);
taskHandlerInterceptor.updateAuthCodes(); return Response._success("修改用户状态成功");
});
// Refresh the in-memory identity view only after the database commits.
if (result.contains("\"result\":\"success\"")) taskHandlerInterceptor.updateAuthCodes();
return result;
} catch (RuntimeException error) {
log.warn("修改用户状态失败: {}", error.getClass().getSimpleName());
return Response._failure("修改用户状态失败");
} }
else{
response.failure("该用户不存在");
}
return response.toJSONString();
} }
public String getAllUser() { public String getAllUser() {
Response response = Response.generateResponse(); Response response = Response.generateResponse();
User[] users = userMapper.selectAllUser(); User[] users = userMapper.selectAllUser();
response.success(new ObjectMapper().valueToTree(users).toString()); response.success(objectMapper.valueToTree(users));
return response.toJSONString(); return response.toJSONString();
} }
@@ -135,4 +164,28 @@ public class UserService{
public int getUserId(String AuthCode){ public int getUserId(String AuthCode){
return userMapper.selectUserByAuthCode(AuthCode).getId(); return userMapper.selectUserByAuthCode(AuthCode).getId();
} }
/**
* 管理员判定。项目没有角色表,历史上以 id=3(狮子)作为管理员,
* 前端也是用 userId === 3 判断;这里把这个约定收敛到一处,
* 权限相关的服务端校验统一走它。
*/
public static final int ADMIN_USER_ID = 3;
public boolean isAdmin(String AuthCode){
if(AuthCode == null)
return false;
User user = userMapper.selectUserByAuthCode(AuthCode);
return user != null && user.getId() == ADMIN_USER_ID;
}
private String coordinated(Supplier<String> action) {
Lock lock = subscriptionState == null ? null : subscriptionState.writeLock();
if (lock != null) lock.lock();
try {
return transactions == null ? action.get() : transactions.execute(ignored -> action.get());
} finally {
if (lock != null) lock.unlock();
}
}
} }
@@ -1,7 +1,7 @@
package com.lion.lionwebsite.Service; package com.lion.lionwebsite.Service;
import com.fasterxml.jackson.databind.ObjectMapper; import tools.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode; import tools.jackson.databind.node.ObjectNode;
import com.lion.lionwebsite.Domain.GalleryTask; import com.lion.lionwebsite.Domain.GalleryTask;
import com.lion.lionwebsite.Util.CustomUtil; import com.lion.lionwebsite.Util.CustomUtil;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
@@ -0,0 +1,45 @@
package com.lion.lionwebsite.Util;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.DumperOptions;
import java.util.*;
import java.util.regex.Pattern;
/** YAML semantics rather than indentation determine nodes and references. */
public final class ClashSubscriptionParser {
private static final Pattern MULTIPLIER = Pattern.compile("(\\d+(?:\\.\\d+)?)x\\s*$", Pattern.CASE_INSENSITIVE);
private ClashSubscriptionParser() { }
public static String transform(List<String> lines, boolean filter, double threshold) {
Object loaded = new Yaml().load(String.join("\n", lines));
if (!(loaded instanceof Map<?, ?> source)) throw new IllegalArgumentException("Clash 上游必须为 YAML 对象");
if (!(source.get("proxies") instanceof List<?> proxies)) throw new IllegalArgumentException("Clash 上游缺少 proxies 列表");
Map<Object, Object> output = new LinkedHashMap<>(source);
Set<String> removed = new HashSet<>();
List<Object> kept = new ArrayList<>();
for (Object node : proxies) {
if (!(node instanceof Map<?, ?> fields) || !(fields.get("name") instanceof String name))
throw new IllegalArgumentException("Clash 节点缺少名称");
var multiplier = MULTIPLIER.matcher(name);
if (filter && multiplier.find() && Double.parseDouble(multiplier.group(1)) > threshold)
removed.add(name);
else kept.add(node);
}
output.put("proxies", kept);
if (source.get("proxy-groups") instanceof List<?> groups) {
List<Object> result = new ArrayList<>();
for (Object group : groups) {
if (!(group instanceof Map<?, ?> fields)) throw new IllegalArgumentException("Clash 分组必须为对象");
Map<Object, Object> copy = new LinkedHashMap<>(fields);
if (fields.get("proxies") instanceof List<?> references)
copy.put("proxies", references.stream().filter(ref -> !removed.contains(ref)).toList());
result.add(copy);
}
output.put("proxy-groups", result);
}
DumperOptions options = new DumperOptions();
options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK);
options.setPrettyFlow(true);
return new Yaml(options).dump(output);
}
}
@@ -1,6 +1,6 @@
package com.lion.lionwebsite.Util; package com.lion.lionwebsite.Util;
import com.fasterxml.jackson.databind.ObjectMapper; import tools.jackson.databind.ObjectMapper;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
import lombok.Data; import lombok.Data;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
@@ -1,119 +1,86 @@
package com.lion.lionwebsite.Util; package com.lion.lionwebsite.Util;
import cn.hutool.core.util.StrUtil;
import cn.hutool.core.util.URLUtil;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse; import jakarta.servlet.http.HttpServletResponse;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.catalina.connector.ClientAbortException; import org.apache.catalina.connector.ClientAbortException;
import org.springframework.http.ContentDisposition;
import org.springframework.http.HttpHeaders; import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpRange;
import java.io.BufferedOutputStream; import java.io.*;
import java.io.File; import java.nio.charset.StandardCharsets;
import java.io.IOException; import java.util.List;
import java.io.RandomAccessFile;
@Slf4j @Slf4j
public class FileDownload { public class FileDownload {
public static void export(HttpServletRequest request, HttpServletResponse response, String path) { public static void export(HttpServletRequest request, HttpServletResponse response, String path) {
File file = new File(path); File file = new File(path);
if (!file.isFile()) {
String fileName = file.getName(); response.setStatus(HttpServletResponse.SC_NOT_FOUND);
return;
String range = request.getHeader(HttpHeaders.RANGE); }
// Size and content refer to the same opened file, even if a cache is replaced.
String rangeSeparator = "-"; try (RandomAccessFile input = new RandomAccessFile(file, "r")) {
// 开始下载位置 long size = input.length();
long startByte = 0; long start = 0;
// 结束下载位置 long end = size - 1;
long endByte = file.length() - 1; boolean partial = false;
String range = request.getHeader(HttpHeaders.RANGE);
// 如果是断点续传 if (range != null && range.startsWith("bytes=")) {
if (range != null && range.contains("bytes=") && range.contains(rangeSeparator)) { try {
// 设置响应状态码为 206 List<HttpRange> ranges = HttpRange.parseRanges(range);
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT); // Multiple ranges are intentionally ignored; send the full representation.
if (ranges.size() == 1) {
range = range.substring(range.lastIndexOf("=") + 1).trim(); if (size == 0) throw new IllegalArgumentException("empty file");
String[] ranges = range.split(rangeSeparator); start = ranges.getFirst().getRangeStart(size);
try { end = ranges.getFirst().getRangeEnd(size);
// 判断 range 的类型 if (start < 0 || start >= size || end < start)
if (ranges.length == 1) { throw new IllegalArgumentException("unsatisfiable range");
// 类型一:bytes=-2343 partial = true;
if (range.startsWith(rangeSeparator)) {
endByte = Long.parseLong(ranges[0]);
}
// 类型二:bytes=2343-
else if (range.endsWith(rangeSeparator)) {
startByte = Long.parseLong(ranges[0]);
} }
} catch (IllegalArgumentException e) {
response.setStatus(HttpServletResponse.SC_REQUESTED_RANGE_NOT_SATISFIABLE);
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes */" + size);
response.setContentLengthLong(0);
return;
} }
// 类型三:bytes=22-2343
else if (ranges.length == 2) {
startByte = Long.parseLong(ranges[0]);
endByte = Long.parseLong(ranges[1]);
}
} catch (NumberFormatException e) {
// 传参不规范,则直接返回所有内容
startByte = 0;
endByte = file.length() - 1;
} }
} else { long remaining = end - start + 1;
// 没有 ranges 即全部一次性传输,需要用 200 状态码,这一行应该可以省掉,因为默认返回是 200 状态码 response.setStatus(partial ? HttpServletResponse.SC_PARTIAL_CONTENT : HttpServletResponse.SC_OK);
response.setStatus(HttpServletResponse.SC_OK); response.setHeader(HttpHeaders.ACCEPT_RANGES, "bytes");
} String mime = request.getServletContext().getMimeType(file.getName());
response.setContentType(mime == null ? "application/octet-stream" : mime);
//要下载的长度(endByte 为总长度 -1,这时候要加回去) response.setHeader(HttpHeaders.CONTENT_DISPOSITION,
long contentLength = endByte - startByte + 1; ContentDisposition.inline().filename(file.getName(), StandardCharsets.UTF_8).build().toString());
//文件类型 response.setContentLengthLong(remaining);
String contentType = request.getServletContext().getMimeType(fileName); if (partial)
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes " + start + "-" + end + "/" + size);
if (StrUtil.isEmpty(contentType)) { if ("HEAD".equalsIgnoreCase(request.getMethod()))
contentType = "attachment"; return;
} input.seek(start);
// 关闭响应流:Tomcat 为大响应体创建的临时文件会在流关闭时删除,
response.setHeader(HttpHeaders.ACCEPT_RANGES, "bytes"); // 不关闭会一直堆积到 Full GC。flushBuffer 只能把内容送出去,不能触发清理。
response.setHeader(HttpHeaders.CONTENT_TYPE, contentType); try (BufferedOutputStream output = new BufferedOutputStream(response.getOutputStream())) {
// 这里文件名换你想要的,inline 表示浏览器可以直接使用 byte[] buffer = new byte[8192];
// 参考资料:https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Headers/Content-Disposition while (remaining > 0) {
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, contentType + ";filename=\"" + URLUtil.encode(fileName) + "\""); int count = input.read(buffer, 0, (int) Math.min(buffer.length, remaining));
response.setHeader(HttpHeaders.CONTENT_LENGTH, String.valueOf(contentLength)); if (count == -1)
// [要下载的开始位置]-[结束位置]/[文件总大小] throw new EOFException("File changed during download");
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes " + startByte + rangeSeparator + endByte + "/" + file.length()); output.write(buffer, 0, count);
remaining -= count;
BufferedOutputStream outputStream;
//已传送数据大小
long transmitted = 0;
try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
try {
outputStream = new BufferedOutputStream(response.getOutputStream());
byte[] buff = new byte[4096];
int len = 0;
randomAccessFile.seek(startByte);
while ((transmitted + len) <= contentLength && (len = randomAccessFile.read(buff)) != -1) {
outputStream.write(buff, 0, len);
transmitted += len;
// 本地测试, 防止下载速度过快
// Thread.sleep(1);
} }
// 处理不足 buff.length 部分 output.flush();
if (transmitted < contentLength) {
len = randomAccessFile.read(buff, 0, (int) (contentLength - transmitted));
outputStream.write(buff, 0, len);
}
outputStream.flush();
response.flushBuffer(); response.flushBuffer();
randomAccessFile.close();
// log.trace("下载完毕: {}-{}, 已传输 {}", startByte, endByte, transmitted);
} catch (ClientAbortException e) {
// ignore 用户停止下载
// log.trace("用户停止下载: {}-{}, 已传输 {}", startByte, endByte, transmitted);
} catch (IOException e) {
log.error("文件下载IO错误: {}", path, e);
} }
} catch (ClientAbortException e) {
// The client cancelled its download.
} catch (IOException e) { } catch (IOException e) {
log.warn("关闭RandomAccessFile失败: {}", path, e); log.warn("文件下载失败: {}", path, e);
if (!response.isCommitted()) {
response.reset();
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
}
} }
} }
} }
@@ -0,0 +1,97 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.Gallery;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;
import java.util.*;
import java.util.regex.Pattern;
/** Pure parsing: upstream requests and download submission live outside this class. */
public final class GalleryPageParser {
private GalleryPageParser() { }
/** 归档页 #db 单元格开头:分辨率标签 + 正数体积 + 受支持单位;后缀费用文本(Free、50GP)忽略。 */
private static final Pattern RESOLUTION_ENTRY =
Pattern.compile("^(Original|(\\d+)x)\\s+(\\d+(?:\\.\\d+)?)\\s*([A-Za-z]+)(?=\\s|$)", Pattern.CASE_INSENSITIVE);
public static Gallery metadata(Document document, String link, int gid) throws IOException {
String title = document.select("#gn").text();
if (title.isBlank()) title = document.select("#gj").text();
if (title.isBlank()) throw new IOException("上游页面缺少画廊名称,可能是错误页");
Map<String, String> fields = new HashMap<>();
for (Element row : document.select("#gdd tr")) {
var cells = row.select("td");
if (cells.size() < 2) continue;
String name = cells.get(0).text().replace(":", "").strip().toLowerCase(Locale.ROOT);
fields.put(name, cells.get(1).text().strip());
}
var pages = Pattern.compile("\\d+").matcher(fields.getOrDefault("length", ""));
if (!pages.find()) throw new IOException("上游页面缺少画廊页数");
Gallery result = new Gallery();
result.setGid(gid); result.setLink(link); result.setName(title + " [" + gid + "]");
result.setCreateTime(System.currentTimeMillis() / 1000); result.setProceeding(0);
result.setLanguage(fields.getOrDefault("language", "").replace("\u00a0", " "));
try {
result.setPages(Integer.parseInt(pages.group()));
result.setFileSize(CustomUtil.stringToFileSize(fields.getOrDefault("file size", "").replace(" ", "")));
} catch (RuntimeException failure) { throw new IOException("上游画廊页数或文件大小格式不支持", failure); }
if (result.getPages() <= 0) throw new IOException("上游画廊页数必须为正数");
result.setDisplayFileSize(CustomUtil.fileSizeToString(result.getFileSize()));
Element thumbnail = document.selectFirst("#gd1 [style*=url]");
var image = Pattern.compile("url\\(\\s*['\"]?(.*?)['\"]?\\s*\\)").matcher(thumbnail == null ? "" : thumbnail.attr("style"));
String imageUrl = image.find() ? image.group(1) : document.select("#gd1 img").attr("src");
result.setThumb_link(imageUrl.replace("https://s.exhentai.org", ""));
return result;
}
public static String archiveLink(Document document) throws IOException {
for (Element anchor : document.select("#gd5 a")) {
var matcher = Pattern.compile("https?://[^'\"\\s]+").matcher(anchor.attr("onclick"));
if (matcher.find()) return matcher.group();
String href = anchor.absUrl("href");
if (!href.isBlank() && (anchor.text().toLowerCase(Locale.ROOT).contains("download") || href.contains("archive")))
return href;
}
throw new IOException("上游页面缺少归档入口");
}
/**
* 解析归档页 #db 的可选分辨率。每个单元格合并后的文本形如 {@code Original 36.31 MiB Free}、
* {@code 800x 5.65 MiB Free}、{@code 2560x N/A N/A},单段布局形如 {@code 1280x 12.5 MiB},
* 都由同一条锚定规则覆盖;只看开头(分辨率 + 正数体积 + 单位),故 N/A 与费用文本不匹配,
* 配对也不会跨 {@code <td>} 借用相邻格。体积须能被 {@link CustomUtil#stringToFileSize(String)}
* 解析为正数,宽度须为正且不溢出。结果按宽度升序、{@code Original} 最后;无可用项抛 IOException。
*/
public static Map<String, String> resolutions(Document document) throws IOException {
TreeMap<Integer, String> sizes = new TreeMap<>();
String original = null;
for (Element cell : document.select("#db td")) {
var matcher = RESOLUTION_ENTRY.matcher(cell.text().replace('\u00a0', ' ').strip());
if (!matcher.find()) continue;
String size = matcher.group(3) + matcher.group(4);
try {
if (CustomUtil.stringToFileSize(size) <= 0) continue;
} catch (RuntimeException unsupported) {
continue;
}
if (matcher.group(1).equalsIgnoreCase("Original")) {
original = size;
continue;
}
int width;
try {
width = Integer.parseInt(matcher.group(2));
} catch (NumberFormatException outOfRange) {
continue;
}
if (width <= 0) continue;
sizes.put(width, size);
}
Map<String, String> result = new LinkedHashMap<>();
sizes.forEach((width, size) -> result.put(width + "x", size));
if (original != null) result.put("Original", original);
if (result.isEmpty()) throw new IOException("上游页面没有可用分辨率");
return result;
}
}
@@ -0,0 +1,74 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.GalleryForQuery;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Pattern;
/** Search entries support table and thumbnail layouts; metadata excludes title/tag text. */
public final class GallerySearchParser {
private GallerySearchParser() { }
public static List<GalleryForQuery> parse(Document document) throws IOException {
List<GalleryForQuery> result = new ArrayList<>();
for (Element row : document.select("tr, .gl1t")) {
if (row.tagName().equals("tr") && !row.select(".gl1t").isEmpty()) continue;
Element link = row.selectFirst("a[href*=/g/]");
if (link == null) continue;
String href = link.attr("href");
Integer gid = GalleryUtil.parseGid(href);
if (gid == null) continue;
GalleryForQuery gallery = new GalleryForQuery();
gallery.setGid(gid.toString()); gallery.setLink(href);
Element title = row.selectFirst(".glink");
String name = title == null ? "" : title.text();
if (name.isBlank()) {
for (Element candidate : row.select("a[href*=/g/]")) {
if (!candidate.text().isBlank()) {
name = candidate.text();
break;
}
}
}
gallery.setName(name);
Element type = row.selectFirst(".cn, .cs");
Element firstCell = row.selectFirst("td");
gallery.setType(type != null ? type.text() : firstCell == null ? "" : firstCell.text());
Element image = row.selectFirst("img");
String thumbnail = image == null ? "" : image.attr("src");
if (thumbnail.isBlank() || thumbnail.startsWith("data:")) thumbnail = image == null ? "" : image.attr("data-src");
gallery.setThumbnailUrl(thumbnail.replace("https://s.exhentai.org", ""));
gallery.setPage(pageCount(row, gid));
Element date = row.selectFirst("[id^=posted_]");
if (date != null) gallery.setUploadTime(date.text());
else {
var dates = Pattern.compile("\\d{4}-\\d{2}-\\d{2}(?:\\s+\\d{2}:\\d{2})?").matcher(row.text());
gallery.setUploadTime(dates.find() ? dates.group() : "");
}
if (gallery.getName().isBlank()) throw new IOException("上游搜索结果缺少名称 gid=" + gid);
result.add(gallery);
}
return result;
}
private static int pageCount(Element row, int gid) throws IOException {
Element metadata = row.clone();
// Extended rows can put title and metadata in the same cell: remove only
// the title links/tags, retaining their containing cells and page counts.
metadata.select("a[href*=/g/], .glink, .gt, .gtl, .gtd, .glthumb").remove();
var expression = Pattern.compile("^(\\d+)\\s+pages?$", Pattern.CASE_INSENSITIVE);
for (Element element : metadata.getAllElements()) {
var pages = expression.matcher(element.text().strip());
if (!pages.matches()) continue;
try {
int count = Integer.parseInt(pages.group(1));
if (count > 0) return count;
} catch (NumberFormatException invalid) {
throw new IOException("上游搜索结果页数格式不支持 gid=" + gid, invalid);
}
}
throw new IOException("上游搜索结果缺少页数 gid=" + gid);
}
}
@@ -1,18 +1,23 @@
package com.lion.lionwebsite.Util; package com.lion.lionwebsite.Util;
import com.fasterxml.jackson.databind.JsonNode; import tools.jackson.databind.JsonNode;
import com.lion.lionwebsite.Domain.Gallery; import com.lion.lionwebsite.Domain.Gallery;
import com.lion.lionwebsite.Domain.ImageKeyCache; import com.lion.lionwebsite.Domain.ImageKeyCache;
import com.lion.lionwebsite.Exception.ResolutionNotMatchException; import com.lion.lionwebsite.Exception.ResolutionNotMatchException;
import org.apache.http.HttpEntity; import java.nio.file.*;
import org.apache.http.client.entity.EntityBuilder; import java.util.concurrent.TimeUnit;
import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.http.client.methods.HttpGet; import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.http.client.methods.HttpPost; import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.http.entity.ContentType; import org.apache.hc.client5.http.entity.EntityBuilder;
import org.apache.http.entity.mime.MultipartEntityBuilder; import org.apache.hc.client5.http.entity.UrlEncodedFormEntity;
import org.apache.http.impl.client.CloseableHttpClient; import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients; import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.message.BasicNameValuePair;
import org.apache.hc.core5.util.Timeout;
import org.jsoup.Jsoup; import org.jsoup.Jsoup;
import org.jsoup.nodes.Document; import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element; import org.jsoup.nodes.Element;
@@ -24,8 +29,9 @@ import java.io.BufferedReader;
import java.io.File; import java.io.File;
import java.io.IOException; import java.io.IOException;
import java.io.InputStreamReader; import java.io.InputStreamReader;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.util.*; import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicInteger;
import static com.lion.lionwebsite.Util.CustomUtil.objectMapper; import static com.lion.lionwebsite.Util.CustomUtil.objectMapper;
@@ -37,14 +43,31 @@ public class GalleryUtil {
static String POST = "post"; static String POST = "post";
static String GET = "get"; static String GET = "get";
static String FORM_DATA = "formData"; static String FORM_URLENCODED = "formUrlEncoded";
static String JSON = "json"; static String JSON = "json";
static ConcurrentHashMap<String, String> gid2MpvKey = new ConcurrentHashMap<>(); /**
* gid → mpvkey 的内存缓存。key 会轮换,代码里对失配也做了刷新兜底,
* 因此这里只需限制为有界 LRU,避免长期运行无上限增长。
*/
static final Map<String, String> gid2MpvKey = Collections.synchronizedMap(
new LinkedHashMap<>(64, 0.75f, true) {
@Override
protected boolean removeEldestEntry(Map.Entry<String, String> eldest) {
return size() > 2048;
}
});
/** Reusable HTTP client —不要每次请求新建 */ /** Reusable HTTP client —不要每次请求新建 */
private static final CloseableHttpClient httpClient = HttpClients.createDefault(); private static final CloseableHttpClient httpClient = HttpClients.custom()
// An upstream download POST has no idempotency key; never replay it on 429/503.
.disableAutomaticRetries()
.setDefaultRequestConfig(RequestConfig.custom()
.setConnectTimeout(Timeout.ofMilliseconds(5_000))
.setConnectionRequestTimeout(Timeout.ofMilliseconds(5_000))
// HttpClient 5 将 socket 读超时改名为 responseTimeout。
.setResponseTimeout(Timeout.ofMilliseconds(15_000)).build()).build();
/** E-Hentai Cookie, injected from application.yaml via CustomBean */ /** E-Hentai Cookie, injected from application.yaml via CustomBean */
private static String ehentaiCookie = ""; private static String ehentaiCookie = "";
@@ -64,6 +87,12 @@ public class GalleryUtil {
* @throws ResolutionNotMatchException 没有目标分辨率 * @throws ResolutionNotMatchException 没有目标分辨率
*/ */
public static Gallery parse(String url, boolean isDownload, String targetResolution) throws IOException, ResolutionNotMatchException { public static Gallery parse(String url, boolean isDownload, String targetResolution) throws IOException, ResolutionNotMatchException {
PreparedGallery prepared = prepare(url);
if (prepared == null) return null;
return isDownload ? submit(prepared, targetResolution) : prepared.gallery();
}
public static PreparedGallery prepare(String url) throws IOException {
String origin = verifyLink(url); String origin = verifyLink(url);
if(origin == null){ if(origin == null){
@@ -71,66 +100,30 @@ public class GalleryUtil {
} }
//初始化图片 //初始化图片
Gallery gallery = new Gallery(); Integer gid = parseGid(url);
gallery.setLink(url); if (gid == null) {
gallery.setCreateTime(System.currentTimeMillis()/1000); log.warn("链接无法解析出 gid,按无效链接处理");
gallery.setGid(Integer.parseInt(url.split("/")[4])); return null;
gallery.setProceeding(0); }
Document galleryDoc = Jsoup.parse(requests(url, GET, null, null), url);
//访问图片页面 Gallery gallery = GalleryPageParser.metadata(galleryDoc, url, gid);
String galleryPage = requests(url, GET, null, null); String archiveLink = GalleryPageParser.archiveLink(galleryDoc);
Document galleryDoc = Jsoup.parse(galleryPage);
//收集图片基本信息
gallery.setName(galleryDoc.select("#gn").text() + " [" + gallery.getGid() + ']');
gallery.setLanguage(galleryDoc.select("#gdd > table > tbody > tr:nth-child(4) > td.gdt2").text().replace("%nbps", ""));
gallery.setPages(Integer.parseInt(galleryDoc.select("#gdd > table > tbody > tr:nth-child(6) > td.gdt2").text().split(" ")[0]));
gallery.setFileSize(CustomUtil.stringToFileSize(galleryDoc.select("#gdd > table > tbody > tr:nth-child(5) > td.gdt2").text().replace(" ", "")));
gallery.setDisplayFileSize(CustomUtil.fileSizeToString(gallery.getFileSize()));
String thumb_link = galleryDoc.select("#gd1 > div").getFirst().attr("style");
gallery.setThumb_link(thumb_link.substring(thumb_link.indexOf("url(") + 4, thumb_link.lastIndexOf(")")).replace("https://s.exhentai.org", ""));
//查找下载页面链接并初始化参数
String download_link = galleryDoc.select("#gd5 > p:nth-child(2) > a").attr("onclick").split("'")[1];
HashMap<String, String> headers = new HashMap<>(); HashMap<String, String> headers = new HashMap<>();
headers.put("origin", origin); headers.put("referer", url);
Document downloadDoc = Jsoup.parse(requests(archiveLink, GET, headers, null), archiveLink);
String download_link = downloadDoc.select("#hathdl_form").attr("abs:action");
if (download_link.isBlank()) throw new IOException("上游页面缺少下载提交表单");
gallery.setAvailableResolution(GalleryPageParser.resolutions(downloadDoc));
gallery.setStatus("等待确认下载");
return new PreparedGallery(gallery, download_link, origin);
}
public static Gallery submit(PreparedGallery prepared, String targetResolution) throws IOException, ResolutionNotMatchException {
Gallery gallery = prepared.gallery();
String download_link = prepared.formUrl();
HashMap<String, String> headers = new HashMap<>();
headers.put("origin", prepared.origin());
headers.put("referer", download_link); headers.put("referer", download_link);
//访问下载页面,获取分辨率
String downloadPage = requests(download_link, GET, headers, null);
Document downloadDoc = Jsoup.parse(downloadPage);
Elements resolutions = downloadDoc.select("#db > div > table > tbody > tr > td");
download_link = downloadDoc.select("#hathdl_form").attr("action");
Map<String, String> availableResolution = new LinkedHashMap<>();
Map<String, String> tempResolutions = new LinkedHashMap<>();
ArrayList<Integer> tempResolutionKeys = new ArrayList<>();
for(Element element: resolutions){
String[] parameter = element.select("td > p").text().split(" ");
if(parameter[1].contains("N/A"))
continue;
tempResolutions.put(parameter[0], parameter[1] + parameter[2]);
if(parameter[0].contains("x")){
tempResolutionKeys.add(Integer.valueOf(parameter[0].replace("x", "")));
}
}
tempResolutionKeys.sort(Comparator.comparingInt(o -> o));
for(Integer resolution: tempResolutionKeys){
availableResolution.put(resolution.toString()+"x", tempResolutions.get(resolution+"x"));
}
availableResolution.put("Original", tempResolutions.get("Original"));
gallery.setAvailableResolution(availableResolution);
//如果不是下载,则直接返回
if(!isDownload){
gallery.setStatus("等待确认下载");
return gallery;
}
//如果目标分辨率不存在,抛出错误 //如果目标分辨率不存在,抛出错误
if(!gallery.getAvailableResolution().containsKey(targetResolution)){ if(!gallery.getAvailableResolution().containsKey(targetResolution)){
log.warn("目标分辨率不存在,可用分辨率: {}", gallery.getAvailableResolution()); log.warn("目标分辨率不存在,可用分辨率: {}", gallery.getAvailableResolution());
@@ -138,15 +131,16 @@ public class GalleryUtil {
} }
gallery.setResolution(targetResolution); gallery.setResolution(targetResolution);
gallery.setFileSize(CustomUtil.stringToFileSize(availableResolution.get(targetResolution).replace(" ", ""))); gallery.setFileSize(CustomUtil.stringToFileSize(gallery.getAvailableResolution().get(targetResolution).replace(" ", "")));
gallery.setDisplayFileSize(CustomUtil.fileSizeToString(gallery.getFileSize())); gallery.setDisplayFileSize(CustomUtil.fileSizeToString(gallery.getFileSize()));
//提交下载请求 //提交下载请求
HashMap<String, String> body = new HashMap<>(); HashMap<String, String> body = new HashMap<>();
body.put("payload", FORM_DATA); // The upstream form has no enctype: match the browser's default encoding.
body.put("payload", FORM_URLENCODED);
body.put("hathdl_xres", targetResolution.replace("x", "").replace("Original", "org")); body.put("hathdl_xres", targetResolution.replace("x", "").replace("Original", "org"));
downloadPage = requests(download_link, POST, headers, body); String downloadPage = requests(download_link, POST, headers, body);
downloadDoc = Jsoup.parse(downloadPage); Document downloadDoc = Jsoup.parse(downloadPage);
//判断下载请求是否提交成功 //判断下载请求是否提交成功
if(downloadDoc.select("#db > p:nth-child(2) > strong").text().startsWith("#")) if(downloadDoc.select("#db > p:nth-child(2) > strong").text().startsWith("#"))
@@ -169,14 +163,12 @@ public class GalleryUtil {
if(content.trim().isEmpty()) if(content.trim().isEmpty())
return null; return null;
Document document = Jsoup.parse(content); Document document = Jsoup.parse(content);
if(document.select("body > script").size() < 2) String script = document.select("script").stream().map(Element::data).collect(java.util.stream.Collectors.joining("\n"));
return null; if (!script.contains("imagelist") || !script.contains("mpvkey")) return null;
Element script = document.select("body > script").get(1);
String[] scripts = script.html().split("\n");
ArrayList<ImageKeyCache> imageKeyCaches = new ArrayList<>(); ArrayList<ImageKeyCache> imageKeyCaches = new ArrayList<>();
AtomicInteger page = new AtomicInteger(1); AtomicInteger page = new AtomicInteger(1);
gid2MpvKey.put(gid, scripts[1].split("=")[1].replace(";", "").replace("\"", "").replace(" ", "")); gid2MpvKey.put(gid, JavaScriptAssignments.string(script, "mpvkey"));
JsonNode nodes = objectMapper.readValue(scripts[2].replace("var imagelist = ", ""), JsonNode.class); JsonNode nodes = objectMapper.readTree(JavaScriptAssignments.array(script, "imagelist"));
nodes.forEach((n) -> { nodes.forEach((n) -> {
ImageKeyCache imageKeyCache = new ImageKeyCache(); ImageKeyCache imageKeyCache = new ImageKeyCache();
imageKeyCache.setGid(gid); imageKeyCache.setGid(gid);
@@ -187,12 +179,28 @@ public class GalleryUtil {
return imageKeyCaches; return imageKeyCaches;
} }
/**
* 解析 mpv 页里的 imagelist 行。
* 该行是 JS 赋值语句(`var imagelist = [...];`),行尾分号属 JavaScript 语法而非 JSON,
* 因此先剥掉「var imagelist = 」前缀与行尾分号,再交给 JSON 解析。
* Jackson 3 默认开启 FAIL_ON_TRAILING_TOKENS,若把分号留给它会在解析时报错。
*/
private static JsonNode parseImagelist(String scriptLine) {
String json = scriptLine.replace("var imagelist = ", "").trim();
if (json.endsWith(";"))
json = json.substring(0, json.length() - 1).trim();
return objectMapper.readValue(json, JsonNode.class);
}
public static String getMpvKey(String url){ public static String getMpvKey(String url){
String gid = String.valueOf(parseGid(url)); String gid = String.valueOf(parseGid(url));
return gid2MpvKey.computeIfAbsent(gid, k -> { String key = gid2MpvKey.get(gid);
if (key == null) {
// refreshMpvKey writes the cache itself; never call it inside computeIfAbsent.
refreshMpvKey(url); refreshMpvKey(url);
return gid2MpvKey.get(k); key = gid2MpvKey.get(gid);
}); }
return key;
} }
public static void refreshMpvKey(String url) { public static void refreshMpvKey(String url) {
@@ -205,15 +213,17 @@ public class GalleryUtil {
content = requests(mpvUrl, "get", header, null); content = requests(mpvUrl, "get", header, null);
}catch (Exception e){ }catch (Exception e){
log.error("刷新mpvKey失败, url: {}", url, e); log.error("刷新mpvKey失败, url: {}", url, e);
gid2MpvKey.put(parseGid(url) + "", null); gid2MpvKey.remove(parseGid(url) + "");
return; return;
} }
Document document = Jsoup.parse(content); try {
Element script = document.select("body > script").get(1); String scripts = Jsoup.parse(content).select("script").stream().map(Element::data)
String[] scripts = script.html().split("\n"); .collect(java.util.stream.Collectors.joining("\n"));
String mpvKey = scripts[1].split("=")[1].replace(";", "").replace("\"", "").replace(" ", ""); gid2MpvKey.put(parseGid(url) + "", JavaScriptAssignments.string(scripts, "mpvkey"));
gid2MpvKey.put(parseGid(url) + "", mpvKey); } catch (IOException e) {
log.info("刷新key:{}", mpvKey); gid2MpvKey.remove(parseGid(url) + "");
log.warn("上游mpv脚本无法解析,缓存已失效");
}
} }
public static String getImageUrl(String mpvKey, ImageKeyCache imageKeyCache) { public static String getImageUrl(String mpvKey, ImageKeyCache imageKeyCache) {
@@ -240,15 +250,38 @@ public class GalleryUtil {
} }
public static String convertImg(String imagePath, String suffix){ public static String convertImg(String imagePath, String suffix){
Runtime rt = Runtime.getRuntime(); Path source = Path.of(imagePath);
Path target = source.resolveSibling(source.getFileName().toString().replaceFirst("\\Q" + suffix + "\\E$", ".avif"));
if (source.equals(target)) return imagePath;
Path temporary = null;
Process process = null;
try { try {
Process exec = rt.exec(new String[]{"convert", imagePath, imagePath.replace(suffix, ".avif")}); temporary = Files.createTempFile(target.toAbsolutePath().getParent(), ".convert-", ".avif");
exec.waitFor(); process = new ProcessBuilder("convert", source.toString(), temporary.toString())
boolean ignored = new File(imagePath).delete(); .redirectErrorStream(true).redirectOutput(ProcessBuilder.Redirect.DISCARD).start();
return imagePath.replace(suffix, ".avif"); if (!process.waitFor(60, TimeUnit.SECONDS))
} catch (IOException| InterruptedException e) { throw new IOException("图片转换超时");
log.error("文件{}转换失败", imagePath, e); if (process.exitValue() != 0 || Files.size(temporary) == 0)
throw new IOException("图片转换失败");
try {
Files.move(temporary, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(temporary, target, StandardCopyOption.REPLACE_EXISTING);
}
Files.deleteIfExists(source);
return target.toString();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return null; return null;
} catch (IOException e) {
log.warn("文件{}转换失败", imagePath, e);
return null;
} finally {
if (process != null && process.isAlive()) process.destroyForcibly();
if (temporary != null) {
try { Files.deleteIfExists(temporary); }
catch (IOException e) { log.warn("清理图片转换临时文件失败", e); }
}
} }
} }
@@ -264,9 +297,18 @@ public class GalleryUtil {
if(url.length() < 40) if(url.length() < 40)
return null; return null;
if(!url.contains("/g/") || !url.contains("exhentai")) try {
URI uri = URI.create(url);
String scheme = uri.getScheme();
if ((!"https".equalsIgnoreCase(scheme) && !"http".equalsIgnoreCase(scheme))
|| !"exhentai.org".equalsIgnoreCase(uri.getHost())
|| uri.getRawUserInfo() != null || uri.getPort() != -1
|| uri.getPath() == null || !uri.getPath().matches("/g/[0-9]+/[A-Za-z0-9]+/?"))
return null;
return scheme.toLowerCase(Locale.ROOT) + "://exhentai.org";
} catch (IllegalArgumentException invalidUrl) {
return null; return null;
return ""; }
} }
/** /**
@@ -301,42 +343,62 @@ public class GalleryUtil {
httpPost.addHeader(header.getKey(), header.getValue()); httpPost.addHeader(header.getKey(), header.getValue());
if(body != null) { if(body != null) {
// Keep the caller's form intact; payload is a local encoding selector.
body = new HashMap<>(body);
String payload; String payload;
if((payload = body.remove("payload")).equals(JSON)) { if((payload = body.remove("payload")).equals(JSON)) {
EntityBuilder entityBuilder = EntityBuilder.create(); EntityBuilder entityBuilder = EntityBuilder.create();
entityBuilder.setContentType(ContentType.APPLICATION_JSON); entityBuilder.setContentType(ContentType.APPLICATION_JSON);
entityBuilder.setText(objectMapper.writeValueAsString(body)); entityBuilder.setText(objectMapper.writeValueAsString(body));
httpPost.setEntity(entityBuilder.build()); httpPost.setEntity(entityBuilder.build());
} else if(payload.equals(FORM_DATA)) { } else if(payload.equals(FORM_URLENCODED)) {
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create(); var fields = body.entrySet().stream()
body.forEach(multipartEntityBuilder::addTextBody); .map(entry -> new BasicNameValuePair(entry.getKey(), entry.getValue())).toList();
httpPost.setEntity(multipartEntityBuilder.build()); httpPost.setEntity(new UrlEncodedFormEntity(fields, StandardCharsets.UTF_8));
} }
} }
httpResponse = httpClient.execute(httpPost); httpResponse = httpClient.execute(httpPost);
} }
HttpEntity responseEntity = httpResponse.getEntity(); try (httpResponse) {
int statusCode = httpResponse.getStatusLine().getStatusCode(); HttpEntity responseEntity = httpResponse.getEntity();
StringBuilder stringBuilder = new StringBuilder(); int statusCode = httpResponse.getCode();
StringBuilder stringBuilder = new StringBuilder();
if(statusCode == 200){ if (statusCode == 200 && responseEntity != null) {
BufferedReader reader = new BufferedReader(new InputStreamReader(responseEntity.getContent())); try (BufferedReader reader = new BufferedReader(new InputStreamReader(responseEntity.getContent(), StandardCharsets.UTF_8))) {
String str; String str;
while((str = reader.readLine()) != null) while ((str = reader.readLine()) != null)
stringBuilder.append(str).append("\n"); stringBuilder.append(str).append("\n");
} else{ }
log.warn("{}:{}", url, statusCode); } else {
// Never log gallery tokens, cookies or an arbitrary upstream error body.
throw new IOException("上游 HTTP " + statusCode + " (" + method.toUpperCase(Locale.ROOT)
+ " " + URI.create(url).getHost() + ")");
}
return stringBuilder.toString();
} }
httpResponse.close();
return stringBuilder.toString();
} }
/**
* 从链接里安全提取 gid。
*
* <p>历史实现只捕 {@code IndexOutOfBoundsException}:非数字段会抛
* {@code NumberFormatException},{@code link == null} 会抛 NPE,两者都会在
* 「无 @ControllerAdvice」的项目里穿透成 500。这里与
* {@code GalleryManageService.parseGidFromLink} 保持一致的宽松语义:
* 任何畸形输入都返回 null,由调用方转成业务失败。
*/
public static Integer parseGid(String link){ public static Integer parseGid(String link){
if (link == null)
return null;
String[] parts = link.split("/g/");
if (parts.length < 2)
return null;
String[] segments = parts[1].split("/");
if (segments.length == 0)
return null;
try { try {
return Integer.parseInt(link.split("/g/")[1].split("/")[0]); return Integer.parseInt(segments[0]);
}catch (IndexOutOfBoundsException e){ } catch (NumberFormatException e) {
return null; return null;
} }
} }
@@ -0,0 +1,67 @@
package com.lion.lionwebsite.Util;
import java.io.*;
import java.net.URI;
import java.net.URLConnection;
import java.nio.file.*;
import java.util.Locale;
import java.util.concurrent.Callable;
/** Downloads into private temporary files and publishes only completed images. */
public final class ImageFileCache {
private static final SingleFlight<Path, Path> downloads = new SingleFlight<>();
private ImageFileCache() { }
public static Path find(Path directory, String name) {
for (String suffix : new String[]{".avif", ".gif"}) {
Path path = directory.resolve(name + suffix);
if (Files.isRegularFile(path) && path.toFile().length() > 0)
return path;
}
return null;
}
public static Path get(Path directory, String name, Callable<String> sourceUrl) throws Exception {
Path key = directory.resolve(name).toAbsolutePath().normalize();
return downloads.run(key, () -> {
Path cached = find(directory, name);
if (cached != null) return cached;
Files.createDirectories(directory);
String url = sourceUrl.call();
if (url == null) throw new IOException("图片地址不存在");
URI source = new URI(url);
boolean gif = source.getPath().toLowerCase(Locale.ROOT).endsWith(".gif");
String suffix = gif ? ".gif" : ".img";
Path temporary = Files.createTempFile(directory, ".download-", suffix);
Path converted = null;
try {
URLConnection connection = source.toURL().openConnection();
connection.setConnectTimeout(5_000);
connection.setReadTimeout(15_000);
try (InputStream input = connection.getInputStream();
OutputStream output = Files.newOutputStream(temporary)) {
input.transferTo(output);
}
if (Files.size(temporary) == 0) throw new IOException("图片内容为空");
if (gif) {
converted = temporary;
} else {
String result = GalleryUtil.convertImg(temporary.toString(), suffix);
if (result == null) throw new IOException("图片转换失败");
converted = Path.of(result);
}
Path target = directory.resolve(name + (gif ? ".gif" : ".avif"));
try {
Files.move(converted, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(converted, target, StandardCopyOption.REPLACE_EXISTING);
}
return target;
} finally {
Files.deleteIfExists(temporary);
if (converted != null && !converted.equals(temporary)) Files.deleteIfExists(converted);
}
});
}
}
@@ -0,0 +1,34 @@
package com.lion.lionwebsite.Util;
import java.io.IOException;
import java.util.regex.Pattern;
/** Extracts data assignments without depending on script position or line layout. */
public final class JavaScriptAssignments {
private JavaScriptAssignments() { }
public static String string(String source, String name) throws IOException {
var matcher = Pattern.compile("\\b" + Pattern.quote(name) + "\\s*=\\s*(['\"])(.*?)\\1", Pattern.DOTALL).matcher(source);
if (!matcher.find()) throw new IOException("上游脚本缺少 " + name);
return matcher.group(2);
}
public static String array(String source, String name) throws IOException {
var matcher = Pattern.compile("\\b" + Pattern.quote(name) + "\\s*=\\s*\\[").matcher(source);
if (!matcher.find()) throw new IOException("上游脚本缺少 " + name);
int start = matcher.end() - 1, depth = 0; char quote = 0; boolean escaped = false;
for (int i = start; i < source.length(); i++) {
char c = source.charAt(i);
if (quote != 0) {
if (escaped) escaped = false;
else if (c == '\\') escaped = true;
else if (c == quote) quote = 0;
continue;
}
if (c == '"' || c == '\'') quote = c;
else if (c == '[') depth++;
else if (c == ']' && --depth == 0) return source.substring(start, i + 1);
}
throw new IOException("上游脚本的 " + name + " 不完整");
}
}
@@ -0,0 +1,107 @@
package com.lion.lionwebsite.Util;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Duration;
import java.time.Instant;
import java.util.HexFormat;
/**
* 机器人一键登录票据:与 PersonalHub 共享密钥的 HMAC 自签串。
*
* <p>形状为 {@code v1.<签发秒级时间戳>.<HMAC-SHA256(密钥, "v1.<时间戳>") 的十六进制>}。
* 机器人本地用同一把密钥签出,LionWebsite 侧只做验签与新鲜度判断,因此不需要
* 机器人到主站的网络调用,两端任一侧临时不可达都不影响链接生成。
*
* <p>设计取舍:票据不携带一次性状态,时间窗内可重放(用户明确接受这一点)。
* 窗口外的票据一律拒绝,所以泄露的链接在 {@code ticket-ttl-seconds} 之后自动失效。
*
* <p>票据只签发给主人:密钥只配置在主人自己的机器人上,不随用户分发。
*/
@Component
@Slf4j
public class PanelLoginTicket {
private static final String VERSION = "v1";
private static final String ALGORITHM = "HmacSHA256";
private final byte[] secret;
private final Duration ttl;
public PanelLoginTicket(
@Value("${personal.login.secret:}") String secret,
@Value("${personal.login.ticket-ttl-seconds:300}") long ttlSeconds) {
this.secret = secret == null ? new byte[0] : secret.getBytes(StandardCharsets.UTF_8);
this.ttl = Duration.ofSeconds(Math.max(30, ttlSeconds));
if (this.secret.length == 0)
log.warn("未配置 personal.login.secret,一键登录票据将一律拒绝");
}
/** 密钥未配置时无法签发也无法校验,调用方据此给出可读提示。 */
public boolean configured() {
return secret.length > 0;
}
public String issue() {
return issue(Instant.now());
}
/** 供测试注入固定时钟。 */
public String issue(Instant now) {
if (!configured())
throw new IllegalStateException("未配置 personal.login.secret");
String stamp = String.valueOf(now.getEpochSecond());
return VERSION + "." + stamp + "." + sign(stamp);
}
public boolean verify(String ticket) {
return verify(ticket, Instant.now());
}
public boolean verify(String ticket, Instant now) {
if (!configured() || ticket == null)
return false;
String[] parts = ticket.split("\\.", -1);
if (parts.length != 3 || !VERSION.equals(parts[0]) || parts[1].isEmpty() || parts[2].isEmpty())
return false;
if (!MessageDigest.isEqual(sign(parts[1]).getBytes(StandardCharsets.UTF_8),
parts[2].getBytes(StandardCharsets.UTF_8)))
return false;
long issuedAt;
try {
issuedAt = Long.parseLong(parts[1]);
} catch (NumberFormatException e) {
return false;
}
// 只接受 [now - ttl, now]:未来时间戳一律拒绝,避免伪造者用远期时间换取长期有效。
long ageSeconds = now.getEpochSecond() - issuedAt;
return ageSeconds >= 0 && ageSeconds <= ttl.getSeconds();
}
public long ttlSeconds() {
return ttl.getSeconds();
}
private String sign(String stamp) {
try {
Mac mac = Mac.getInstance(ALGORITHM);
mac.init(new SecretKeySpec(secret, ALGORITHM));
return HexFormat.of().formatHex(
mac.doFinal((VERSION + "." + stamp).getBytes(StandardCharsets.UTF_8)));
} catch (Exception e) {
// 算法名固定且密钥非空,正常不会走到这里;一旦发生必须视为校验失败。
log.error("计算登录票据签名失败", e);
return "";
}
}
}
@@ -0,0 +1,6 @@
package com.lion.lionwebsite.Util;
import com.lion.lionwebsite.Domain.Gallery;
/** A read-only upstream preparation; creating this value never submits a download. */
public record PreparedGallery(Gallery gallery, String formUrl, String origin) { }
@@ -1,7 +1,7 @@
package com.lion.lionwebsite.Util; package com.lion.lionwebsite.Util;
import com.fasterxml.jackson.databind.JsonNode; import tools.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ObjectNode; import tools.jackson.databind.node.ObjectNode;
public class Response { public class Response {
@@ -19,7 +19,8 @@ public class Response {
} }
public String get(String key){ public String get(String key){
return result.get(key).asText(); JsonNode node = result.get(key);
return node == null ? null : node.asText();
} }
public void setData(String data){ public void setData(String data){
@@ -57,7 +58,8 @@ public class Response {
} }
public String getData(){ public String getData(){
return result.get("data").asText(); JsonNode node = result.get("data");
return node == null ? null : node.asText();
} }
/** /**
@@ -69,8 +71,10 @@ public class Response {
return getData(); return getData();
} }
/** 未设置 result 键时视为失败,而不是抛 NPE。 */
public boolean isSuccess(){ public boolean isSuccess(){
return result.get("result").asText().equals("success"); JsonNode node = result.get("result");
return node != null && "success".equals(node.asText());
} }
@@ -0,0 +1,30 @@
package com.lion.lionwebsite.Util;
import java.util.concurrent.*;
/** Concurrent callers for the same key share one in-flight operation, including its failure. */
public final class SingleFlight<K, V> {
private final ConcurrentHashMap<K, CompletableFuture<V>> running = new ConcurrentHashMap<>();
public V run(K key, Callable<V> operation) throws Exception {
CompletableFuture<V> mine = new CompletableFuture<>();
CompletableFuture<V> existing = running.putIfAbsent(key, mine);
if (existing != null) {
try { return existing.get(); }
catch (ExecutionException e) {
if (e.getCause() instanceof Exception cause) throw cause;
throw new IllegalStateException(e.getCause());
}
}
try {
V result = operation.call();
mine.complete(result);
return result;
} catch (Exception | Error e) {
mine.completeExceptionally(e);
throw e;
} finally {
running.remove(key, mine);
}
}
}
@@ -0,0 +1,328 @@
package com.lion.lionwebsite.Util;
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.AtomicMoveNotSupportedException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.security.MessageDigest;
import java.util.Comparator;
import java.util.HexFormat;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Stream;
/**
* 订阅缓存的分代存储:v2ray 与 clash 两份内容必须同时生效。
*
* <p>旧实现把两份文件各自原子替换,第二份失败时用户会拿到「新 v2ray + 旧 clash」的
* 混合版本。这里把两份文件写进同一个不可变分代目录,校验通过后用一次原子替换发布
* 指针(manifest),因此抓取、解析、写入或发布任何一步失败,都只会保留上一个完整分代。
*
* <p>目录结构:
* <pre>
* &lt;cacheRoot&gt;/&lt;accountId&gt;/current 指针(原子替换,最后发布)
* &lt;cacheRoot&gt;/&lt;accountId&gt;/generations/&lt;generation&gt;/v2ray.txt
* &lt;cacheRoot&gt;/&lt;accountId&gt;/generations/&lt;generation&gt;/clash.yaml
* &lt;cacheRoot&gt;/&lt;accountId&gt;/v2ray.txt + clash.yaml 旧结构,两份都在才按 legacy 读
* </pre>
*
* <p>读取始终从指针指向的同一个分代里取两份文件,所以拿到的一定是同版本内容。
*/
@Slf4j
public class SubscriptionCacheStore {
private static final String MANIFEST_FILE = "current";
private static final String GENERATION_DIR = "generations";
private static final String V2_FILE = "v2ray.txt";
private static final String CLASH_FILE = "clash.yaml";
/** 保留目录名,结构上永远不存在,用于表达「当前没有可分发版本」。 */
private static final String UNAVAILABLE_DIR = ".unavailable";
private static final String MANIFEST_GENERATION = "generation";
private static final String MANIFEST_V2_SHA = "v2-sha256";
private static final String MANIFEST_CLASH_SHA = "clash-sha256";
/** 旧结构读出来的版本的 generation 标记;真实分代名是定宽时间戳,不会与之碰撞。 */
private static final String LEGACY_GENERATION = "legacy";
/** 保留分代数:当前与上一个,足够回退且不会无限增长。 */
private static final int RETAINED_GENERATIONS = 2;
private static final AtomicLong SEQUENCE = new AtomicLong();
private final Path root;
public SubscriptionCacheStore(String cacheRoot) {
this(Paths.get(cacheRoot));
}
public SubscriptionCacheStore(Path root) {
this.root = root;
}
/** 同一个分代目录里读出的一对内容;两份文件必然同版本。 */
public record Pair(String generation, byte[] v2, byte[] clash) { }
public Path accountDir(int accountId) {
return root.resolve(String.valueOf(accountId));
}
/** 旧结构的账号根文件路径。仅供兼容测试定位 legacy 文件,不作为回退目标。 */
public Path legacyPath(int accountId, String client) {
return accountDir(accountId).resolve(fileName(client));
}
/**
* 没有完整可分发版本时返回的哨兵路径。
*
* <p>它保证不存在,因此调用方用 {@code Files.isRegularFile} 判定就会走「订阅暂不可用」
* 分支,而不会被半份 legacy 文件骗过。
*/
public Path unavailablePath(int accountId, String client) {
return accountDir(accountId).resolve(UNAVAILABLE_DIR).resolve(fileName(client));
}
private static String fileName(String client) {
return "v2".equals(client) ? V2_FILE : CLASH_FILE;
}
/**
* 写入一份新分代并在最后发布指针。
*
* <p>失败时抛出异常,且不改变指针与上一个分代,调用方据此保留上次的完整版本。
*/
public Pair publish(int accountId, byte[] v2, byte[] clash) throws IOException {
Path account = accountDir(accountId);
Files.createDirectories(account);
String generation = nextGeneration();
Path dir = account.resolve(GENERATION_DIR).resolve(generation);
boolean published = false;
try {
Files.createDirectories(dir);
writeFile(dir.resolve(V2_FILE), v2);
writeFile(dir.resolve(CLASH_FILE), clash);
// 发布前校验:两份文件都必须能按写入内容读回,避免把半截内容挂上指针。
verify(dir.resolve(V2_FILE), v2);
verify(dir.resolve(CLASH_FILE), clash);
publishManifest(account, generation, sha256(v2), sha256(clash));
published = true;
} finally {
if (!published)
deleteRecursively(dir);
}
prune(account, generation);
removeLegacy(account);
return new Pair(generation, v2, clash);
}
/** 读取当前发布内容;只有构成一整个有效版本时才返回。 */
public Optional<Pair> read(int accountId) {
return Optional.ofNullable(load(accountDir(accountId)));
}
/**
* 解析当前应分发的单个文件。
*
* <p>与 {@link #read}、{@link #hasComplete} 使用同一份配对契约:只有两份文件
* 齐全且哈希校验通过时才返回路径,因此分发路径不会发出半份或已损坏的缓存。
*/
public Optional<Path> resolve(int accountId, String client) {
Path account = accountDir(accountId);
Pair pair = load(account);
if (pair == null)
return Optional.empty();
Path path = LEGACY_GENERATION.equals(pair.generation())
? account.resolve(fileName(client))
: account.resolve(GENERATION_DIR).resolve(pair.generation()).resolve(fileName(client));
return Files.isRegularFile(path) ? Optional.of(path) : Optional.empty();
}
/** 是否已有可分发的一对内容(分代两份都在且哈希匹配,或旧结构两份都在)。 */
public boolean hasComplete(int accountId) {
return load(accountDir(accountId)) != null;
}
/** 删除该账号的全部缓存(指针、分代、旧结构文件),使后续请求判定为无缓存。 */
public void invalidate(int accountId) {
Path account = accountDir(accountId);
deleteIfExists(account.resolve(MANIFEST_FILE));
deleteRecursively(account.resolve(GENERATION_DIR));
deleteIfExists(account.resolve(V2_FILE));
deleteIfExists(account.resolve(CLASH_FILE));
}
/** 写入接缝:测试覆写以模拟第二份文件写入失败。 */
protected void writeFile(Path target, byte[] data) throws IOException {
atomicWrite(target, data);
}
/** 指针发布接缝:测试覆写以模拟 manifest 发布失败。 */
protected void publishManifest(Path accountDir, String generation, String v2Sha256, String clashSha256)
throws IOException {
atomicWrite(accountDir.resolve(MANIFEST_FILE),
manifestText(generation, v2Sha256, clashSha256).getBytes(StandardCharsets.UTF_8));
}
/**
* 读取「一整个版本」:一次读入指针快照,再按该快照定位分代并校验双文件哈希。
*
* <p>指针只读一次,因此并发 {@link #publish} 切换指针时不会出现「用新分代名配旧哈希」
* 或哈希字段缺失导致的空指针;要么拿到完整的新版本,要么拿到完整的旧版本。
*/
private Pair load(Path account) {
Properties manifest = readManifest(account);
if (manifest == null) {
// 指针文件不存在才是「旧结构」。指针存在但读不出来说明它已损坏,同样按无缓存处理。
return Files.isRegularFile(account.resolve(MANIFEST_FILE)) ? null : loadLegacy(account);
}
Pair generation = loadGeneration(account, manifest);
if (generation != null)
return generation;
// 指针存在却读不出完整版本(半写、被篡改、内容缺失)时不得回退到 legacy:
// 那会发出一个比指针更旧的版本,掩盖损坏。此时按无可用缓存处理。
return null;
}
private Pair loadGeneration(Path account, Properties manifest) {
String generation = manifest.getProperty(MANIFEST_GENERATION);
if (generation == null || generation.isBlank() || generation.contains("/") || generation.contains("..")) {
log.warn("订阅缓存指针内容非法");
return null;
}
String v2Sha256 = manifest.getProperty(MANIFEST_V2_SHA);
String clashSha256 = manifest.getProperty(MANIFEST_CLASH_SHA);
if (v2Sha256 == null || clashSha256 == null) {
log.warn("订阅缓存指针缺少哈希字段 generation={}", generation);
return null;
}
Path dir = account.resolve(GENERATION_DIR).resolve(generation);
byte[] v2 = readFile(dir.resolve(V2_FILE));
byte[] clash = readFile(dir.resolve(CLASH_FILE));
if (v2 == null || clash == null) {
log.warn("订阅缓存分代内容缺失 generation={}", generation);
return null;
}
if (!matches(v2, v2Sha256) || !matches(clash, clashSha256)) {
log.warn("订阅缓存分代内容校验失败 generation={}", generation);
return null;
}
return new Pair(generation, v2, clash);
}
/** 旧结构:只有账号根下两份文件都在才构成一个有效版本。 */
private Pair loadLegacy(Path account) {
byte[] v2 = readFile(account.resolve(V2_FILE));
byte[] clash = readFile(account.resolve(CLASH_FILE));
if (v2 == null || clash == null)
return null;
return new Pair(LEGACY_GENERATION, v2, clash);
}
/** 指针读取接缝:包内可见,便于测试统计「每次查询只读一次指针」。 */
Properties readManifest(Path account) {
Path manifest = account.resolve(MANIFEST_FILE);
if (!Files.isRegularFile(manifest))
return null;
try {
Properties properties = new Properties();
properties.load(new StringReader(Files.readString(manifest, StandardCharsets.UTF_8)));
return properties;
} catch (IOException e) {
log.warn("读取订阅缓存指针失败: {}", e.getMessage());
return null;
}
}
private static byte[] readFile(Path path) {
try {
return Files.isRegularFile(path) ? Files.readAllBytes(path) : null;
} catch (IOException e) {
log.warn("读取订阅缓存文件失败 path={} error={}", path, e.getMessage());
return null;
}
}
private static void verify(Path path, byte[] expected) throws IOException {
byte[] written = Files.readAllBytes(path);
if (written.length != expected.length || !MessageDigest.isEqual(written, expected))
throw new IOException("订阅缓存写入校验失败 " + path.getFileName());
}
private static boolean matches(byte[] content, String expectedSha256) {
return expectedSha256 != null && expectedSha256.equals(sha256(content));
}
private static String manifestText(String generation, String v2Sha256, String clashSha256) {
return MANIFEST_GENERATION + "=" + generation + "\n"
+ MANIFEST_V2_SHA + "=" + v2Sha256 + "\n"
+ MANIFEST_CLASH_SHA + "=" + clashSha256 + "\n";
}
/** 分代名定宽可排序:时间戳在前保证单调,序号消除同毫秒碰撞。 */
private static String nextGeneration() {
return String.format("%013d-%06d", System.currentTimeMillis(), SEQUENCE.incrementAndGet() % 1_000_000);
}
private void prune(Path account, String current) {
Path dir = account.resolve(GENERATION_DIR);
List<Path> generations;
try (Stream<Path> entries = Files.list(dir)) {
generations = entries.filter(Files::isDirectory)
.sorted(Comparator.comparing(path -> path.getFileName().toString(), Comparator.reverseOrder()))
.toList();
} catch (IOException e) {
log.warn("清理订阅缓存分代失败 path={} error={}", dir, e.getMessage());
return;
}
for (int index = RETAINED_GENERATIONS; index < generations.size(); index++) {
Path stale = generations.get(index);
if (!stale.getFileName().toString().equals(current))
deleteRecursively(stale);
}
}
/** 分代发布成功后旧结构文件被取代,避免两份来源同时存在。 */
private void removeLegacy(Path account) {
deleteIfExists(account.resolve(V2_FILE));
deleteIfExists(account.resolve(CLASH_FILE));
}
private static void atomicWrite(Path target, byte[] data) throws IOException {
Path temp = target.resolveSibling(target.getFileName() + ".tmp");
Files.write(temp, data);
try {
Files.move(temp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(temp, target, StandardCopyOption.REPLACE_EXISTING);
}
}
private static void deleteIfExists(Path path) {
try {
Files.deleteIfExists(path);
} catch (IOException e) {
log.warn("删除订阅缓存文件失败 path={} error={}", path, e.getMessage());
}
}
private static void deleteRecursively(Path path) {
if (!Files.exists(path))
return;
try (Stream<Path> entries = Files.walk(path)) {
entries.sorted(Comparator.reverseOrder()).forEach(SubscriptionCacheStore::deleteIfExists);
} catch (IOException e) {
log.warn("删除订阅缓存目录失败 path={} error={}", path, e.getMessage());
}
}
private static String sha256(byte[] input) {
try {
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(input));
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
}
@@ -0,0 +1,65 @@
package com.lion.lionwebsite.Util;
import java.util.List;
/**
* 抓取上游订阅时伪装的客户端身份。
*
* <p>此前请求只带 HttpClient 的默认头,实测上游收到的是
* {@code User-agent: Apache-HttpClient/5.6.4 (Java/25.0.4)},
* 对订阅站点而言等同于「这是一个 Java 程序在批量拉取」。
* 这里按订阅格式给出与真实客户端一致的常见头组合。
*
* <p>注意:只能伪装到 HTTP 头这一层。真实 Clash/Mihomo 使用 Go 的 TLS 栈与
* HTTP/2,TLS 指纹与 Java HttpClient 仍有差异;若上游做主动指纹识别,
* 需要另外引入真实客户端进程,本枚举解决不了。
*/
public enum SubscriptionClientProfile {
MIHOMO("mihomo/v1.19.2", "application/yaml, */*", "zh-CN,zh;q=0.9"),
CLASH_VERGE("clash-verge/v2.0.3", "application/yaml, */*", "zh-CN,zh;q=0.9"),
CLASH_WINDOWS("ClashforWindows/0.20.39", "*/*", "zh-CN,zh;q=0.9"),
V2RAYN("v2rayN/6.45", "*/*", "zh-CN,zh;q=0.9"),
V2RAYNG("v2rayNG/1.9.16", "*/*", "zh-CN,zh;q=0.9"),
SHADOWROCKET("Shadowrocket/2.2.39", "*/*", "zh-CN,zh;q=0.9");
static final List<SubscriptionClientProfile> CLASH_PROFILES = List.of(MIHOMO, CLASH_VERGE, CLASH_WINDOWS);
static final List<SubscriptionClientProfile> V2_PROFILES = List.of(V2RAYN, V2RAYNG, SHADOWROCKET);
private final String userAgent;
private final String accept;
private final String acceptLanguage;
SubscriptionClientProfile(String userAgent, String accept, String acceptLanguage) {
this.userAgent = userAgent;
this.accept = accept;
this.acceptLanguage = acceptLanguage;
}
public String userAgent() {
return userAgent;
}
public String accept() {
return accept;
}
public String acceptLanguage() {
return acceptLanguage;
}
/**
* 按账号与订阅格式确定性地选择一个身份。
*
* <p>刻意不使用随机数:同一个账号每次都用同一个客户端,才符合真实用户的样子;
* 逐请求更换 User-Agent 本身就是更明显的机器特征。不同账号使用不同客户端,
* 又便于在必要时按账号区分上游流量。
*
* @param accountId 子账号 ID,可为 null(视为 0)
* @param clash true 表示 Clash 格式,false 表示 V2Ray 格式
*/
public static SubscriptionClientProfile forAccount(Integer accountId, boolean clash) {
List<SubscriptionClientProfile> candidates = clash ? CLASH_PROFILES : V2_PROFILES;
int id = accountId == null ? 0 : accountId;
return candidates.get(Math.floorMod(id, candidates.size()));
}
}
+39 -1
View File
@@ -1,5 +1,15 @@
server: server:
port: 8888 port: 8888
servlet:
session:
# nginx 的 location / 会把浏览器请求改写成后端 /personal/…,而浏览器地址栏仍是
# /user 这类路径。若沿用容器按请求路径推导的 Cookie Path(/personal),浏览器
# 判定 /user 不匹配就不会带上会话,面板会一直 401。这里固定为 /。
cookie:
path: /
http-only: true
same-site: lax
timeout: 14d
tomcat: tomcat:
max-swallow-size: 10000MB max-swallow-size: 10000MB
http2: http2:
@@ -12,6 +22,12 @@ spring:
datasource-cache: datasource-cache:
driver-class-name: org.sqlite.JDBC driver-class-name: org.sqlite.JDBC
jdbc-url: jdbc:sqlite:cache.db jdbc-url: jdbc:sqlite:cache.db
# 定时任务默认只有 1 个线程。订阅刷新一轮最多 2 个账号、每个 2 次请求(各 15 秒超时),
# 最坏可占用约 1 分钟;单线程会让连接自检、cookie 检测等任务在此期间全部排队。
task:
scheduling:
pool:
size: 2
mvc: mvc:
view: view:
prefix: /resources/templates/ prefix: /resources/templates/
@@ -38,6 +54,19 @@ local:
subscription: subscription:
cache-root: sub/accounts cache-root: sub/accounts
# 分散刷新:每个账号在 window-hours 内至少刷新一次,两两之间至少相隔 min-gap-minutes;
# max-per-tick 限制单个 tick 刷新几个账号,避免重启后一批到期账号同时开火。
refresh:
window-hours: 24
min-gap-minutes: 60
# 5 分钟一个轻量 tick,只处理已到期账号。
tick-interval-ms: 300000
# 重启后延迟 2 分钟再开始,先让应用完成启动。
initial-delay-ms: 120000
max-per-tick: 2
# 失败后的重试间隔,必须小于 window-hours。
retry-delay-minutes: 60
stale-alert-multiplier: 2.0
upstream: upstream:
# Use environment variables or an external config file in production. # Use environment variables or an external config file in production.
v2-url-template: "https://aaaa.gay/link/{key}?client=v2" v2-url-template: "https://aaaa.gay/link/{key}?client=v2"
@@ -47,7 +76,16 @@ subscription:
standby: standby:
sync-enabled: "${SUBSCRIPTION_STANDBY_SYNC_ENABLED:false}" sync-enabled: "${SUBSCRIPTION_STANDBY_SYNC_ENABLED:false}"
sync-secret: "${SUBSCRIPTION_SYNC_SECRET:}" sync-secret: "${SUBSCRIPTION_SYNC_SECRET:}"
retry-interval-ms: 60000 # 快照只在订阅内容变化时推送(另在节点上线时补推一次),没有周期性重发。
# 备机的过期语义是「主站失联」,由节点按「收到主站的任何消息」判定新鲜度,
# 主站每 30 分钟的可用性检查即可持续刷新,无需为续期周期性传输快照。
bot: bot:
token: "5222939329:AAHa6l9ZuVVdNSDLPI_H-c8O_VgeOEw5plA" token: "5222939329:AAHa6l9ZuVVdNSDLPI_H-c8O_VgeOEw5plA"
# 一键登录:机器人用同一密钥自签 HMAC 票据,主站只做验签与时间窗校验。
# secret 必须通过环境变量注入,不得写入仓库;为空时所有票据一律拒绝。
personal:
login:
secret: "${PERSONAL_LOGIN_SECRET:}"
ticket-ttl-seconds: 300
@@ -0,0 +1,169 @@
package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Interceptor.HumanInterceptor;
import com.lion.lionwebsite.Interceptor.PersonalInterceptor;
import com.lion.lionwebsite.Interceptor.TaskHandlerInterceptor;
import com.lion.lionwebsite.Service.WebSocketService;
import org.junit.jupiter.api.Test;
import org.springframework.web.servlet.HandlerInterceptor;
import org.springframework.web.servlet.config.annotation.InterceptorRegistration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
/**
* 拦截器与 WebSocket 的注册关系。
* 这是整个应用的「安全地图」:若某条路径漏挂鉴权拦截器,接口就会在无授权码
* 的情况下对外可用(9/8 那批改动正是栽在 /GalleryManage/** 的漏挂上),
* 所以这里把路径与拦截器的对应关系显式锁进测试。
*/
class ConfigurationWiringTest {
/** 记录 addInterceptors 调用顺序与实际挂载路径的假注册表。 */
private static final class RecordingRegistry extends InterceptorRegistry {
final List<String> order = new ArrayList<>();
final Map<String, List<String>> paths = new LinkedHashMap<>();
@Override
public InterceptorRegistration addInterceptor(HandlerInterceptor interceptor) {
String name = interceptor.getClass().getSimpleName();
order.add(name);
return new RecordingRegistration(name, paths);
}
}
/** 只记录路径,其余注册动作不做真实处理。 */
private static final class RecordingRegistration extends InterceptorRegistration {
private final String name;
private final Map<String, List<String>> paths;
RecordingRegistration(String name, Map<String, List<String>> paths) {
super(mock(HandlerInterceptor.class));
this.name = name;
this.paths = paths;
}
@Override
public InterceptorRegistration addPathPatterns(String... patterns) {
paths.computeIfAbsent(name, k -> new ArrayList<>()).addAll(List.of(patterns));
return this;
}
}
private static RecordingRegistry registryOf(InterceptorConfiguration config) {
RecordingRegistry registry = new RecordingRegistry();
config.addInterceptors(registry);
return registry;
}
/** TaskHandlerInterceptor 必须覆盖 /GalleryManage 全子路径与 /validate。 */
@Test
void taskHandlerGuardsGalleryManageAndValidate() {
var config = new InterceptorConfiguration(mock(TaskHandlerInterceptor.class));
var registry = registryOf(config);
List<String> guarded = registry.paths.get("TaskHandlerInterceptor");
assertNotNull(guarded, "TaskHandlerInterceptor 必须被注册");
assertTrue(guarded.contains("/GalleryManage"), "精确路径必须挂着");
assertTrue(guarded.contains("/GalleryManage/**"), "子路径必须挂着(历史漏挂点)");
assertTrue(guarded.contains("/validate"));
}
/** PersonalInterceptor 必须覆盖 /personal/** 与 /remote/**。 */
@Test
void personalInterceptorGuardsPrivateAreas() {
var config = new InterceptorConfiguration(mock(TaskHandlerInterceptor.class));
var registry = registryOf(config);
List<String> guarded = registry.paths.get("PersonalInterceptor");
assertNotNull(guarded, "PersonalInterceptor 必须被注册");
assertTrue(guarded.contains("/personal/**"));
assertTrue(guarded.contains("/remote/**"));
}
/** HumanInterceptor 只管首页与移动端入口。 */
@Test
void humanInterceptorGuardsEntryPointsOnly() {
var config = new InterceptorConfiguration(mock(TaskHandlerInterceptor.class));
var registry = registryOf(config);
assertEquals(List.of("/", "/mobile"), registry.paths.get("HumanInterceptor"));
}
/** @Bean 暴露的拦截器类型必须与注册时一致(写错类型会让鉴权静默失效)。 */
@Test
void exposedInterceptorBeansHaveExpectedTypes() {
var config = new InterceptorConfiguration(mock(TaskHandlerInterceptor.class));
assertInstanceOf(PersonalInterceptor.class, config.getPersonalInterceptor());
assertInstanceOf(HumanInterceptor.class, config.getHumanInterceptor());
}
/** WebSocket 处理器必须挂在 /ws/,并放开跨域(前端部署在不同源)。 */
@Test
void websocketHandlerIsRegisteredAtWsPath() {
var service = new WebSocketService();
var config = new WebsocketConfiguration(service);
var registration = mock(org.springframework.web.socket.config.annotation.WebSocketHandlerRegistration.class,
RETURNS_SELF);
var registry = mock(org.springframework.web.socket.config.annotation.WebSocketHandlerRegistry.class);
when(registry.addHandler(any(), anyString())).thenReturn(registration);
config.registerWebSocketHandlers(registry);
verify(registry).addHandler(eq(service), eq("/ws/"));
verify(registration).setAllowedOriginPatterns("*");
}
// ---------- CORS ----------
/** 暴露 CorsRegistry 的 protected 配置表,便于断言实际生效的规则。 */
private static final class InspectableCorsRegistry
extends org.springframework.web.servlet.config.annotation.CorsRegistry {
@Override
public java.util.Map<String, org.springframework.web.cors.CorsConfiguration> getCorsConfigurations() {
return super.getCorsConfigurations();
}
}
/** CORS 配置必须允许凭据,且不得使用通配来源(两者不能同时成立)。 */
@Test
void corsAllowsCredentialsWithOriginPatterns() {
var registry = new InspectableCorsRegistry();
new CorsConfig().addCorsMappings(registry);
var configs = registry.getCorsConfigurations();
assertEquals(1, configs.size(), "应只注册一条 /** 的映射");
assertTrue(configs.containsKey("/**"));
var mapping = configs.get("/**");
assertTrue(mapping.getAllowCredentials(), "前端带 Cookie 时需要允许凭据");
assertTrue(mapping.getAllowedMethods().containsAll(List.of("GET", "POST", "PUT", "DELETE")),
"四种方法都应放开");
assertEquals(List.of("*"), mapping.getAllowedHeaders());
}
/**
* allowCredentials + allowedOrigins("*") 是非法组合(Spring 会抛异常),
* 因此必须走 allowedOriginPatterns。这里锁定该实现方式不被改回去。
*/
@Test
void corsUsesOriginPatternsRatherThanWildcardOrigins() {
var registry = new InspectableCorsRegistry();
new CorsConfig().addCorsMappings(registry);
var mapping = registry.getCorsConfigurations().get("/**");
assertTrue(mapping.getAllowedOriginPatterns().contains("*"),
"来源应通过 allowedOriginPatterns 放开");
assertNull(mapping.getAllowedOrigins(),
"不应设置 allowedOrigins,否则与 allowCredentials 冲突");
}
}
@@ -0,0 +1,318 @@
package com.lion.lionwebsite.Configuration;
import com.lion.lionwebsite.Dao.normal.GalleryMapper;
import com.lion.lionwebsite.Dao.normal.UserMapper;
import org.junit.jupiter.api.Test;
import org.mybatis.spring.mapper.MapperFactoryBean;
import org.mybatis.spring.mapper.MapperScannerConfigurer;
import org.springframework.aot.hint.MemberCategory;
import org.springframework.aot.hint.RuntimeHints;
import org.springframework.aot.hint.predicate.RuntimeHintsPredicates;
import org.springframework.beans.factory.support.RegisteredBean;
import org.springframework.beans.factory.support.RootBeanDefinition;
import java.lang.reflect.Method;
import java.util.Collection;
import static org.junit.jupiter.api.Assertions.*;
/**
* GraalVM 原生镜像的 MyBatis 运行时提示(AOT hints)。
*
* 这些代码在 JVM 模式下几乎不执行,但一旦原生构建,缺少 hint 就会在运行期抛
* ClassNotFoundException / 反射失败——而这类问题只在原生产物上暴露,回归成本极高。
* 因此这里直接调用这些处理器,断言「该注册的反射与代理都注册了」。
*/
class MyBatisNativeConfigurationTest {
/** 运行期提示注册器应把 MyBatis 关键类全部登记为可反射,并注册 dtd/xsd 资源。 */
@Test
void runtimeHintsRegistrarRegistersMyBatisReflectionAndResources() {
RuntimeHints hints = new RuntimeHints();
new MyBatisNativeConfiguration.MyBaitsRuntimeHintsRegistrar()
.registerHints(hints, getClass().getClassLoader());
// 语言驱动与日志实现等都会被 MyBatis 反射实例化
assertTrue(RuntimeHintsPredicates.reflection()
.onType(org.apache.ibatis.scripting.xmltags.XMLLanguageDriver.class).test(hints),
"XMLLanguageDriver 应可反射");
assertTrue(RuntimeHintsPredicates.reflection()
.onType(org.apache.ibatis.logging.slf4j.Slf4jImpl.class).test(hints),
"Slf4jImpl 应可反射");
assertTrue(RuntimeHintsPredicates.reflection()
.onType(org.apache.ibatis.session.SqlSessionFactory.class).test(hints));
assertTrue(RuntimeHintsPredicates.reflection()
.onType(java.util.ArrayList.class).test(hints),
"集合类型也应登记(MyBatis 需要实例化)");
// XML 映射文件的 DTD/XSD 解析依赖这些资源
assertTrue(RuntimeHintsPredicates.resource()
.forResource("org/apache/ibatis/builder/xml/mybatis-3-mapper.dtd").test(hints));
assertTrue(RuntimeHintsPredicates.resource()
.forResource("org/apache/ibatis/builder/xml/mybatis-3-config.xsd").test(hints));
}
/** 反射条目应包含全部成员类别,而不只是构造器。 */
@Test
void reflectionHintsIncludeAllMemberCategories() {
RuntimeHints hints = new RuntimeHints();
new MyBatisNativeConfiguration.MyBaitsRuntimeHintsRegistrar()
.registerHints(hints, getClass().getClassLoader());
assertTrue(RuntimeHintsPredicates.reflection()
.onType(org.apache.ibatis.session.SqlSessionFactory.class)
.withMemberCategory(MemberCategory.INVOKE_DECLARED_METHODS).test(hints),
"应登记方法调用权限");
assertTrue(RuntimeHintsPredicates.reflection()
.onType(org.apache.ibatis.session.SqlSessionFactory.class)
.withMemberCategory(MemberCategory.ACCESS_DECLARED_FIELDS).test(hints),
"应登记字段访问权限");
}
// ---------- MyBatisMapperTypeUtils ----------
/** 返回类型解析:泛型 T 应解析成 mapper 接口声明的具体类型。 */
@Test
void resolveReturnClassUnwrapsGenerics() throws Exception {
Method genericList = Holder.class.getMethod("genericList");
assertEquals(String.class,
MyBatisNativeConfiguration.MyBatisMapperTypeUtils
.resolveReturnClass(Holder.class, genericList));
Method plain = Holder.class.getMethod("plain");
assertEquals(int.class,
MyBatisNativeConfiguration.MyBatisMapperTypeUtils
.resolveReturnClass(Holder.class, plain));
}
/** Map 泛型取 value 类型(index 1),其他泛型取第一个参数。 */
@Test
void resolveReturnClassPrefersMapValueType() throws Exception {
assertEquals(Integer.class,
MyBatisNativeConfiguration.MyBatisMapperTypeUtils
.resolveReturnClass(Holder.class, Holder.class.getMethod("genericMap")));
}
/** 数组返回类型应解析为组件类型。 */
@Test
void resolveReturnClassUnwrapsArrays() throws Exception {
assertEquals(String.class,
MyBatisNativeConfiguration.MyBatisMapperTypeUtils
.resolveReturnClass(Holder.class, Holder.class.getMethod("arrayReturn")));
}
/** 参数类型解析:所有参数类都应收集到。 */
@Test
void resolveParameterClassesCollectsEveryParameter() throws Exception {
Collection<Class<?>> params = MyBatisNativeConfiguration.MyBatisMapperTypeUtils
.resolveParameterClasses(Holder.class, Holder.class.getMethod("twoArgs", String.class, Integer.class));
assertTrue(params.contains(String.class));
assertTrue(params.contains(Integer.class));
}
/** 无参数方法应得到空集合而不是异常。 */
@Test
void resolveParameterClassesHandlesNoArguments() throws Exception {
Collection<Class<?>> params = MyBatisNativeConfiguration.MyBatisMapperTypeUtils
.resolveParameterClasses(Holder.class, Holder.class.getMethod("plain"));
assertTrue(params.isEmpty(), "无参方法应返回空集合");
}
// ---------- MapperFactoryBean 后置处理器 ----------
/**
* 泛型未解析时,应把 mapper 接口注入为构造器泛型参数并设置 targetType,
* 从而避免容器提前初始化 MapperFactoryBean。
*/
@Test
void factoryBeanPostProcessorResolvesUnresolvableMapperType() {
var processor = new MyBatisNativeConfiguration.MyBatisMapperFactoryBeanPostProcessor();
processor.setBeanFactory(new org.springframework.beans.factory.support.DefaultListableBeanFactory());
RootBeanDefinition definition = new RootBeanDefinition(MapperFactoryBean.class);
definition.setTargetType(org.springframework.core.ResolvableType.forClass(MapperFactoryBean.class));
definition.getPropertyValues().add("mapperInterface", GalleryMapper.class);
processor.postProcessMergedBeanDefinition(definition, MapperFactoryBean.class, "galleryMapper");
assertEquals(GalleryMapper.class,
definition.getConstructorArgumentValues().getGenericArgumentValue(Class.class).getValue(),
"mapper 接口应被注入为构造器泛型参数");
assertFalse(definition.getResolvableType().hasUnresolvableGenerics(),
"targetType 设置后泛型应可解析");
}
/** 非 MapperFactoryBean 的 bean 不应被改动。 */
@Test
void factoryBeanPostProcessorIgnoresOtherBeans() {
var processor = new MyBatisNativeConfiguration.MyBatisMapperFactoryBeanPostProcessor();
processor.setBeanFactory(new org.springframework.beans.factory.support.DefaultListableBeanFactory());
RootBeanDefinition definition = new RootBeanDefinition(String.class);
processor.postProcessMergedBeanDefinition(definition, String.class, "someString");
assertTrue(definition.getConstructorArgumentValues().isEmpty(),
"无关 bean 不应被注入构造器参数");
}
/** 取不到映射器接口时应静默返回,不影响启动。 */
@Test
void factoryBeanPostProcessorToleratesMissingMapperInterface() {
var processor = new MyBatisNativeConfiguration.MyBatisMapperFactoryBeanPostProcessor();
processor.setBeanFactory(new org.springframework.beans.factory.support.DefaultListableBeanFactory());
RootBeanDefinition definition = new RootBeanDefinition(MapperFactoryBean.class);
definition.setTargetType(org.springframework.core.ResolvableType.forClass(MapperFactoryBean.class));
// 故意不设置 mapperInterface 属性
assertDoesNotThrow(() ->
processor.postProcessMergedBeanDefinition(definition, MapperFactoryBean.class, "broken"));
}
// ---------- AOT 处理器 ----------
/** MapperScannerConfigurer 必须被排除在 AOT 处理之外(否则会被提前实例化)。 */
@Test
void aotProcessorExcludesMapperScannerConfigurer() {
var processor = new MyBatisNativeConfiguration.MyBatisBeanFactoryInitializationAotProcessor();
RegisteredBean registered = registeredBean("configurer", MapperScannerConfigurer.class);
assertTrue(processor.isExcludedFromAotProcessing(registered));
}
@Test
void aotProcessorDoesNotExcludeOrdinaryBeans() {
var processor = new MyBatisNativeConfiguration.MyBatisBeanFactoryInitializationAotProcessor();
RegisteredBean registered = registeredBean("service", GalleryMapper.class);
assertFalse(processor.isExcludedFromAotProcessing(registered));
}
/**
* 没有 MapperFactoryBean 时不应产出 AOT contribution(返回 null),
* 避免为无 MyBatis 的上下文生成多余代码。
*/
@Test
void aotProcessorReturnsNullWithoutMappers() {
var processor = new MyBatisNativeConfiguration.MyBatisBeanFactoryInitializationAotProcessor();
// 空工厂:没有任何 MapperFactoryBean
var beanFactory = new org.springframework.beans.factory.support.DefaultListableBeanFactory();
assertNull(processor.processAheadOfTime(beanFactory));
}
/**
* 存在 MapperFactoryBean 时应产出 contribution,并在应用时注册
* mapper 接口的反射、JDK 代理与同名 XML 资源。
*/
@Test
void aotProcessorRegistersMapperProxyAndResources() {
var processor = new MyBatisNativeConfiguration.MyBatisBeanFactoryInitializationAotProcessor();
// 真实工厂:注册一个 MapperFactoryBean,getBeanNamesForType 会带上 & 前缀
var beanFactory = new org.springframework.beans.factory.support.DefaultListableBeanFactory();
RootBeanDefinition definition = new RootBeanDefinition(MapperFactoryBean.class);
definition.getPropertyValues().add("mapperInterface", UserMapper.class);
beanFactory.registerBeanDefinition("userMapper", definition);
var contribution = processor.processAheadOfTime(beanFactory);
assertNotNull(contribution, "有 mapper 时应产出 AOT contribution");
RuntimeHints hints = new RuntimeHints();
contribution.applyTo(new StubGenerationContext(hints), new NoOpInitializationCode());
assertTrue(RuntimeHintsPredicates.reflection().onType(UserMapper.class).test(hints),
"mapper 接口应登记反射");
assertTrue(RuntimeHintsPredicates.proxies().forInterfaces(UserMapper.class).test(hints),
"mapper 接口应登记 JDK 代理");
assertTrue(RuntimeHintsPredicates.resource()
.forResource("com/lion/lionwebsite/Dao/normal/UserMapper.xml").test(hints),
"同名 XML 映射文件应登记");
}
@Test
void subscriptionEventMapperRegistersEventAndBindingReflection() {
var processor = new MyBatisNativeConfiguration.MyBatisBeanFactoryInitializationAotProcessor();
var factory = new org.springframework.beans.factory.support.DefaultListableBeanFactory();
var definition = new RootBeanDefinition(MapperFactoryBean.class);
definition.getPropertyValues().add("mapperInterface", com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper.class);
factory.registerBeanDefinition("subscriptionEventMapper", definition);
var hints = new RuntimeHints();
var contribution = processor.processAheadOfTime(factory);
assertNotNull(contribution);
contribution.applyTo(new StubGenerationContext(hints), new NoOpInitializationCode());
assertTrue(RuntimeHintsPredicates.reflection().onType(com.lion.lionwebsite.Domain.SubscriptionEvent.class).test(hints));
assertTrue(RuntimeHintsPredicates.reflection().onType(com.lion.lionwebsite.Domain.SubscriptionEventBinding.class).test(hints));
assertTrue(RuntimeHintsPredicates.proxies().forInterfaces(com.lion.lionwebsite.Dao.normal.SubscriptionEventMapper.class).test(hints));
}
/** applyTo 只用 runtimeHints,初始化代码侧给一个记录型空实现即可。 */
private static final class NoOpInitializationCode
implements org.springframework.beans.factory.aot.BeanFactoryInitializationCode {
final java.util.List<org.springframework.aot.generate.MethodReference> initializers = new java.util.ArrayList<>();
@Override
public org.springframework.aot.generate.GeneratedMethods getMethods() {
throw new UnsupportedOperationException("测试不断言生成方法");
}
@Override
public org.springframework.javapoet.ClassName getClassName() {
return org.springframework.javapoet.ClassName.bestGuess("com.example.Generated");
}
@Override
public void addInitializer(org.springframework.aot.generate.MethodReference methodReference) {
initializers.add(methodReference);
}
}
/**
* RegisteredBean.of(beanFactory, name) 要求该 bean 已注册在工厂里,
* 因此先用一个真实的 DefaultListableBeanFactory 注册定义再取。
*/
private static RegisteredBean registeredBean(String name, Class<?> type) {
var factory = new org.springframework.beans.factory.support.DefaultListableBeanFactory();
factory.registerBeanDefinition(name, new RootBeanDefinition(type));
return RegisteredBean.of(factory, name);
}
/** 只提供 applyTo 所需能力的极简生成上下文替身(只用到 runtimeHints)。 */
private record StubGenerationContext(RuntimeHints hints)
implements org.springframework.aot.generate.GenerationContext {
@Override
public RuntimeHints getRuntimeHints() {
return hints;
}
@Override
public org.springframework.aot.generate.GeneratedClasses getGeneratedClasses() {
throw new UnsupportedOperationException("测试不需要生成类");
}
@Override
public org.springframework.aot.generate.GeneratedFiles getGeneratedFiles() {
throw new UnsupportedOperationException("测试不需要生成文件");
}
@Override
public org.springframework.aot.generate.GenerationContext withName(String name) {
return this;
}
}
/** 承载各类返回/参数类型的样例接口,用于驱动类型解析。 */
@SuppressWarnings("unused")
interface Holder {
int plain();
String arrayReturn();
java.util.List<String> genericList();
java.util.Map<String, Integer> genericMap();
void twoArgs(String a, Integer b);
}
}
@@ -0,0 +1,143 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.SubService;
import com.lion.lionwebsite.Service.UserService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
/**
* /personal/subBind 与 /personal/user 的路由契约。
* 两个控制器都是直通转调,测试价值在于路径/方法/参数的映射正确
* (方法写错会让前端拿到 405,参数名写错会静默传 null)。
*/
class AccountControllersTest {
private SubService subService;
private UserService userService;
private MockMvc mockMvc;
@BeforeEach
void setUp() {
subService = mock(SubService.class);
userService = mock(UserService.class);
}
private MockMvc subMvc() {
return MockMvcBuilders.standaloneSetup(new SubController(subService)).build();
}
private MockMvc userMvc() {
return MockMvcBuilders.standaloneSetup(new UserController(userService)).build();
}
// ---------- SubController ----------
@Test
void subBindEndpointsRouteToService() throws Exception {
var mvc = subMvc();
when(subService.insertSubBind("alice", 1)).thenReturn("{\"result\":\"success\"}");
mvc.perform(post("/personal/subBind/").param("user", "alice").param("accountId", "1"))
.andExpect(status().isOk());
verify(subService).insertSubBind("alice", 1);
mvc.perform(put("/personal/subBind/").param("user", "alice"));
verify(subService).resetKey("alice");
mvc.perform(get("/personal/subBind/all"));
verify(subService).selectAllSubBind();
mvc.perform(get("/personal/subBind/allRecord"));
verify(subService).SelectAllSubUpdateRecord();
mvc.perform(delete("/personal/subBind/").param("user", "alice"));
verify(subService).deleteSubBind("alice");
mvc.perform(put("/personal/subBind/alice/account").param("accountId", "3"));
verify(subService).rebind("alice", 3);
}
/** 账号增删改查与刷新。 */
@Test
void subscriptionAccountEndpointsRouteToService() throws Exception {
var mvc = subMvc();
mvc.perform(get("/personal/subBind/accounts"));
verify(subService).listSubscriptionAccounts();
mvc.perform(post("/personal/subBind/accounts")
.param("name", "n").param("upstreamKey", "k"));
verify(subService).insertSubscriptionAccount("n", "k", true, true);
mvc.perform(put("/personal/subBind/accounts/7")
.param("name", "n2").param("upstreamKey", "k2"));
verify(subService).updateSubscriptionAccount(7, "n2", "k2", true, true);
mvc.perform(post("/personal/subBind/accounts/7/refresh"));
verify(subService).refreshSubscriptionAccount(7);
mvc.perform(delete("/personal/subBind/accounts/7"));
verify(subService).deleteSubscriptionAccount(7);
}
/** filterHighMultiplier / enabled 的默认值为 true,显式传 false 必须被尊重。 */
@Test
void accountFlagsHonourExplicitValuesAndDefaults() throws Exception {
var mvc = subMvc();
mvc.perform(post("/personal/subBind/accounts")
.param("name", "n").param("upstreamKey", "k")
.param("filterHighMultiplier", "false").param("enabled", "false"));
verify(subService).insertSubscriptionAccount("n", "k", false, false);
// 不传两个开关时使用默认 true(与前端表单默认勾选一致)
mvc.perform(put("/personal/subBind/accounts/9")
.param("name", "n").param("upstreamKey", "k"));
verify(subService).updateSubscriptionAccount(9, "n", "k", true, true);
}
// ---------- UserController ----------
@Test
void userEndpointsRouteToService() throws Exception {
var mvc = userMvc();
mvc.perform(get("/personal/user"));
verify(userService).getAllUser();
mvc.perform(post("/personal/user")
.param("targetAuthCode", "code").param("username", "alice"));
verify(userService).addAuthCode("code", "alice");
mvc.perform(put("/personal/user/AuthCode")
.param("targetAuthCode", "old").param("newAuthCode", "new"));
verify(userService).alterAuthCode("old", "new");
mvc.perform(put("/personal/user/Username")
.param("targetAuthCode", "code").param("newUsername", "bob"));
verify(userService).alterUsername("code", "bob");
mvc.perform(delete("/personal/user").param("targetAuthCode", "code"));
verify(userService).deleteAuthCode("code");
mvc.perform(put("/personal/user/status")
.param("AuthCode", "code").param("isEnable", "false"));
verify(userService).alterStatus("code", false);
}
/** 停用状态的布尔绑定:不传时按 false 处理,这是 Spring 的既有行为。 */
@Test
void userStatusDefaultsToFalseWhenFlagOmitted() throws Exception {
var mvc = userMvc();
mvc.perform(put("/personal/user/status").param("AuthCode", "code"));
verify(userService).alterStatus("code", false);
}
}
@@ -0,0 +1,293 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.CollectService;
import com.lion.lionwebsite.Service.GalleryManageService;
import com.lion.lionwebsite.Service.RemoteService;
import com.lion.lionwebsite.Service.UserService;
import com.lion.lionwebsite.Util.Response;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
/**
* /GalleryManage 的 HTTP 契约。
* 这里重点是「参数不全时必须在控制器层就拦住」,不能被透传到服务层去碰数据库;
* 以及 type 分发到正确的查询方法(写错就查错人)。
*/
class GalleryManageControllerTest {
private GalleryManageService galleryManageService;
private CollectService collectService;
private UserService userService;
private RemoteService remoteService;
private MockMvc mockMvc;
@BeforeEach
void setUp() {
galleryManageService = mock(GalleryManageService.class);
collectService = mock(CollectService.class);
userService = mock(UserService.class);
remoteService = mock(RemoteService.class);
mockMvc = MockMvcBuilders.standaloneSetup(new GalleryManageController(
galleryManageService, collectService, userService, remoteService))
// standaloneSetup 默认按 ISO-8859-1 输出字符串,会把中文写成 "?";
// 生产环境 Spring Boot 用 UTF-8,这里对齐以免断言到被破坏的响应体。
.setMessageConverters(new org.springframework.http.converter.StringHttpMessageConverter(
java.nio.charset.StandardCharsets.UTF_8))
.build();
}
// ---------- create_task ----------
/** link 为空必须在控制器层拒绝,不能进服务层。 */
@Test
void createTaskRejectsMissingLink() throws Exception {
mockMvc.perform(post("/GalleryManage").param("targetResolution", "1280x720"))
.andExpect(status().isOk())
.andExpect(content().string(containsFailure()));
verifyNoInteractions(galleryManageService);
}
@Test
void createTaskRejectsMissingResolution() throws Exception {
mockMvc.perform(post("/GalleryManage").param("link", "https://exhentai.org/g/1/abc/"))
.andExpect(status().isOk())
.andExpect(content().string(containsFailure()));
verifyNoInteractions(galleryManageService);
}
@Test
void createTaskDelegatesWhenParametersComplete() throws Exception {
when(galleryManageService.createTask(anyString(), anyString(), anyString()))
.thenReturn("{\"result\":\"success\"}");
mockMvc.perform(post("/GalleryManage")
.param("link", "https://exhentai.org/g/1/abc/")
.param("targetResolution", "1280x720")
.param("AuthCode", "code"))
.andExpect(status().isOk())
.andExpect(content().string("{\"result\":\"success\"}"));
verify(galleryManageService).createTask("https://exhentai.org/g/1/abc/", "1280x720", "code");
}
// ---------- selectGallery 的 type 分发 ----------
/**
* 缺 type 时回「参数不全」,但注意执行顺序:控制器先解析授权码,再校验 type。
* 也就是说非法授权码会在更早处失败(生产上由 TaskHandlerInterceptor 先拒掉),
* 这个顺序本身是契约的一部分,故一并锁定。
*/
@Test
void selectGalleryRejectsMissingTypeAfterResolvingUser() throws Exception {
when(userService.getUserId("code")).thenReturn(7);
var result = mockMvc.perform(get("/GalleryManage").param("AuthCode", "code"))
.andExpect(status().isOk())
.andReturn();
assertTrue(body(result).contains("failure"), "实际输出: " + body(result));
verify(userService).getUserId("code");
verifyNoInteractions(galleryManageService);
}
/** 未知 type 应回「参数错误」,不得落到任何查询分支。 */
@Test
void selectGalleryRejectsUnknownType() throws Exception {
when(userService.getUserId("code")).thenReturn(7);
var result = mockMvc.perform(get("/GalleryManage")
.param("type", "bogus").param("AuthCode", "code"))
.andExpect(status().isOk())
.andReturn();
// MockMvc 默认按 ISO-8859-1 解码响应体,中文需显式按 UTF-8 还原
assertTrue(body(result).contains("参数错误"), "实际输出: " + body(result));
verifyNoInteractions(galleryManageService);
}
/** 每种 type 都要路由到对应方法,且 all/downloader 必须带上解析出的 userId/授权码。 */
@Test
void selectGalleryRoutesEachTypeToItsQuery() throws Exception {
when(userService.getUserId("code")).thenReturn(7);
mockMvc.perform(get("/GalleryManage").param("type", "link")
.param("param", "https://e/g/1/").param("AuthCode", "code"));
verify(galleryManageService).selectTaskByLink("https://e/g/1/");
mockMvc.perform(get("/GalleryManage").param("type", "gid")
.param("param", "123").param("AuthCode", "code"));
verify(galleryManageService).selectTaskByGid(123);
mockMvc.perform(get("/GalleryManage").param("type", "all").param("AuthCode", "code"));
verify(galleryManageService).selectAllGallery(7);
mockMvc.perform(get("/GalleryManage").param("type", "name")
.param("param", "sakura").param("AuthCode", "code"));
verify(galleryManageService).selectGalleryByName("sakura");
mockMvc.perform(get("/GalleryManage").param("type", "downloader").param("AuthCode", "code"));
verify(galleryManageService).selectGalleryByDownloader("code");
}
/** gid 非数字应抛绑定异常而不是静默查 0 号。 */
@Test
void selectGalleryRejectsNonNumericGid() throws Exception {
when(userService.getUserId("code")).thenReturn(7);
try {
mockMvc.perform(get("/GalleryManage")
.param("type", "gid").param("param", "not-a-number").param("AuthCode", "code"));
fail("非数字 gid 应抛出 NumberFormatException");
} catch (Exception e) {
assertInstanceOf(NumberFormatException.class, e.getCause() == null ? e : e.getCause());
}
}
// ---------- deleteTask ----------
// ---------- 下载人查询 ----------
/** 缺 gid 时在控制器层拦住,不进服务层。 */
@Test
void selectDownloaderRejectsMissingGid() throws Exception {
mockMvc.perform(get("/GalleryManage/downloader").param("AuthCode", "code"))
.andExpect(status().isOk())
.andExpect(content().string(containsFailure()));
verifyNoInteractions(galleryManageService);
}
/** 正常请求必须把 gid 与授权码原样传给服务层。 */
@Test
void selectDownloaderDelegatesWithGidAndAuthCode() throws Exception {
when(galleryManageService.selectDownloaderByGid(500, "code")).thenReturn("{}");
mockMvc.perform(get("/GalleryManage/downloader").param("gid", "500").param("AuthCode", "code"))
.andExpect(status().isOk());
verify(galleryManageService).selectDownloaderByGid(500, "code");
}
@Test
void deleteTaskRejectsMissingGid() throws Exception {
mockMvc.perform(delete("/GalleryManage").param("AuthCode", "code"))
.andExpect(status().isOk())
.andExpect(content().string(containsFailure()));
verifyNoInteractions(galleryManageService);
}
@Test
void deleteTaskDelegatesWithGidAndAuthCode() throws Exception {
when(galleryManageService.deleteGalleryByGid(55, "code")).thenReturn("{\"result\":\"success\"}");
mockMvc.perform(delete("/GalleryManage")
.param("gid", "55").param("AuthCode", "code"))
.andExpect(status().isOk())
.andExpect(content().string("{\"result\":\"success\"}"));
verify(galleryManageService).deleteGalleryByGid(55, "code");
}
// ---------- 收藏 ----------
/** 收藏使用授权码解析出的 userId,而不是请求里的任意值。 */
@Test
void collectUsesResolvedUserId() throws Exception {
when(userService.getUserId("code")).thenReturn(7);
when(collectService.collectGallery(9, 7)).thenReturn("{\"result\":\"success\"}");
mockMvc.perform(post("/GalleryManage/collect")
.param("gid", "9").param("AuthCode", "code"))
.andExpect(status().isOk());
verify(collectService).collectGallery(9, 7);
}
@Test
void disCollectUsesResolvedUserId() throws Exception {
when(userService.getUserId("code")).thenReturn(7);
when(collectService.disCollectGallery(9, 7)).thenReturn("{\"result\":\"success\"}");
mockMvc.perform(post("/GalleryManage/disCollect")
.param("gid", "9").param("AuthCode", "code"))
.andExpect(status().isOk());
verify(collectService).disCollectGallery(9, 7);
}
// ---------- 其余直通接口 ----------
@Test
void simpleEndpointsDelegateToService() throws Exception {
mockMvc.perform(post("/GalleryManage/reconnect"));
verify(galleryManageService).reconnect();
mockMvc.perform(post("/GalleryManage/test"));
verify(remoteService).checkAvailability();
mockMvc.perform(get("/GalleryManage/weekUsedAmount"));
verify(galleryManageService).getWeekUsedAmount();
mockMvc.perform(post("/GalleryManage/cache").param("url", "https://e/g/1/x/"));
verify(galleryManageService).cacheImagesKey("https://e/g/1/x/");
mockMvc.perform(post("/GalleryManage/reset"));
verify(galleryManageService).resetUndone();
}
@Test
void retryRejectsMissingGidButAcceptsPresentOne() throws Exception {
mockMvc.perform(post("/GalleryManage/retry"))
.andExpect(status().isOk())
.andExpect(content().string(containsFailure()));
verify(galleryManageService, never()).retryGallery(anyInt());
mockMvc.perform(post("/GalleryManage/retry").param("gid", "12"));
verify(galleryManageService).retryGallery(12);
}
/** 在线图片接口返回 Callable(异步),控制器必须原样交回而不立即执行。 */
@Test
void onlineImageReturnsCallableUnresolved() throws Exception {
java.util.concurrent.atomic.AtomicBoolean invoked = new java.util.concurrent.atomic.AtomicBoolean();
when(galleryManageService.getCachedImage(eq("123"), eq(2), any(), any()))
.thenReturn(() -> {
invoked.set(true);
return null;
});
mockMvc.perform(get("/GalleryManage/onlineImage/2").param("gid", "123"))
.andExpect(request().asyncStarted());
verify(galleryManageService).getCachedImage(eq("123"), eq(2),
any(HttpServletRequest.class), any(HttpServletResponse.class));
}
private static org.hamcrest.Matcher<String> containsFailure() {
return org.hamcrest.Matchers.containsString("\"failure\"");
}
/** 按 UTF-8 还原响应体(MockMvc 默认字符集会把中文解成乱码)。 */
private static String body(org.springframework.test.web.servlet.MvcResult result) throws Exception {
return result.getResponse().getContentAsString(java.nio.charset.StandardCharsets.UTF_8);
}
/** 供断言使用的极小响应体,避免测试里散落魔法字符串。 */
@SuppressWarnings("unused")
private static String failureBody(String reason) {
return Response._failure(reason);
}
}
@@ -0,0 +1,125 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.LocalService;
import com.lion.lionwebsite.Service.PersonalService;
import com.lion.lionwebsite.Interceptor.PersonalInterceptor;
import com.lion.lionwebsite.Util.PanelLoginTicket;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.mock.web.MockHttpSession;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
/**
* /personal 的 HTTP 契约。
* 文件管理与分享接口下线后,这里只剩订阅更新时间、家里 IP 与留言转发;
* 权限由 PersonalInterceptor 另外把关(见 PersonalInterceptorTest)。
*/
class PersonalControllerTest {
private PersonalService personalService;
private LocalService localService;
private PanelLoginTicket panelLoginTicket;
private MockMvc mockMvc;
private static final String SECRET = "test-secret-for-login-ticket";
@BeforeEach
void setUp() {
personalService = mock(PersonalService.class);
localService = mock(LocalService.class);
// 用真实票据实现(不是 mock),这样验签与时间窗的开销也在链路里被覆盖。
panelLoginTicket = new PanelLoginTicket(SECRET, 300);
mockMvc = MockMvcBuilders
.standaloneSetup(new PersonalController(personalService, localService, panelLoginTicket))
.build();
}
@Test
void indexRedirectsToIndexPage() throws Exception {
mockMvc.perform(get("/personal/"))
.andExpect(status().is3xxRedirection())
.andExpect(redirectedUrl("/index"));
}
// ---------- 一键登录 ----------
/** 合法票据:建立会话并跳到面板入口。 */
@Test
void loginAcceptsFreshTicketAndStartsSession() throws Exception {
var result = mockMvc.perform(get("/personal/login").param("t", panelLoginTicket.issue()))
.andExpect(status().is3xxRedirection())
.andExpect(redirectedUrl("/index"))
.andReturn();
var session = (MockHttpSession) result.getRequest().getSession(false);
assertNotNull(session, "登录成功必须建立会话");
assertTrue(Boolean.TRUE.equals(session.getAttribute(PersonalInterceptor.SESSION_ATTRIBUTE)));
}
/** 票据缺失、被篡改、或来自未来的时间戳,都必须退回提示页。 */
@Test
void loginRejectsInvalidTickets() throws Exception {
String valid = panelLoginTicket.issue();
for (String bad : new String[]{null, "", "v1.1.2", valid + "x", valid.replace(".", "")}) {
var request = get("/personal/login");
if (bad != null)
request = request.param("t", bad);
mockMvc.perform(request)
.andExpect(status().is3xxRedirection())
.andExpect(redirectedUrl("/denied"))
.andExpect(result -> assertNull(result.getRequest().getSession(false),
"非法票据不得建立会话"));
}
}
/** 退出必须销毁会话。 */
@Test
void logoutInvalidatesSession() throws Exception {
var session = new MockHttpSession();
session.setAttribute(PersonalInterceptor.SESSION_ATTRIBUTE, Boolean.TRUE);
mockMvc.perform(get("/personal/login/logout").session(session))
.andExpect(status().is3xxRedirection())
.andExpect(redirectedUrl("/denied"));
assertTrue(session.isInvalid(), "退出后会话必须失效");
}
@Test
void simpleOperationsDelegateWithTheirParameters() throws Exception {
mockMvc.perform(get("/personal/lastUpdate"));
verify(personalService).lastUpdate();
mockMvc.perform(get("/personal/ip"));
verify(personalService).getIp();
mockMvc.perform(post("/personal/message2me").param("message", "hello"));
verify(personalService).message2me("hello");
}
/** 手动更新订阅:成功与失败必须映射成不同的 result,前端据此提示。 */
@Test
void updateSubReflectsServiceOutcome() throws Exception {
when(localService.updateSub()).thenReturn(true);
var ok = mockMvc.perform(post("/personal/updateSub"))
.andExpect(status().isOk())
.andReturn();
assertTrue(ok.getResponse().getContentAsString().contains("\"result\":\"success\""));
when(localService.updateSub()).thenReturn(false);
var failed = mockMvc.perform(post("/personal/updateSub"))
.andExpect(status().isOk())
.andReturn();
assertTrue(failed.getResponse().getContentAsString().contains("\"result\":\"failure\""));
verify(localService, times(2)).updateSub();
}
}
@@ -0,0 +1,173 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Domain.User;
import com.lion.lionwebsite.Service.PublicService;
import com.lion.lionwebsite.Service.QueryService;
import com.lion.lionwebsite.Service.RemoteService;
import com.lion.lionwebsite.Service.SubService;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import static org.junit.jupiter.api.Assertions.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
/**
* 公开路由的 HTTP 契约:路径、方法、参数绑定与响应内容。
* 全部用 standaloneSetup(不加载 Spring 上下文、不连数据库、不占端口),
* 拦截器另行单测,这里只锁控制器自身的分发与转发行为。
*/
class PublicControllerTest {
private PublicService publicService;
private RemoteService remoteService;
private SubService subService;
private QueryService queryService;
private MockMvc mockMvc;
@BeforeEach
void setUp() {
publicService = mock(PublicService.class);
remoteService = mock(RemoteService.class);
subService = mock(SubService.class);
queryService = mock(QueryService.class);
mockMvc = MockMvcBuilders
.standaloneSetup(new PublicController(publicService, remoteService, subService, queryService))
.build();
}
@Test
void indexRedirectsToIndexPage() throws Exception {
mockMvc.perform(get("/"))
.andExpect(status().is3xxRedirection())
.andExpect(redirectedUrl("/index"));
}
/** 有 X-Forwarded-For 时以它为准(反向代理后的真实客户端 IP)。 */
@Test
void ipPrefersForwardedForHeader() throws Exception {
mockMvc.perform(get("/ip").header("X-Forwarded-For", "203.0.113.7"))
.andExpect(status().isOk())
.andExpect(content().string("203.0.113.7"));
verify(publicService, never()).logIpAddress(anyString());
}
/** 没有转发头时回退到 remoteAddr。 */
@Test
void ipFallsBackToRemoteAddress() throws Exception {
mockMvc.perform(get("/ip").with(request -> {
request.setRemoteAddr("198.51.100.9");
return request;
}))
.andExpect(status().isOk())
.andExpect(content().string("198.51.100.9"));
}
/** 只有 auth=ip 时才记录家里 IP,其他取值不得写库。 */
@Test
void ipOnlyLogsWhenAuthIsIp() throws Exception {
mockMvc.perform(get("/ip").param("auth", "ip").header("X-Forwarded-For", "203.0.113.7"))
.andExpect(status().isOk());
verify(publicService).logIpAddress("203.0.113.7");
mockMvc.perform(get("/ip").param("auth", "other").header("X-Forwarded-For", "203.0.113.8"))
.andExpect(status().isOk());
verify(publicService, times(1)).logIpAddress(anyString());
}
/** 身份直接内嵌为 JSON 对象,保留 result/data 字段。 */
@Test
void validateReturnsIdentityAndNodeAvailability() throws Exception {
when(publicService.getUserId("code")).thenReturn(new User(7, "code", "alice", null, true));
when(remoteService.isDead()).thenReturn(false);
mockMvc.perform(post("/validate").param("AuthCode", "code"))
.andExpect(status().isOk())
.andExpect(content().string(org.hamcrest.Matchers.containsString("\"userId\":7")))
.andExpect(content().string(org.hamcrest.Matchers.containsString("\"username\":\"alice\"")))
.andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAvailable\":true")))
// 普通用户 isAdmin=false,前端据此隐藏下载人信息。
.andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAdmin\":false")));
}
/** 管理员(id=3)必须带上 isAdmin=true,前端据此显示下载人信息与筛选。 */
@Test
void validateMarksAdminAccount() throws Exception {
when(publicService.getUserId("admin")).thenReturn(new User(3, "admin", "狮子", null, true));
when(remoteService.isDead()).thenReturn(false);
mockMvc.perform(post("/validate").param("AuthCode", "admin"))
.andExpect(status().isOk())
.andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAdmin\":true")));
}
/**
* 用户名含引号/反斜杠时,内层 data 仍必须是合法 JSON。
* 回归:原先手工 String.format 拼接会在这种输入下产出非法 JSON,
* 前端 JSON.parse(res.data.data) 会直接失败,用户卡在登录态。
*/
@Test
void validateEscapesSpecialCharactersInUsername() throws Exception {
when(publicService.getUserId("code")).thenReturn(new User(7, "code", "a\"b\\c", null, true));
when(remoteService.isDead()).thenReturn(false);
var result = mockMvc.perform(post("/validate").param("AuthCode", "code"))
.andExpect(status().isOk())
.andReturn();
// 直接读取结构化 data,特殊字符保持原样。
var outer = com.lion.lionwebsite.Util.CustomUtil.objectMapper
.readTree(result.getResponse().getContentAsString());
var inner = outer.get("data");
assertEquals("a\"b\\c", inner.get("username").asText(), "特殊字符必须原样保留且 JSON 合法");
}
/** 存储节点掉线时 isAvailable 必须为 false,前端据此提示。 */
@Test
void validateReportsUnavailableWhenNodeIsDead() throws Exception {
when(publicService.getUserId("code")).thenReturn(new User(7, "code", "alice", null, true));
when(remoteService.isDead()).thenReturn(true);
mockMvc.perform(post("/validate").param("AuthCode", "code"))
.andExpect(status().isOk())
.andExpect(content().string(org.hamcrest.Matchers.containsString("\"isAvailable\":false")));
}
@Test
void alterAuthCodeDelegatesToService() throws Exception {
when(publicService.alterAuthCode("old", "new")).thenReturn("{\"result\":\"success\"}");
mockMvc.perform(put("/AuthCode").param("AuthCode", "old").param("newAuthCode", "new"))
.andExpect(status().isOk())
.andExpect(content().string("{\"result\":\"success\"}"));
verify(publicService).alterAuthCode("old", "new");
}
/** 订阅分发必须把 client 与 key 原样交给服务层(含路径中的 key)。 */
@Test
void publicSubPassesClientAndKeyToService() throws Exception {
mockMvc.perform(get("/sub/v2/abcd1234"))
.andExpect(status().isOk());
verify(subService).updateSub(any(HttpServletResponse.class), any(HttpServletRequest.class),
eq("v2"), eq("abcd1234"));
}
@Test
void ehThumbnailDelegatesToQueryService() throws Exception {
mockMvc.perform(get("/GalleryManage/ehThumbnail").param("path", "123/abc.jpg"))
.andExpect(status().isOk());
verify(queryService).getEhThumbnail(eq("123/abc.jpg"),
any(HttpServletRequest.class), any(HttpServletResponse.class));
}
}
@@ -0,0 +1,57 @@
package com.lion.lionwebsite.Controller;
import com.lion.lionwebsite.Service.QueryService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
/**
* /query 搜索入口的路由契约:三个可选参数(keyword/prev/next)必须原样透传,
* 因为分页完全依赖 prev/next 拼 URL。
*/
class QueryControllerTest {
private QueryService queryService;
private MockMvc mockMvc;
@BeforeEach
void setUp() {
queryService = mock(QueryService.class);
mockMvc = MockMvcBuilders.standaloneSetup(new QueryController(queryService)).build();
}
@Test
void queryForwardsAllThreeParameters() throws Exception {
when(queryService.query("sakura", "P", "N")).thenReturn("{\"result\":\"success\"}");
mockMvc.perform(get("/query")
.param("keyword", "sakura").param("prev", "P").param("next", "N"))
.andExpect(status().isOk())
.andExpect(content().string("{\"result\":\"success\"}"));
verify(queryService).query("sakura", "P", "N");
}
/** 分页参数缺省时应传 null(服务层据此决定是否拼分页参数)。 */
@Test
void queryPassesNullsForAbsentPagination() throws Exception {
mockMvc.perform(get("/query").param("keyword", "sakura"))
.andExpect(status().isOk());
verify(queryService).query("sakura", null, null);
}
/** 完全不带参数也应正常到达服务层,由它决定如何应对空关键词。 */
@Test
void queryWithoutKeywordReachesService() throws Exception {
mockMvc.perform(get("/query"))
.andExpect(status().isOk());
verify(queryService).query(null, null, null);
}
}

Some files were not shown because too many files have changed in this diff Show More