Compare commits
29
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
973904be79 | ||
|
|
25b71ad7d1 | ||
|
|
e125b07c13 | ||
|
|
31ceb4d5e5 | ||
|
|
cbd8c3a3b9 | ||
|
|
c26853bc9f | ||
|
|
0aae2da92f | ||
|
|
8e23231dc3 | ||
|
|
80b82121a1 | ||
|
|
77f817d402 | ||
|
|
6a97e403ba | ||
|
|
c76af43b7c | ||
|
|
ced322123a | ||
|
|
de1e81d9b0 | ||
|
|
c27bdbf026 | ||
|
|
9798f0541a | ||
|
|
cbfd634f0d | ||
|
|
942aecf27c | ||
|
|
922e7a2a61 | ||
|
|
347f2bec14 | ||
|
|
ad1d96290c | ||
|
|
ce21d8724e | ||
|
|
8a56e9726f | ||
|
|
3ab82509a0 | ||
|
|
6447a74e0c | ||
|
|
b350b5bda9 | ||
|
|
83476dded2 | ||
|
|
b1a631f8b4 | ||
|
|
684cb608e2 |
@@ -37,3 +37,6 @@ build/
|
||||
### Mac OS ###
|
||||
.DS_Store
|
||||
/.idea/encodings.xml
|
||||
|
||||
# 本地运行与测试日志
|
||||
run.out
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# storageNode 内置订阅网关
|
||||
|
||||
8889 的 BackupSubServer 直接承载网关;无需独立进程或端口。空的
|
||||
`SubscriptionPrimaryUrl` 保留旧快照模式;生产设 `https://lionwebsite.xyz`,
|
||||
`SubscriptionEventDir` 默认 `/root/gallery/storageNode/subscription-events`,
|
||||
共享 `SubscriptionSyncSecret`,启用网关且无密钥会拒绝启动。
|
||||
|
||||
Nginx `/sub/` 始终代理本机8889,并覆盖 `X-Real-IP $remote_addr`,保留原始
|
||||
User-Agent/Range。只有 loopback 对端可以提供 X-Real-IP;公网直连8889
|
||||
忽略所有代理来源 IP,X-Forwarded-For 始终不使用。
|
||||
|
||||
每个有效 v2/cat GET/HEAD 优先请求固定主站 origin,连接1秒、整次含完整正文3秒;
|
||||
不传外部认证头、不跟随重定向,正文最大16MiB。网络错误、超时、5xx及3xx转备用快照;
|
||||
400/403/404等其他状态透传,不使用旧缓存。GET/HEAD、Range和内容类型/范围头保留。
|
||||
失败后熔断10秒,期间只读本地,到期只允许一个探测,成功后恢复逐请求访问主站。
|
||||
|
||||
每次请求建立UUID事件,字段为eventId/keyHash/client/clientIp/userAgent/occurredAt。
|
||||
只存公开key的SHA256,不保存公开key、正文或密钥。UA保留合法Latin-1原始前缀,去除控制字符、非Latin-1替换为?,最大2048字符且UTF8<=2048bytes,
|
||||
并在实际base64url事件头超过4096字符时继续截断;发主站UA与事件一致。
|
||||
GET和HEAD均携带签名事件头;HEAD不会记日志。
|
||||
|
||||
签名请求头为 X-Subscription-Event(base64url无padding紧凑JSON)、
|
||||
X-Subscription-Timestamp(epoch秒)、X-Subscription-Signature(HMAC-SHA256 hex)。
|
||||
签名输入 `v1\n<timestamp>\n<sha256(raw JSON bytes)>`,UTF8。
|
||||
主站已记但响应超时,也用原eventId回传,由主站receipt去重。
|
||||
|
||||
备用快照200/206 GET必须先写事件文件并fsync/原子rename/目录fsync才返回;
|
||||
落盘失败或队列超限返回503,HEAD/404/416/503不写成功事件。
|
||||
目录700,文件600。上限100000条/256MiB,保留16MiB磁盘余量;达到限制拒绝新成功更新,
|
||||
不会静默丢弃旧记录。启动恢复所有json,元数据损坏会拒绝网关初始化,需先保留文件并排查。
|
||||
|
||||
后台单线程每10秒最多回传1批,100条且原始body<=256KiB,POST
|
||||
`/internal/subscription-events`,body `{"events":[...]}`。签名使用同一规则对原始body计算,
|
||||
不含事件header。只清理本批 `{"acknowledged":["<eventId>"]}` 明确确认的事件,
|
||||
并同步目录。网络/5xx/非法JSON/未确认/未知ID均保留原ID重试。上传不持有熔断器锁。
|
||||
进程关闭会释放HTTP client和后台线程;Main shutdown hook管理完整服务生命周期。
|
||||
|
||||
`/health/subscription` 保持现有快照字段;`/health/subscription-gateway`
|
||||
另报pendingCount/oldestAgeMillis/circuit/lastUploadAt/lastUploadState,无用户信息。
|
||||
|
||||
验证必须在部署手册指定的构建VM运行,不能在工作机执行Java编译/测试:
|
||||
`mvn -B test`。`SubscriptionGatewayTest` 使用真实本地HTTP主站和节点入口覆盖签名、
|
||||
主站优先、fallback/持久化、ack/重启/同ID、HEAD、404、Range、timeout、熔断/单探测、
|
||||
后台上传隔离、来源IP、路径编码、磁盘错误、批次字节上限、redirect和不完整/超大正文。
|
||||
新增持久化协议全用Jackson树节点,不新增反射DTO。
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
# storageNode 项目结构
|
||||
|
||||
## 概述
|
||||
分布式存储/下载节点服务端,用于画廊网站。职责包括:从远程源下载图片集、压缩为 ZIP 归档、通过 HTTP 提供下载(支持断点续传)、管理代理订阅配置(V2Ray/Clash),并通过 Netty 自定义 TCP 协议与中心服务器通信。
|
||||
|
||||
- **Group ID:** `org.lion`
|
||||
- **Version:** `1.0`
|
||||
- **Java 版本:** 21 字节码(生产使用 GraalVM JDK 25)
|
||||
- **构建工具:** Maven(单模块)
|
||||
- **编译目标:** 当前以 JVM/JAR + `lib/` 运行;保留 GraalVM 原生配置但尚未完成 JDK 25 原生验证
|
||||
|
||||
---
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
storageNode/
|
||||
├── pom.xml # Maven 构建配置
|
||||
└── src/
|
||||
├── main/
|
||||
│ ├── java/
|
||||
│ │ └── lion/
|
||||
│ │ ├── Main.java # 程序入口
|
||||
│ │ ├── storageNode.java # Netty TCP 核心节点
|
||||
│ │ ├── CustomUtil.java # 工具方法(HTTP 通知、端口查找、ObjectMapper)
|
||||
│ │ ├── MultiThreadedHTTPServer.java # HTTP 文件下载服务(端口 8888)
|
||||
│ │ ├── Config/
|
||||
│ │ │ └── Config.java # 加载 config.properties
|
||||
│ │ ├── Domain/
|
||||
│ │ │ └── GalleryTask.java # 下载任务领域模型
|
||||
│ │ ├── ErrorCode/
|
||||
│ │ │ └── ErrorCode.java # 错误码常量
|
||||
│ │ ├── Externel/
|
||||
│ │ │ └── BackupSubServer.java # 代理订阅文件服务(端口 8889)
|
||||
│ │ ├── Message/
|
||||
│ │ │ ├── AbstractMessage.java # 消息基类 + 消息类型常量
|
||||
│ │ │ ├── MessageCodec.java # Netty 编解码器(ByteBuf ↔ AbstractMessage,JSON 格式)
|
||||
│ │ │ └── Main/
|
||||
│ │ │ ├── AvailableCheckMessage.java # 可用性检查(type=8)
|
||||
│ │ │ ├── DeleteGalleryMessage.java # 删除画廊请求(type=3)
|
||||
│ │ │ ├── DownloadPostMessage.java # 提交下载任务(type=1)
|
||||
│ │ │ ├── DownloadStatusMessage.java # 上报任务状态(type=2)
|
||||
│ │ │ ├── IdentityMessage.java # 身份认证握手(type=6)
|
||||
│ │ │ ├── MaintainMessage.java # 心跳维持(type=7)
|
||||
│ │ │ └── ResponseMessage.java # 通用响应(type=0)
|
||||
│ │ └── Service/
|
||||
│ │ ├── DeleteService.java # 删除画廊目录
|
||||
│ │ ├── DownloadCheckService.java # 下载监控与压缩服务
|
||||
│ │ ├── PrimaryChannelTracker.java # 主站通道引用登记与自愈
|
||||
│ │ └── RekickPolicy.java # 主动重新唤起主站的限流判定
|
||||
│ └── resources/
|
||||
│ ├── config.properties # DouNai 订阅地址配置
|
||||
│ ├── simplelogger.properties # SLF4J 日志配置(输出到 run.out)
|
||||
│ └── reflect-config.json # GraalVM 反射配置(Jackson 序列化)
|
||||
└── test/
|
||||
└── java/ # 协议编解码、节点消息处理与上报循环、
|
||||
# 订阅快照与分发、下载/压缩恢复、
|
||||
# 主站通道自愈、删除与工具方法测试
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 关键依赖
|
||||
|
||||
| 依赖 | 版本 | 用途 |
|
||||
|---|---|---|
|
||||
| `io.netty:netty-all` | 4.1.138.Final | TCP 服务端/客户端 |
|
||||
| `com.fasterxml.jackson.core:jackson-databind` | 2.22.2 | JSON 序列化 |
|
||||
| `org.projectlombok:lombok` | 1.18.48 | 简化样板代码(`@Data`, `@Slf4j`) |
|
||||
| `ch.qos.logback:logback-classic` | 1.5.38 | 日志实现 |
|
||||
| `cn.hutool:hutool-all` | 5.8.47 | 文件操作、ZIP 压缩、HTTP 请求 |
|
||||
| `org.apache.commons:commons-compress` | 1.28.0 | 压缩归档 |
|
||||
| `org.graalvm.buildtools:native-maven-plugin` | 1.1.8 | 原生镜像配置(待验证) |
|
||||
|
||||
---
|
||||
|
||||
## 启动流程
|
||||
|
||||
1. `Main.main()` → 调用 `boot()`(遗留的 Netty Bootstrap),然后执行 `Config.loadConfig()`
|
||||
2. 在新线程中启动 `BackupSubServer`(端口 8889)— 提供代理订阅文件下载
|
||||
3. 在新线程中启动 `MultiThreadedHTTPServer`(端口 8888)— 提供画廊 ZIP 文件下载
|
||||
4. 主线程创建 `storageNode()` 实例(阻塞构造函数,永不返回):
|
||||
- 从端口 26321 开始查找空闲端口,绑定 Netty TCP 服务端
|
||||
- 作为 TCP 客户端连接 `lionwebsite.xyz:26322~26342`,唤醒中心服务器
|
||||
- 启动 `DownloadCheckService` 和 5 秒定时任务 `mainThread()`
|
||||
|
||||
---
|
||||
|
||||
## 通信协议(Netty TCP)
|
||||
|
||||
自定义协议格式:`[1字节类型] + [4字节长度] + [JSON 负载]`
|
||||
|
||||
| 类型字节 | 消息类 | 方向 |
|
||||
|---|---|---|
|
||||
| 0 | `ResponseMessage` | 响应 |
|
||||
| 1 | `DownloadPostMessage` | 服务端 → 节点 |
|
||||
| 2 | `DownloadStatusMessage` | 节点 → 服务端 |
|
||||
| 3 | `DeleteGalleryMessage` | 服务端 → 节点 |
|
||||
| 6 | `IdentityMessage` | 握手 |
|
||||
| 7 | `MaintainMessage` | 心跳 |
|
||||
| 8 | `AvailableCheckMessage` | 可用性检查 |
|
||||
| 9 | `SubscriptionSnapshotMessage` | 主站 → 节点,完整订阅备机快照 |
|
||||
|
||||
---
|
||||
|
||||
## HTTP 服务
|
||||
|
||||
### MultiThreadedHTTPServer(端口 8888)
|
||||
- 提供压缩后的画廊 ZIP 文件下载
|
||||
- 仅接受来自 `lionwebsite.xyz` IP 的连接
|
||||
- 支持 HTTP 206 Partial Content(断点续传)
|
||||
- 参数:`AuthCode`(管理员访问)、`gid`(画廊 ID)
|
||||
|
||||
### BackupSubServer(端口 8889)
|
||||
- 提供 V2Ray 和 Clash 代理订阅文件
|
||||
- 不直接访问上游;主站完成下载和倍率过滤后,通过 Netty 类型 9 推送完整快照
|
||||
- 按公开 Key 的 SHA-256 查找用户绑定的子账号,未知 Key 返回 404
|
||||
- 快照经 SHA-256 和 HMAC 校验后原子落盘,主站离线时继续分发最后成功版本
|
||||
- 文件存储于 `sub/snapshots/{revision}/accounts/{accountId}/`
|
||||
- 没有有效快照或快照超过最大有效期时返回 503,不回退旧共享订阅
|
||||
- `GET /health/subscription` 提供不含 Key 和订阅正文的快照状态
|
||||
|
||||
---
|
||||
|
||||
## 核心服务
|
||||
|
||||
### DownloadCheckService
|
||||
- 扫描下载目录(`/root/gallery/hentai/download/`)监控进度
|
||||
- 通过检测 `galleryinfo.txt` 文件判断下载完成
|
||||
- 将完成的下载任务移入压缩队列
|
||||
- 后台线程每 5 秒执行 ZIP 压缩,完成后删除源目录
|
||||
|
||||
### DeleteService
|
||||
- 按名称删除画廊目录
|
||||
- 失败时返回 `ErrorCode.IO_ERROR` 或 `ErrorCode.FILE_NOT_FOUND`
|
||||
|
||||
### PrimaryChannelTracker
|
||||
- 登记「哪条通道是主站」,供任务状态上报与心跳使用
|
||||
- 只有主站会发的消息类型(身份/任务下发/探活/订阅快照)才可用于认领引用;
|
||||
备机的身份消息与节点自己的出站类型都不算证据
|
||||
- 首选通道断开时立即回退到其它仍可用的已认证通道;引用被清空时,
|
||||
已认证通道上的下一条消息即可恢复它,避免任务状态静默停止上报
|
||||
|
||||
### RekickPolicy
|
||||
- 判定「有待上报任务、却无可用通道」时是否应主动重新唤起主站
|
||||
- 以 30 秒为最小间隔限流,既能快速自愈,又不会在主站确实离线时形成重连风暴
|
||||
|
||||
---
|
||||
|
||||
## 配置说明
|
||||
|
||||
### config.properties(从 `/root/gallery/storageNode/config.properties` 加载)
|
||||
```properties
|
||||
SubscriptionSyncEnabled=false
|
||||
SubscriptionSyncSecret=
|
||||
SubscriptionDataDir=/root/gallery/storageNode/sub
|
||||
SubscriptionMaxStaleSeconds=604800
|
||||
SubscriptionMaxPayloadBytes=52428800
|
||||
SubscriptionHttpPort=8889
|
||||
SubscriptionHttpWorkers=4
|
||||
SubscriptionSocketTimeoutMs=10000
|
||||
```
|
||||
|
||||
生产同步密钥优先通过 `SUBSCRIPTION_SYNC_SECRET` 环境变量提供,不得提交到仓库或写入日志。
|
||||
|
||||
### simplelogger.properties
|
||||
- 日志级别:`info`
|
||||
- 时间戳格式:`yyyy-MM-dd HH:mm:ss`
|
||||
- 输出文件:`run.out`
|
||||
|
||||
---
|
||||
|
||||
## 架构说明
|
||||
|
||||
- 单模块 Maven 项目,现有 62 个测试用例:协议编解码(MessageCodec)、节点消息处理与
|
||||
状态上报循环、备机订阅分发 HTTP(8889)、订阅快照存取、下载/压缩恢复与 gid 匹配、
|
||||
主站通道自愈、删除与工具方法
|
||||
- 硬编码文件系统路径(`/root/gallery/...`)→ 仅限 Linux 部署
|
||||
- 外部连接:`lionwebsite.xyz`、`personal.lionwebsite.xyz`、`aaaa.gay`
|
||||
- GraalVM 原生镜像编译,包含 Jackson 反射配置
|
||||
- 大量使用 Lombok(`@Data`、`@Slf4j`)
|
||||
- 使用 Hutool 工具库处理文件/ZIP/HTTP 操作
|
||||
@@ -9,65 +9,113 @@
|
||||
<version>1.0</version>
|
||||
|
||||
<properties>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
<maven.compiler.release>21</maven.compiler.release>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.fasterxml.jackson</groupId>
|
||||
<artifactId>jackson-bom</artifactId>
|
||||
<version>2.22.3</version>
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>io.netty</groupId>
|
||||
<artifactId>netty-all</artifactId>
|
||||
<version>4.1.101.Final</version>
|
||||
<!--
|
||||
只保留实际用到的模块。原先的 netty-all 是聚合 pom,会拖进 35 个模块
|
||||
(含 osx/aarch64/riscv64 的 epoll/kqueue native、codec-http3/mqtt/redis/
|
||||
smtp/stomp/xml/protobuf、transport-rxtx/sctp/udt 等),
|
||||
而本节点只用 ServerBootstrap/Bootstrap/NioEventLoopGroup/
|
||||
NioServerSocketChannel/NioSocketChannel/ByteBuf/ByteToMessageCodec/
|
||||
LengthFieldBasedFrameDecoder。
|
||||
transport 提供 Bootstrap/EventLoop/Channel,codec 提供
|
||||
ByteToMessageCodec 与 LengthFieldBasedFrameDecoder。
|
||||
buffer/common/resolver 由它们传递引入。这里不用 handler:
|
||||
节点侧未使用 LoggingHandler 等 handler 模块的类。
|
||||
-->
|
||||
<artifactId>netty-transport</artifactId>
|
||||
<version>4.1.138.Final</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>io.netty</groupId>
|
||||
<artifactId>netty-codec</artifactId>
|
||||
<version>4.1.138.Final</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.fasterxml.jackson.core</groupId>
|
||||
<artifactId>jackson-databind</artifactId>
|
||||
<version>2.15.2</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>1.18.30</version>
|
||||
<version>1.18.48</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
<version>2.0.9</version>
|
||||
<version>2.0.19</version>
|
||||
</dependency>
|
||||
|
||||
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-simple</artifactId>
|
||||
<version>2.0.7</version>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
<version>1.5.38</version>
|
||||
</dependency>
|
||||
|
||||
|
||||
<dependency>
|
||||
<groupId>cn.hutool</groupId>
|
||||
<artifactId>hutool-all</artifactId>
|
||||
<version>5.8.26</version>
|
||||
<version>5.8.47</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.commons</groupId>
|
||||
<artifactId>commons-compress</artifactId>
|
||||
<version>1.25.0</version>
|
||||
<version>1.28.0</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.httpcomponents</groupId>
|
||||
<artifactId>httpclient</artifactId>
|
||||
<version>4.5.14</version>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter</artifactId>
|
||||
<version>5.14.4</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<version>3.2.5</version>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<version>3.13.0</version>
|
||||
<configuration>
|
||||
<annotationProcessorPaths>
|
||||
<path>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>1.18.48</version>
|
||||
</path>
|
||||
</annotationProcessorPaths>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- <plugin>-->
|
||||
<!-- <groupId>org.apache.maven.plugins</groupId>-->
|
||||
<!-- <artifactId>maven-assembly-plugin</artifactId>-->
|
||||
@@ -94,21 +142,22 @@
|
||||
<plugin>
|
||||
<groupId>org.graalvm.buildtools</groupId>
|
||||
<artifactId>native-maven-plugin</artifactId>
|
||||
<version>0.9.28</version>
|
||||
<version>1.1.8</version>
|
||||
<configuration>
|
||||
<mainClass>lion.Main</mainClass>
|
||||
<imageName>storageNode</imageName>
|
||||
<buildArgs>
|
||||
<arg>--add-modules=jdk.httpserver</arg>
|
||||
<arg>-H:+ReportExceptionStackTraces</arg>
|
||||
<arg>--gc=G1</arg>
|
||||
<arg>--enable-url-protocols=https</arg>
|
||||
<arg>-H:IncludeResources="simplelogger.properties"</arg>
|
||||
<arg>--initialize-at-build-time=org.slf4j.simple.SimpleLogger,org.slf4j.simple.SimpleLoggerFactory,org.slf4j.simple.SimpleLoggerConfiguration</arg>
|
||||
<arg>-H:IncludeResources="logback.xml"</arg>
|
||||
<arg>--initialize-at-build-time=ch.qos.logback.classic,ch.qos.logback.core,ch.qos.logback.classic.pattern,ch.qos.logback.core.pattern</arg>
|
||||
<arg>-H:ReflectionConfigurationFiles=src/main/resources/reflect-config.json</arg>
|
||||
</buildArgs>
|
||||
<metadataRepository>
|
||||
<enabled>true</enabled>
|
||||
</metadataRepository>
|
||||
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
|
||||
@@ -3,21 +3,96 @@ package lion.Config;
|
||||
import java.io.FileInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Properties;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
@Slf4j
|
||||
public class Config {
|
||||
public static String DouNaiV2ray;
|
||||
public static String DouNaiClash;
|
||||
public static final String CONFIG_PATH = "/root/gallery/storageNode/config.properties";
|
||||
|
||||
public static boolean subscriptionSyncEnabled;
|
||||
public static String subscriptionSyncSecret;
|
||||
public static String subscriptionDataDir;
|
||||
public static long subscriptionMaxStaleSeconds;
|
||||
public static int subscriptionMaxPayloadBytes;
|
||||
public static int subscriptionHttpPort;
|
||||
public static int subscriptionHttpWorkers;
|
||||
public static int subscriptionSocketTimeoutMs;
|
||||
/** 空值保留快照直读模式;设置后启用进程内主站优先网关。 */
|
||||
public static String subscriptionPrimaryUrl = "";
|
||||
public static String subscriptionEventDir = "/root/gallery/storageNode/subscription-events";
|
||||
|
||||
/** 下载服务的管理员口令;为空表示 8888 的管理员直取文件能力关闭。 */
|
||||
public static String adminDownloadCode = "";
|
||||
|
||||
/** 管理员直取文件时允许访问的根目录,越界一律拒绝。 */
|
||||
public static String adminDownloadRoot = "/root/gallery/gallery";
|
||||
|
||||
/**
|
||||
* 可靠任务协议的持久化目录。
|
||||
*
|
||||
* <p>默认独立于订阅数据目录:任务日志与订阅快照的生命周期、备份策略和损坏影响面都不同,
|
||||
* 混在一处会让其中一方的清理或迁移误伤另一方。
|
||||
*/
|
||||
public static String taskJournalDir = "/root/gallery/storageNode/task-journal";
|
||||
|
||||
public static void loadConfig(){
|
||||
loadConfig(CONFIG_PATH);
|
||||
}
|
||||
|
||||
/**
|
||||
* 从指定文件加载配置。
|
||||
*
|
||||
* <p>拆出带路径的版本是为了能直接对「默认值填充」和「启用同步却没配密钥就拒绝启动」
|
||||
* 写断言,不必依赖生产路径存在。文件缺失不是致命错误(保留内置默认值),
|
||||
* 但会记录明确日志,避免以错误配置静默运行。
|
||||
*/
|
||||
public static void loadConfig(String path){
|
||||
Properties prop = new Properties();
|
||||
|
||||
try (InputStream input = new FileInputStream("/root/gallery/storageNode/config.properties")) {
|
||||
prop.load(input);
|
||||
DouNaiV2ray = prop.getProperty("DouNaiV2ray");
|
||||
DouNaiClash = prop.getProperty("DouNaiClash");
|
||||
} catch (IOException ex) {
|
||||
ex.printStackTrace();
|
||||
if (!Files.isRegularFile(Path.of(path))) {
|
||||
log.warn("配置文件不存在:{},使用内置默认值", path);
|
||||
apply(prop);
|
||||
return;
|
||||
}
|
||||
|
||||
try (InputStream input = new FileInputStream(path)) {
|
||||
prop.load(input);
|
||||
} catch (IOException ex) {
|
||||
log.error("加载配置失败:{}", ex.getMessage());
|
||||
apply(prop);
|
||||
return;
|
||||
}
|
||||
apply(prop);
|
||||
}
|
||||
|
||||
private static void apply(Properties prop){
|
||||
subscriptionSyncEnabled = Boolean.parseBoolean(value(prop, "SubscriptionSyncEnabled", "false"));
|
||||
subscriptionSyncSecret = System.getenv().getOrDefault("SUBSCRIPTION_SYNC_SECRET",
|
||||
value(prop, "SubscriptionSyncSecret", ""));
|
||||
subscriptionDataDir = value(prop, "SubscriptionDataDir", "/root/gallery/storageNode/sub");
|
||||
subscriptionMaxStaleSeconds = Long.parseLong(value(prop, "SubscriptionMaxStaleSeconds", "604800"));
|
||||
subscriptionMaxPayloadBytes = Integer.parseInt(value(prop, "SubscriptionMaxPayloadBytes", "52428800"));
|
||||
subscriptionHttpPort = Integer.parseInt(value(prop, "SubscriptionHttpPort", "8889"));
|
||||
subscriptionHttpWorkers = Integer.parseInt(value(prop, "SubscriptionHttpWorkers", "4"));
|
||||
subscriptionSocketTimeoutMs = Integer.parseInt(value(prop, "SubscriptionSocketTimeoutMs", "10000"));
|
||||
subscriptionPrimaryUrl = value(prop, "SubscriptionPrimaryUrl", "");
|
||||
subscriptionEventDir = value(prop, "SubscriptionEventDir", "/root/gallery/storageNode/subscription-events");
|
||||
if (!subscriptionPrimaryUrl.isBlank() && subscriptionSyncSecret.isBlank())
|
||||
throw new IllegalStateException("启用订阅网关时必须配置 SUBSCRIPTION_SYNC_SECRET");
|
||||
// 口令不再写死在源码里:优先取环境变量,其次取配置文件,都为空即关闭该能力。
|
||||
adminDownloadCode = System.getenv().getOrDefault("STORAGE_DOWNLOAD_ADMIN_CODE",
|
||||
value(prop, "AdminDownloadCode", ""));
|
||||
adminDownloadRoot = value(prop, "AdminDownloadRoot", "/root/gallery/gallery");
|
||||
taskJournalDir = value(prop, "TaskJournalDir", "/root/gallery/storageNode/task-journal");
|
||||
if (subscriptionSyncEnabled && subscriptionSyncSecret.isBlank())
|
||||
throw new IllegalStateException("启用订阅同步时必须配置 SUBSCRIPTION_SYNC_SECRET");
|
||||
}
|
||||
|
||||
private static String value(Properties prop, String key, String fallback) {
|
||||
return prop.getProperty(key, fallback).trim();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,27 +3,35 @@ package lion;
|
||||
import cn.hutool.http.HttpRequest;
|
||||
import cn.hutool.http.HttpResponse;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import lombok.Data;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.ServerSocket;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
|
||||
|
||||
@Data
|
||||
/**
|
||||
* 通用工具:外部通知、空闲端口探测,以及共享的 {@link ObjectMapper}。
|
||||
*
|
||||
* <p>原先手写的 HTTP 响应工具({@code getRequestHeader} / {@code sendErrorResponse} /
|
||||
* {@code sendFileRange})已随 8888 文件服务迁移到 JDK {@code HttpServer} 而移除;
|
||||
* 文件与 Range 的语义现在由 {@code lion.Http.FileResponseFactory} 统一负责。
|
||||
*/
|
||||
@Slf4j
|
||||
public class CustomUtil {
|
||||
|
||||
public static AtomicInteger counter = new AtomicInteger();
|
||||
|
||||
public static ObjectMapper objectMapper = new ObjectMapper();
|
||||
|
||||
public static void notifyMe(String message) {
|
||||
String url = "https://personal.lionwebsite.xyz/message2me?AuthCode=alone&message=" + message;
|
||||
HttpRequest request = HttpRequest.post(url);
|
||||
// 画廊名可能含空格、'&'、'#' 或非 ASCII,必须编码后再拼进查询串,否则会截断或串参数。
|
||||
String url = "https://personal.lionwebsite.xyz/message2me?AuthCode=alone&message="
|
||||
+ URLEncoder.encode(message == null ? "" : message, StandardCharsets.UTF_8);
|
||||
// 外呼必须有超时;调用方虽有专用守护线程,但不设超时仍可能长期占住它。
|
||||
HttpRequest request = HttpRequest.post(url).timeout(5_000);
|
||||
request.header("User-Agent", "Mozilla/5.0");
|
||||
try(HttpResponse response = request.execute()) {
|
||||
if(response.getStatus() != 200) {
|
||||
System.out.println("通知失败, status code:" + response.getStatus() + ", message:" + message);
|
||||
log.warn("通知失败, status code:{}, message:{}", response.getStatus(), message);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -31,7 +39,6 @@ public class CustomUtil {
|
||||
public static int _findIdlePort(int port) {
|
||||
for(int i=port; i<65535; i++){
|
||||
try(ServerSocket ignored = new ServerSocket(i)){
|
||||
ignored.close();
|
||||
return i;
|
||||
}catch (IOException ignored) {
|
||||
}
|
||||
|
||||
@@ -6,22 +6,22 @@ import lombok.Data;
|
||||
|
||||
@Data
|
||||
public class GalleryTask {
|
||||
public static byte DOWNLOADING = 1;
|
||||
public static byte DOWNLOAD_COMPLETE = 2;
|
||||
public static byte COMPRESSING = 3;
|
||||
public static byte COMPRESS_COMPLETE = 4;
|
||||
public static final byte DOWNLOADING = 1;
|
||||
public static final byte DOWNLOAD_COMPLETE = 2;
|
||||
public static final byte COMPRESSING = 3;
|
||||
public static final byte COMPRESS_COMPLETE = 4;
|
||||
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
private String name;
|
||||
private volatile String name;
|
||||
|
||||
private int gid;
|
||||
|
||||
private byte status;
|
||||
private volatile byte status;
|
||||
|
||||
private int proceeding;
|
||||
private volatile int proceeding;
|
||||
|
||||
@JsonIgnore
|
||||
private String path;
|
||||
private volatile String path;
|
||||
|
||||
@JsonIgnore
|
||||
public boolean is_download_complete(){
|
||||
|
||||
@@ -1,290 +1,155 @@
|
||||
package lion.Externel;
|
||||
|
||||
import lion.CustomUtil;
|
||||
import lion.Http.FileResponseFactory;
|
||||
import lion.Http.HttpFileServer;
|
||||
import lion.Http.HttpRequestHandler;
|
||||
import lion.Http.HttpRequestModel;
|
||||
import lion.Http.HttpResponseModel;
|
||||
import lion.Service.SubscriptionSnapshotStore;
|
||||
import lion.Service.SubscriptionGateway;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.http.HttpEntity;
|
||||
import org.apache.http.client.methods.CloseableHttpResponse;
|
||||
import org.apache.http.client.methods.HttpGet;
|
||||
import org.apache.http.impl.client.CloseableHttpClient;
|
||||
import org.apache.http.impl.client.HttpClients;
|
||||
|
||||
import java.io.*;
|
||||
import java.net.*;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Paths;
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import static lion.Config.Config.DouNaiClash;
|
||||
import static lion.Config.Config.DouNaiV2ray;
|
||||
import java.io.IOException;
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* 备机订阅分发服务(端口由配置提供,历史上为 8889)。
|
||||
*
|
||||
* <p>外壳已从手工 {@code ServerSocket} 改为与下载服务相同的 JDK
|
||||
* {@code com.sun.net.httpserver.HttpServer}({@link HttpFileServer})。路径与语义保持不变:
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code /health/subscription}:返回快照状态 JSON;</li>
|
||||
* <li>{@code /sub/v2/<publicKey>}:返回 v2ray 订阅;{@code /sub/cat/<publicKey>}:返回 Clash;</li>
|
||||
* <li>只有 {@code GET}/{@code HEAD},其他方法 405;</li>
|
||||
* <li>无有效快照(unavailable/expired)返回 503,未命中 Key 返回 404,过短的 Key 直接 404;</li>
|
||||
* <li>Range 支持 206、不可满足区间 416,语义统一由 {@link FileResponseFactory} 提供。</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>{@link #run()} 保留为旧入口({@code Main} 用它起线程),内部改为阻塞等待,直到线程被中断。
|
||||
*/
|
||||
@Slf4j
|
||||
public class BackupSubServer {
|
||||
public final class BackupSubServer implements Runnable, AutoCloseable {
|
||||
|
||||
public static void main(String[] args) {
|
||||
updateSub();
|
||||
ScheduledExecutorService threadPool = Executors.newScheduledThreadPool(1);
|
||||
threadPool.scheduleAtFixedRate(BackupSubServer::updateSub, 0, 12, TimeUnit.HOURS);
|
||||
/** 公开 Key 的合法长度下限,过短直接按不存在处理,避免无谓查表。 */
|
||||
private static final int MIN_KEY_LENGTH = 6;
|
||||
private static final int MAX_KEY_LENGTH = 512;
|
||||
|
||||
String ip = "";
|
||||
try(ServerSocket serverSocket = new ServerSocket(8889)) {
|
||||
log.info("Sub Server listening on port {}", 8889);
|
||||
while (true) {
|
||||
Socket clientSocket = serverSocket.accept();
|
||||
ip = clientSocket.getInetAddress().getHostAddress();
|
||||
log.info("Client connected:{}", ip);
|
||||
// 线程池处理下载请求
|
||||
handleClientRequest(clientSocket);
|
||||
}
|
||||
} catch (IOException e) {
|
||||
log.error("处理http请求时出错,IP:{},ERROR:{}", ip, e.getMessage());
|
||||
}
|
||||
private static final String V2_CLIENT = "v2";
|
||||
private static final String CLASH_CLIENT = "cat";
|
||||
|
||||
private final SubscriptionSnapshotStore snapshotStore;
|
||||
private final int port;
|
||||
private final int workerCount;
|
||||
private final SubscriptionGateway gateway;
|
||||
private volatile HttpFileServer server;
|
||||
|
||||
public BackupSubServer(SubscriptionSnapshotStore snapshotStore, int port, int workerCount) {
|
||||
this(snapshotStore, port, workerCount, null);
|
||||
}
|
||||
|
||||
public static void updateSub(){
|
||||
File DouNaiClashFile = new File("sub/DouNaiClash.txt");
|
||||
File DouNaiV2rayFile = new File("sub/DouNaiV2ray.txt");
|
||||
File directory = new File("sub");
|
||||
/** gateway 由同一宿主管理生命周期;null 保持历史快照模式。 */
|
||||
public BackupSubServer(SubscriptionSnapshotStore snapshotStore, int port, int workerCount,
|
||||
SubscriptionGateway gateway) {
|
||||
this.snapshotStore = Objects.requireNonNull(snapshotStore);
|
||||
this.port = port;
|
||||
this.workerCount = Math.max(1, workerCount);
|
||||
this.gateway = gateway;
|
||||
}
|
||||
|
||||
if(!directory.isDirectory())
|
||||
/** 旧入口:绑定端口、开始接收请求,并阻塞到线程被中断。 */
|
||||
@Override
|
||||
public void run() {
|
||||
try {
|
||||
Files.createDirectory(Paths.get("sub"));
|
||||
} catch (IOException e) {
|
||||
log.error("create directory error:{}", e.getMessage());
|
||||
}
|
||||
|
||||
List<String> DouNaiClash_profile;
|
||||
|
||||
//下载豆奶v2ray订阅
|
||||
try(FileWriter writer = new FileWriter(DouNaiV2rayFile)) {
|
||||
String DouNaiV2rayRaw = Get(DouNaiV2ray).getFirst();
|
||||
String[] v2rayPlain = new String(Base64.getDecoder().decode(DouNaiV2rayRaw)).split("\n");
|
||||
StringBuilder stringBuilder = new StringBuilder();
|
||||
Pattern pattern = Pattern.compile("-?\\d+(\\.\\d+)?");
|
||||
|
||||
//过滤高倍率节点
|
||||
for(String node: v2rayPlain){
|
||||
String name = URLDecoder.decode(node.split("#")[1], StandardCharsets.UTF_8);
|
||||
if(name.contains("流量")){
|
||||
Matcher matcher = pattern.matcher(name.substring(name.indexOf("(") + 1, name.indexOf(")")));
|
||||
|
||||
if (matcher.find()) {
|
||||
// 将匹配到的数字添加到列表中
|
||||
float ratio = Float.parseFloat(matcher.group());
|
||||
if(ratio <= 2) {
|
||||
stringBuilder.append(node).append("\n");
|
||||
continue;
|
||||
}
|
||||
}
|
||||
stringBuilder.append(node).append("\n");
|
||||
}
|
||||
else{
|
||||
stringBuilder.append(node).append("\n");
|
||||
}
|
||||
}
|
||||
writer.write(new String(Base64.getEncoder().encode(stringBuilder.toString().getBytes(StandardCharsets.UTF_8))));
|
||||
|
||||
log.info("load DouNai v2ray complete");
|
||||
}catch (IOException e){
|
||||
log.error("load DouNai v2ray failure: {}", e.getMessage());
|
||||
}
|
||||
|
||||
//下载豆奶clash订阅
|
||||
try(FileWriter writer = new FileWriter(DouNaiClashFile)) {
|
||||
DouNaiClash_profile = Get(DouNaiClash);
|
||||
//过滤高倍率节点
|
||||
ArrayList<String> clashProcessed = new ArrayList<>();
|
||||
boolean isProxies = false;
|
||||
boolean skip = false;
|
||||
for(String line: DouNaiClash_profile){
|
||||
if(line.equals("proxies:"))
|
||||
isProxies = true;
|
||||
else if(line.equals("proxy-groups:") && isProxies)
|
||||
isProxies = false;
|
||||
|
||||
if(isProxies) {
|
||||
if (line.contains("name"))
|
||||
skip = line.contains("流量");
|
||||
if (!skip)
|
||||
clashProcessed.add(line);
|
||||
}
|
||||
else
|
||||
if (!line.contains("流量"))
|
||||
clashProcessed.add(line);
|
||||
}
|
||||
|
||||
for(String line: clashProcessed)
|
||||
writer.write(line + "\n");
|
||||
|
||||
log.info("load DouNai clash complete");
|
||||
}catch (IOException e){
|
||||
log.error("load DouNai clash failure: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
public static ArrayList<String> Get(String url) throws IOException {
|
||||
CloseableHttpClient httpClient = HttpClients.createDefault();
|
||||
CloseableHttpResponse httpResponse;
|
||||
HttpGet httpGet = new HttpGet(url);
|
||||
|
||||
httpResponse = httpClient.execute(httpGet);
|
||||
|
||||
HttpEntity responseEntity = httpResponse.getEntity();
|
||||
int statusCode = httpResponse.getStatusLine().getStatusCode();
|
||||
ArrayList<String> temp = new ArrayList<>();
|
||||
|
||||
if (statusCode == 200) {
|
||||
BufferedReader reader = new BufferedReader(new InputStreamReader(responseEntity.getContent()));
|
||||
String str;
|
||||
while ((str = reader.readLine()) != null)
|
||||
temp.add(str);
|
||||
}
|
||||
|
||||
httpClient.close();
|
||||
httpResponse.close();
|
||||
return temp;
|
||||
}
|
||||
|
||||
private static void handleClientRequest(Socket clientSocket) {
|
||||
String fileName = "";
|
||||
HttpFileServer running = startServer();
|
||||
server = running;
|
||||
log.info("备机订阅服务监听端口 {}", running.port());
|
||||
while (!Thread.currentThread().isInterrupted()) {
|
||||
try {
|
||||
BufferedReader requestReader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
|
||||
String requestLine = requestReader.readLine();
|
||||
|
||||
// Parse the request line to get the method and path
|
||||
String[] requestParts = requestLine.split(" ");
|
||||
String method = requestParts[0];
|
||||
Map<String, String> paramMap = parseRequestLine(requestParts[1]);//path
|
||||
if(paramMap == null){
|
||||
sendErrorResponse(clientSocket, "404");
|
||||
return;
|
||||
}
|
||||
log.info(Arrays.toString(requestParts));
|
||||
|
||||
// Only handle GET requests
|
||||
if (method.equals("GET")) {
|
||||
// Set the file path for download
|
||||
File file = new File(fileName);
|
||||
switch (paramMap.get("Client")) {
|
||||
case "v2" -> file = new File("sub/DouNaiV2ray.txt");
|
||||
case "cat" -> file = new File("sub/DouNaiClash.txt");
|
||||
}
|
||||
fileName = file.getName();
|
||||
log.info(file.getAbsolutePath());
|
||||
// Check if the file exists and is readable
|
||||
if (file.exists() && file.isFile() && file.canRead()) {
|
||||
// Get the file length
|
||||
long fileLength = file.length();
|
||||
|
||||
// Get the range information for resuming download
|
||||
long startByte = 0;
|
||||
long endByte = fileLength - 1;
|
||||
String rangeHeader = getRequestHeader(requestReader);
|
||||
if (rangeHeader != null && rangeHeader.startsWith("bytes=")) {
|
||||
String[] rangeValues = rangeHeader.substring(6).split("-");
|
||||
startByte = Long.parseLong(rangeValues[0]);
|
||||
if (rangeValues.length > 1 && !rangeValues[1].isEmpty()) {
|
||||
endByte = Long.parseLong(rangeValues[1]);
|
||||
Thread.sleep(60_000);
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
|
||||
// Send the HTTP response headers
|
||||
OutputStream responseStream = clientSocket.getOutputStream();
|
||||
PrintWriter responseWriter = new PrintWriter(responseStream, true);
|
||||
responseWriter.println("HTTP/1.1 206 Partial Content");
|
||||
responseWriter.println("Content-Type: application/octet-stream");
|
||||
responseWriter.println("Accept-Ranges: bytes");
|
||||
responseWriter.println("Content-Length: " + (endByte - startByte + 1));
|
||||
responseWriter.println("Content-Range: bytes " + startByte + "-" + endByte + "/" + fileLength);
|
||||
responseWriter.println();
|
||||
|
||||
// Send the file content
|
||||
try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
|
||||
randomAccessFile.seek(startByte);
|
||||
byte[] buffer = new byte[1024];
|
||||
int bytesRead;
|
||||
long bytesRemaining = endByte - startByte + 1;
|
||||
while (bytesRemaining > 0 && (bytesRead = randomAccessFile.read(buffer, 0, (int) Math.min(buffer.length, bytesRemaining))) != -1) {
|
||||
responseStream.write(buffer, 0, bytesRead);
|
||||
bytesRemaining -= bytesRead;
|
||||
}
|
||||
}catch (SocketException ignore){
|
||||
|
||||
}
|
||||
|
||||
// Close the response output stream
|
||||
responseStream.close();
|
||||
} else {
|
||||
// File not found or not readable, send 404 response
|
||||
sendErrorResponse(clientSocket, "404 Not Found");
|
||||
}
|
||||
} else {
|
||||
// Non-GET requests, send 501 response
|
||||
sendErrorResponse(clientSocket, "501 Not Implemented");
|
||||
}
|
||||
|
||||
// Close the request reader and client socket
|
||||
requestReader.close();
|
||||
clientSocket.close();
|
||||
running.close();
|
||||
} catch (IOException e) {
|
||||
log.error("处理文件下载时出错,IP:{}, 文件:{}, ERROR:{}", clientSocket.getInetAddress().getHostAddress(), fileName, e.getMessage());
|
||||
log.error("备机订阅服务停止: {}", e.getMessage());
|
||||
} finally {
|
||||
close();
|
||||
}
|
||||
}
|
||||
|
||||
private static String getRequestHeader(BufferedReader requestReader) throws IOException {
|
||||
String line;
|
||||
while ((line = requestReader.readLine()) != null) {
|
||||
if (line.trim().isEmpty()) {
|
||||
break;
|
||||
/** 建好并启动服务;调用方负责 {@link HttpFileServer#close()}。 */
|
||||
public HttpFileServer startServer() throws IOException {
|
||||
HttpFileServer started = HttpFileServer.create(port, "/", gateway == null ? handler(snapshotStore) : gateway, workerCount);
|
||||
started.start();
|
||||
server = started;
|
||||
return started;
|
||||
}
|
||||
|
||||
if (line.startsWith("Range" + ":")) {
|
||||
return line.substring("Range".length() + 1).trim();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@Override
|
||||
public void close() {
|
||||
if (server != null) server.close();
|
||||
if (gateway != null) gateway.close();
|
||||
}
|
||||
|
||||
public static Map<String, String> parseRequestLine(String requestLine) {
|
||||
Map<String, String> pathParams = new HashMap<>();
|
||||
|
||||
if(requestLine == null)
|
||||
return null;
|
||||
|
||||
String path;
|
||||
if(requestLine.contains("?"))
|
||||
path = requestLine.split("\\?")[0];
|
||||
else
|
||||
path = requestLine;
|
||||
|
||||
String[] vars = path.split("/");
|
||||
|
||||
if(vars.length < 4)
|
||||
return null;
|
||||
|
||||
pathParams.put("Key", vars[3]);
|
||||
pathParams.put("Client", vars[2]);
|
||||
|
||||
if(!pathParams.get("Client").equals("cat") && !pathParams.get("Client").equals("v2"))
|
||||
return null;
|
||||
|
||||
if(pathParams.get("Key").length()<6)
|
||||
return null;
|
||||
|
||||
return pathParams;
|
||||
/** 当前运行中的外壳,便于宿主统一关闭;未启动时为 null。 */
|
||||
public HttpFileServer server() {
|
||||
return server;
|
||||
}
|
||||
|
||||
private static void sendErrorResponse(Socket clientSocket, String statusCode) throws IOException {
|
||||
OutputStream responseStream = clientSocket.getOutputStream();
|
||||
PrintWriter responseWriter = new PrintWriter(responseStream, true);
|
||||
responseWriter.println("HTTP/1.1 " + statusCode);
|
||||
responseWriter.println("Content-Type: text/html");
|
||||
responseWriter.println();
|
||||
responseWriter.println("<h1>" + statusCode + "</h1>");
|
||||
responseStream.close();
|
||||
/** 构建路由处理器,便于测试直接驱动而不绑定端口。 */
|
||||
public static HttpRequestHandler handler(SubscriptionSnapshotStore snapshotStore) {
|
||||
Objects.requireNonNull(snapshotStore, "snapshotStore");
|
||||
return request -> route(snapshotStore, request);
|
||||
}
|
||||
|
||||
private static HttpResponseModel route(SubscriptionSnapshotStore snapshotStore, HttpRequestModel request) {
|
||||
if (!HttpRequestModel.GET.equals(request.method()) && !request.isHead())
|
||||
return plain(405, "text/plain", "Method Not Allowed");
|
||||
|
||||
String path = request.decodedPath() == null ? "" : request.decodedPath();
|
||||
if ("/health/subscription".equals(path)) {
|
||||
try {
|
||||
byte[] body = CustomUtil.objectMapper.writeValueAsBytes(snapshotStore.status());
|
||||
return json(200, body);
|
||||
} catch (IOException e) {
|
||||
log.error("序列化订阅状态失败: {}", e.getMessage());
|
||||
return plain(500, "text/plain", "Internal Server Error");
|
||||
}
|
||||
}
|
||||
|
||||
String[] segments = path.split("/", -1);
|
||||
// "" / "sub" / <v2|cat> / <key>
|
||||
if (segments.length != 4 || !"sub".equals(segments[1])
|
||||
|| !(V2_CLIENT.equals(segments[2]) || CLASH_CLIENT.equals(segments[2])))
|
||||
return plain(404, "text/plain", "Not Found");
|
||||
|
||||
// decodedPath 已由外壳解码一次(保留字面量 '+'),这里直接使用,避免二次解码。
|
||||
String key = segments[3];
|
||||
if (key.length() < MIN_KEY_LENGTH || key.length() > MAX_KEY_LENGTH)
|
||||
return plain(404, "text/plain", "Not Found");
|
||||
|
||||
SubscriptionSnapshotStore.Lookup lookup = snapshotStore.lookup(segments[2], key);
|
||||
if (lookup == null) {
|
||||
SubscriptionSnapshotStore.Status status = snapshotStore.status();
|
||||
boolean unavailable = "unavailable".equals(status.state()) || "expired".equals(status.state());
|
||||
return unavailable
|
||||
? plain(503, "text/plain", "Service Unavailable")
|
||||
: plain(404, "text/plain", "Not Found");
|
||||
}
|
||||
|
||||
String contentType = V2_CLIENT.equals(segments[2]) ? "text/plain; charset=utf-8" : "text/yaml; charset=utf-8";
|
||||
return FileResponseFactory.forBytes(request.rangeHeader(), lookup.content(), contentType, null);
|
||||
}
|
||||
|
||||
private static HttpResponseModel json(int status, byte[] body) {
|
||||
return FileResponseFactory.forBytes(null, body, "application/json; charset=utf-8", null);
|
||||
}
|
||||
|
||||
private static HttpResponseModel plain(int status, String contentType, String body) {
|
||||
return HttpResponseModel.text(status, contentType, body);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
package lion.Http;
|
||||
|
||||
import java.util.Locale;
|
||||
|
||||
/**
|
||||
* HTTP {@code Range} 头的解析与 {@code Content-Range} 计算。
|
||||
*
|
||||
* <p>只处理单区间(与旧实现的 {@code split(",", 2)[0]} 行为一致),但补齐旧实现缺失的
|
||||
* 后缀区间 {@code bytes=-n} 与开放结尾 {@code bytes=a-},并把「无法满足」与「语法非法」
|
||||
* 分成两种结果:前者回 416,后者按 RFC 忽略 Range 返回完整实体。
|
||||
*/
|
||||
public final class ByteRange {
|
||||
|
||||
/** 解析结论。 */
|
||||
public enum Kind {
|
||||
/** 忽略 Range(没有该头、语法非法或不受支持),按完整实体返回 200。 */
|
||||
FULL,
|
||||
/** 返回 206 与指定区间。 */
|
||||
PARTIAL,
|
||||
/** 语法合法但落在实体之外,返回 416。 */
|
||||
UNSATISFIABLE
|
||||
}
|
||||
|
||||
/** 一次 Range 解析的结果;{@code start}/{@code endInclusive} 仅在 {@link Kind#PARTIAL} 时有意义。 */
|
||||
public record Resolution(Kind kind, long start, long endInclusive) {
|
||||
public boolean partial() {
|
||||
return kind == Kind.PARTIAL;
|
||||
}
|
||||
|
||||
public boolean unsatisfiable() {
|
||||
return kind == Kind.UNSATISFIABLE;
|
||||
}
|
||||
|
||||
/** 区间字节数。 */
|
||||
public long length() {
|
||||
return endInclusive - start + 1;
|
||||
}
|
||||
|
||||
/** 形如 {@code bytes 4-9/10} 的响应头值。 */
|
||||
public String contentRange(long totalLength) {
|
||||
return "bytes " + start + "-" + endInclusive + "/" + totalLength;
|
||||
}
|
||||
}
|
||||
|
||||
private static final Resolution FULL = new Resolution(Kind.FULL, 0, -1);
|
||||
private static final Resolution UNSATISFIABLE = new Resolution(Kind.UNSATISFIABLE, 0, -1);
|
||||
|
||||
private ByteRange() {
|
||||
}
|
||||
|
||||
/**
|
||||
* 解析 Range 头。
|
||||
*
|
||||
* @param headerValue 原始 {@code Range} 头,可为 null
|
||||
* @param fileLength 实体总长度
|
||||
*/
|
||||
public static Resolution parse(String headerValue, long fileLength) {
|
||||
if (headerValue == null)
|
||||
return FULL;
|
||||
String value = headerValue.trim();
|
||||
if (!value.toLowerCase(Locale.ROOT).startsWith("bytes="))
|
||||
return FULL;
|
||||
value = value.substring("bytes=".length()).trim();
|
||||
// 多区间只取第一段,与旧实现一致。
|
||||
int comma = value.indexOf(',');
|
||||
if (comma >= 0)
|
||||
value = value.substring(0, comma).trim();
|
||||
if (value.isEmpty())
|
||||
return FULL;
|
||||
|
||||
int dash = value.indexOf('-');
|
||||
if (dash < 0)
|
||||
return FULL;
|
||||
String first = value.substring(0, dash).trim();
|
||||
String second = value.substring(dash + 1).trim();
|
||||
|
||||
if (first.isEmpty()) {
|
||||
// 后缀区间:bytes=-n 取最后 n 字节。
|
||||
long suffix = parseLong(second);
|
||||
if (suffix < 0)
|
||||
return FULL;
|
||||
if (suffix == 0 || fileLength == 0)
|
||||
return UNSATISFIABLE;
|
||||
return partial(Math.max(0, fileLength - suffix), fileLength - 1);
|
||||
}
|
||||
|
||||
long start = parseLong(first);
|
||||
if (start < 0)
|
||||
return FULL;
|
||||
if (start >= fileLength)
|
||||
return UNSATISFIABLE;
|
||||
|
||||
long end;
|
||||
if (second.isEmpty()) {
|
||||
end = fileLength - 1;
|
||||
} else {
|
||||
end = parseLong(second);
|
||||
if (end < 0)
|
||||
return FULL;
|
||||
if (end < start)
|
||||
return UNSATISFIABLE;
|
||||
end = Math.min(end, fileLength - 1);
|
||||
}
|
||||
return partial(start, end);
|
||||
}
|
||||
|
||||
private static Resolution partial(long start, long endInclusive) {
|
||||
return new Resolution(Kind.PARTIAL, start, endInclusive);
|
||||
}
|
||||
|
||||
/** 非数字、空串与溢出一律按「无法解析」处理,由调用方退回完整实体。 */
|
||||
private static long parseLong(String value) {
|
||||
if (value == null || value.isEmpty())
|
||||
return -1;
|
||||
try {
|
||||
return Long.parseLong(value);
|
||||
} catch (NumberFormatException notANumber) {
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package lion.Http;
|
||||
|
||||
import java.io.File;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 把「文件 + Range 头」变成响应模型,统一 200/206/416 的语义。
|
||||
*
|
||||
* <p>下载(8888)与订阅(8889)两条链路共用,避免又出现两套手写 Range 逻辑。
|
||||
*/
|
||||
public final class FileResponseFactory {
|
||||
|
||||
private FileResponseFactory() {
|
||||
}
|
||||
|
||||
/** 无 Content-Disposition 的文件响应。 */
|
||||
public static HttpResponseModel forFile(String rangeHeader, File file, String contentDispositionFileName) {
|
||||
return forFile(rangeHeader, file, contentDispositionFileName, "application/octet-stream");
|
||||
}
|
||||
|
||||
/**
|
||||
* 内存正文的区间响应(订阅内容走这里)。
|
||||
*
|
||||
* <p>与 {@link #forFile} 共用 {@link ByteRange} 的 200/206/416 判定,避免两套实现漂移。
|
||||
*/
|
||||
public static HttpResponseModel forBytes(String rangeHeader, byte[] content, String contentType,
|
||||
String contentDispositionFileName) {
|
||||
byte[] body = content == null ? new byte[0] : content;
|
||||
String disposition = contentDispositionFileName == null
|
||||
? null : contentDisposition(contentDispositionFileName);
|
||||
|
||||
ByteRange.Resolution range = ByteRange.parse(rangeHeader, body.length);
|
||||
if (range.unsatisfiable())
|
||||
return HttpResponseModel.status(416);
|
||||
|
||||
if (range.partial()) {
|
||||
int from = (int) range.start();
|
||||
int to = (int) range.endInclusive();
|
||||
byte[] slice = java.util.Arrays.copyOfRange(body, from, to + 1);
|
||||
return HttpResponseModel.bytes(206, slice, contentType, disposition,
|
||||
Map.of("Accept-Ranges", "bytes", "Content-Range", range.contentRange(body.length)));
|
||||
}
|
||||
return HttpResponseModel.bytes(200, body, contentType, disposition, Map.of("Accept-Ranges", "bytes"));
|
||||
}
|
||||
|
||||
/**
|
||||
* 构造文件响应。
|
||||
*
|
||||
* <p>完整下载必须回 200;命中 Range 才回 206 并带 {@code Content-Range};
|
||||
* 起点越界回 416。GET 与 HEAD 用同一套判定,HEAD 由外壳负责省略正文。
|
||||
*/
|
||||
public static HttpResponseModel forFile(String rangeHeader, File file, String contentDispositionFileName,
|
||||
String contentType) {
|
||||
long length = file.length();
|
||||
String disposition = contentDispositionFileName == null ? null : contentDisposition(contentDispositionFileName);
|
||||
Map<String, String> headers = new LinkedHashMap<>();
|
||||
headers.put("Accept-Ranges", "bytes");
|
||||
|
||||
ByteRange.Resolution range = ByteRange.parse(rangeHeader, length);
|
||||
if (range.unsatisfiable())
|
||||
return HttpResponseModel.status(416);
|
||||
|
||||
if (range.partial()) {
|
||||
headers.put("Content-Range", range.contentRange(length));
|
||||
return HttpResponseModel.file(206, file, range.start(), range.length(), contentType, disposition, headers);
|
||||
}
|
||||
return HttpResponseModel.file(200, file, 0, length, contentType, disposition, headers);
|
||||
}
|
||||
|
||||
/**
|
||||
* HTTP 响应头不能直接包含中文等 Unicode 字符。保留 ASCII 兼容名,并用
|
||||
* RFC 5987 的 UTF-8 filename* 传递完整文件名;控制字符、引号和反斜杠仍会清理。
|
||||
*/
|
||||
static String contentDisposition(String fileName) {
|
||||
StringBuilder safe = new StringBuilder();
|
||||
StringBuilder ascii = new StringBuilder();
|
||||
if (fileName != null) fileName.codePoints().forEach(codePoint -> {
|
||||
int sanitized = Character.isISOControl(codePoint) || codePoint == '"' || codePoint == '\\'
|
||||
? '_' : codePoint;
|
||||
safe.appendCodePoint(sanitized);
|
||||
ascii.append(sanitized <= 0x7e ? (char) sanitized : '_');
|
||||
});
|
||||
String disposition = "attachment; filename=\"" + ascii + "\"";
|
||||
if (safe.toString().equals(ascii.toString())) return disposition;
|
||||
String encoded = URLEncoder.encode(safe.toString(), StandardCharsets.UTF_8)
|
||||
.replace("+", "%20").replace("*", "%2A");
|
||||
return disposition + "; filename*=UTF-8''" + encoded;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
package lion.Http;
|
||||
|
||||
import java.io.File;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
import java.util.function.LongSupplier;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
/**
|
||||
* gid → 归档 ZIP 的缓存索引,替代「每个下载请求都遍历整个画廊根目录」。
|
||||
*
|
||||
* <p>目录名形如 {@code <标题> [<gid>]},归档为同目录同名 {@code .zip}(与
|
||||
* DownloadCheckService 的命名和匹配规则一致)。目录名与归档随时可能新增或变化,
|
||||
* 因此采用「按需扫描建表 + 扫描频率受限 + 可显式失效」:
|
||||
*
|
||||
* <ul>
|
||||
* <li>在扫描有效间隔内:O(1) 查表返回,且仍校验文件是可读普通文件;</li>
|
||||
* <li>扫描频率有界:距上次扫描超过
|
||||
* {@code rescanIntervalMillis} 时下一次查询才重扫。因此归档完成后新出现的目录无需外部
|
||||
* 干预即可被发现,而未知 gid 也无法把扫描频率推高到「每个请求一次」;</li>
|
||||
* <li>{@link #invalidate()}:强制下次查询重扫,供批量归档/删除后立即失效;</li>
|
||||
* <li>索引规模只覆盖画廊根目录这一层,受目录数量约束。</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>磁盘访问经 {@link Supplier} 注入,测试可用内存清单替代真实文件系统。
|
||||
*/
|
||||
public final class GalleryArchiveIndex {
|
||||
|
||||
/** 返回画廊根目录下的一层子目录;可能为 null 或空。 */
|
||||
private final Supplier<File[]> subdirectoryLister;
|
||||
|
||||
/** 查询时允许重扫的最小间隔,避免未知 gid 造成扫描风暴。 */
|
||||
private final long rescanIntervalMillis;
|
||||
|
||||
private final LongSupplier clock;
|
||||
|
||||
private final ReentrantLock lock = new ReentrantLock();
|
||||
|
||||
/** gid → 归档文件;一次扫描重建。 */
|
||||
private final Map<Integer, File> index = new HashMap<>();
|
||||
|
||||
private boolean indexed;
|
||||
private long lastScanAtMillis;
|
||||
|
||||
public GalleryArchiveIndex(File galleryRoot, long rescanIntervalMillis) {
|
||||
this(rescanIntervalMillis, System::currentTimeMillis, () -> galleryRoot.listFiles(File::isDirectory));
|
||||
}
|
||||
|
||||
GalleryArchiveIndex(long rescanIntervalMillis, LongSupplier clock, Supplier<File[]> subdirectoryLister) {
|
||||
this.rescanIntervalMillis = Math.max(0, rescanIntervalMillis);
|
||||
this.clock = Objects.requireNonNull(clock, "clock");
|
||||
this.subdirectoryLister = Objects.requireNonNull(subdirectoryLister, "subdirectoryLister");
|
||||
}
|
||||
|
||||
/**
|
||||
* 查找 gid 对应的可读归档。
|
||||
*
|
||||
* @return 归档文件;不存在或不可读时返回 null
|
||||
*/
|
||||
public File find(String gid) {
|
||||
Integer key = parseGid(gid);
|
||||
if (key == null)
|
||||
return null;
|
||||
|
||||
lock.lock();
|
||||
try {
|
||||
long now = clock.getAsLong();
|
||||
if (!indexed || now - lastScanAtMillis >= rescanIntervalMillis)
|
||||
rebuild(now);
|
||||
|
||||
return usable(index.get(key));
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
/** 归档新增或删除后调用,使下一次查询重新扫描。 */
|
||||
public void invalidate() {
|
||||
lock.lock();
|
||||
try {
|
||||
index.clear();
|
||||
indexed = false;
|
||||
lastScanAtMillis = 0;
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
/** 索引中的 gid 数量,用于测试与状态观测。 */
|
||||
public int size() {
|
||||
lock.lock();
|
||||
try {
|
||||
return index.size();
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
/** 调用方必须持有 {@link #lock};{@code now} 作为扫描时刻一并记录。 */
|
||||
private void rebuild(long now) {
|
||||
index.clear();
|
||||
File[] directories = subdirectoryLister.get();
|
||||
if (directories != null) {
|
||||
for (File directory : directories) {
|
||||
if (directory == null)
|
||||
continue;
|
||||
List<Integer> gids = extractGids(directory.getName());
|
||||
if (gids.isEmpty())
|
||||
continue;
|
||||
File zip = new File(directory, directory.getName() + ".zip");
|
||||
if (!zip.isFile())
|
||||
continue;
|
||||
// 与原 findGalleryZipByGid 一致:同一 gid 以扫描到的第一个目录为准。
|
||||
for (Integer gid : gids)
|
||||
index.putIfAbsent(gid, zip);
|
||||
}
|
||||
}
|
||||
indexed = true;
|
||||
lastScanAtMillis = now;
|
||||
}
|
||||
|
||||
private static File usable(File file) {
|
||||
return file != null && file.isFile() && file.canRead() ? file : null;
|
||||
}
|
||||
|
||||
/** gid 解析失败或不是整数时返回 null。 */
|
||||
static Integer parseGid(String gid) {
|
||||
if (gid == null || gid.isBlank())
|
||||
return null;
|
||||
try {
|
||||
return Integer.parseInt(gid.trim());
|
||||
} catch (NumberFormatException notANumber) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 收集目录名里所有可匹配的 gid。
|
||||
*
|
||||
* <p>原 {@code findGalleryZipByGid} 对请求 gid 用 {@code lastIndexOf("[" + gid)} 定位,
|
||||
* 并要求标记后的下一个字符是 {@code ']'} 或 {@code '-'}。由于查找的是完整的
|
||||
* {@code "[<gid>"} 前缀,目录名中每一处「数字组紧跟 ] 或 -」都对应一个可命中的 gid,
|
||||
* 因此这里全部提取,再由调用方按目录顺序做 first-wins,行为与逐请求扫描一致。
|
||||
*/
|
||||
static List<Integer> extractGids(String directoryName) {
|
||||
List<Integer> gids = new ArrayList<>();
|
||||
if (directoryName == null)
|
||||
return gids;
|
||||
for (int marker = directoryName.indexOf('['); marker >= 0; marker = directoryName.indexOf('[', marker + 1)) {
|
||||
int digitEnd = marker + 1;
|
||||
while (digitEnd < directoryName.length() && Character.isDigit(directoryName.charAt(digitEnd)))
|
||||
digitEnd++;
|
||||
if (digitEnd == marker + 1 || digitEnd >= directoryName.length())
|
||||
continue;
|
||||
char next = directoryName.charAt(digitEnd);
|
||||
if (next != ']' && next != '-')
|
||||
continue;
|
||||
Integer gid = parseGid(directoryName.substring(marker + 1, digitEnd));
|
||||
if (gid != null)
|
||||
gids.add(gid);
|
||||
}
|
||||
return gids;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package lion.Http;
|
||||
|
||||
import lion.Config.Config;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.File;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
|
||||
/**
|
||||
* 8888 文件下载路由:把请求解析成要发送的文件与区间,再由外壳写出响应。
|
||||
*
|
||||
* <p>与旧实现保持一致的判定:
|
||||
*
|
||||
* <ul>
|
||||
* <li>只接受 {@code GET} 与 {@code HEAD},其他方法 501;</li>
|
||||
* <li>缺少 {@code AuthCode} 参数一律 403;</li>
|
||||
* <li>管理员口令命中时按 {@code AdminDownloadRoot} 约束直取文件,越界按 404;</li>
|
||||
* <li>否则优先按 {@code gid} 查归档;查不到再按链接里的文件名兼容旧链接;</li>
|
||||
* <li>文件不存在或不可读一律 404。</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>这里不负责 Range 头解析与状态码 206/416/200 的选择:这些交给
|
||||
* {@link FileResponseFactory},以便同时被下载与订阅两条链路复用。
|
||||
*/
|
||||
@Slf4j
|
||||
public final class GalleryDownloadHandler implements HttpRequestHandler {
|
||||
|
||||
/** 旧下载链接没有 gid 时按路径中的文件名回到这个根目录下查找。 */
|
||||
private static final String LEGACY_GALLERY_ROOT = "/root/gallery/gallery";
|
||||
|
||||
/** 兜底用的不存在路径:让「按文件名找不到归档」走统一的 404 分支。 */
|
||||
private static final File FILE_NOT_FOUND = new File("/root/abc");
|
||||
|
||||
private final GalleryArchiveIndex archiveIndex;
|
||||
|
||||
public GalleryDownloadHandler(GalleryArchiveIndex archiveIndex) {
|
||||
this.archiveIndex = archiveIndex;
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpResponseModel handle(HttpRequestModel request) {
|
||||
if (!HttpRequestModel.GET.equals(request.method()) && !request.isHead())
|
||||
return HttpResponseModel.text(501, "text/html", "<h1>501 Not Implemented</h1>");
|
||||
|
||||
String authCode = request.parameter("AuthCode");
|
||||
if (authCode == null)
|
||||
return HttpResponseModel.text(403, "text/html", "<h1>403 Forbidden</h1>");
|
||||
|
||||
File file = isAdminRequest(authCode) ? resolveAdminRequest(request) : resolveGalleryRequest(request);
|
||||
if (file == null || !file.exists() || !file.isFile() || !file.canRead())
|
||||
return HttpResponseModel.text(404, "text/html", "<h1>404 Not Found</h1>");
|
||||
|
||||
String fileName = file.getName();
|
||||
log.info("提供下载:{}", file.getAbsolutePath());
|
||||
return FileResponseFactory.forFile(request.rangeHeader(), file, fileName);
|
||||
}
|
||||
|
||||
private File resolveAdminRequest(HttpRequestModel request) {
|
||||
File file = resolveAdminFile(request.decodedPath());
|
||||
if (file == null)
|
||||
log.warn("管理员下载越界,已拒绝:{}", request.decodedPath());
|
||||
return file;
|
||||
}
|
||||
|
||||
private File resolveGalleryRequest(HttpRequestModel request) {
|
||||
String gid = request.parameter("gid");
|
||||
File file = gid == null ? null : archiveIndex.find(gid);
|
||||
if (file != null)
|
||||
return file;
|
||||
|
||||
// 兼容没有 gid 参数的旧下载链接:按链接中的文件名找回归档。
|
||||
String requestPath = request.decodedPath();
|
||||
if (requestPath != null && requestPath.contains(".")) {
|
||||
String name = requestPath.substring(0, requestPath.lastIndexOf('.'));
|
||||
return new File(LEGACY_GALLERY_ROOT + name + "/" + name + ".zip");
|
||||
}
|
||||
return FILE_NOT_FOUND;
|
||||
}
|
||||
|
||||
/**
|
||||
* 该请求是否走管理员直取文件通道。
|
||||
*
|
||||
* <p>口令来自配置(环境变量 STORAGE_DOWNLOAD_ADMIN_CODE 优先,其次配置文件
|
||||
* AdminDownloadCode)。未配置即关闭该能力;比较用固定时间算法。
|
||||
*/
|
||||
public static boolean isAdminRequest(String authCode) {
|
||||
String expected = Config.adminDownloadCode;
|
||||
if (expected == null || expected.isBlank() || authCode == null)
|
||||
return false;
|
||||
return MessageDigest.isEqual(expected.getBytes(StandardCharsets.UTF_8),
|
||||
authCode.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
/**
|
||||
* 把管理员的请求路径解析为受配置根目录约束的文件。
|
||||
*
|
||||
* <p>用真实路径做前缀判断,因此 {@code ..} 与指向根目录之外的符号链接都会被拒。
|
||||
* 文件不存在(含越界)返回 null,由调用方回 404。
|
||||
*/
|
||||
public static File resolveAdminFile(String requestPath) {
|
||||
if (requestPath == null || requestPath.isBlank())
|
||||
return null;
|
||||
try {
|
||||
Path root = Path.of(Config.adminDownloadRoot).toRealPath();
|
||||
// 兼容两种调用:绝对路径(主站历史上就是这么传的)与相对根目录的路径。
|
||||
Path raw = Path.of(requestPath);
|
||||
Path candidate = (raw.isAbsolute() ? raw : root.resolve(raw)).normalize();
|
||||
if (!candidate.startsWith(root))
|
||||
return null;
|
||||
// 必须落到真实路径再判断一次:这样指向根目录之外的符号链接也会被拒。
|
||||
Path resolved = candidate.toRealPath();
|
||||
return resolved.startsWith(root) && resolved.toFile().isFile() ? resolved.toFile() : null;
|
||||
} catch (Exception e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,265 @@
|
||||
package lion.Http;
|
||||
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.OutputStream;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.URLDecoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.ArrayBlockingQueue;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.ThreadFactory;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
/**
|
||||
* 基于 JDK {@code com.sun.net.httpserver.HttpServer} 的 HTTP 外壳。
|
||||
*
|
||||
* <p>取代原来的手工 {@code ServerSocket} 实现:请求行/请求头解析、HTTP/1.1 分帧与
|
||||
* 连接保活交给 JDK,本类只负责把 {@link HttpExchange} 适配成
|
||||
* {@link HttpRequestModel},调用 {@link HttpRequestHandler},再把
|
||||
* {@link HttpResponseModel} 写回。
|
||||
*
|
||||
* <p>下载(8888)与订阅(8889)共用这一个外壳,只是注入的处理器不同。
|
||||
*/
|
||||
@Slf4j
|
||||
public final class HttpFileServer implements AutoCloseable {
|
||||
|
||||
/** 默认队列深度:请求超出线程数后先排队,队列满则退化为背压。 */
|
||||
private static final int DEFAULT_QUEUE_DEPTH = 64;
|
||||
|
||||
private final HttpServer server;
|
||||
private final ExecutorService executor;
|
||||
private final int port;
|
||||
private volatile boolean stopped;
|
||||
|
||||
private HttpFileServer(HttpServer server, ExecutorService executor, int port) {
|
||||
this.server = server;
|
||||
this.executor = executor;
|
||||
this.port = port;
|
||||
}
|
||||
|
||||
/**
|
||||
* 绑定端口并创建上下文(尚未开始接收请求)。
|
||||
*
|
||||
* @param port 监听端口;0 表示由系统分配,便于测试
|
||||
* @param path 上下文路径,例如 {@code "/"}
|
||||
* @param handler 请求处理器
|
||||
* @param workerCount 工作线程数上限
|
||||
*/
|
||||
public static HttpFileServer create(int port, String path, HttpRequestHandler handler, int workerCount)
|
||||
throws IOException {
|
||||
return create(port, path, handler, workerCount, DEFAULT_QUEUE_DEPTH);
|
||||
}
|
||||
|
||||
/**
|
||||
* 绑定端口并创建上下文,显式指定排队深度。
|
||||
*
|
||||
* <p>执行器为有界 {@link ThreadPoolExecutor}:固定线程数 + {@link ArrayBlockingQueue},
|
||||
* 队列满时用 {@link ThreadPoolExecutor.CallerRunsPolicy} 在提交线程(JDK HttpServer 的
|
||||
* dispatcher)上直接执行,形成对 accept 的背压,而不是无上限堆积任务或静默丢弃请求。
|
||||
*/
|
||||
public static HttpFileServer create(int port, String path, HttpRequestHandler handler,
|
||||
int workerCount, int queueDepth) throws IOException {
|
||||
Objects.requireNonNull(handler, "handler");
|
||||
int workers = Math.max(1, workerCount);
|
||||
// backlog 只影响内核半连接队列,取一个足够大的固定值即可。
|
||||
HttpServer server = HttpServer.create(new InetSocketAddress(port), 128);
|
||||
ExecutorService executor = new ThreadPoolExecutor(
|
||||
workers, workers, 0L, TimeUnit.MILLISECONDS,
|
||||
new ArrayBlockingQueue<>(Math.max(1, queueDepth)),
|
||||
namedDaemonThreads("storage-http"),
|
||||
new ThreadPoolExecutor.CallerRunsPolicy());
|
||||
server.setExecutor(executor);
|
||||
server.createContext(path, exchange -> serve(exchange, handler));
|
||||
return new HttpFileServer(server, executor, server.getAddress().getPort());
|
||||
}
|
||||
|
||||
/** 开始接收请求。 */
|
||||
public void start() {
|
||||
server.start();
|
||||
}
|
||||
|
||||
/** 实际绑定到的端口({@code create} 传 0 时用于取回随机端口)。 */
|
||||
public int port() {
|
||||
return port;
|
||||
}
|
||||
|
||||
/** 执行器,供包内测试校验有界队列与背压策略。 */
|
||||
ExecutorService executor() {
|
||||
return executor;
|
||||
}
|
||||
|
||||
/**
|
||||
* 停止接收请求并释放线程。
|
||||
*
|
||||
* <p>{@code HttpServer.stop} 最多等待 1 秒让在途交换结束,随后强制关闭连接;
|
||||
* 线程池在本方法内收敛,可安全重复调用。
|
||||
*/
|
||||
@Override
|
||||
public void close() {
|
||||
if (stopped)
|
||||
return;
|
||||
stopped = true;
|
||||
try {
|
||||
server.stop(1);
|
||||
} finally {
|
||||
executor.shutdownNow();
|
||||
}
|
||||
}
|
||||
|
||||
/** 单次请求处理:适配 → 路由 → 写响应;任何异常都不得打死外壳。 */
|
||||
private static void serve(HttpExchange exchange, HttpRequestHandler handler) {
|
||||
try (exchange) {
|
||||
HttpResponseModel response;
|
||||
try {
|
||||
response = handler.handle(toRequestModel(exchange));
|
||||
} catch (Exception failure) {
|
||||
log.error("处理请求失败: {} {}", exchange.getRequestMethod(), exchange.getRequestURI(), failure);
|
||||
response = HttpResponseModel.text(500, "text/html", "<h1>500 Internal Server Error</h1>");
|
||||
}
|
||||
if (response.closeWithoutResponse()) {
|
||||
// 不调用 sendResponseHeaders,try-with-resources 关掉交换即直接断开 TCP。
|
||||
log.info("拒绝来源 {} 的请求: {}", exchange.getRemoteAddress(), exchange.getRequestURI());
|
||||
return;
|
||||
}
|
||||
write(exchange, response);
|
||||
} catch (IOException e) {
|
||||
// 客户端提前断开属于正常情况,记录后不再尝试写响应。
|
||||
log.debug("写响应失败: {}", e.getMessage());
|
||||
} catch (Exception e) {
|
||||
log.error("处理请求时发生未预期错误", e);
|
||||
}
|
||||
}
|
||||
|
||||
static HttpRequestModel toRequestModel(HttpExchange exchange) {
|
||||
String rawPath = exchange.getRequestURI().getRawPath();
|
||||
String decodedPath = decodePath(rawPath);
|
||||
Map<String, String> requestHeaders = new LinkedHashMap<>();
|
||||
exchange.getRequestHeaders().forEach((key, values) -> {
|
||||
if (!values.isEmpty()) requestHeaders.put(key, values.getFirst());
|
||||
});
|
||||
return new HttpRequestModel(
|
||||
exchange.getRequestMethod(),
|
||||
rawPath,
|
||||
decodedPath,
|
||||
parseQuery(exchange.getRequestURI().getRawQuery()),
|
||||
exchange.getRequestHeaders().getFirst("Range"),
|
||||
remoteAddress(exchange), requestHeaders);
|
||||
}
|
||||
|
||||
/** 对端 IP;取不到时返回 null,交由上层按既有策略处理。 */
|
||||
static String remoteAddress(HttpExchange exchange) {
|
||||
InetSocketAddress remote = exchange.getRemoteAddress();
|
||||
return remote == null || remote.getAddress() == null ? null : remote.getAddress().getHostAddress();
|
||||
}
|
||||
|
||||
/** 解析查询串;值内的 {@code =} 不得截断,重复键取最后一次。 */
|
||||
public static Map<String, String> parseQuery(String rawQuery) {
|
||||
Map<String, String> parameters = new LinkedHashMap<>();
|
||||
if (rawQuery == null || rawQuery.isEmpty())
|
||||
return parameters;
|
||||
for (String pair : rawQuery.split("&")) {
|
||||
if (pair.isEmpty())
|
||||
continue;
|
||||
int equals = pair.indexOf('=');
|
||||
if (equals < 0)
|
||||
continue;
|
||||
parameters.put(decodeQuery(pair.substring(0, equals)), decodeQuery(pair.substring(equals + 1)));
|
||||
}
|
||||
return parameters;
|
||||
}
|
||||
|
||||
/**
|
||||
* 路径解码:只做百分号解码,{@code +} 保留为字面量。
|
||||
*
|
||||
* <p>路径不是 {@code application/x-www-form-urlencoded},{@code +} 在路径里就是普通字符
|
||||
* (例如文件名 {@code plus+name.zip}),不能按空格处理。这里先把 {@code +} 预转义成
|
||||
* {@code %2B} 再交给 {@link URLDecoder},从而只保留其百分号解码能力。
|
||||
*/
|
||||
public static String decodePath(String value) {
|
||||
if (value == null)
|
||||
return null;
|
||||
return decodeQuery(value.replace("+", "%2B"));
|
||||
}
|
||||
|
||||
/** 查询串解码:标准表单语义,{@code +} 表示空格。非法序列退回原文。 */
|
||||
public static String decodeQuery(String value) {
|
||||
if (value == null)
|
||||
return null;
|
||||
try {
|
||||
return URLDecoder.decode(value, StandardCharsets.UTF_8);
|
||||
} catch (IllegalArgumentException malformed) {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 写响应。
|
||||
*
|
||||
* <p>HEAD 必须显式写 {@code Content-Length} 并以 {@code -1} 结束,否则 JDK 实现
|
||||
* 不会给出长度;任何响应都不写正文,避免客户端读到错位的数据。
|
||||
*/
|
||||
static void write(HttpExchange exchange, HttpResponseModel response) throws IOException {
|
||||
boolean head = "HEAD".equalsIgnoreCase(exchange.getRequestMethod());
|
||||
long bodyLength = response.hasFile() ? response.bodyLength()
|
||||
: (response.body() == null ? 0 : response.body().length);
|
||||
|
||||
var headers = exchange.getResponseHeaders();
|
||||
if (response.contentType() != null)
|
||||
headers.set("Content-Type", response.contentType());
|
||||
if (response.contentDisposition() != null)
|
||||
headers.set("Content-Disposition", response.contentDisposition());
|
||||
for (Map.Entry<String, String> header : response.headers().entrySet())
|
||||
headers.set(header.getKey(), header.getValue());
|
||||
|
||||
if (head || bodyLength == 0) {
|
||||
// HEAD 与空正文:手动给出长度,再以 -1 结尾确保不写正文。
|
||||
if (head && !headers.containsKey("Content-Length"))
|
||||
headers.set("Content-Length", Long.toString(bodyLength));
|
||||
exchange.sendResponseHeaders(response.status(), -1);
|
||||
return;
|
||||
}
|
||||
|
||||
exchange.sendResponseHeaders(response.status(), bodyLength);
|
||||
if (response.hasFile())
|
||||
writeFile(exchange, response);
|
||||
else
|
||||
exchange.getResponseBody().write(response.body());
|
||||
}
|
||||
|
||||
private static void writeFile(HttpExchange exchange, HttpResponseModel response) throws IOException {
|
||||
long remaining = response.bodyLength();
|
||||
if (remaining <= 0)
|
||||
return;
|
||||
byte[] buffer = new byte[64 * 1024];
|
||||
try (java.io.RandomAccessFile file = new java.io.RandomAccessFile(response.bodyFile(), "r");
|
||||
OutputStream out = exchange.getResponseBody()) {
|
||||
file.seek(response.bodyStart());
|
||||
int bytesRead;
|
||||
while (remaining > 0
|
||||
&& (bytesRead = file.read(buffer, 0, (int) Math.min(buffer.length, remaining))) != -1) {
|
||||
out.write(buffer, 0, bytesRead);
|
||||
remaining -= bytesRead;
|
||||
}
|
||||
out.flush();
|
||||
}
|
||||
}
|
||||
|
||||
private static ThreadFactory namedDaemonThreads(String prefix) {
|
||||
AtomicInteger counter = new AtomicInteger();
|
||||
return runnable -> {
|
||||
Thread thread = new Thread(runnable, prefix + "-" + counter.incrementAndGet());
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
};
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package lion.Http;
|
||||
|
||||
/**
|
||||
* 传输无关的请求处理器:输入请求视图,输出响应模型。
|
||||
*/
|
||||
@FunctionalInterface
|
||||
public interface HttpRequestHandler {
|
||||
|
||||
HttpResponseModel handle(HttpRequestModel request);
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
package lion.Http;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* 与传输实现无关的请求视图。
|
||||
*
|
||||
* <p>在 JDK {@code HttpServer} 的 {@code HttpExchange} 之上再抽一层,使路由与业务判定
|
||||
* 可以被单元测试直接驱动,不必每次都起真实端口;真实集成测试仍走 HTTP。
|
||||
*/
|
||||
public final class HttpRequestModel {
|
||||
|
||||
/** {@code GET},唯一允许下载的方法。 */
|
||||
public static final String GET = "GET";
|
||||
/** {@code HEAD},与 GET 同语义但不回正文。 */
|
||||
public static final String HEAD = "HEAD";
|
||||
|
||||
private final String method;
|
||||
private final String rawPath;
|
||||
private final String decodedPath;
|
||||
private final Map<String, String> queryParameters;
|
||||
private final String rangeHeader;
|
||||
private final String clientAddress;
|
||||
private final Map<String, String> headers;
|
||||
|
||||
public HttpRequestModel(String method, String rawPath, String decodedPath,
|
||||
Map<String, String> queryParameters, String rangeHeader) {
|
||||
this(method, rawPath, decodedPath, queryParameters, rangeHeader, null);
|
||||
}
|
||||
|
||||
public HttpRequestModel(String method, String rawPath, String decodedPath,
|
||||
Map<String, String> queryParameters, String rangeHeader, String clientAddress) {
|
||||
this(method, rawPath, decodedPath, queryParameters, rangeHeader, clientAddress, Map.of());
|
||||
}
|
||||
|
||||
public HttpRequestModel(String method, String rawPath, String decodedPath,
|
||||
Map<String, String> queryParameters, String rangeHeader, String clientAddress,
|
||||
Map<String, String> headers) {
|
||||
this.method = Objects.requireNonNull(method, "method");
|
||||
this.rawPath = rawPath;
|
||||
this.decodedPath = decodedPath;
|
||||
this.queryParameters = queryParameters == null
|
||||
? Collections.emptyMap()
|
||||
: Collections.unmodifiableMap(new LinkedHashMap<>(queryParameters));
|
||||
this.rangeHeader = rangeHeader;
|
||||
this.clientAddress = clientAddress;
|
||||
Map<String, String> normalized = new LinkedHashMap<>();
|
||||
if (headers != null)
|
||||
headers.forEach((key, value) -> normalized.put(key.toLowerCase(java.util.Locale.ROOT), value));
|
||||
this.headers = Collections.unmodifiableMap(normalized);
|
||||
}
|
||||
|
||||
public String method() {
|
||||
return method;
|
||||
}
|
||||
|
||||
/** 未解码的请求路径,用于保持旧管理员直取逻辑对路径原文的处理。 */
|
||||
public String rawPath() {
|
||||
return rawPath;
|
||||
}
|
||||
|
||||
/** 百分号解码后的请求路径。 */
|
||||
public String decodedPath() {
|
||||
return decodedPath;
|
||||
}
|
||||
|
||||
/** 查询参数,值已解码;重复键取最后一次出现的值。 */
|
||||
public Map<String, String> queryParameters() {
|
||||
return queryParameters;
|
||||
}
|
||||
|
||||
/** 单值查询参数,缺失返回 null。 */
|
||||
public String parameter(String name) {
|
||||
return queryParameters.get(name);
|
||||
}
|
||||
|
||||
/** 原始 {@code Range} 头,缺失返回 null。 */
|
||||
public String rangeHeader() {
|
||||
return rangeHeader;
|
||||
}
|
||||
|
||||
public boolean isHead() {
|
||||
return HEAD.equals(method);
|
||||
}
|
||||
|
||||
/** 对端 IP;由外壳填充,直接驱动业务逻辑时可为 null。 */
|
||||
/** 单值请求头,名称大小写不敏感。 */
|
||||
public String header(String name) {
|
||||
return headers.get(name.toLowerCase(java.util.Locale.ROOT));
|
||||
}
|
||||
|
||||
public String clientAddress() {
|
||||
return clientAddress;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
package lion.Http;
|
||||
|
||||
import java.io.File;
|
||||
import java.util.Collections;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 与传输实现无关的响应模型:要么是最多一个区间的文件实体,要么是一段简单文本。
|
||||
*/
|
||||
public final class HttpResponseModel {
|
||||
|
||||
private final int status;
|
||||
private final byte[] body;
|
||||
private final File bodyFile;
|
||||
private final long bodyStart;
|
||||
private final long bodyLength;
|
||||
private final String contentType;
|
||||
private final String contentDisposition;
|
||||
private final Map<String, String> headers;
|
||||
private final boolean closeWithoutResponse;
|
||||
|
||||
private HttpResponseModel(int status, byte[] body, File bodyFile, long bodyStart, long bodyLength,
|
||||
String contentType, String contentDisposition, Map<String, String> headers,
|
||||
boolean closeWithoutResponse) {
|
||||
this.status = status;
|
||||
this.body = body;
|
||||
this.bodyFile = bodyFile;
|
||||
this.bodyStart = bodyStart;
|
||||
this.bodyLength = bodyLength;
|
||||
this.contentType = contentType;
|
||||
this.contentDisposition = contentDisposition;
|
||||
this.headers = headers;
|
||||
this.closeWithoutResponse = closeWithoutResponse;
|
||||
}
|
||||
|
||||
/** 简单状态响应,正文可为空。 */
|
||||
public static HttpResponseModel text(int status, String contentType, String body) {
|
||||
byte[] bytes = body == null ? new byte[0] : body.getBytes(java.nio.charset.StandardCharsets.UTF_8);
|
||||
return new HttpResponseModel(status, bytes, null, 0, bytes.length, contentType, null, Map.of(), false);
|
||||
}
|
||||
|
||||
/** 仅状态码与空正文。 */
|
||||
public static HttpResponseModel status(int status) {
|
||||
return text(status, null, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* 不回任何响应并关闭连接。
|
||||
*
|
||||
* <p>用于来源 IP 不在白名单的请求:与旧实现一致,既不泄露服务存在,也不给出可探测
|
||||
* 的差异。这不属于 HTTP 响应,外壳必须直接断开。
|
||||
*/
|
||||
public static HttpResponseModel rejected() {
|
||||
return new HttpResponseModel(0, null, null, 0, 0, null, null, Map.of(), true);
|
||||
}
|
||||
|
||||
/** 文件实体;{@code start}/{@code length} 描述要发送的字节区间。 */
|
||||
public static HttpResponseModel file(int status, File file, long start, long length,
|
||||
String contentType, String contentDisposition,
|
||||
Map<String, String> headers) {
|
||||
return new HttpResponseModel(status, null, file, start, length, contentType, contentDisposition,
|
||||
headers == null ? Map.of() : Collections.unmodifiableMap(new LinkedHashMap<>(headers)), false);
|
||||
}
|
||||
|
||||
/** 内存正文并携带附加响应头(订阅内容用)。 */
|
||||
public static HttpResponseModel bytes(int status, byte[] body, String contentType, String contentDisposition,
|
||||
Map<String, String> headers) {
|
||||
byte[] content = body == null ? new byte[0] : body;
|
||||
return new HttpResponseModel(status, content, null, 0, content.length, contentType, contentDisposition,
|
||||
headers == null ? Map.of() : Collections.unmodifiableMap(new LinkedHashMap<>(headers)), false);
|
||||
}
|
||||
|
||||
public int status() {
|
||||
return status;
|
||||
}
|
||||
|
||||
/** 是否以文件为正文。 */
|
||||
public boolean hasFile() {
|
||||
return bodyFile != null;
|
||||
}
|
||||
|
||||
public File bodyFile() {
|
||||
return bodyFile;
|
||||
}
|
||||
|
||||
public long bodyStart() {
|
||||
return bodyStart;
|
||||
}
|
||||
|
||||
public long bodyLength() {
|
||||
return bodyLength;
|
||||
}
|
||||
|
||||
/** 内存正文;文件响应时为 null。 */
|
||||
public byte[] body() {
|
||||
return body;
|
||||
}
|
||||
|
||||
public String contentType() {
|
||||
return contentType;
|
||||
}
|
||||
|
||||
public String contentDisposition() {
|
||||
return contentDisposition;
|
||||
}
|
||||
|
||||
/** 附加响应头(Content-Range、Accept-Ranges 等)。 */
|
||||
public Map<String, String> headers() {
|
||||
return headers;
|
||||
}
|
||||
|
||||
/** 是否需要直接关闭连接而不写响应。 */
|
||||
public boolean closeWithoutResponse() {
|
||||
return closeWithoutResponse;
|
||||
}
|
||||
}
|
||||
@@ -3,17 +3,37 @@ package lion;
|
||||
import io.netty.bootstrap.Bootstrap;
|
||||
import lion.Config.Config;
|
||||
import lion.Externel.BackupSubServer;
|
||||
import lion.Service.SubscriptionSnapshotStore;
|
||||
import lion.Service.SubscriptionGateway;
|
||||
import java.io.IOException;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.nio.file.Paths;
|
||||
|
||||
@Slf4j
|
||||
public class Main {
|
||||
|
||||
public static void main(String[] args) {
|
||||
boot();
|
||||
Config.loadConfig();
|
||||
new Thread(() -> BackupSubServer.main(null)).start();
|
||||
SubscriptionSnapshotStore snapshotStore = new SubscriptionSnapshotStore(
|
||||
java.nio.file.Paths.get(Config.subscriptionDataDir), Config.subscriptionSyncSecret,
|
||||
Config.subscriptionMaxStaleSeconds, Config.subscriptionMaxPayloadBytes);
|
||||
snapshotStore.load();
|
||||
try {
|
||||
SubscriptionGateway gateway = Config.subscriptionPrimaryUrl.isBlank() ? null
|
||||
: new SubscriptionGateway(snapshotStore, Config.subscriptionPrimaryUrl,
|
||||
Config.subscriptionSyncSecret, Paths.get(Config.subscriptionEventDir));
|
||||
BackupSubServer subscriptions = new BackupSubServer(snapshotStore, Config.subscriptionHttpPort,
|
||||
Config.subscriptionHttpWorkers, gateway);
|
||||
Runtime.getRuntime().addShutdownHook(new Thread(subscriptions::close, "subscription-shutdown"));
|
||||
new Thread(subscriptions, "subscription-backup-http").start();
|
||||
} catch (IOException failure) {
|
||||
throw new IllegalStateException("无法初始化订阅事件持久化目录", failure);
|
||||
}
|
||||
new Thread(() -> MultiThreadedHTTPServer.main(null)).start();
|
||||
new storageNode();
|
||||
// 启用可靠任务协议:任务与待 ACK 报告持久化到独立目录,重启后恢复。
|
||||
new storageNode(snapshotStore, Paths.get(Config.taskJournalDir), true);
|
||||
}
|
||||
|
||||
public static void boot(){
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
package lion.Message;
|
||||
|
||||
/**
|
||||
* 所有协议消息的基类。
|
||||
*
|
||||
* <p>{@code messageType}/{@code messageId} 一直是公开字段,这里补上标准的 getter/setter,
|
||||
* 让两端测试都能用属性访问。这些访问器生成的 JSON 属性名与原有字段名完全相同,
|
||||
* 因此线上字节没有任何变化。
|
||||
*
|
||||
* <p>刻意不用 Lombok:{@code @Data} 会为继承它的消息类生成 equals/hashCode/toString,
|
||||
* 改变既有消息的相等语义。这里只补最小必要的访问器。
|
||||
*/
|
||||
public class AbstractMessage {
|
||||
|
||||
public static final byte DOWNLOAD_POST_MESSAGE = 1;
|
||||
@@ -14,7 +24,39 @@ public class AbstractMessage {
|
||||
|
||||
public static final byte MAINTAIN_MESSAGE = 7;
|
||||
|
||||
public static final byte AVAILABLE_CHECK_MESSAGE = 8;
|
||||
|
||||
public static final byte SUBSCRIPTION_SNAPSHOT_MESSAGE = 9;
|
||||
|
||||
/** 协议能力协商(v2,见 protocol-v2.md)。 */
|
||||
public static final byte PROTOCOL_HELLO_MESSAGE = 10;
|
||||
|
||||
/** 可靠任务命令(v2):稳定 operationId 的 DOWNLOAD/DELETE。 */
|
||||
public static final byte TASK_COMMAND_MESSAGE = 11;
|
||||
|
||||
/** 可靠任务报告(v2):带 reportId/revision 的不可变快照。 */
|
||||
public static final byte TASK_REPORT_MESSAGE = 12;
|
||||
|
||||
/** 可靠任务报告确认(v2)。 */
|
||||
public static final byte TASK_REPORT_ACK_MESSAGE = 13;
|
||||
|
||||
public byte messageType;
|
||||
|
||||
public int messageId;
|
||||
|
||||
public byte getMessageType() {
|
||||
return messageType;
|
||||
}
|
||||
|
||||
public void setMessageType(byte messageType) {
|
||||
this.messageType = messageType;
|
||||
}
|
||||
|
||||
public int getMessageId() {
|
||||
return messageId;
|
||||
}
|
||||
|
||||
public void setMessageId(int messageId) {
|
||||
this.messageId = messageId;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lion.Message.AbstractMessage;
|
||||
import lombok.Data;
|
||||
|
||||
@Data
|
||||
public class AvailableCheckMessage extends AbstractMessage {
|
||||
{
|
||||
messageType = AVAILABLE_CHECK_MESSAGE;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lion.Message.AbstractMessage;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* 协议能力协商(type=10)。
|
||||
*
|
||||
* <p>主站在身份消息之后发送本消息;节点收到 {@code protocolVersion >= 2} 后回一条同样的
|
||||
* hello,随后才允许使用 type 11/12/13 的可靠任务消息。
|
||||
*
|
||||
* <p>字段刻意保持最小:旧节点/旧主站的 Jackson 会拒绝未知字段,因此旧消息类型一律不得
|
||||
* 增加字段,新增能力只能通过新类型承载。
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class ProtocolHelloMessage extends AbstractMessage {
|
||||
{
|
||||
messageType = PROTOCOL_HELLO_MESSAGE;
|
||||
}
|
||||
|
||||
/** 当前节点实现的协议版本。 */
|
||||
public static final int VERSION = 2;
|
||||
|
||||
/** 可靠任务协议的默认能力标记。 */
|
||||
public static final String RELIABLE_TASK_V2 = "reliable-task-v2";
|
||||
|
||||
private int protocolVersion;
|
||||
|
||||
private List<String> features = new ArrayList<>();
|
||||
|
||||
public static ProtocolHelloMessage v2() {
|
||||
ProtocolHelloMessage hello = new ProtocolHelloMessage();
|
||||
hello.protocolVersion = VERSION;
|
||||
hello.features = new ArrayList<>(List.of(RELIABLE_TASK_V2));
|
||||
return hello;
|
||||
}
|
||||
|
||||
/** 对端是否声明了可靠任务协议。 */
|
||||
public boolean supportsReliableTask() {
|
||||
return protocolVersion >= VERSION
|
||||
&& features != null
|
||||
&& features.contains(RELIABLE_TASK_V2);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class SubscriptionAccountSnapshot {
|
||||
private Integer accountId;
|
||||
private boolean enabled;
|
||||
private boolean filterHighMultiplier;
|
||||
private String v2ContentBase64;
|
||||
private String v2Sha256;
|
||||
private String clashContentBase64;
|
||||
private String clashSha256;
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class SubscriptionBindingSnapshot {
|
||||
private String publicKeySha256;
|
||||
private Integer accountId;
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
import lombok.ToString;
|
||||
import lion.Message.AbstractMessage;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
@ToString(exclude = {"payloadBase64", "signature"})
|
||||
public class SubscriptionSnapshotMessage extends AbstractMessage {
|
||||
{
|
||||
messageType = SUBSCRIPTION_SNAPSHOT_MESSAGE;
|
||||
}
|
||||
|
||||
private int schemaVersion;
|
||||
private String revision;
|
||||
private long generatedAt;
|
||||
private String payloadBase64;
|
||||
private String payloadSha256;
|
||||
private String signature;
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class SubscriptionSnapshotPayload {
|
||||
private int schemaVersion;
|
||||
private List<SubscriptionAccountSnapshot> accounts = new ArrayList<>();
|
||||
private List<SubscriptionBindingSnapshot> bindings = new ArrayList<>();
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonIgnore;
|
||||
import lion.Message.AbstractMessage;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/**
|
||||
* 可靠任务命令(type=11,主站 → 节点)。
|
||||
*
|
||||
* <p>{@code operationId} 对同一逻辑操作稳定不变:主站重试、断线重发必须复用同一个 id。
|
||||
* 节点按 operationId 去重,绝不重复执行已成功的副作用。
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class TaskCommandMessage extends AbstractMessage {
|
||||
{
|
||||
messageType = TASK_COMMAND_MESSAGE;
|
||||
}
|
||||
|
||||
public static final String ACTION_DOWNLOAD = "DOWNLOAD";
|
||||
public static final String ACTION_DELETE = "DELETE";
|
||||
|
||||
private String operationId;
|
||||
|
||||
/** {@link #ACTION_DOWNLOAD} 或 {@link #ACTION_DELETE}。 */
|
||||
private String action;
|
||||
|
||||
private int gid;
|
||||
|
||||
private String name;
|
||||
|
||||
/**
|
||||
* 必须 {@link JsonIgnore}:Jackson 会把 {@code isDownload()} 当成属性 {@code download} 一并
|
||||
* 序列化,而对端主站使用更严格的 Jackson,默认拒绝未知字段,握手会直接失败。
|
||||
*/
|
||||
@JsonIgnore
|
||||
public boolean isDownload() {
|
||||
return ACTION_DOWNLOAD.equals(action);
|
||||
}
|
||||
|
||||
@JsonIgnore
|
||||
public boolean isDelete() {
|
||||
return ACTION_DELETE.equals(action);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonIgnore;
|
||||
import lion.Message.AbstractMessage;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
/**
|
||||
* 可靠任务报告的确认(type=13,主站 → 节点)。
|
||||
*
|
||||
* <p>主站只在报告落库并提交后回 {@link #RESULT_SUCCESS};落库失败不得回成功。节点收到成功
|
||||
* ACK 才允许移除对应版本的快照,收到非成功结果则保留并重试。
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class TaskReportAckMessage extends AbstractMessage {
|
||||
{
|
||||
messageType = TASK_REPORT_ACK_MESSAGE;
|
||||
}
|
||||
|
||||
/** 报告已提交落库,节点可安全移除该版本快照。 */
|
||||
public static final byte RESULT_SUCCESS = 0;
|
||||
|
||||
/** 主站未提交(例如数据库失败),节点必须保留快照并重试。 */
|
||||
public static final byte RESULT_REJECTED = 1;
|
||||
|
||||
private String reportId;
|
||||
|
||||
private byte result;
|
||||
|
||||
/** 同 {@link TaskCommandMessage#isDownload()}:避免被序列化成未知字段 {@code success}。 */
|
||||
@JsonIgnore
|
||||
public boolean isSuccess() {
|
||||
return result == RESULT_SUCCESS;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package lion.Message.Main;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.Message.AbstractMessage;
|
||||
import lombok.Data;
|
||||
import lombok.NoArgsConstructor;
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 可靠任务状态报告(type=12,节点 → 主站)。
|
||||
*
|
||||
* <p>每条报告有稳定的 {@code reportId} 与单调递增的 {@code revision}。主站按 reportId 幂等
|
||||
* 落库、在提交后回 {@link TaskReportAckMessage};节点只有在收到匹配版本的 ACK 后才移除对应
|
||||
* 快照。重发必须复用同一 reportId 与 revision。
|
||||
*/
|
||||
@Data
|
||||
@NoArgsConstructor
|
||||
public class TaskReportMessage extends AbstractMessage {
|
||||
{
|
||||
messageType = TASK_REPORT_MESSAGE;
|
||||
}
|
||||
|
||||
private String reportId;
|
||||
|
||||
private long revision;
|
||||
|
||||
private GalleryTask[] galleryTasks;
|
||||
|
||||
/** gid → 该任务的活跃操作 operationId,供主站丢弃过期或已删除任务的报告。 */
|
||||
private Map<Integer, String> operationIds = new LinkedHashMap<>();
|
||||
}
|
||||
@@ -5,6 +5,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import io.netty.buffer.ByteBuf;
|
||||
import io.netty.channel.ChannelHandlerContext;
|
||||
import io.netty.handler.codec.ByteToMessageCodec;
|
||||
import io.netty.handler.codec.CorruptedFrameException;
|
||||
import io.netty.handler.codec.EncoderException;
|
||||
import lion.CustomUtil;
|
||||
import lion.Message.Main.*;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
@@ -15,6 +17,9 @@ import java.util.List;
|
||||
@Slf4j
|
||||
public class MessageCodec extends ByteToMessageCodec<AbstractMessage> {
|
||||
|
||||
/** 单帧负载上限,与上游 LengthFieldBasedFrameDecoder 的 100MB 保持一致。 */
|
||||
private static final int MAX_FRAME_BYTES = 100_000_000;
|
||||
|
||||
ObjectMapper objectMapper;
|
||||
|
||||
public MessageCodec(){
|
||||
@@ -23,9 +28,15 @@ public class MessageCodec extends ByteToMessageCodec<AbstractMessage> {
|
||||
|
||||
@Override
|
||||
protected void encode(ChannelHandlerContext channelHandlerContext, AbstractMessage abstractMessage, ByteBuf byteBuf) {
|
||||
// 先序列化、再写帧头:否则中途失败会留下只有类型字节、没有长度和 JSON 的半帧,
|
||||
// 对端的分帧器会为了等长度前缀一直挂住这条连接。
|
||||
byte[] bytes;
|
||||
try {
|
||||
bytes = objectMapper.writeValueAsBytes(abstractMessage);
|
||||
} catch (Exception e) {
|
||||
throw new EncoderException("序列化消息失败:" + e.getMessage(), e);
|
||||
}
|
||||
byteBuf.writeByte(abstractMessage.messageType);
|
||||
|
||||
byte[] bytes = objectMapper.valueToTree(abstractMessage).toString().getBytes(StandardCharsets.UTF_8);
|
||||
byteBuf.writeInt(bytes.length);
|
||||
byteBuf.writeBytes(bytes);
|
||||
}
|
||||
@@ -34,6 +45,8 @@ public class MessageCodec extends ByteToMessageCodec<AbstractMessage> {
|
||||
protected void decode(ChannelHandlerContext channelHandlerContext, ByteBuf byteBuf, List<Object> list) throws Exception {
|
||||
byte messageType = byteBuf.readByte();
|
||||
int length = byteBuf.readInt();
|
||||
if (length < 0 || length > MAX_FRAME_BYTES)
|
||||
throw new CorruptedFrameException("非法的帧长度: " + length);
|
||||
byte[] bytes = new byte[length];
|
||||
byteBuf.readBytes(bytes);
|
||||
final String metadata = new String(bytes, StandardCharsets.UTF_8);
|
||||
@@ -45,6 +58,12 @@ public class MessageCodec extends ByteToMessageCodec<AbstractMessage> {
|
||||
case AbstractMessage.DELETE_GALLERY_MESSAGE -> objectMapper.readValue(metadata, DeleteGalleryMessage.class);
|
||||
case AbstractMessage.IDENTITY_MESSAGE -> objectMapper.readValue(metadata, IdentityMessage.class);
|
||||
case AbstractMessage.MAINTAIN_MESSAGE -> objectMapper.readValue(metadata, MaintainMessage.class);
|
||||
case AbstractMessage.AVAILABLE_CHECK_MESSAGE -> objectMapper.readValue(metadata, AvailableCheckMessage.class);
|
||||
case AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE -> objectMapper.readValue(metadata, SubscriptionSnapshotMessage.class);
|
||||
case AbstractMessage.PROTOCOL_HELLO_MESSAGE -> objectMapper.readValue(metadata, ProtocolHelloMessage.class);
|
||||
case AbstractMessage.TASK_COMMAND_MESSAGE -> objectMapper.readValue(metadata, TaskCommandMessage.class);
|
||||
case AbstractMessage.TASK_REPORT_MESSAGE -> objectMapper.readValue(metadata, TaskReportMessage.class);
|
||||
case AbstractMessage.TASK_REPORT_ACK_MESSAGE -> objectMapper.readValue(metadata, TaskReportAckMessage.class);
|
||||
default -> null;
|
||||
};
|
||||
|
||||
|
||||
@@ -1,219 +1,141 @@
|
||||
package lion;
|
||||
|
||||
import lion.Config.Config;
|
||||
import lion.Http.GalleryArchiveIndex;
|
||||
import lion.Http.GalleryDownloadHandler;
|
||||
import lion.Http.HttpFileServer;
|
||||
import lion.Http.HttpRequestHandler;
|
||||
import lion.Http.HttpResponseModel;
|
||||
import lion.Http.HttpRequestModel;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.*;
|
||||
import java.net.*;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashMap;
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.net.InetAddress;
|
||||
import java.net.UnknownHostException;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
|
||||
/**
|
||||
* 8888 画廊下载服务入口。
|
||||
*
|
||||
* <p>HTTP 外壳已改为 JDK {@code com.sun.net.httpserver.HttpServer}(见
|
||||
* {@link HttpFileServer}),本类只保留启动逻辑、来源 IP 判定,以及供既有测试使用的
|
||||
* 兼容入口。端口仍为 {@value #PORT}。
|
||||
*/
|
||||
@Slf4j
|
||||
public class MultiThreadedHTTPServer {
|
||||
private static final int PORT = 8888;
|
||||
private static final int BUFFER_SIZE = 1024;
|
||||
public final class MultiThreadedHTTPServer {
|
||||
|
||||
/** 对外下载端口;改动会同时影响 Nginx 转发配置,保持固定。 */
|
||||
public static final int PORT = 8888;
|
||||
|
||||
/** 工作线程上限,避免无界线程池在异常流量下耗尽资源。 */
|
||||
private static final int WORKER_THREADS = 16;
|
||||
|
||||
/** 未命中后允许重扫归档目录的最小间隔,让新归档可见但不会被未知 gid 拖成扫描风暴。 */
|
||||
private static final long ARCHIVE_INDEX_RESCAN_MILLIS = 5_000L;
|
||||
|
||||
private static final String GALLERY_ROOT = "/root/gallery/gallery";
|
||||
|
||||
private MultiThreadedHTTPServer() {
|
||||
}
|
||||
|
||||
/**
|
||||
* 兼容旧入口:在独立线程里启动服务并阻塞,直到当前线程被中断。
|
||||
*
|
||||
* <p>历史上由 {@code Main} 用 {@code new Thread(() -> main(null))} 调用,因此这里
|
||||
* 不能立即返回;同时不能阻止 JVM 退出,故服务线程全部为守护线程。
|
||||
*/
|
||||
public static void main(String[] args) {
|
||||
ExecutorService threadPool = Executors.newCachedThreadPool();
|
||||
String real_ip;
|
||||
try{
|
||||
real_ip = InetAddress.getByName("lionwebsite.xyz").getHostAddress();
|
||||
} catch (UnknownHostException ignored){
|
||||
real_ip = "207.60.50.74";
|
||||
}
|
||||
try(ServerSocket serverSocket = new ServerSocket(PORT)) {
|
||||
log.info("Server listening on port {}", PORT);
|
||||
while (true) {
|
||||
Socket clientSocket = serverSocket.accept();
|
||||
String ip = clientSocket.getInetAddress().getHostAddress();
|
||||
if(ip.equals(real_ip)){
|
||||
log.info("Client connected");
|
||||
// 线程池处理下载请求
|
||||
threadPool.submit(() -> handleClientRequest(clientSocket));
|
||||
}else{
|
||||
log.info("unknown ip: " + ip);
|
||||
clientSocket.close();
|
||||
}
|
||||
|
||||
}
|
||||
} catch (IOException e) {
|
||||
log.error("处理http请求时出错,IP:{},ERROR:{}", real_ip, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static void handleClientRequest(Socket clientSocket) {
|
||||
String fileName = "";
|
||||
String realIp;
|
||||
try {
|
||||
BufferedReader requestReader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
|
||||
String requestLine = requestReader.readLine();
|
||||
|
||||
// Parse the request line to get the method and path
|
||||
String[] requestParts = requestLine.split(" ");
|
||||
String method = requestParts[0];
|
||||
Map<String, String> paramMap = parseRequestLine(requestParts[1]);//path
|
||||
log.info(Arrays.toString(requestParts));
|
||||
|
||||
// Only handle GET requests
|
||||
if (method.equals("GET")) {
|
||||
// Set the file path for download
|
||||
File file;
|
||||
if(paramMap.get("AuthCode") != null)
|
||||
if(paramMap.get("AuthCode").equals("alone")){
|
||||
String path = URLDecoder.decode(requestParts[1].split("\\?")[0], StandardCharsets.UTF_8);
|
||||
file = new File(path);
|
||||
}
|
||||
else {
|
||||
String filePath = "/root/gallery/gallery";
|
||||
String path = URLDecoder.decode(requestParts[1].split("\\?")[0], StandardCharsets.UTF_8);
|
||||
if(!path.contains(".")){
|
||||
file = new File("/root/abc");
|
||||
}else {
|
||||
String name = path.substring(0, path.lastIndexOf('.'));
|
||||
name = filePath + name + "/" + name + ".zip";
|
||||
file = new File(name);
|
||||
|
||||
//该文件不存在
|
||||
if(!file.isFile()){
|
||||
String gid = paramMap.get("gid");
|
||||
|
||||
//文件不存在的情况下gid也不存在,直接404
|
||||
if(gid == null)
|
||||
file = new File("/root/abc");
|
||||
|
||||
//gid存在的情况下尝试查找对应的文件
|
||||
else {
|
||||
File[] galleryDirectories = (new File(filePath)).listFiles();
|
||||
|
||||
assert galleryDirectories != null;
|
||||
for (File galleryDirectory : galleryDirectories)
|
||||
if (galleryDirectory.getName().contains(gid)) {
|
||||
file = new File(galleryDirectory.getAbsolutePath(), galleryDirectory.getName() + ".zip");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
else{
|
||||
sendErrorResponse(clientSocket, "403 Forbidden");
|
||||
return;
|
||||
}
|
||||
fileName = file.getName();
|
||||
log.info(file.getAbsolutePath());
|
||||
// Check if the file exists and is readable
|
||||
if (file.exists() && file.isFile() && file.canRead()) {
|
||||
// Get the file length
|
||||
long fileLength = file.length();
|
||||
|
||||
// Get the range information for resuming download
|
||||
long startByte = 0;
|
||||
long endByte = fileLength - 1;
|
||||
String rangeHeader = getRequestHeader(requestReader);
|
||||
if (rangeHeader != null && rangeHeader.startsWith("bytes=")) {
|
||||
String[] rangeValues = rangeHeader.substring(6).split("-");
|
||||
startByte = Long.parseLong(rangeValues[0]);
|
||||
if (rangeValues.length > 1 && !rangeValues[1].isEmpty()) {
|
||||
endByte = Long.parseLong(rangeValues[1]);
|
||||
}
|
||||
realIp = InetAddress.getByName("lionwebsite.xyz").getHostAddress();
|
||||
} catch (UnknownHostException ignored) {
|
||||
realIp = "207.60.50.74";
|
||||
}
|
||||
|
||||
// Send the HTTP response headers
|
||||
OutputStream responseStream = clientSocket.getOutputStream();
|
||||
PrintWriter responseWriter = new PrintWriter(responseStream, true);
|
||||
responseWriter.println("HTTP/1.1 206 Partial Content");
|
||||
responseWriter.println("Content-Type: application/octet-stream");
|
||||
responseWriter.println("Accept-Ranges: bytes");
|
||||
responseWriter.println("Content-Length: " + (endByte - startByte + 1));
|
||||
responseWriter.println("Content-Range: bytes " + startByte + "-" + endByte + "/" + fileLength);
|
||||
responseWriter.println("Content-Disposition: attachment; filename=\"" + fileName + "\"");
|
||||
responseWriter.println();
|
||||
|
||||
// Send the file content
|
||||
try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
|
||||
randomAccessFile.seek(startByte);
|
||||
byte[] buffer = new byte[BUFFER_SIZE];
|
||||
int bytesRead;
|
||||
long bytesRemaining = endByte - startByte + 1;
|
||||
while (bytesRemaining > 0 && (bytesRead = randomAccessFile.read(buffer, 0, (int) Math.min(buffer.length, bytesRemaining))) != -1) {
|
||||
responseStream.write(buffer, 0, bytesRead);
|
||||
bytesRemaining -= bytesRead;
|
||||
try (HttpFileServer server = createServer(realIp)) {
|
||||
server.start();
|
||||
log.info("下载服务已启动,监听端口 {}", server.port());
|
||||
while (!Thread.currentThread().isInterrupted()) {
|
||||
try {
|
||||
Thread.sleep(60_000);
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}catch (SocketException ignore){
|
||||
|
||||
}
|
||||
|
||||
// Close the response output stream
|
||||
responseStream.close();
|
||||
} else {
|
||||
// File not found or not readable, send 404 response
|
||||
sendErrorResponse(clientSocket, "404 Not Found");
|
||||
}
|
||||
} else {
|
||||
// Non-GET requests, send 501 response
|
||||
sendErrorResponse(clientSocket, "501 Not Implemented");
|
||||
}
|
||||
|
||||
// Close the request reader and client socket
|
||||
requestReader.close();
|
||||
clientSocket.close();
|
||||
} catch (IOException e) {
|
||||
log.error("处理文件下载时出错,IP:{}, 文件:{}, ERROR:{}", clientSocket.getInetAddress().getHostAddress(), fileName, e.getMessage());
|
||||
log.error("启动下载服务失败: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static String getRequestHeader(BufferedReader requestReader) throws IOException {
|
||||
String line;
|
||||
while ((line = requestReader.readLine()) != null) {
|
||||
if (line.trim().isEmpty()) {
|
||||
break;
|
||||
/**
|
||||
* 建好(但尚未启动)下载服务。
|
||||
*
|
||||
* @param allowedClientIp 唯一允许的来源 IP;为 null 时不限制来源
|
||||
*/
|
||||
public static HttpFileServer createServer(String allowedClientIp) throws IOException {
|
||||
GalleryArchiveIndex archiveIndex = new GalleryArchiveIndex(
|
||||
new File(GALLERY_ROOT), ARCHIVE_INDEX_RESCAN_MILLIS);
|
||||
HttpRequestHandler handler = withClientFilter(new GalleryDownloadHandler(archiveIndex), allowedClientIp);
|
||||
return HttpFileServer.create(PORT, "/", handler, WORKER_THREADS);
|
||||
}
|
||||
|
||||
if (line.startsWith("Range" + ":")) {
|
||||
return line.substring("Range".length() + 1).trim();
|
||||
/**
|
||||
* 在业务处理器之前插入来源校验:非白名单 IP 直接关闭连接。
|
||||
*
|
||||
* <p>与旧实现一致:被拒的连接不返回任何响应,避免向未知来源暴露服务存在。
|
||||
* 对端 IP 取不到时(例如本地测试直接驱动、或代理未传)默认放行,行为与旧实现
|
||||
* 只在「拿到 IP 且不等于白名单」时拒绝相同。
|
||||
*/
|
||||
static HttpRequestHandler withClientFilter(HttpRequestHandler delegate, String allowedClientIp) {
|
||||
return request -> {
|
||||
String clientIp = request.clientAddress();
|
||||
if (allowedClientIp != null && clientIp != null && !allowedClientIp.equals(clientIp)) {
|
||||
log.info("拒绝未知来源: {}", clientIp);
|
||||
return HttpResponseModel.rejected();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
return delegate.handle(request);
|
||||
};
|
||||
}
|
||||
|
||||
// ---- 以下为既有测试依赖的兼容入口,转发到 lion.Http 的实现 ----
|
||||
|
||||
/** @see GalleryDownloadHandler#isAdminRequest(String) */
|
||||
static boolean isAdminRequest(String authCode) {
|
||||
return GalleryDownloadHandler.isAdminRequest(authCode);
|
||||
}
|
||||
|
||||
/** @see GalleryDownloadHandler#resolveAdminFile(String) */
|
||||
static File resolveAdminFile(String requestPath) {
|
||||
return GalleryDownloadHandler.resolveAdminFile(requestPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* 解析请求行中的路径与查询参数。
|
||||
*
|
||||
* @return 含 {@code path} 的映射;请求行为 null 时返回 null
|
||||
*/
|
||||
public static Map<String, String> parseRequestLine(String requestLine) {
|
||||
Map<String, String> queryParams = new HashMap<>();
|
||||
|
||||
if(requestLine == null)
|
||||
if (requestLine == null)
|
||||
return null;
|
||||
|
||||
if (requestLine.contains("?")) {
|
||||
String[] requestParts = requestLine.split("\\?");
|
||||
String path = requestParts[0];
|
||||
queryParams.put("path", path);
|
||||
|
||||
// Extract the query string after the '?' character
|
||||
String queryString = requestParts[1];
|
||||
String[] paramPairs = queryString.split("&");
|
||||
|
||||
// Split the query string into individual parameter key-value pairs
|
||||
for (String paramPair : paramPairs) {
|
||||
String[] keyValue = paramPair.split("=");
|
||||
if (keyValue.length == 2) {
|
||||
String key = URLDecoder.decode(keyValue[0], StandardCharsets.UTF_8);
|
||||
String value = URLDecoder.decode(keyValue[1], StandardCharsets.UTF_8);
|
||||
queryParams.put(key, value);
|
||||
}
|
||||
}
|
||||
}else{
|
||||
queryParams.put("path", requestLine);
|
||||
}
|
||||
return queryParams;
|
||||
Map<String, String> parameters = lion.Http.HttpFileServer.parseQuery(
|
||||
requestLine.contains("?") ? requestLine.substring(requestLine.indexOf('?') + 1) : null);
|
||||
String path = requestLine.contains("?") ? requestLine.substring(0, requestLine.indexOf('?')) : requestLine;
|
||||
// 路径里的 '+' 是字面量(如 plus+name.zip),只有查询值才把 '+' 当空格。
|
||||
parameters.put("path", lion.Http.HttpFileServer.decodePath(path));
|
||||
return parameters;
|
||||
}
|
||||
|
||||
private static void sendErrorResponse(Socket clientSocket, String statusCode) throws IOException {
|
||||
OutputStream responseStream = clientSocket.getOutputStream();
|
||||
PrintWriter responseWriter = new PrintWriter(responseStream, true);
|
||||
responseWriter.println("HTTP/1.1 " + statusCode);
|
||||
responseWriter.println("Content-Type: text/html");
|
||||
responseWriter.println();
|
||||
responseWriter.println("<h1>" + statusCode + "</h1>");
|
||||
responseStream.close();
|
||||
/** 兼容入口:对单个连接按请求模型处理。 */
|
||||
static void handleClientRequest(java.net.Socket clientSocket) {
|
||||
throw new UnsupportedOperationException(
|
||||
"手工 socket 处理已移除,请使用 createServer 或 HttpFileServer");
|
||||
}
|
||||
|
||||
/** 请求模型入口,便于测试直接驱动(保留旧签名语义)。 */
|
||||
static lion.Http.HttpResponseModel handle(HttpRequestModel request, String allowedClientIp) {
|
||||
GalleryArchiveIndex archiveIndex = new GalleryArchiveIndex(
|
||||
new File(GALLERY_ROOT), ARCHIVE_INDEX_RESCAN_MILLIS);
|
||||
return withClientFilter(new GalleryDownloadHandler(archiveIndex), allowedClientIp).handle(request);
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import lion.ErrorCode.ErrorCode;
|
||||
import cn.hutool.core.io.FileUtil;
|
||||
|
||||
import java.io.File;
|
||||
import java.nio.file.Path;
|
||||
|
||||
public class DeleteService {
|
||||
public static byte deleteAll(String path){
|
||||
@@ -16,4 +17,23 @@ public class DeleteService {
|
||||
else
|
||||
return ErrorCode.IO_ERROR;
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除某个根目录下的子目录,且保证不越界。
|
||||
*
|
||||
* <p>画廊名来自主站下发的消息,直接拼接根路径会让 {@code ../} 或绝对路径
|
||||
* 操纵到根目录之外(例如删掉节点上任意目录)。这里按规范化后的路径判断包含关系,
|
||||
* 越界与空名一律按「找不到」处理,绝不落到磁盘删除。
|
||||
*/
|
||||
public static byte deleteWithin(String rootPath, String galleryName){
|
||||
if (galleryName == null || galleryName.isBlank())
|
||||
return ErrorCode.FILE_NOT_FOUND;
|
||||
|
||||
Path root = new File(rootPath).toPath().toAbsolutePath().normalize();
|
||||
Path target = root.resolve(galleryName).normalize();
|
||||
if (target.equals(root) || !target.startsWith(root))
|
||||
return ErrorCode.FILE_NOT_FOUND;
|
||||
|
||||
return deleteAll(target.toString());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,26 +1,24 @@
|
||||
package lion.Service;
|
||||
|
||||
import io.netty.channel.Channel;
|
||||
import io.netty.channel.DefaultEventLoop;
|
||||
import io.netty.channel.EventLoop;
|
||||
import io.netty.util.concurrent.Promise;
|
||||
import lion.CustomUtil;
|
||||
import lion.Domain.GalleryTask;
|
||||
import cn.hutool.core.io.FileUtil;
|
||||
import cn.hutool.core.util.ZipUtil;
|
||||
import lion.Message.AbstractMessage;
|
||||
import lombok.Data;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.*;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.ScheduledThreadPoolExecutor;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
|
||||
import java.nio.file.*;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.zip.CRC32;
|
||||
import java.util.zip.ZipEntry;
|
||||
import java.util.zip.ZipFile;
|
||||
|
||||
@Slf4j
|
||||
@Data
|
||||
public class DownloadCheckService {
|
||||
Map<Integer, GalleryTask> queue;
|
||||
|
||||
@@ -30,24 +28,91 @@ public class DownloadCheckService {
|
||||
|
||||
ScheduledThreadPoolExecutor convert_thread;
|
||||
|
||||
ArrayList<GalleryTask> compress_queue;
|
||||
final ArrayList<GalleryTask> compress_queue;
|
||||
|
||||
Channel node;
|
||||
/**
|
||||
* 正在压缩(已被压缩线程认领)的 gid。
|
||||
*
|
||||
* <p>删除命令与压缩线程会争用同一份画廊目录:删除请求若在压缩写入归档的过程中
|
||||
* 移走源目录,会留下半个归档或让压缩失败。这里的约定是删除先查该集合,命中即回
|
||||
* 可重试的「忙」,等压缩结束后由主站重试,而不是并发破坏数据。
|
||||
*/
|
||||
final Set<Integer> activeCompressions = ConcurrentHashMap.newKeySet();
|
||||
|
||||
HashMap<Integer, Promise<AbstractMessage>> promises;
|
||||
final Map<Integer, Long> retryAfter = new ConcurrentHashMap<>();
|
||||
|
||||
EventLoop eventLoop;
|
||||
/**
|
||||
* 已校验通过的归档缓存:路径 → (归档大小, 校验时刻)。
|
||||
*
|
||||
* <p>归档一旦落盘就不再变化(发布用 ATOMIC_MOVE),因此同一路径只有在大小变化时
|
||||
* 才需要重新校验。没有它时,每次 {@link #addToQueue} 都会把候选 ZIP 整包读取并
|
||||
* 重算 CRC——主站重连触发 resetUndone 时会对所有未完成任务走一遍,纯属浪费磁盘 I/O。
|
||||
*/
|
||||
final Map<String, VerifiedArchive> verifiedArchives = new ConcurrentHashMap<>();
|
||||
|
||||
public DownloadCheckService(Map<Integer, GalleryTask> queue, HashMap<Integer, Promise<AbstractMessage>> promises){
|
||||
private static final long ARCHIVE_REVERIFY_MILLIS = 3_600_000L;
|
||||
|
||||
private record VerifiedArchive(long size, long verifiedAt) {}
|
||||
|
||||
/**
|
||||
* 状态调度使用的共享监视器就是本实例对象本身。
|
||||
*
|
||||
* <p>删除命令与下载扫描争用同一份任务队列与磁盘目录,两者必须真正互斥:<b>只检查
|
||||
* 「是否正在压缩」的状态判定不足以构成互斥</b>。扫描线程可能在「标 COMPRESSING」与
|
||||
* 「进压缩队列」之间被抢占,而删除此刻看到的队列与 active 集合都还没有该 gid,于是删除
|
||||
* 先把画廊删掉,扫描随后把删除前选中的旧对象排进压缩队列,压缩线程又把归档重新写出来——
|
||||
* 已删除的画廊被「复活」并上报给主站。
|
||||
*
|
||||
* <p>因此 {@link #downloadCheck()}、{@link #addToQueue}、{@link #removeGid}、
|
||||
* {@link #isCompressing} 的状态调度,以及 {@link #compress()} 对队列的「领取」,全部在
|
||||
* 本监视器下完成;{@code ReliableTaskService.executeDelete} 也用同一监视器把
|
||||
* 「查忙 → 删磁盘 → 清队列」包成一个原子段。
|
||||
*
|
||||
* <p>锁定顺序(唯一方向,避免死锁):调用方锁 → 本实例监视器 → {@code compress_queue}。
|
||||
* 真正的压缩 I/O 不在监视器内执行,删除的互斥由 {@link #activeCompressions} 标记保证。
|
||||
*
|
||||
* <p>共享监视器就是 {@code this}({@code DownloadCheckService} 实例)本身,因此
|
||||
* {@code ReliableTaskService.executeDelete} 用 {@code synchronized (downloadCheckService)}
|
||||
* 即可与本类的状态调度取到同一把锁。
|
||||
*/
|
||||
|
||||
/**
|
||||
* 完成通知的专用线程。
|
||||
*
|
||||
* <p>{@link CustomUtil#notifyMe} 是同步 HTTP,原先在 {@link #addToQueue} 里直接调用,
|
||||
* 而 addToQueue 由 Netty IO 线程持节点主锁执行,一次慢请求就会把整个节点的状态上报
|
||||
* 与删除处理一起卡住。改为异步投递后,外呼延迟不再占用锁。
|
||||
*/
|
||||
private final ExecutorService notificationExecutor;
|
||||
|
||||
/** 是否发送完成通知;仅生产构造(启动调度器)时启用,单元测试保持静默不外呼。 */
|
||||
private final boolean notificationsEnabled;
|
||||
|
||||
public DownloadCheckService(Map<Integer, GalleryTask> queue){
|
||||
this(queue, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* 可注入构造:{@code startScheduler=false} 时不启动压缩调度器,也不发送完成通知,
|
||||
* 供离线测试与「只跑旧协议」的节点实例使用。
|
||||
*/
|
||||
public DownloadCheckService(Map<Integer, GalleryTask> queue, boolean startScheduler){
|
||||
this.queue = queue;
|
||||
this.promises = promises;
|
||||
eventLoop = new DefaultEventLoop();
|
||||
compress_queue = new ArrayList<>(0);
|
||||
this.notificationsEnabled = startScheduler;
|
||||
this.notificationExecutor = Executors.newSingleThreadExecutor(r -> {
|
||||
Thread thread = new Thread(r, "storage-node-notify");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
if (startScheduler) {
|
||||
convert_thread = new ScheduledThreadPoolExecutor(1);
|
||||
convert_thread.scheduleAtFixedRate(this::compress, 0, 5, TimeUnit.SECONDS);
|
||||
convert_thread.scheduleWithFixedDelay(this::compress, 0, 5, TimeUnit.SECONDS);
|
||||
}
|
||||
}
|
||||
|
||||
public boolean downloadCheck(){
|
||||
synchronized (this) {
|
||||
if(queue.isEmpty())
|
||||
return false;
|
||||
log.info("下载检查:{}", Arrays.toString(queue.values().toArray()));
|
||||
@@ -72,7 +137,7 @@ public class DownloadCheckService {
|
||||
|
||||
while(fileIterator.hasNext()){
|
||||
File file = fileIterator.next();
|
||||
if(!file.getName().contains(String.valueOf(galleryTask.getGid())))
|
||||
if(!file.isDirectory() || !matchesGid(file.getName(), galleryTask.getGid()))
|
||||
continue;
|
||||
|
||||
galleryTask.setStatus(GalleryTask.DOWNLOADING);
|
||||
@@ -98,77 +163,300 @@ public class DownloadCheckService {
|
||||
|
||||
//压缩队列
|
||||
for(GalleryTask galleryTask: queue.values())
|
||||
if (galleryTask.is_download_complete()) {
|
||||
if (galleryTask.is_download_complete()
|
||||
&& System.currentTimeMillis() >= retryAfter.getOrDefault(galleryTask.getGid(), 0L)) {
|
||||
// 必须以队列里的当前对象为准:本轮遍历的引用可能已被删除流程 removeGid
|
||||
// 移除(删除已把磁盘清掉),把旧对象排进压缩队列会让已删除的画廊复活。
|
||||
GalleryTask current = queue.get(galleryTask.getGid());
|
||||
if (current != galleryTask)
|
||||
continue;
|
||||
galleryTask.setStatus(GalleryTask.COMPRESSING);
|
||||
synchronized (compress_queue) {
|
||||
compress_queue.add(galleryTask);
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 压缩线程:将压缩队列的任务复制一份,进行转换
|
||||
*/
|
||||
public void compress() {
|
||||
if(compress_queue.isEmpty())
|
||||
ArrayList<GalleryTask> galleryTasks;
|
||||
// 领取与扫描入队共用同一监视器:原子地把队列交给压缩线程并标记 active,
|
||||
// 使删除再也看不到「已标 COMPRESSING 但尚未 active」的空档。
|
||||
synchronized (this) {
|
||||
if (compress_queue.isEmpty())
|
||||
return;
|
||||
ReentrantLock reentrantLock = new ReentrantLock();
|
||||
reentrantLock.lock();
|
||||
ArrayList<GalleryTask> galleryTasks = new ArrayList<>(compress_queue);
|
||||
galleryTasks = new ArrayList<>(compress_queue);
|
||||
compress_queue.clear();
|
||||
reentrantLock.unlock();
|
||||
for (GalleryTask claimed : galleryTasks)
|
||||
activeCompressions.add(claimed.getGid());
|
||||
}
|
||||
for (GalleryTask galleryTask : galleryTasks) {
|
||||
Path temporary = null;
|
||||
try {
|
||||
log.info("开始压缩:{}", galleryTask.getName());
|
||||
//创建文件夹
|
||||
File file = new File(storagePath + galleryTask.getName());
|
||||
if (file.isDirectory() || file.mkdirs()) {
|
||||
log.info("{}文件夹创建成功", galleryTask.getName());
|
||||
} else {
|
||||
log.error("{}文件夹创建失败", galleryTask.getName());
|
||||
continue;
|
||||
Path directory = Paths.get(storagePath, galleryTask.getName());
|
||||
Files.createDirectories(directory);
|
||||
Path archive = directory.resolve(galleryTask.getName() + ".zip");
|
||||
temporary = Files.createTempFile(directory, ".compress-", ".zip.part");
|
||||
ZipUtil.zip(galleryTask.getPath(), temporary.toString());
|
||||
if (!isValidArchive(temporary.toFile()))
|
||||
throw new IOException("压缩包校验失败");
|
||||
// Publish only a closed, verified archive. A crash leaves a .part file.
|
||||
try {
|
||||
Files.move(temporary, archive, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
|
||||
} catch (AtomicMoveNotSupportedException e) {
|
||||
Files.move(temporary, archive, StandardCopyOption.REPLACE_EXISTING);
|
||||
}
|
||||
|
||||
//生成压缩包
|
||||
ZipUtil.zip(galleryTask.getPath(), storagePath + galleryTask.getName() + "/" + galleryTask.getName() + ".zip");
|
||||
log.info("{}压缩完成", galleryTask.getName());
|
||||
|
||||
FileUtil.del(galleryTask.getPath());
|
||||
temporary = null;
|
||||
galleryTask.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
} catch (Exception e){
|
||||
log.error("{}压缩失败:{}", galleryTask, e.getMessage());
|
||||
retryAfter.remove(galleryTask.getGid());
|
||||
if (!FileUtil.del(galleryTask.getPath()))
|
||||
log.warn("压缩已完成,但源目录清理失败: {}", galleryTask.getPath());
|
||||
log.info("{}压缩完成", galleryTask.getName());
|
||||
} catch (Exception e) {
|
||||
// Keep the source and restore an existing, retryable protocol state.
|
||||
if (!galleryTask.is_compress_complete()) {
|
||||
retryAfter.put(galleryTask.getGid(), System.currentTimeMillis() + 30_000);
|
||||
galleryTask.setStatus(GalleryTask.DOWNLOAD_COMPLETE);
|
||||
}
|
||||
log.error("{}压缩或清理失败,源文件保留,稍后可重试", galleryTask.getName(), e);
|
||||
} finally {
|
||||
activeCompressions.remove(galleryTask.getGid());
|
||||
if (temporary != null) {
|
||||
try { Files.deleteIfExists(temporary); }
|
||||
catch (IOException e) { log.warn("清理压缩临时文件失败: {}", temporary, e); }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 检查改任务是否为已完成任务,如已完成则返回true,若未完成则加入队列
|
||||
* @return true if compress complete, false otherwise
|
||||
* 该 gid 是否正在压缩(含已排队待压缩)。
|
||||
*
|
||||
* <p>删除与压缩是互斥的:命中时调用方应回可重试的「忙」,而不是继续动磁盘。
|
||||
*/
|
||||
public boolean addToQueue(GalleryTask galleryTask){
|
||||
//是否含有名字,进行中任务一般有名字,没有名字则肯定为初始任务,存在名字至少在下载路径出现过
|
||||
if(galleryTask.getName() == null || galleryTask.getName().isEmpty()){
|
||||
queue.putIfAbsent(galleryTask.getGid(), galleryTask);
|
||||
public boolean isCompressing(int gid){
|
||||
synchronized (this) {
|
||||
if (activeCompressions.contains(gid))
|
||||
return true;
|
||||
for (GalleryTask queued : compress_queue)
|
||||
if (queued.getGid() == gid)
|
||||
return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
//查询hah下载路径中,是否存在该任务下载路径,存在则为下载中或下载完成任务,加入队列
|
||||
if(new File(downloadPath + galleryTask.getGid()).isDirectory()){
|
||||
queue.putIfAbsent(galleryTask.getGid(), galleryTask);
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* 从压缩队列与任务队列中摘除该 gid。
|
||||
*
|
||||
* <p>用于删除成功后清理本节点内部状态,避免已删除任务继续被扫描或压缩。
|
||||
*/
|
||||
public void removeGid(int gid){
|
||||
synchronized (this) {
|
||||
queue.remove(gid);
|
||||
retryAfter.remove(gid);
|
||||
compress_queue.removeIf(task -> task.getGid() == gid);
|
||||
}
|
||||
}
|
||||
|
||||
//查询存放路径中是否含有该任务的压缩包,存在则为下载完成任务
|
||||
if(new File(storagePath + galleryTask.getName() + "/" + galleryTask.getName() + ".zip").exists()){
|
||||
galleryTask.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
CustomUtil.notifyMe(String.format("任务:%s在添加时已下载完成,更新任务状态", galleryTask.getName()));
|
||||
/**
|
||||
* 关闭本服务持有的线程资源。
|
||||
*
|
||||
* <p>压缩调度器与通知线程都是守护线程,进程退出不会因此挂住;但测试与将来可能的
|
||||
* 热重启必须能显式释放,否则每个实例都会留下一个仍在跑压缩的线程。
|
||||
*/
|
||||
public void shutdown(){
|
||||
if (convert_thread != null)
|
||||
convert_thread.shutdownNow();
|
||||
notificationExecutor.shutdownNow();
|
||||
}
|
||||
|
||||
static boolean isValidArchive(File file) {
|
||||
if (!file.isFile())
|
||||
return false;
|
||||
try (ZipFile zip = new ZipFile(file)) {
|
||||
if (zip.size() == 0)
|
||||
return false;
|
||||
byte[] buffer = new byte[8192];
|
||||
Enumeration<? extends ZipEntry> entries = zip.entries();
|
||||
while (entries.hasMoreElements()) {
|
||||
ZipEntry entry = entries.nextElement();
|
||||
if (entry.isDirectory())
|
||||
continue;
|
||||
CRC32 crc = new CRC32();
|
||||
long size = 0;
|
||||
try (InputStream input = zip.getInputStream(entry)) {
|
||||
int count;
|
||||
while ((count = input.read(buffer)) != -1) {
|
||||
crc.update(buffer, 0, count);
|
||||
size += count;
|
||||
}
|
||||
}
|
||||
if (size != entry.getSize() || crc.getValue() != entry.getCrc())
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
} catch (IOException e) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 受理或刷新一个任务,并按 gid 用磁盘事实重建它的状态。
|
||||
*
|
||||
* <p>与扫描、删除共用同一监视器:这里既要读磁盘目录,又要写任务队列,如果与删除交叉执行,
|
||||
* 会出现「删除已清空磁盘,入队仍按删除前的目录刷新出下载完成」的矛盾状态。
|
||||
*
|
||||
* <p>重建顺序是「归档优先」:gid 稳定而目录名不稳定(下载器会追加后缀),因此先按 gid
|
||||
* 找到校验有效的 ZIP 判为压缩完成,再回退到下载目录判断下载中/下载完成。只有归档有效才算
|
||||
* 终态,损坏或空归档一律不得被当成完成。
|
||||
*/
|
||||
public GalleryTask addToQueue(GalleryTask galleryTask){
|
||||
return addToQueue(galleryTask, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* 受理或刷新一个任务的内部实现。
|
||||
*
|
||||
* <p>{@code notifyOnAlreadyStored=false} 供启动恢复使用:恢复时对每个任务做一次磁盘事实
|
||||
* 重建,命中的是「重启前就已经存在的归档」,而不是主站刚刚新下的任务,不该为它再发一遍
|
||||
* 「添加时已下载完成」通知。
|
||||
*/
|
||||
GalleryTask addToQueue(GalleryTask galleryTask, boolean notifyOnAlreadyStored){
|
||||
synchronized (this) {
|
||||
GalleryTask active = queue.get(galleryTask.getGid());
|
||||
// 只有真正「正在压缩」才短路:压缩线程正持有该对象写归档,替换它会丢状态。
|
||||
if (active != null && active.is_compressing())
|
||||
return active;
|
||||
retryAfter.remove(galleryTask.getGid());
|
||||
// A reconnect can resend a task whose name is stale (for example, the
|
||||
// downloader appended a resolution suffix). Resolve completed archives
|
||||
// by gid first, because the gid is stable while the directory name is not.
|
||||
File storedDirectory = findStoredDirectoryByGid(galleryTask.getGid());
|
||||
if(storedDirectory != null){
|
||||
// 已有有效归档即为终态;保留队列里那个仍然活跃的对象,避免同一 gid 出现两个实例。
|
||||
GalleryTask target = active == null ? galleryTask : active;
|
||||
if (active != null)
|
||||
queue.remove(galleryTask.getGid());
|
||||
target.setName(storedDirectory.getName());
|
||||
target.setPath(storedDirectory.getPath());
|
||||
target.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
if (notifyOnAlreadyStored)
|
||||
notifyAlreadyStored(target.getName());
|
||||
return target;
|
||||
}
|
||||
|
||||
// If the task is still downloading, retain the actual directory name so
|
||||
// subsequent progress and compression use the same identity.
|
||||
File downloadingDirectory = findDirectoryByGid(new File(downloadPath), galleryTask.getGid());
|
||||
if(downloadingDirectory != null){
|
||||
GalleryTask queuedTask = queue.get(galleryTask.getGid());
|
||||
if(queuedTask == null){
|
||||
refreshDownloadingTask(galleryTask, downloadingDirectory);
|
||||
GalleryTask existing = queue.putIfAbsent(galleryTask.getGid(), galleryTask);
|
||||
return existing == null ? galleryTask : existing;
|
||||
}
|
||||
if(!queuedTask.is_compressing() && !queuedTask.is_compress_complete())
|
||||
refreshDownloadingTask(queuedTask, downloadingDirectory);
|
||||
return queuedTask;
|
||||
}
|
||||
|
||||
GalleryTask queuedTask = queue.putIfAbsent(galleryTask.getGid(), galleryTask);
|
||||
return queuedTask == null ? galleryTask : queuedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private void refreshDownloadingTask(GalleryTask galleryTask, File downloadingDirectory){
|
||||
galleryTask.setName(downloadingDirectory.getName());
|
||||
File[] pages = downloadingDirectory.listFiles((dir, name) -> !name.equals("galleryinfo.txt"));
|
||||
galleryTask.setProceeding(pages == null ? 0 : pages.length);
|
||||
if(new File(downloadingDirectory, "galleryinfo.txt").isFile()){
|
||||
galleryTask.setStatus(GalleryTask.DOWNLOAD_COMPLETE);
|
||||
galleryTask.setPath(downloadingDirectory.getPath());
|
||||
}else{
|
||||
galleryTask.setStatus(GalleryTask.DOWNLOADING);
|
||||
}
|
||||
}
|
||||
|
||||
private File findStoredDirectoryByGid(int gid){
|
||||
File storageDirectory = new File(storagePath);
|
||||
File[] directories = storageDirectory.listFiles(File::isDirectory);
|
||||
if(directories == null)
|
||||
return null;
|
||||
|
||||
for(File directory : directories){
|
||||
if(matchesGid(directory.getName(), gid)
|
||||
&& isVerifiedArchive(new File(directory, directory.getName() + ".zip")))
|
||||
return directory;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 按 gid 在存储目录中解析真实归档目录名。
|
||||
*
|
||||
* <p>删除命令需要确切目录名:主站下发的 name 可能带下载器追加的分辨率后缀,直接用它会
|
||||
* 删错对象或报找不到。先按 gid 找已归档目录,再回退到下载目录,最后才用主站给的名字。
|
||||
*/
|
||||
public String findDirectoryNameByGid(int gid){
|
||||
File stored = findStoredDirectoryByGid(gid);
|
||||
if (stored != null)
|
||||
return stored.getName();
|
||||
File downloading = findDirectoryByGid(new File(downloadPath), gid);
|
||||
return downloading == null ? null : downloading.getName();
|
||||
}
|
||||
|
||||
/**
|
||||
* 压缩包校验带缓存:归档按大小不变,校验结果可复用。
|
||||
* 大小变化(重新压缩)或距上次校验超过一小时才重算,避免重复整包读取。
|
||||
*/
|
||||
private boolean isVerifiedArchive(File archive){
|
||||
if(!archive.isFile())
|
||||
return false;
|
||||
long size = archive.length();
|
||||
long now = System.currentTimeMillis();
|
||||
VerifiedArchive cached = verifiedArchives.get(archive.getPath());
|
||||
if(cached != null && cached.size() == size && now - cached.verifiedAt() < ARCHIVE_REVERIFY_MILLIS)
|
||||
return true;
|
||||
if(!isValidArchive(archive)){
|
||||
verifiedArchives.remove(archive.getPath());
|
||||
return false;
|
||||
}
|
||||
verifiedArchives.put(archive.getPath(), new VerifiedArchive(size, now));
|
||||
return true;
|
||||
}
|
||||
|
||||
private void notifyAlreadyStored(String taskName){
|
||||
if(!notificationsEnabled)
|
||||
return;
|
||||
notificationExecutor.execute(() ->
|
||||
CustomUtil.notifyMe(String.format("任务:%s在添加时已下载完成,更新任务状态", taskName)));
|
||||
}
|
||||
|
||||
//异常情况,发送通知
|
||||
CustomUtil.notifyMe(String.format("任务:%s存在名字,但是下载路径为空且不存在压缩包", galleryTask.getName()));
|
||||
private File findDirectoryByGid(File parentDirectory, int gid){
|
||||
File[] directories = parentDirectory.listFiles(File::isDirectory);
|
||||
if(directories == null)
|
||||
return null;
|
||||
|
||||
for(File directory : directories)
|
||||
if(matchesGid(directory.getName(), gid))
|
||||
return directory;
|
||||
return null;
|
||||
}
|
||||
|
||||
private boolean matchesGid(String name, int gid){
|
||||
String gidMarker = "[" + gid;
|
||||
int markerIndex = name.lastIndexOf(gidMarker);
|
||||
if(markerIndex < 0)
|
||||
return false;
|
||||
|
||||
int suffixIndex = markerIndex + gidMarker.length();
|
||||
return suffixIndex < name.length()
|
||||
&& (name.charAt(suffixIndex) == ']' || name.charAt(suffixIndex) == '-');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package lion.Service;
|
||||
|
||||
/**
|
||||
* 节点侧命令结果码。
|
||||
*
|
||||
* <p>前两个值与主站 {@code Error.ErrorCode} 保持一致,直接复用旧协议的语义;
|
||||
* {@link #BUSY_COMPRESSING} 是可靠任务协议新增的可重试拒绝:目标正在压缩,此刻删除或
|
||||
* 覆盖会把半成品归档暴露出去,主站应以同一 operationId 稍后重试,节点不会记为永久成功。
|
||||
*/
|
||||
public class NodeErrorCode {
|
||||
|
||||
/** 命令成功。 */
|
||||
public static final byte SUCCESS = 0;
|
||||
|
||||
/** 立即重试不一定成功,但状态可能变化,属可重试失败。 */
|
||||
public static final byte IO_ERROR = 1;
|
||||
|
||||
/** 目标不存在。对删除而言等价于「已达成目标」,是终态。 */
|
||||
public static final byte FILE_NOT_FOUND = 2;
|
||||
|
||||
/** 目标正在压缩,当前无法处理;可重试,不得记为永久结果。 */
|
||||
public static final byte BUSY_COMPRESSING = 3;
|
||||
|
||||
/** 命令本身不合法(缺 operationId 或未知 action),重试不会改变结果。 */
|
||||
public static final byte INVALID_COMMAND = 4;
|
||||
|
||||
/** 终态结果才写入回执,重试直接回放,不再执行副作用。 */
|
||||
public static boolean isTerminal(byte result) {
|
||||
return result == SUCCESS || result == FILE_NOT_FOUND || result == INVALID_COMMAND;
|
||||
}
|
||||
|
||||
private NodeErrorCode() {
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package lion.Service;
|
||||
|
||||
import io.netty.channel.Channel;
|
||||
import io.netty.util.AttributeKey;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
/**
|
||||
* 「哪条通道是主站」的唯一登记处。
|
||||
*
|
||||
* <p>历史缺陷:主站重连期间可能出现多条已认证通道,节点先引用 X,随后被后建立的 Y
|
||||
* 覆盖;Y 断开时把引用清空,而仍可用的 X 继续发送心跳与任务消息,节点却因为引用为空
|
||||
* 不再上报任何任务状态。主站侧的存活探测又能收到 X 上的响应,双方于是都判定
|
||||
* 「连接正常」,未完成任务的状态就此永久卡住。
|
||||
*
|
||||
* <p>因此把引用管理收敛到这里,并保留全部已认证通道:首选通道一旦失效,立即回退到
|
||||
* 其它仍可用的已认证通道;引用被误清时,任何一条已认证通道上的消息也能把它恢复。
|
||||
* 这样「有已认证通道在,就一定能上报」成为不变式,不必等人工重连或下一个探测周期。
|
||||
*
|
||||
* <p>「已认证」的口径是「该通道发来过只有主站会发的消息类型」。事故中出问题的那条通道
|
||||
* 正是靠可用性探测消息证明自己仍在工作的,若只认 {@code IdentityMessage},这条通道
|
||||
* 就永远无法重新取得引用。节点数据端口本就只应面向主站开放,凭证由网络边界提供。
|
||||
*/
|
||||
@Slf4j
|
||||
public class PrimaryChannelTracker {
|
||||
|
||||
static final AttributeKey<Boolean> AUTHENTICATED = AttributeKey.valueOf("primaryAuthenticated");
|
||||
|
||||
/** 全部已认证通道,用于在首选通道失效时立即回退。 */
|
||||
private final Set<Channel> authenticated = ConcurrentHashMap.newKeySet();
|
||||
|
||||
private volatile Channel primary;
|
||||
|
||||
/** 收到只有主站会发的消息:认证该通道;若当前引用不是它,则改指向它。 */
|
||||
public void onPrimaryMessage(Channel channel) {
|
||||
markAuthenticated(channel);
|
||||
if (primary == null) {
|
||||
primary = channel;
|
||||
log.info("主站通道引用已建立");
|
||||
} else if (channel != primary) {
|
||||
switchTo(channel);
|
||||
}
|
||||
}
|
||||
|
||||
private void markAuthenticated(Channel channel) {
|
||||
if (Boolean.TRUE.equals(channel.attr(AUTHENTICATED).get()))
|
||||
return;
|
||||
channel.attr(AUTHENTICATED).set(Boolean.TRUE);
|
||||
channel.closeFuture().addListener(ignored -> unregister(channel));
|
||||
authenticated.add(channel);
|
||||
}
|
||||
|
||||
/** 通道退出:移出已认证集合,必要时把引用移交给其它仍可用的通道。 */
|
||||
public void unregister(Channel channel) {
|
||||
authenticated.remove(channel);
|
||||
if (channel != primary)
|
||||
return;
|
||||
primary = firstUsable();
|
||||
}
|
||||
|
||||
private void switchTo(Channel channel) {
|
||||
log.info("主站通道引用已切换到仍在工作的通道");
|
||||
primary = channel;
|
||||
}
|
||||
|
||||
public Channel current() {
|
||||
Channel channel = primary;
|
||||
if (channel != null && channel.isActive())
|
||||
return channel;
|
||||
// 首选通道已失效:立即回退,避免出现「有可用通道却无人上报」的静默状态。
|
||||
Channel fallback = firstUsable();
|
||||
if (fallback != null && fallback != channel) {
|
||||
log.info("主站首选通道不可用,回退到其它已认证通道");
|
||||
primary = fallback;
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
/** 引用是否存在且仍可写。 */
|
||||
public boolean usable() {
|
||||
return current() != null;
|
||||
}
|
||||
|
||||
private Channel firstUsable() {
|
||||
for (Channel channel : authenticated)
|
||||
if (channel.isActive())
|
||||
return channel;
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package lion.Service;
|
||||
|
||||
/**
|
||||
* 「是否该主动重新唤起主站」的判定。
|
||||
*
|
||||
* <p>节点持有待上报的任务状态却没有可用通道时,只等主站下一个探测周期可能长达半小时;
|
||||
* 期间用户在下载器里看到的是任务永远停在「已提交」。这里按固定间隔限流地允许重连,
|
||||
* 既能在几十秒内自愈,又不会在主站确实离线时形成重连风暴。
|
||||
*
|
||||
* <p>把判定单独拆出来是为了能直接对时间轴写断言,不必启动 Netty。
|
||||
*/
|
||||
public class RekickPolicy {
|
||||
|
||||
/** 两次主动唤起之间的最小间隔。 */
|
||||
static final long MIN_INTERVAL_MILLIS = 30_000L;
|
||||
|
||||
private long lastAttemptMillis;
|
||||
|
||||
/**
|
||||
* 本轮是否应当唤起主站。
|
||||
*
|
||||
* @param pendingReports 是否有待上报的任务状态
|
||||
* @param channelUsable 是否已存在可用通道
|
||||
* @param nowMillis 当前时间
|
||||
*/
|
||||
public boolean shouldRekick(boolean pendingReports, boolean channelUsable, long nowMillis) {
|
||||
if (!pendingReports || channelUsable)
|
||||
return false;
|
||||
if (lastAttemptMillis != 0 && nowMillis - lastAttemptMillis < MIN_INTERVAL_MILLIS)
|
||||
return false;
|
||||
lastAttemptMillis = nowMillis;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,318 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.Message.Main.TaskCommandMessage;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* 可靠任务协议在节点侧的执行与状态编排。
|
||||
*
|
||||
* <p>它把三样东西绑在一起:任务队列(运行期事实)、{@link TaskJournal}(持久事实)、
|
||||
* {@link TaskReporter}(上报与 ACK 生命周期)。命令先查回执再执行、成功副作用先落盘再回执;
|
||||
* 报告先落盘再发送、只有匹配版本的 ACK 才清理任务。
|
||||
*
|
||||
* <p>所有对日志与队列的修改都在本类的锁下完成,保证「落盘」与「内存状态」不会分叉。
|
||||
*/
|
||||
@Slf4j
|
||||
public class ReliableTaskService {
|
||||
|
||||
private final TaskJournal journal;
|
||||
|
||||
private final TaskReporter reporter;
|
||||
|
||||
private final DownloadCheckService downloadCheckService;
|
||||
|
||||
/** 任务队列与日志共享的真实来源;节点扫描会就地更新其中的对象。 */
|
||||
private final Map<Integer, GalleryTask> queue;
|
||||
|
||||
private final Path storagePath;
|
||||
|
||||
/** gid → 当前生效的下发操作 id,随命令更新并写进报告。 */
|
||||
private final Map<Integer, String> operationIds = new LinkedHashMap<>();
|
||||
|
||||
private final Object lock = new Object();
|
||||
|
||||
public ReliableTaskService(Path journalDirectory, Map<Integer, GalleryTask> queue,
|
||||
DownloadCheckService downloadCheckService, Path storagePath) {
|
||||
this(new TaskJournal(journalDirectory), queue, downloadCheckService, storagePath);
|
||||
}
|
||||
|
||||
ReliableTaskService(TaskJournal journal, Map<Integer, GalleryTask> queue,
|
||||
DownloadCheckService downloadCheckService, Path storagePath) {
|
||||
this.journal = journal;
|
||||
this.reporter = new TaskReporter(journal);
|
||||
this.queue = queue;
|
||||
this.downloadCheckService = downloadCheckService;
|
||||
this.storagePath = storagePath;
|
||||
}
|
||||
|
||||
public TaskReporter reporter() {
|
||||
return reporter;
|
||||
}
|
||||
|
||||
public TaskJournal journal() {
|
||||
return journal;
|
||||
}
|
||||
|
||||
/**
|
||||
* 启动恢复。
|
||||
*
|
||||
* <p>先读回日志,再按磁盘事实重建每个任务的状态,最后并入内存队列。
|
||||
*
|
||||
* <p>这里必须用「有效 ZIP / 下载目录」的真实校验来重建,而不是把中断的压缩简单改成
|
||||
* 「下载完成」了事:{@code compress()} 已经原子发布 ZIP 并删掉源目录之后、日志尚未落盘时
|
||||
* 崩溃,重启后就是一个「日志说压缩中、磁盘上只剩归档、下载目录空无一物」的三方矛盾。
|
||||
* 若只降级为下载完成,扫描会因下载目录为空而提前返回、stableop 重投又只回放回执,
|
||||
* 任务就永久停在等待压缩。按 gid 命中有效归档即判为压缩完成(同时补回 name/path),
|
||||
* 命中源目录则恢复下载中/下载完成与进度,两者都没有时保留任务、不捏造完成。
|
||||
*
|
||||
* <p>恢复只重建本地事实,绝不重新提交上游下载;operationId 与待 ACK 报告原样保留,
|
||||
* 任务的 revision 过滤仍由 {@link #applyAck} 负责。
|
||||
*/
|
||||
public void restore() throws IOException {
|
||||
journal.load();
|
||||
synchronized (lock) {
|
||||
for (TaskJournal.JournalTask stored : journal.tasks().values()) {
|
||||
GalleryTask task = stored.toGalleryTask();
|
||||
if (task.getStatus() == GalleryTask.COMPRESSING)
|
||||
// 退回到可被扫描重排的状态;此处尚未入队,addToQueue 不会因「正在压缩」短路。
|
||||
task.setStatus(GalleryTask.DOWNLOAD_COMPLETE);
|
||||
// 用磁盘事实重建:有效归档 → 压缩完成;源目录 → 下载中/完成(带进度);
|
||||
// 都没有时保留任务原状。恢复期的命中不应再发「添加时已完成」通知。
|
||||
GalleryTask restored = downloadCheckService.addToQueue(task, false);
|
||||
queue.putIfAbsent(restored.getGid(), restored);
|
||||
if (stored.operationId != null)
|
||||
operationIds.put(stored.gid, stored.operationId);
|
||||
}
|
||||
// 把重建出的磁盘事实立即落盘:日志停在 COMPRESSING 时它必须被新的压缩完成状态与更高
|
||||
// revision 取代。这既让后续报告不再带着过期的压缩中状态上报,也让崩溃前生成的旧快照
|
||||
// ACK 在 removeTaskIfRevision 处因版本过期而被拒绝,清不掉刚恢复的新事实。
|
||||
journal.syncTasks(List.copyOf(queue.values()), operationIds);
|
||||
log.info("节点重启恢复: 任务={}, 待 ACK 报告={}", queue.size(), journal.pendingReports().size());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 执行一条可靠任务命令。
|
||||
*
|
||||
* <p>返回的字节就是回给主站的结果码。已终态成功的 operationId 直接回放原结果,
|
||||
* 不再触碰磁盘或队列;可重试失败不写回执,主站以同一 operationId 重试即可。
|
||||
*/
|
||||
public byte handleCommand(TaskCommandMessage command) {
|
||||
if (command == null || command.getOperationId() == null || command.getOperationId().isBlank()
|
||||
|| command.getAction() == null)
|
||||
return NodeErrorCode.INVALID_COMMAND;
|
||||
|
||||
synchronized (lock) {
|
||||
TaskJournal.CommandReceipt receipt = journal.receipt(command.getOperationId());
|
||||
if (receipt != null) {
|
||||
log.info("命令已执行过,回放回执: operationId={}, result={}", command.getOperationId(), receipt.result);
|
||||
return receipt.result;
|
||||
}
|
||||
|
||||
try {
|
||||
if (command.isDownload())
|
||||
return executeDownload(command);
|
||||
if (command.isDelete())
|
||||
return executeDelete(command);
|
||||
return NodeErrorCode.INVALID_COMMAND;
|
||||
} catch (IOException e) {
|
||||
// 落盘失败时绝不能把副作用当成成功:回可重试错误,主站会以同一 id 再来。
|
||||
log.error("可靠命令落盘失败,回可重试错误: operationId={}", command.getOperationId(), e);
|
||||
return NodeErrorCode.IO_ERROR;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private byte executeDownload(TaskCommandMessage command) throws IOException {
|
||||
GalleryTask incoming = new GalleryTask();
|
||||
incoming.setGid(command.getGid());
|
||||
incoming.setName(command.getName());
|
||||
|
||||
GalleryTask active = downloadCheckService.addToQueue(incoming);
|
||||
operationIds.put(active.getGid(), command.getOperationId());
|
||||
queue.putIfAbsent(active.getGid(), active);
|
||||
|
||||
// 先把入队事实写进日志,再写回执:回执一旦落盘就代表「已受理」,重启后必须还能看到任务。
|
||||
journal.bumpAndPutTask(active, command.getOperationId());
|
||||
journal.putReceipt(TaskJournal.CommandReceipt.of(
|
||||
command.getOperationId(), TaskCommandMessage.ACTION_DOWNLOAD,
|
||||
active.getGid(), active.getName(), NodeErrorCode.SUCCESS));
|
||||
return NodeErrorCode.SUCCESS;
|
||||
}
|
||||
|
||||
private byte executeDelete(TaskCommandMessage command) throws IOException {
|
||||
int gid = command.getGid();
|
||||
|
||||
// 删除的「查忙 → 删磁盘 → 清队列」必须与扫描的「选任务 → 标压缩中 → 排进压缩队列」
|
||||
// 原子互斥。只查一次 isCompressing 会在「已标 COMPRESSING、尚未进压缩队列」的窗口里
|
||||
// 判定为不忙:删除把画廊删掉,扫描随后仍把删除前选中的旧对象排进压缩队列,压缩线程
|
||||
// 又把归档重新写出来,已删除的画廊被复活并上报。整段持同一监视器后该窗口不复存在。
|
||||
synchronized (downloadCheckService) {
|
||||
// 压缩中删除会破坏正在进行中的归档写入,先回可重试的「忙」,不写回执。
|
||||
if (downloadCheckService.isCompressing(gid)) {
|
||||
log.info("任务正在压缩,删除请求延后重试: gid={}", gid);
|
||||
return NodeErrorCode.BUSY_COMPRESSING;
|
||||
}
|
||||
|
||||
String directoryName = resolveDirectoryName(gid, command.getName());
|
||||
byte result = directoryName == null
|
||||
? NodeErrorCode.FILE_NOT_FOUND
|
||||
: deleteGallery(directoryName);
|
||||
|
||||
if (!NodeErrorCode.isTerminal(result))
|
||||
return result;
|
||||
|
||||
downloadCheckService.removeGid(gid);
|
||||
queue.remove(gid);
|
||||
operationIds.remove(gid);
|
||||
journal.removeTask(gid);
|
||||
journal.putReceipt(TaskJournal.CommandReceipt.of(
|
||||
command.getOperationId(), TaskCommandMessage.ACTION_DELETE,
|
||||
gid, directoryName, result));
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
private byte deleteGallery(String name) {
|
||||
File target = storagePath.resolve(name).toFile();
|
||||
if (!target.isDirectory())
|
||||
return NodeErrorCode.FILE_NOT_FOUND;
|
||||
return DeleteService.deleteWithin(storagePath.toString(), name);
|
||||
}
|
||||
|
||||
/** 优先按 gid 解析真实目录名;主站的名称可能因下载器追加后缀而失真。 */
|
||||
private String resolveDirectoryName(int gid, String fallbackName) {
|
||||
String byGid = downloadCheckService.findDirectoryNameByGid(gid);
|
||||
if (byGid != null)
|
||||
return byGid;
|
||||
if (fallbackName != null && !fallbackName.isBlank()
|
||||
&& storagePath.resolve(fallbackName).toFile().isDirectory())
|
||||
return fallbackName;
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 把扫描后的队列状态对齐到日志。返回是否产生了需要上报的变化。
|
||||
*/
|
||||
public boolean refreshJournalFromQueue() throws IOException {
|
||||
synchronized (lock) {
|
||||
return journal.syncTasks(List.copyOf(queue.values()), operationIds);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 生成一份待发送报告并先落盘。无任务可报时返回空。
|
||||
*
|
||||
* <p>若当前日志中的任务版本已被某份待 ACK 快照完整覆盖,则不重复生成:断线重连时应当
|
||||
* 原样重发旧快照,而不是为同一批内容再堆一份新快照,否则主站要处理无限增长的重复报告。
|
||||
*/
|
||||
public Optional<TaskJournal.ReportSnapshot> buildReport() throws IOException {
|
||||
synchronized (lock) {
|
||||
// 报告必须反映最新磁盘事实:扫描线程会就地更新运行期任务,先对齐日志再取快照,
|
||||
// 否则会发出一份落后于磁盘的状态。重复调用是幂等的,已同步时不会产生写入。
|
||||
journal.syncTasks(List.copyOf(queue.values()), operationIds);
|
||||
if (journal.tasks().isEmpty())
|
||||
return Optional.empty();
|
||||
if (isCoveredByPendingSnapshot())
|
||||
return Optional.empty();
|
||||
TaskJournal.ReportSnapshot snapshot = reporter.persistSnapshot(
|
||||
List.copyOf(journal.tasks().values()), operationIds);
|
||||
return Optional.of(snapshot);
|
||||
}
|
||||
}
|
||||
|
||||
/** 当前日志里的每个任务版本是否都已被某份待 ACK 快照覆盖。 */
|
||||
private boolean isCoveredByPendingSnapshot() {
|
||||
List<TaskJournal.ReportSnapshot> pending = List.copyOf(journal.pendingReports().values());
|
||||
if (pending.isEmpty())
|
||||
return false;
|
||||
for (TaskJournal.JournalTask task : journal.tasks().values()) {
|
||||
boolean found = false;
|
||||
for (TaskJournal.ReportSnapshot snapshot : pending)
|
||||
for (TaskJournal.JournalTask covered : snapshot.tasks)
|
||||
if (covered.gid == task.gid && covered.revision == task.revision) {
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
if (!found)
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* 应用主站 ACK。
|
||||
*
|
||||
* <p>ACK 只代表「主站已落库收到这份报告」,不代表任务已经完成。因此这里只移除真正的终态
|
||||
* 任务({@link GalleryTask#COMPRESS_COMPLETE});处于下载中、压缩中等进行中的任务即使被
|
||||
* 确认,也必须留在队列与日志里继续推进,否则会在下载尚未结束时被提前遗忘。
|
||||
*
|
||||
* <p>还要额外比对运行期状态与快照内容:扫描线程在锁外就地更新 {@link GalleryTask},
|
||||
* 一份迟到的 ACK 可能在任务又前进之后才到。若只比 revision,扫描尚未落盘的那次变化
|
||||
* 会让旧 ACK 误删更新过的任务,所以状态与进度也必须一致才允许移除。
|
||||
*
|
||||
* <p>返回被移除的 gid;重复、失败、乱序或非终态的 ACK 不产生任何移除。
|
||||
*/
|
||||
public Set<Integer> applyAck(String reportId, byte result) throws IOException {
|
||||
synchronized (lock) {
|
||||
TaskJournal.ReportSnapshot snapshot = reporter.settleAck(reportId, result == 0);
|
||||
if (snapshot == null)
|
||||
return Set.of();
|
||||
|
||||
Set<Integer> cleared = new LinkedHashSet<>();
|
||||
for (TaskJournal.JournalTask acked : snapshot.tasks) {
|
||||
GalleryTask runtime = queue.get(acked.gid);
|
||||
// 只有压缩完成是终态;进行中的任务必须保留。
|
||||
if (runtime == null || runtime.getStatus() != GalleryTask.COMPRESS_COMPLETE)
|
||||
continue;
|
||||
// 快照确认后任务又被改动(例如进度前进)时不得按旧内容删除。
|
||||
if (runtime.getStatus() != acked.status || runtime.getProceeding() != acked.proceeding)
|
||||
continue;
|
||||
String currentOperation = operationIds.get(acked.gid);
|
||||
if (currentOperation != null && acked.operationId != null
|
||||
&& !currentOperation.equals(acked.operationId))
|
||||
continue;
|
||||
if (journal.removeTaskIfRevision(acked.gid, acked.revision)) {
|
||||
queue.remove(acked.gid);
|
||||
operationIds.remove(acked.gid);
|
||||
cleared.add(acked.gid);
|
||||
}
|
||||
}
|
||||
if (!cleared.isEmpty())
|
||||
log.info("主站已确认完成的任务,节点清理: {}", cleared);
|
||||
return cleared;
|
||||
}
|
||||
}
|
||||
|
||||
/** 尚未收到 ACK 的报告,供断线恢复后重发。 */
|
||||
public List<TaskJournal.ReportSnapshot> pendingReports() {
|
||||
synchronized (lock) {
|
||||
return List.copyOf(journal.pendingReports().values());
|
||||
}
|
||||
}
|
||||
|
||||
/** 当前活跃的下载操作 id 快照,供报告携带。 */
|
||||
public Map<Integer, String> operationIds() {
|
||||
synchronized (lock) {
|
||||
return new LinkedHashMap<>(operationIds);
|
||||
}
|
||||
}
|
||||
|
||||
/** 队列中当前待上报(尚未被 ACK 确认)的任务 gid。 */
|
||||
public Set<Integer> journalGids() {
|
||||
synchronized (lock) {
|
||||
return new LinkedHashSet<>(journal.tasks().keySet());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package lion.Service;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import lion.CustomUtil;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.channels.FileChannel;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.nio.file.StandardOpenOption;
|
||||
import java.nio.file.attribute.PosixFilePermissions;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import java.util.TreeMap;
|
||||
import java.util.UUID;
|
||||
|
||||
/** 每个成功的备用 GET 先同步落盘;只在主站明确确认后删除。无公开 Key 和订阅正文。 */
|
||||
public final class SubscriptionEventJournal {
|
||||
private static final long MAX_BYTES = 256L * 1024 * 1024;
|
||||
private static final int MAX_EVENTS = 100_000;
|
||||
private static final int MAX_EVENT_BYTES = 16 * 1024;
|
||||
private static final long RESERVE_BYTES = 16L * 1024 * 1024;
|
||||
private final Path directory;
|
||||
private final TreeMap<String, Pending> events = new TreeMap<>();
|
||||
private long bytes;
|
||||
|
||||
public record Pending(String filename, String eventId, long occurredAt, byte[] json) { }
|
||||
|
||||
public SubscriptionEventJournal(Path directory) throws IOException {
|
||||
this.directory = directory;
|
||||
Files.createDirectories(directory);
|
||||
if (Files.isSymbolicLink(directory)) throw new IOException("Subscription journal cannot be a symlink");
|
||||
Files.setPosixFilePermissions(directory, PosixFilePermissions.fromString("rwx------"));
|
||||
try (var files = Files.list(directory)) {
|
||||
for (Path file : files.filter(p -> p.getFileName().toString().endsWith(".json")).toList()) {
|
||||
if (!Files.isRegularFile(file, java.nio.file.LinkOption.NOFOLLOW_LINKS)
|
||||
|| Files.size(file) > MAX_EVENT_BYTES)
|
||||
throw new IOException("Invalid subscription event file");
|
||||
byte[] json = Files.readAllBytes(file);
|
||||
JsonNode value = CustomUtil.objectMapper.readTree(json);
|
||||
Pending pending = pending(file.getFileName().toString(), value, json);
|
||||
Files.setPosixFilePermissions(file, PosixFilePermissions.fromString("rw-------"));
|
||||
events.put(pending.filename(), pending);
|
||||
bytes += json.length;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static Pending pending(String filename, JsonNode value, byte[] json) throws IOException {
|
||||
try {
|
||||
String eventId = UUID.fromString(value.path("eventId").asText()).toString();
|
||||
long occurredAt = value.path("occurredAt").asLong(-1);
|
||||
if (!value.isObject() || !value.path("occurredAt").isIntegralNumber() || occurredAt <= 0
|
||||
|| !filename.equals(filename(occurredAt, eventId))
|
||||
|| !value.path("keyHash").isTextual() || !value.path("keyHash").asText().matches("[a-f0-9]{64}")
|
||||
|| !value.path("client").isTextual() || !Set.of("v2", "cat").contains(value.path("client").asText())
|
||||
|| !value.path("clientIp").isTextual() || value.path("clientIp").asText().length() > 45
|
||||
|| !value.path("clientIp").asText().matches("[0-9a-fA-F:.]+")
|
||||
|| !value.path("userAgent").isTextual() || value.path("userAgent").asText().length() > 2048)
|
||||
throw new IllegalArgumentException("Invalid event metadata");
|
||||
Set<String> fields = Set.of("eventId", "keyHash", "client", "clientIp", "userAgent", "occurredAt");
|
||||
var fieldNames = value.fieldNames();
|
||||
while (fieldNames.hasNext()) if (!fields.contains(fieldNames.next()))
|
||||
throw new IllegalArgumentException("Unknown event field");
|
||||
return new Pending(filename, eventId, occurredAt, json);
|
||||
} catch (IllegalArgumentException failure) {
|
||||
throw new IOException("Invalid subscription event metadata", failure);
|
||||
}
|
||||
}
|
||||
|
||||
private static String filename(long time, String id) {
|
||||
return String.format(java.util.Locale.ROOT, "%019d-%s.json", time, id);
|
||||
}
|
||||
|
||||
public synchronized void append(JsonNode event) throws IOException {
|
||||
byte[] json = CustomUtil.objectMapper.writeValueAsBytes(event);
|
||||
Pending pending = pending(filename(event.path("occurredAt").asLong(), event.path("eventId").asText()), event, json);
|
||||
if (events.containsKey(pending.filename())) return;
|
||||
if (json.length > MAX_EVENT_BYTES || events.size() >= MAX_EVENTS || bytes + json.length > MAX_BYTES
|
||||
|| Files.getFileStore(directory).getUsableSpace() < RESERVE_BYTES + json.length)
|
||||
throw new IOException("Subscription event journal capacity exceeded");
|
||||
Path target = directory.resolve(pending.filename());
|
||||
Path temporary = directory.resolve(pending.filename() + ".tmp");
|
||||
try {
|
||||
try (FileChannel channel = FileChannel.open(temporary, Set.of(StandardOpenOption.CREATE_NEW,
|
||||
StandardOpenOption.WRITE), PosixFilePermissions.asFileAttribute(
|
||||
PosixFilePermissions.fromString("rw-------")))) {
|
||||
ByteBuffer buffer = ByteBuffer.wrap(json);
|
||||
while (buffer.hasRemaining()) channel.write(buffer);
|
||||
channel.force(true);
|
||||
}
|
||||
Files.move(temporary, target, StandardCopyOption.ATOMIC_MOVE);
|
||||
events.put(pending.filename(), pending);
|
||||
bytes += json.length;
|
||||
forceDirectory();
|
||||
} finally {
|
||||
Files.deleteIfExists(temporary);
|
||||
}
|
||||
}
|
||||
|
||||
public synchronized List<Pending> oldest(int limit) {
|
||||
return new ArrayList<>(events.values().stream().limit(limit).toList());
|
||||
}
|
||||
|
||||
/** 未确认或本批以外的 ID 绝不清理;目录同步失败时保留内存项,下轮继续幂等重试。 */
|
||||
public synchronized void acknowledge(List<Pending> batch, Set<String> acknowledged) throws IOException {
|
||||
List<Pending> deleted = new ArrayList<>();
|
||||
for (Pending pending : batch) {
|
||||
if (acknowledged.contains(pending.eventId()) && events.containsKey(pending.filename())) {
|
||||
Files.deleteIfExists(directory.resolve(pending.filename()));
|
||||
deleted.add(pending);
|
||||
}
|
||||
}
|
||||
if (deleted.isEmpty()) return;
|
||||
forceDirectory();
|
||||
for (Pending pending : deleted) {
|
||||
events.remove(pending.filename());
|
||||
bytes -= pending.json().length;
|
||||
}
|
||||
}
|
||||
|
||||
private void forceDirectory() throws IOException {
|
||||
try (FileChannel channel = FileChannel.open(directory, StandardOpenOption.READ)) {
|
||||
channel.force(true);
|
||||
}
|
||||
}
|
||||
|
||||
public synchronized int size() { return events.size(); }
|
||||
public synchronized long oldestAgeMillis() {
|
||||
return events.isEmpty() ? 0 : Math.max(0, System.currentTimeMillis() - events.firstEntry().getValue().occurredAt());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,338 @@
|
||||
package lion.Service;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import lion.CustomUtil;
|
||||
import lion.Externel.BackupSubServer;
|
||||
import lion.Http.HttpRequestHandler;
|
||||
import lion.Http.HttpRequestModel;
|
||||
import lion.Http.HttpResponseModel;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.io.IOException;
|
||||
import java.net.InetAddress;
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Duration;
|
||||
import java.util.Base64;
|
||||
import java.util.HashSet;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.Flow;
|
||||
import java.util.concurrent.ScheduledExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
/** storageNode 内置的主站优先订阅入口及离线访问记录回传器。 */
|
||||
@Slf4j
|
||||
public final class SubscriptionGateway implements HttpRequestHandler, AutoCloseable {
|
||||
private static final int MAX_RESPONSE_BYTES = 16 * 1024 * 1024;
|
||||
private static final int MAX_BATCH_BYTES = 256 * 1024;
|
||||
private final HttpRequestHandler local;
|
||||
private final URI origin;
|
||||
private final byte[] secret;
|
||||
private final SubscriptionEventJournal journal;
|
||||
private final HttpClient client;
|
||||
private final Duration timeout;
|
||||
private final long cooldownMillis;
|
||||
private final ScheduledExecutorService uploader;
|
||||
private final Object uploadLock = new Object();
|
||||
private long openUntil;
|
||||
private long generation;
|
||||
private boolean halfOpen;
|
||||
private volatile long lastUploadAt;
|
||||
private volatile String lastUploadState = "idle";
|
||||
private volatile boolean closed;
|
||||
|
||||
public SubscriptionGateway(SubscriptionSnapshotStore store, String primaryUrl, String secret, Path eventDir)
|
||||
throws IOException {
|
||||
this(store, primaryUrl, secret, eventDir, Duration.ofSeconds(3), 10_000, true);
|
||||
}
|
||||
|
||||
/** 可配置时间仅供真实 HTTP 测试;生产固定 1s 连接/3s 总时限/10s 熔断。 */
|
||||
public SubscriptionGateway(SubscriptionSnapshotStore store, String primaryUrl, String secret, Path eventDir,
|
||||
Duration timeout, long cooldownMillis, boolean backgroundUpload) throws IOException {
|
||||
this.local = BackupSubServer.handler(store);
|
||||
this.origin = validateOrigin(primaryUrl);
|
||||
if (secret == null || secret.isBlank()) throw new IllegalArgumentException("Subscription gateway secret required");
|
||||
this.secret = secret.getBytes(StandardCharsets.UTF_8);
|
||||
this.journal = new SubscriptionEventJournal(eventDir);
|
||||
this.timeout = timeout;
|
||||
this.cooldownMillis = cooldownMillis;
|
||||
this.client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(1))
|
||||
.followRedirects(HttpClient.Redirect.NEVER).build();
|
||||
this.uploader = Executors.newSingleThreadScheduledExecutor(r -> {
|
||||
Thread thread = new Thread(r, "subscription-events-upload");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
if (backgroundUpload) uploader.scheduleWithFixedDelay(this::uploadSafely, 10, 10, TimeUnit.SECONDS);
|
||||
}
|
||||
|
||||
private static URI validateOrigin(String value) {
|
||||
URI uri = URI.create(value);
|
||||
if (!("http".equals(uri.getScheme()) || "https".equals(uri.getScheme()))
|
||||
|| uri.getHost() == null || uri.getUserInfo() != null || uri.getQuery() != null
|
||||
|| uri.getFragment() != null || !(uri.getPath().isEmpty() || "/".equals(uri.getPath())))
|
||||
throw new IllegalArgumentException("SubscriptionPrimaryUrl must be a fixed HTTP(S) origin");
|
||||
return uri;
|
||||
}
|
||||
|
||||
@Override
|
||||
public HttpResponseModel handle(HttpRequestModel request) {
|
||||
String path = request.decodedPath();
|
||||
if ("/health/subscription-gateway".equals(path)) {
|
||||
if (!HttpRequestModel.GET.equals(request.method()) && !request.isHead()) return HttpResponseModel.status(405);
|
||||
try {
|
||||
return HttpResponseModel.bytes(200, CustomUtil.objectMapper.writeValueAsBytes(status()),
|
||||
"application/json; charset=utf-8", null, Map.of());
|
||||
} catch (IOException failure) {
|
||||
return HttpResponseModel.status(500);
|
||||
}
|
||||
}
|
||||
if (!HttpRequestModel.GET.equals(request.method()) && !request.isHead()) return local.handle(request);
|
||||
String[] parts = path == null ? new String[0] : path.split("/", -1);
|
||||
if (parts.length != 4 || !parts[0].isEmpty() || !"sub".equals(parts[1])
|
||||
|| !("v2".equals(parts[2]) || "cat".equals(parts[2]))
|
||||
|| parts[3].length() < 6 || parts[3].length() > 512)
|
||||
return local.handle(request);
|
||||
// 内部路径不能成为 URI 目录跳转或带控制字符的请求。
|
||||
if (parts[3].codePoints().anyMatch(c -> Character.isISOControl(c))) return HttpResponseModel.status(404);
|
||||
ObjectNode event;
|
||||
try {
|
||||
event = event(request, parts[2], parts[3]);
|
||||
} catch (IOException failure) {
|
||||
return HttpResponseModel.status(503);
|
||||
}
|
||||
Attempt attempt = acquire();
|
||||
if (attempt != null) {
|
||||
try {
|
||||
HttpResponseModel response = primary(request, parts, event);
|
||||
if (response.status() < 500 && !(response.status() >= 300 && response.status() < 400)) {
|
||||
recovered(attempt);
|
||||
return response;
|
||||
}
|
||||
failed(attempt);
|
||||
} catch (Exception failure) {
|
||||
if (failure instanceof InterruptedException) Thread.currentThread().interrupt();
|
||||
failed(attempt);
|
||||
// 不记录 URL/公开 key/密钥,失败原因只暴露类别。
|
||||
log.debug("订阅主站请求失败: {}", failure.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
HttpResponseModel fallback = local.handle(request);
|
||||
if (!request.isHead() && (fallback.status() == 200 || fallback.status() == 206)) {
|
||||
try {
|
||||
journal.append(event);
|
||||
} catch (IOException failure) {
|
||||
log.error("订阅离线记录无法持久化,拒绝本次更新: {}", failure.getClass().getSimpleName());
|
||||
return HttpResponseModel.text(503, "text/plain", "Subscription access journal unavailable");
|
||||
}
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private ObjectNode event(HttpRequestModel request, String format, String key) throws IOException {
|
||||
ObjectNode event = CustomUtil.objectMapper.createObjectNode();
|
||||
event.put("eventId", UUID.randomUUID().toString());
|
||||
event.put("keyHash", sha256(key.getBytes(StandardCharsets.UTF_8)));
|
||||
event.put("client", format);
|
||||
event.put("clientIp", clientIp(request));
|
||||
String rawAgent = request.header("User-Agent");
|
||||
if (rawAgent == null) rawAgent = "";
|
||||
// HTTP/1 header值使用Latin-1。正常原始UA保持不变;移除控制字符,
|
||||
// 非Latin-1字符替换为?,避免非法header导致每次请求都只能fallback。
|
||||
StringBuilder normalized = new StringBuilder();
|
||||
rawAgent.codePoints().filter(c -> c >= 32 && c != 127)
|
||||
.limit(2048).forEach(c -> normalized.appendCodePoint(c <= 255 ? c : '?'));
|
||||
String agent = normalized.toString();
|
||||
while (agent.getBytes(StandardCharsets.UTF_8).length > 2048)
|
||||
agent = agent.substring(0, agent.length() - 1);
|
||||
event.put("userAgent", agent);
|
||||
event.put("occurredAt", System.currentTimeMillis());
|
||||
// 签名事件头的 base64url 必须 <=4096。保留能装下的原始前缀,避免切断代理对。
|
||||
while (Base64.getUrlEncoder().withoutPadding().encodeToString(
|
||||
CustomUtil.objectMapper.writeValueAsBytes(event)).length() > 4096) {
|
||||
agent = agent.substring(0, agent.offsetByCodePoints(agent.length(), -1));
|
||||
event.put("userAgent", agent);
|
||||
}
|
||||
return event;
|
||||
}
|
||||
|
||||
public static String clientIp(HttpRequestModel request) {
|
||||
InetAddress remote = literalAddress(request.clientAddress());
|
||||
if (remote != null && remote.isLoopbackAddress()) {
|
||||
InetAddress forwarded = literalAddress(request.header("X-Real-IP"));
|
||||
if (forwarded != null) return forwarded.getHostAddress();
|
||||
}
|
||||
return remote == null ? "0.0.0.0" : remote.getHostAddress();
|
||||
}
|
||||
|
||||
private static InetAddress literalAddress(String value) {
|
||||
if (value == null || value.length() > 45 || !value.matches("[0-9a-fA-F:.]+")) return null;
|
||||
if (!value.contains(":")) {
|
||||
if (!value.matches("[0-9]{1,3}(\\.[0-9]{1,3}){3}")) return null;
|
||||
for (String part : value.split("\\.")) if (Integer.parseInt(part) > 255) return null;
|
||||
}
|
||||
try { return InetAddress.getByName(value); } catch (Exception ignored) { return null; }
|
||||
}
|
||||
|
||||
private HttpResponseModel primary(HttpRequestModel request, String[] parts, ObjectNode event) throws Exception {
|
||||
URI target = endpoint("/sub/" + parts[2] + "/" + parts[3]);
|
||||
HttpRequest.Builder builder = HttpRequest.newBuilder(target).timeout(timeout)
|
||||
.method(request.isHead() ? "HEAD" : "GET", HttpRequest.BodyPublishers.noBody())
|
||||
.header("User-Agent", event.path("userAgent").asText());
|
||||
if (request.rangeHeader() != null) builder.header("Range", request.rangeHeader());
|
||||
// HEAD 也必须签名,使主站绕过普通访问记录分支;主站与节点均不记 HEAD。
|
||||
byte[] json = CustomUtil.objectMapper.writeValueAsBytes(event);
|
||||
builder.header("X-Subscription-Event", Base64.getUrlEncoder().withoutPadding().encodeToString(json));
|
||||
sign(builder, json);
|
||||
HttpResponse<byte[]> response = send(builder.build(), MAX_RESPONSE_BYTES);
|
||||
Map<String, String> headers = new LinkedHashMap<>();
|
||||
for (String name : List.of("Content-Range", "Accept-Ranges"))
|
||||
response.headers().firstValue(name).ifPresent(value -> headers.put(name, value));
|
||||
if (request.isHead()) response.headers().firstValue("Content-Length")
|
||||
.ifPresent(value -> headers.put("Content-Length", value));
|
||||
return HttpResponseModel.bytes(response.statusCode(), response.body(),
|
||||
response.headers().firstValue("Content-Type").orElse("application/octet-stream"), null, headers);
|
||||
}
|
||||
|
||||
/** URI 构造器百分号编码单一路径;从不拼接用户 origin/query 或跟随 redirect。 */
|
||||
private URI endpoint(String path) throws URISyntaxException {
|
||||
return new URI(origin.getScheme(), null, origin.getHost(), origin.getPort(), path, null, null);
|
||||
}
|
||||
|
||||
private void sign(HttpRequest.Builder builder, byte[] payload) throws Exception {
|
||||
String timestamp = Long.toString(System.currentTimeMillis() / 1000);
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(secret, "HmacSHA256"));
|
||||
String input = "v1\n" + timestamp + "\n" + sha256(payload);
|
||||
builder.header("X-Subscription-Timestamp", timestamp)
|
||||
.header("X-Subscription-Signature", HexFormat.of().formatHex(mac.doFinal(input.getBytes(StandardCharsets.UTF_8))));
|
||||
}
|
||||
|
||||
private static String sha256(byte[] bytes) {
|
||||
try { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)); }
|
||||
catch (java.security.NoSuchAlgorithmException impossible) { throw new IllegalStateException(impossible); }
|
||||
}
|
||||
|
||||
/** sendAsync 的整次 future 包含正文消费,避免 ofInputStream 绕过读取时限。 */
|
||||
private HttpResponse<byte[]> send(HttpRequest request, int maxBytes) throws Exception {
|
||||
CompletableFuture<HttpResponse<byte[]>> future = client.sendAsync(request,
|
||||
info -> new BoundedSubscriber(maxBytes));
|
||||
try { return future.get(timeout.toMillis(), TimeUnit.MILLISECONDS); }
|
||||
finally { if (!future.isDone()) future.cancel(true); }
|
||||
}
|
||||
|
||||
private static final class BoundedSubscriber implements HttpResponse.BodySubscriber<byte[]> {
|
||||
private final HttpResponse.BodySubscriber<byte[]> delegate = HttpResponse.BodySubscribers.ofByteArray();
|
||||
private final int limit;
|
||||
private int received;
|
||||
private Flow.Subscription subscription;
|
||||
BoundedSubscriber(int limit) { this.limit = limit; }
|
||||
public CompletionStage<byte[]> getBody() { return delegate.getBody(); }
|
||||
public void onSubscribe(Flow.Subscription value) { subscription = value; delegate.onSubscribe(value); }
|
||||
public void onNext(List<ByteBuffer> buffers) {
|
||||
long size = buffers.stream().mapToLong(ByteBuffer::remaining).sum();
|
||||
if (size > limit - received) {
|
||||
subscription.cancel();
|
||||
delegate.onError(new IOException("Subscription HTTP response exceeds limit"));
|
||||
} else { received += (int) size; delegate.onNext(buffers); }
|
||||
}
|
||||
public void onError(Throwable error) { delegate.onError(error); }
|
||||
public void onComplete() { delegate.onComplete(); }
|
||||
}
|
||||
|
||||
private record Attempt(long generation) { }
|
||||
private synchronized Attempt acquire() {
|
||||
if (openUntil == 0) return new Attempt(generation);
|
||||
if (System.currentTimeMillis() < openUntil || halfOpen) return null;
|
||||
halfOpen = true;
|
||||
return new Attempt(generation);
|
||||
}
|
||||
private synchronized void failed(Attempt attempt) {
|
||||
if (attempt.generation() != generation) return;
|
||||
openUntil = System.currentTimeMillis() + cooldownMillis;
|
||||
halfOpen = false;
|
||||
generation++;
|
||||
}
|
||||
private synchronized void recovered(Attempt attempt) {
|
||||
if (attempt.generation() != generation) return;
|
||||
openUntil = 0;
|
||||
halfOpen = false;
|
||||
}
|
||||
|
||||
public Map<String, Object> status() {
|
||||
String circuit;
|
||||
synchronized (this) { circuit = openUntil == 0 ? "closed" : halfOpen ? "half-open" : "open"; }
|
||||
return Map.of("pendingCount", journal.size(), "oldestAgeMillis", journal.oldestAgeMillis(),
|
||||
"circuit", circuit, "lastUploadAt", lastUploadAt, "lastUploadState", lastUploadState);
|
||||
}
|
||||
|
||||
private void uploadSafely() {
|
||||
try { uploadOnce(); }
|
||||
catch (Exception failure) {
|
||||
lastUploadState = "failed";
|
||||
log.debug("订阅访问记录回传失败: {}", failure.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
|
||||
/** 每轮最多一批、100条、256KiB;所有未明确确认的原始 eventId 均留盘。 */
|
||||
public void uploadOnce() throws Exception {
|
||||
synchronized (uploadLock) {
|
||||
if (closed) return;
|
||||
List<SubscriptionEventJournal.Pending> candidates = journal.oldest(100);
|
||||
if (candidates.isEmpty()) { lastUploadState = "idle"; return; }
|
||||
ObjectNode body = CustomUtil.objectMapper.createObjectNode();
|
||||
var array = body.putArray("events");
|
||||
java.util.ArrayList<SubscriptionEventJournal.Pending> batch = new java.util.ArrayList<>();
|
||||
int total = 13; // {"events":[]} + 逗号安全余量
|
||||
for (var pending : candidates) {
|
||||
if (total + pending.json().length + 1 > MAX_BATCH_BYTES) break;
|
||||
array.add(CustomUtil.objectMapper.readTree(pending.json()));
|
||||
batch.add(pending);
|
||||
total += pending.json().length + 1;
|
||||
}
|
||||
if (batch.isEmpty()) throw new IOException("Subscription event exceeds upload batch limit");
|
||||
byte[] json = CustomUtil.objectMapper.writeValueAsBytes(body);
|
||||
HttpRequest.Builder builder = HttpRequest.newBuilder(endpoint("/internal/subscription-events"))
|
||||
.timeout(timeout).header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofByteArray(json));
|
||||
sign(builder, json);
|
||||
HttpResponse<byte[]> response = send(builder.build(), 64 * 1024);
|
||||
if (response.statusCode() != 200) throw new IOException("Subscription event upload rejected");
|
||||
JsonNode result = CustomUtil.objectMapper.readTree(response.body());
|
||||
if (result == null || !result.path("acknowledged").isArray()) throw new IOException("Invalid subscription event acknowledgement");
|
||||
Set<String> ids = new HashSet<>();
|
||||
for (JsonNode id : result.path("acknowledged")) {
|
||||
if (!id.isTextual()) throw new IOException("Invalid subscription event acknowledgement");
|
||||
ids.add(id.asText());
|
||||
}
|
||||
journal.acknowledge(batch, ids);
|
||||
lastUploadAt = System.currentTimeMillis();
|
||||
lastUploadState = "ok";
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
closed = true;
|
||||
uploader.shutdownNow();
|
||||
client.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,369 @@
|
||||
package lion.Service;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import lion.CustomUtil;
|
||||
import lion.Message.Main.SubscriptionAccountSnapshot;
|
||||
import lion.Message.Main.SubscriptionBindingSnapshot;
|
||||
import lion.Message.Main.SubscriptionSnapshotMessage;
|
||||
import lion.Message.Main.SubscriptionSnapshotPayload;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.UncheckedIOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.*;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.zip.GZIPInputStream;
|
||||
|
||||
/**
|
||||
* Last-known-good subscription data used by the standby HTTP server.
|
||||
* The store never downloads upstream subscriptions and never stores upstream keys.
|
||||
*/
|
||||
@Slf4j
|
||||
public final class SubscriptionSnapshotStore {
|
||||
public static final byte APPLY_SUCCESS = 0;
|
||||
public static final byte APPLY_INVALID = 1;
|
||||
public static final byte APPLY_IO_ERROR = 2;
|
||||
public static final byte APPLY_OLD = 3;
|
||||
|
||||
private final Path root;
|
||||
private final byte[] syncSecret;
|
||||
private final long maxStaleMillis;
|
||||
private final int maxPayloadBytes;
|
||||
private final ObjectMapper objectMapper = CustomUtil.objectMapper;
|
||||
private final AtomicReference<Snapshot> current = new AtomicReference<>();
|
||||
/**
|
||||
* 最近一次收到主站存活信号的时刻(本地接收时间)。
|
||||
*
|
||||
* <p>「过期」的语义是「主站已失联」,因此新鲜度必须由「主站的任何存活信号」驱动,
|
||||
* 而不能只看「收到快照」。快照是内容寻址的:内容长期不变时主站没有理由反复推送,
|
||||
* 若以收到快照为准,备机会在内容不动的第 7 天误判过期;反过来,若主站为此定时重发
|
||||
* 整份快照,又只为续期而传输约 346 KiB/次。
|
||||
*
|
||||
* <p>主站本就有常规存活探测(每 30 分钟的 AvailableCheckMessage),节点回它以
|
||||
* ResponseMessage。把该信号纳入新鲜度后:过期 ⇔ 主站失联超过有效期,
|
||||
* 语义精确,且不再需要任何为「续期」而生的专用推送。
|
||||
*
|
||||
* <p>时刻取自节点本地时钟、不参与签名,攻击者无法通过重放旧 revision 续期。
|
||||
*/
|
||||
private volatile long lastPrimaryContactAt;
|
||||
|
||||
public SubscriptionSnapshotStore(Path root, String syncSecret, long maxStaleSeconds, int maxPayloadBytes) {
|
||||
this.root = Objects.requireNonNull(root);
|
||||
this.syncSecret = syncSecret == null ? new byte[0] : syncSecret.getBytes(StandardCharsets.UTF_8);
|
||||
this.maxStaleMillis = Math.max(0, maxStaleSeconds) * 1000L;
|
||||
this.maxPayloadBytes = maxPayloadBytes;
|
||||
}
|
||||
|
||||
public void load() {
|
||||
List<String> candidates = new ArrayList<>();
|
||||
try {
|
||||
Path pointer = root.resolve("current-revision");
|
||||
if (Files.isRegularFile(pointer)) {
|
||||
String revision = Files.readString(pointer, StandardCharsets.UTF_8).trim();
|
||||
if (isRevision(revision)) candidates.add(revision);
|
||||
}
|
||||
Path snapshots = root.resolve("snapshots");
|
||||
if (Files.isDirectory(snapshots)) {
|
||||
try (var stream = Files.list(snapshots)) {
|
||||
stream.filter(Files::isDirectory)
|
||||
.map(path -> path.getFileName().toString())
|
||||
.filter(SubscriptionSnapshotStore::isRevision)
|
||||
.filter(revision -> !candidates.contains(revision))
|
||||
.sorted(Comparator.reverseOrder())
|
||||
.forEach(candidates::add);
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.warn("扫描订阅快照失败: {}", e.getMessage());
|
||||
}
|
||||
for (String revision : candidates) {
|
||||
try {
|
||||
Snapshot snapshot = loadSnapshot(root.resolve("snapshots").resolve(revision));
|
||||
current.set(snapshot);
|
||||
// 重启后还没有收到过主站快照,先以落盘内容的生成时间作为新鲜度起点,
|
||||
// 否则重启即视为「刚同步过」,会让超期快照被错误续期。
|
||||
lastPrimaryContactAt = snapshot.generatedAt();
|
||||
writePointer(revision);
|
||||
log.info("加载订阅快照成功 revision={} accounts={} bindings={}", shortRevision(revision), snapshot.accountCount(), snapshot.bindingCount());
|
||||
return;
|
||||
} catch (Exception e) {
|
||||
log.warn("订阅快照损坏,尝试上一版本 revision={}: {}", shortRevision(revision), e.getMessage());
|
||||
}
|
||||
}
|
||||
current.set(null);
|
||||
log.warn("没有可用的订阅快照,备机订阅暂不可用");
|
||||
}
|
||||
|
||||
public ApplyResult apply(SubscriptionSnapshotMessage message) {
|
||||
Path staging = null;
|
||||
try {
|
||||
if (message == null || message.getSchemaVersion() != 1 || !isRevision(message.getRevision()))
|
||||
return new ApplyResult(APPLY_INVALID, "消息版本或 revision 非法");
|
||||
if (syncSecret.length == 0)
|
||||
return new ApplyResult(APPLY_INVALID, "同步密钥未配置");
|
||||
|
||||
byte[] compressed = decodeBase64(message.getPayloadBase64(), maxPayloadBytes);
|
||||
if (!constantEquals(message.getPayloadSha256(), sha256(compressed)))
|
||||
return new ApplyResult(APPLY_INVALID, "payload SHA-256 校验失败");
|
||||
String signatureInput = message.getSchemaVersion() + "\n" + message.getRevision() + "\n"
|
||||
+ message.getGeneratedAt() + "\n" + message.getPayloadSha256();
|
||||
if (!constantEquals(message.getSignature(), hmac(signatureInput.getBytes(StandardCharsets.UTF_8))))
|
||||
return new ApplyResult(APPLY_INVALID, "快照签名校验失败");
|
||||
|
||||
byte[] payloadBytes = gunzip(compressed, maxPayloadBytes);
|
||||
if (!constantEquals(message.getRevision(), sha256(payloadBytes)))
|
||||
return new ApplyResult(APPLY_INVALID, "revision 与 payload 不一致");
|
||||
SubscriptionSnapshotPayload payload = objectMapper.readValue(payloadBytes, SubscriptionSnapshotPayload.class);
|
||||
SnapshotData data = validatePayload(payload);
|
||||
Snapshot old = current.get();
|
||||
if (old != null) {
|
||||
if (message.getRevision().equals(old.revision())) {
|
||||
// 内容未变但主站仍在同步:刷新新鲜度,避免备机因「内容长期不变」而误判过期。
|
||||
lastPrimaryContactAt = System.currentTimeMillis();
|
||||
return new ApplyResult(APPLY_OLD, "revision 已存在");
|
||||
}
|
||||
if (message.getGeneratedAt() < old.generatedAt())
|
||||
return new ApplyResult(APPLY_OLD, "快照时间早于当前版本");
|
||||
}
|
||||
|
||||
Path snapshots = root.resolve("snapshots");
|
||||
Files.createDirectories(snapshots);
|
||||
staging = snapshots.resolve(".staging-" + message.getRevision());
|
||||
deleteRecursively(staging);
|
||||
Files.createDirectories(staging.resolve("accounts"));
|
||||
for (AccountFiles account : data.accounts.values()) {
|
||||
Path dir = staging.resolve("accounts").resolve(String.valueOf(account.accountId()));
|
||||
Files.createDirectories(dir);
|
||||
Files.write(dir.resolve("v2ray.txt"), account.v2(), StandardOpenOption.CREATE_NEW);
|
||||
Files.write(dir.resolve("clash.yaml"), account.clash(), StandardOpenOption.CREATE_NEW);
|
||||
}
|
||||
Map<String, Object> manifest = new LinkedHashMap<>();
|
||||
manifest.put("revision", message.getRevision());
|
||||
manifest.put("generatedAt", message.getGeneratedAt());
|
||||
// 保留主站签名对应的原始 JSON 字节,重启时不依赖 Jackson 再序列化顺序。
|
||||
manifest.put("payloadBase64", Base64.getEncoder().encodeToString(payloadBytes));
|
||||
Files.write(staging.resolve("manifest.json"), objectMapper.writeValueAsBytes(manifest), StandardOpenOption.CREATE_NEW);
|
||||
|
||||
Path destination = snapshots.resolve(message.getRevision());
|
||||
if (Files.exists(destination))
|
||||
deleteRecursively(destination);
|
||||
atomicMove(staging, destination);
|
||||
writePointer(message.getRevision());
|
||||
|
||||
Snapshot snapshot = new Snapshot(message.getRevision(), message.getGeneratedAt(), data.byKeyHash,
|
||||
data.accounts.size(), data.bindingCount);
|
||||
current.set(snapshot);
|
||||
lastPrimaryContactAt = System.currentTimeMillis();
|
||||
cleanupOldSnapshots(message.getRevision());
|
||||
return new ApplyResult(APPLY_SUCCESS, "同步成功");
|
||||
} catch (Exception e) {
|
||||
log.error("应用订阅快照失败: {}", e.getMessage());
|
||||
return new ApplyResult(APPLY_IO_ERROR, e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage());
|
||||
} finally {
|
||||
if (staging != null && Files.exists(staging)) {
|
||||
try { deleteRecursively(staging); }
|
||||
catch (IOException e) { log.warn("清理订阅快照 staging 失败: {}", staging); }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void writePointer(String revision) throws IOException {
|
||||
Files.createDirectories(root);
|
||||
Path pointerTmp = root.resolve("current-revision.tmp");
|
||||
Files.writeString(pointerTmp, revision + "\n", StandardCharsets.UTF_8,
|
||||
StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
|
||||
atomicMove(pointerTmp, root.resolve("current-revision"));
|
||||
}
|
||||
|
||||
public Lookup lookup(String client, String publicKey) {
|
||||
Snapshot snapshot = current.get();
|
||||
if (snapshot == null || isStale(System.currentTimeMillis()))
|
||||
return null;
|
||||
AccountFiles account = snapshot.byKeyHash().get(sha256(publicKey.getBytes(StandardCharsets.UTF_8)));
|
||||
if (account == null)
|
||||
return null;
|
||||
return new Lookup("v2".equals(client) ? account.v2() : account.clash(), snapshot.revision(), snapshot.generatedAt());
|
||||
}
|
||||
|
||||
public Status status() {
|
||||
Snapshot snapshot = current.get();
|
||||
if (snapshot == null)
|
||||
return new Status("unavailable", null, 0, 0, 0);
|
||||
long now = System.currentTimeMillis();
|
||||
long age = Math.max(0, now - lastPrimaryContactAt);
|
||||
return new Status(isStale(now) ? "expired" : "ready", snapshot.revision(), snapshot.accountCount(), snapshot.bindingCount(), age);
|
||||
}
|
||||
|
||||
/**
|
||||
* 主站是否已失联超过最大有效期。
|
||||
*
|
||||
* <p>由主站的任何存活信号刷新(见 {@link #markPrimaryContact()}),不限于快照。
|
||||
*/
|
||||
private boolean isStale(long now) {
|
||||
return maxStaleMillis > 0 && now - lastPrimaryContactAt > maxStaleMillis;
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录一次「主站仍在」的证据。
|
||||
*
|
||||
* <p>用于快照之外的常规存活信号(例如主站每 30 分钟的可用性检查)。
|
||||
* 只要主站在线,本节点的新鲜度就会持续被刷新,因此备机不会因为
|
||||
* 「订阅内容长期不变、主站没理由重发快照」而被判过期。
|
||||
*/
|
||||
public void markPrimaryContact() {
|
||||
lastPrimaryContactAt = System.currentTimeMillis();
|
||||
}
|
||||
|
||||
private Snapshot loadSnapshot(Path directory) throws IOException {
|
||||
JsonNode manifest = objectMapper.readTree(Files.readAllBytes(directory.resolve("manifest.json")));
|
||||
String revision = manifest.path("revision").asText();
|
||||
long generatedAt = manifest.path("generatedAt").asLong(0);
|
||||
byte[] payloadBytes = decodeBase64(manifest.path("payloadBase64").asText(null), maxPayloadBytes);
|
||||
SubscriptionSnapshotPayload payload = objectMapper.readValue(payloadBytes, SubscriptionSnapshotPayload.class);
|
||||
SnapshotData data = validatePayload(payload);
|
||||
if (!isRevision(revision) || !constantEquals(revision, sha256(payloadBytes)))
|
||||
throw new IOException("快照 manifest revision 校验失败");
|
||||
Map<Integer, AccountFiles> accounts = new HashMap<>();
|
||||
for (SubscriptionAccountSnapshot account : payload.getAccounts()) {
|
||||
Path dir = directory.resolve("accounts").resolve(String.valueOf(account.getAccountId()));
|
||||
byte[] v2 = Files.readAllBytes(dir.resolve("v2ray.txt"));
|
||||
byte[] clash = Files.readAllBytes(dir.resolve("clash.yaml"));
|
||||
if (!constantEquals(account.getV2Sha256(), sha256(v2)) || !constantEquals(account.getClashSha256(), sha256(clash)))
|
||||
throw new IOException("缓存文件校验失败");
|
||||
accounts.put(account.getAccountId(), new AccountFiles(account.getAccountId(), v2, clash));
|
||||
}
|
||||
Map<String, AccountFiles> byKey = new HashMap<>();
|
||||
for (SubscriptionBindingSnapshot binding : payload.getBindings())
|
||||
byKey.put(binding.getPublicKeySha256(), accounts.get(binding.getAccountId()));
|
||||
return new Snapshot(revision, generatedAt, byKey, accounts.size(), payload.getBindings().size());
|
||||
}
|
||||
|
||||
private SnapshotData validatePayload(SubscriptionSnapshotPayload payload) throws IOException {
|
||||
if (payload == null || payload.getSchemaVersion() != 1 || payload.getAccounts() == null || payload.getBindings() == null)
|
||||
throw new IOException("payload 版本或字段非法");
|
||||
if (payload.getAccounts().size() > 100 || payload.getBindings().size() > 10000)
|
||||
throw new IOException("快照条目数量超限");
|
||||
Map<Integer, AccountFiles> accounts = new HashMap<>();
|
||||
for (SubscriptionAccountSnapshot account : payload.getAccounts()) {
|
||||
if (account == null || account.getAccountId() == null || account.getAccountId() <= 0 || !account.isEnabled()
|
||||
|| !isBase64Sha(account.getV2Sha256()) || !isBase64Sha(account.getClashSha256()))
|
||||
throw new IOException("账号字段非法");
|
||||
byte[] v2 = decodeBase64(account.getV2ContentBase64(), maxPayloadBytes);
|
||||
byte[] clash = decodeBase64(account.getClashContentBase64(), maxPayloadBytes);
|
||||
if (!constantEquals(account.getV2Sha256(), sha256(v2)) || !constantEquals(account.getClashSha256(), sha256(clash)))
|
||||
throw new IOException("账号缓存 SHA-256 校验失败");
|
||||
if (accounts.put(account.getAccountId(), new AccountFiles(account.getAccountId(), v2, clash)) != null)
|
||||
throw new IOException("账号 ID 重复");
|
||||
}
|
||||
Map<String, AccountFiles> byKey = new HashMap<>();
|
||||
for (SubscriptionBindingSnapshot binding : payload.getBindings()) {
|
||||
if (binding == null || !isBase64Sha(binding.getPublicKeySha256()) || !accounts.containsKey(binding.getAccountId()))
|
||||
throw new IOException("绑定字段或账号引用非法");
|
||||
if (byKey.put(binding.getPublicKeySha256(), accounts.get(binding.getAccountId())) != null)
|
||||
throw new IOException("公开 Key Hash 重复");
|
||||
}
|
||||
return new SnapshotData(accounts, byKey, payload.getBindings().size());
|
||||
}
|
||||
|
||||
private void cleanupOldSnapshots(String currentRevision) {
|
||||
try {
|
||||
Path snapshots = root.resolve("snapshots");
|
||||
List<Path> dirs;
|
||||
try (var stream = Files.list(snapshots)) {
|
||||
dirs = stream.filter(Files::isDirectory)
|
||||
.filter(p -> !p.getFileName().toString().startsWith(".staging-"))
|
||||
.sorted(Comparator.comparingLong(SubscriptionSnapshotStore::lastModified).reversed())
|
||||
.toList();
|
||||
}
|
||||
dirs.stream().filter(p -> !p.getFileName().toString().equals(currentRevision))
|
||||
.skip(1).forEach(p -> {
|
||||
try { deleteRecursively(p); } catch (IOException e) { log.warn("清理旧订阅快照失败: {}", p); }
|
||||
});
|
||||
} catch (IOException e) {
|
||||
log.warn("扫描旧订阅快照失败: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static byte[] gunzip(byte[] compressed, int limit) throws IOException {
|
||||
try (GZIPInputStream input = new GZIPInputStream(new ByteArrayInputStream(compressed))) {
|
||||
byte[] buffer = new byte[8192];
|
||||
var output = new java.io.ByteArrayOutputStream();
|
||||
int total = 0, read;
|
||||
while ((read = input.read(buffer)) != -1) {
|
||||
total += read;
|
||||
if (total > limit) throw new IOException("解压后 payload 超限");
|
||||
output.write(buffer, 0, read);
|
||||
}
|
||||
return output.toByteArray();
|
||||
}
|
||||
}
|
||||
|
||||
private String hmac(byte[] input) throws Exception {
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(syncSecret, "HmacSHA256"));
|
||||
return hex(mac.doFinal(input));
|
||||
}
|
||||
|
||||
private static byte[] decodeBase64(String value, int maxBytes) throws IOException {
|
||||
if (value == null || value.length() > maxBytes * 2L)
|
||||
throw new IOException("Base64 数据超限");
|
||||
try {
|
||||
byte[] result = Base64.getDecoder().decode(value);
|
||||
if (result.length > maxBytes) throw new IOException("数据超限");
|
||||
return result;
|
||||
} catch (IllegalArgumentException e) {
|
||||
throw new IOException("Base64 数据非法");
|
||||
}
|
||||
}
|
||||
|
||||
private static String sha256(byte[] bytes) {
|
||||
try { return hex(MessageDigest.getInstance("SHA-256").digest(bytes)); }
|
||||
catch (Exception e) { throw new IllegalStateException(e); }
|
||||
}
|
||||
|
||||
private static String hex(byte[] bytes) {
|
||||
return HexFormat.of().formatHex(bytes);
|
||||
}
|
||||
|
||||
private static boolean constantEquals(String expected, String actual) {
|
||||
return expected != null && MessageDigest.isEqual(expected.toLowerCase(Locale.ROOT).getBytes(StandardCharsets.US_ASCII),
|
||||
actual.toLowerCase(Locale.ROOT).getBytes(StandardCharsets.US_ASCII));
|
||||
}
|
||||
|
||||
private static boolean isRevision(String value) { return isBase64Sha(value); }
|
||||
private static boolean isBase64Sha(String value) { return value != null && value.matches("[0-9a-fA-F]{64}"); }
|
||||
private static String shortRevision(String revision) { return revision == null ? null : revision.substring(0, Math.min(12, revision.length())); }
|
||||
|
||||
private static long lastModified(Path path) {
|
||||
try { return Files.getLastModifiedTime(path).toMillis(); }
|
||||
catch (IOException e) { return 0; }
|
||||
}
|
||||
|
||||
private static void atomicMove(Path source, Path target) throws IOException {
|
||||
try { Files.move(source, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); }
|
||||
catch (AtomicMoveNotSupportedException e) { Files.move(source, target, StandardCopyOption.REPLACE_EXISTING); }
|
||||
}
|
||||
|
||||
private static void deleteRecursively(Path path) throws IOException {
|
||||
if (!Files.exists(path)) return;
|
||||
try (var stream = Files.walk(path)) {
|
||||
stream.sorted(Comparator.reverseOrder()).forEach(p -> {
|
||||
try { Files.deleteIfExists(p); } catch (IOException e) { throw new UncheckedIOException(e); }
|
||||
});
|
||||
} catch (UncheckedIOException e) { throw e.getCause(); }
|
||||
}
|
||||
|
||||
public record ApplyResult(byte code, String message) {}
|
||||
public record Lookup(byte[] content, String revision, long generatedAt) {}
|
||||
public record Status(String state, String revision, int accountCount, int bindingCount, long ageMillis) {}
|
||||
private record AccountFiles(Integer accountId, byte[] v2, byte[] clash) {}
|
||||
private record SnapshotData(Map<Integer, AccountFiles> accounts, Map<String, AccountFiles> byKeyHash, int bindingCount) {}
|
||||
private record Snapshot(String revision, long generatedAt, Map<String, AccountFiles> byKeyHash, int accountCount, int bindingCount) {}
|
||||
}
|
||||
@@ -0,0 +1,457 @@
|
||||
package lion.Service;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import lion.CustomUtil;
|
||||
import lion.Domain.GalleryTask;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.AtomicMoveNotSupportedException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.HashSet;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.function.Consumer;
|
||||
|
||||
/**
|
||||
* 任务与报告的持久化日志。
|
||||
*
|
||||
* <p>它承担协议 v2 的「可恢复」部分:任务、命令回执、单调递增的 revision,以及尚未收到
|
||||
* ACK 的报告快照。所有写入都先落盘再对内存生效——节点在写盘前崩溃,主站看不到任何未落盘
|
||||
* 的副作用;写盘后才崩溃,重启时能从磁盘恢复待重发的报告。
|
||||
*
|
||||
* <p>并发约定:本类的方法不是线程安全的,调用方({@link ReliableTaskService})必须在自己的
|
||||
* 锁下调用。这样做是为了让「持久化 + 内存状态更新」在同一个临界区内保持一致。
|
||||
*/
|
||||
@Slf4j
|
||||
public class TaskJournal {
|
||||
|
||||
/** 日志文件固定名,便于运维定位与备份。 */
|
||||
static final String STATE_FILE = "task-journal.json";
|
||||
|
||||
/** 落盘格式版本,供后续不兼容变更识别。 */
|
||||
static final int SCHEMA_VERSION = 1;
|
||||
|
||||
private final Path directory;
|
||||
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
/**
|
||||
* 当前已落盘的状态。
|
||||
*
|
||||
* <p>内存状态只在成功写盘之后才被替换,因此任何一次 I/O 失败都不会让内存领先磁盘:
|
||||
* 调用方拿到 {@link IOException} 时,它与磁盘看到的仍是同一份数据。若允许「先改内存再写盘」,
|
||||
* 写盘失败会留下「内存已受理、磁盘没有」的裂缝,重启后命令回执与任务全部消失,
|
||||
* 主站重试会重复执行副作用。
|
||||
*/
|
||||
private volatile State state = new State();
|
||||
|
||||
public TaskJournal(Path directory) {
|
||||
this(directory, CustomUtil.objectMapper);
|
||||
}
|
||||
|
||||
TaskJournal(Path directory, ObjectMapper objectMapper) {
|
||||
this.directory = directory;
|
||||
this.objectMapper = objectMapper;
|
||||
}
|
||||
|
||||
/**
|
||||
* 从磁盘恢复。目录不存在时视为空日志,并交由调用方决定是否创建。
|
||||
*
|
||||
* <p>文件损坏(截断、非法 JSON)不会被静默吞掉:抛出异常让启动失败,避免带着一份
|
||||
* 空日志继续运行、把主站上「已完成」的任务重新看成未完成而重复上报或重复执行。
|
||||
*/
|
||||
public void load() throws IOException {
|
||||
Path file = directory.resolve(STATE_FILE);
|
||||
if (!Files.isRegularFile(file)) {
|
||||
log.info("任务日志不存在,以空日志启动: {}", file);
|
||||
return;
|
||||
}
|
||||
State loaded;
|
||||
try {
|
||||
loaded = objectMapper.readValue(Files.readAllBytes(file), State.class);
|
||||
} catch (IOException e) {
|
||||
throw new IOException("任务日志损坏,拒绝以不一致状态启动: " + file, e);
|
||||
}
|
||||
// 版本校验必须发生在使用数据之前:更高版本的日志可能含有本程序不认识的语义
|
||||
// (例如新的任务字段或状态),按旧规则解释会静默丢字段。宁可拒绝启动。
|
||||
if (loaded.schemaVersion > SCHEMA_VERSION) {
|
||||
throw new IOException("任务日志 schemaVersion=" + loaded.schemaVersion
|
||||
+ " 高于本节点支持的 " + SCHEMA_VERSION + ",拒绝启动: " + file);
|
||||
}
|
||||
if (loaded.schemaVersion < SCHEMA_VERSION) {
|
||||
log.warn("任务日志 schemaVersion={} 低于当前 {},按兼容方式读取",
|
||||
loaded.schemaVersion, SCHEMA_VERSION);
|
||||
}
|
||||
state.schemaVersion = loaded.schemaVersion;
|
||||
state.revision = loaded.revision;
|
||||
state.tasks = loaded.tasks == null ? new LinkedHashMap<>() : new LinkedHashMap<>(loaded.tasks);
|
||||
state.commandReceipts = loaded.commandReceipts == null ? new LinkedHashMap<>() : new LinkedHashMap<>(loaded.commandReceipts);
|
||||
state.pendingReports = loaded.pendingReports == null ? new LinkedHashMap<>() : new LinkedHashMap<>(loaded.pendingReports);
|
||||
log.info("任务日志已恢复: tasks={}, receipts={}, pendingReports={}, revision={}",
|
||||
state.tasks.size(), state.commandReceipts.size(), state.pendingReports.size(), state.revision);
|
||||
}
|
||||
|
||||
// ---- 任务 ----
|
||||
|
||||
public Map<Integer, JournalTask> tasks() {
|
||||
return state.tasks;
|
||||
}
|
||||
|
||||
public JournalTask task(int gid) {
|
||||
return state.tasks.get(gid);
|
||||
}
|
||||
|
||||
/** 新增或覆盖一个任务记录并落盘。 */
|
||||
public void putTask(JournalTask task) throws IOException {
|
||||
commit(next -> {
|
||||
next.revision = Math.max(next.revision, task.revision);
|
||||
next.tasks.put(task.gid, task);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 为一次任务状态变化同时推进 revision 并落盘。
|
||||
*
|
||||
* <p>revision 与任务记录必须在同一次落盘里生效:若分两次写,节点可能在「revision 已涨
|
||||
* 但任务还没更新」的状态下崩溃,重启后该任务的变化会被判定为已上报而永久丢失。
|
||||
*/
|
||||
public JournalTask bumpAndPutTask(GalleryTask task, String operationId) throws IOException {
|
||||
final JournalTask[] created = new JournalTask[1];
|
||||
commit(next -> {
|
||||
JournalTask journalTask = JournalTask.of(task, operationId, next.revision + 1);
|
||||
next.revision = journalTask.revision;
|
||||
next.tasks.put(journalTask.gid, journalTask);
|
||||
created[0] = journalTask;
|
||||
});
|
||||
return created[0];
|
||||
}
|
||||
|
||||
/** 仅当当前记录版本与期望一致时才移除,避免删掉更新的任务。 */
|
||||
public boolean removeTaskIfRevision(int gid, long revision) throws IOException {
|
||||
JournalTask current = state.tasks.get(gid);
|
||||
if (current == null || current.revision != revision)
|
||||
return false;
|
||||
commit(next -> next.tasks.remove(gid));
|
||||
return true;
|
||||
}
|
||||
|
||||
public void removeTask(int gid) throws IOException {
|
||||
if (state.tasks.containsKey(gid))
|
||||
commit(next -> next.tasks.remove(gid));
|
||||
}
|
||||
|
||||
/**
|
||||
* 用运行期任务队列对齐日志,一次落盘完成全部变化。
|
||||
*
|
||||
* <p>把「扫描出的新状态」批量写进日志,而不是每个任务各写一次:每轮扫描可能同时更新
|
||||
* 多个任务的进度,逐条 flush 会让磁盘写入量随任务数线性放大。所有变化共享同一个新
|
||||
* revision,这也让一份报告能一次覆盖它们。
|
||||
*
|
||||
* @return 是否发生了需要上报的变化
|
||||
*/
|
||||
public boolean syncTasks(Collection<GalleryTask> queueTasks, Map<Integer, String> operationIds) throws IOException {
|
||||
boolean[] mutated = new boolean[1];
|
||||
commit(next -> {
|
||||
List<JournalTask> changed = new ArrayList<>();
|
||||
Set<Integer> present = new HashSet<>();
|
||||
for (GalleryTask task : queueTasks) {
|
||||
present.add(task.getGid());
|
||||
JournalTask current = next.tasks.get(task.getGid());
|
||||
if (current != null
|
||||
&& current.status == task.getStatus()
|
||||
&& current.proceeding == task.getProceeding()
|
||||
&& Objects.equals(current.name, task.getName()))
|
||||
continue;
|
||||
changed.add(JournalTask.of(task, operationIds.get(task.getGid()), next.revision + 1));
|
||||
}
|
||||
// 队列里已不存在的任务(删除、完成后被清理)也要从日志移除,否则会被反复上报。
|
||||
List<Integer> vanished = new ArrayList<>();
|
||||
for (Integer gid : next.tasks.keySet())
|
||||
if (!present.contains(gid))
|
||||
vanished.add(gid);
|
||||
|
||||
if (changed.isEmpty() && vanished.isEmpty())
|
||||
return;
|
||||
|
||||
for (Integer gid : vanished)
|
||||
next.tasks.remove(gid);
|
||||
next.revision = next.revision + 1;
|
||||
for (JournalTask task : changed) {
|
||||
task.revision = next.revision;
|
||||
next.tasks.put(task.gid, task);
|
||||
}
|
||||
mutated[0] = true;
|
||||
});
|
||||
return mutated[0];
|
||||
}
|
||||
|
||||
// ---- revision ----
|
||||
|
||||
public long revision() {
|
||||
return state.revision;
|
||||
}
|
||||
|
||||
/** 取下一个 revision 并立即落盘,保证重启后不会回退。 */
|
||||
public long nextRevision() throws IOException {
|
||||
long[] revision = new long[1];
|
||||
commit(next -> {
|
||||
next.revision = next.revision + 1;
|
||||
revision[0] = next.revision;
|
||||
});
|
||||
return revision[0];
|
||||
}
|
||||
|
||||
// ---- 命令回执(幂等) ----
|
||||
|
||||
public CommandReceipt receipt(String operationId) {
|
||||
return operationId == null ? null : state.commandReceipts.get(operationId);
|
||||
}
|
||||
|
||||
/** 全部命令回执;供测试断言幂等回执的存在与数量。 */
|
||||
public Map<String, CommandReceipt> receipts() {
|
||||
return Map.copyOf(state.commandReceipts);
|
||||
}
|
||||
|
||||
/** 记录命令成功回执并落盘;恢复的消息是「已经执行过,直接回放结果」。 */
|
||||
public void putReceipt(CommandReceipt receipt) throws IOException {
|
||||
commit(next -> next.commandReceipts.put(receipt.operationId, receipt));
|
||||
}
|
||||
|
||||
// ---- 待 ACK 报告 ----
|
||||
|
||||
public Map<String, ReportSnapshot> pendingReports() {
|
||||
return state.pendingReports;
|
||||
}
|
||||
|
||||
public ReportSnapshot pendingReport(String reportId) {
|
||||
return reportId == null ? null : state.pendingReports.get(reportId);
|
||||
}
|
||||
|
||||
/** 持久化一份报告快照;必须在写通道之前调用。 */
|
||||
public void putPendingReport(ReportSnapshot snapshot) throws IOException {
|
||||
commit(next -> next.pendingReports.put(snapshot.reportId, snapshot));
|
||||
}
|
||||
|
||||
/** 按报告的精确版本移除快照;版本不匹配(已有更新)时保留。 */
|
||||
public boolean removePendingReport(String reportId, long revision) throws IOException {
|
||||
ReportSnapshot current = state.pendingReports.get(reportId);
|
||||
if (current == null || current.revision != revision)
|
||||
return false;
|
||||
commit(next -> next.pendingReports.remove(reportId));
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* 写时复制提交:在副本上应用变更,写盘成功后才用副本替换当前状态。
|
||||
*
|
||||
* <p>这是「I/O 失败不得让内存领先磁盘」的实现方式。{@code mutation} 抛异常时状态原样保留;
|
||||
* 写盘抛异常时副本被丢弃,内存仍是最后一次成功落盘的内容。调用方在捕获 {@link IOException}
|
||||
* 后可以安全地把操作当作「未发生」重试,而不会重复执行已生效的副作用。
|
||||
*/
|
||||
private void commit(Consumer<State> mutation) throws IOException {
|
||||
State next = state.copy();
|
||||
mutation.accept(next);
|
||||
next.schemaVersion = SCHEMA_VERSION;
|
||||
writeNewState(next);
|
||||
// 只有文件已完整替换后才发布新状态:此前的任何失败都不改变内存。
|
||||
state = next;
|
||||
}
|
||||
|
||||
private void writeNewState(State next) throws IOException {
|
||||
Files.createDirectories(directory);
|
||||
Path temporary = Files.createTempFile(directory, ".task-journal-", ".tmp");
|
||||
try {
|
||||
objectMapper.writerWithDefaultPrettyPrinter().writeValue(temporary.toFile(), next);
|
||||
try {
|
||||
Files.move(temporary, directory.resolve(STATE_FILE),
|
||||
StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
|
||||
} catch (AtomicMoveNotSupportedException e) {
|
||||
Files.move(temporary, directory.resolve(STATE_FILE), StandardCopyOption.REPLACE_EXISTING);
|
||||
}
|
||||
temporary = null;
|
||||
} finally {
|
||||
if (temporary != null)
|
||||
Files.deleteIfExists(temporary);
|
||||
}
|
||||
}
|
||||
|
||||
/** 日志文件路径,供测试与运维查看。 */
|
||||
public Path file() {
|
||||
return directory.resolve(STATE_FILE);
|
||||
}
|
||||
|
||||
/** 落盘结构;字段名即为磁盘格式,改动需同步版本号。 */
|
||||
public static class State {
|
||||
public int schemaVersion = SCHEMA_VERSION;
|
||||
public long revision;
|
||||
public Map<Integer, JournalTask> tasks = new LinkedHashMap<>();
|
||||
public Map<String, CommandReceipt> commandReceipts = new LinkedHashMap<>();
|
||||
public Map<String, ReportSnapshot> pendingReports = new LinkedHashMap<>();
|
||||
|
||||
/**
|
||||
* 深拷贝,供写时复制使用。
|
||||
*
|
||||
* <p>必须逐层复制到可变条目:只拷 map 的外层会让「新状态」与「旧状态」共享同一个
|
||||
* JournalTask 实例,写盘失败后旧状态里的对象也已被改写,回滚就失效了。
|
||||
*/
|
||||
State copy() {
|
||||
State clone = new State();
|
||||
clone.schemaVersion = schemaVersion;
|
||||
clone.revision = revision;
|
||||
for (Map.Entry<Integer, JournalTask> entry : tasks.entrySet())
|
||||
clone.tasks.put(entry.getKey(), entry.getValue().copy());
|
||||
for (Map.Entry<String, CommandReceipt> entry : commandReceipts.entrySet())
|
||||
clone.commandReceipts.put(entry.getKey(), entry.getValue().copy());
|
||||
for (Map.Entry<String, ReportSnapshot> entry : pendingReports.entrySet())
|
||||
clone.pendingReports.put(entry.getKey(), entry.getValue().copy());
|
||||
return clone;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 一条可恢复的任务记录。
|
||||
*
|
||||
* <p>刻意不直接序列化 {@link GalleryTask}:那是线上协议对象,字段会随协议演进;日志需要
|
||||
* 稳定的自有结构,并在读取时重建任务对象,避免协议对象被截断写入后重启解析失败。
|
||||
*/
|
||||
public static class JournalTask {
|
||||
public int gid;
|
||||
public String name;
|
||||
public byte status;
|
||||
public int proceeding;
|
||||
/** 该任务当前生效的下发操作 id,供报告携带与幂等判断。 */
|
||||
public String operationId;
|
||||
public long revision;
|
||||
|
||||
public JournalTask() {
|
||||
}
|
||||
|
||||
public static JournalTask of(GalleryTask task, String operationId, long revision) {
|
||||
JournalTask journal = new JournalTask();
|
||||
journal.gid = task.getGid();
|
||||
journal.name = task.getName();
|
||||
journal.status = task.getStatus();
|
||||
journal.proceeding = task.getProceeding();
|
||||
journal.operationId = operationId;
|
||||
journal.revision = revision;
|
||||
return journal;
|
||||
}
|
||||
|
||||
/** 重建协议对象;path 属于运行期状态,重启后由扫描重新确认,不写入日志。 */
|
||||
public GalleryTask toGalleryTask() {
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(gid);
|
||||
task.setName(name);
|
||||
task.setStatus(status);
|
||||
task.setProceeding(proceeding);
|
||||
return task;
|
||||
}
|
||||
|
||||
JournalTask copy() {
|
||||
JournalTask clone = new JournalTask();
|
||||
clone.gid = gid;
|
||||
clone.name = name;
|
||||
clone.status = status;
|
||||
clone.proceeding = proceeding;
|
||||
clone.operationId = operationId;
|
||||
clone.revision = revision;
|
||||
return clone;
|
||||
}
|
||||
}
|
||||
|
||||
/** 已成功执行的命令回执,按 operationId 去重。 */
|
||||
public static class CommandReceipt {
|
||||
public String operationId;
|
||||
public String action;
|
||||
public int gid;
|
||||
public String name;
|
||||
/** 回放时使用的响应码,与当初回给主站的一致。 */
|
||||
public byte result;
|
||||
|
||||
public CommandReceipt() {
|
||||
}
|
||||
|
||||
public static CommandReceipt of(String operationId, String action, int gid, String name, byte result) {
|
||||
CommandReceipt receipt = new CommandReceipt();
|
||||
receipt.operationId = operationId;
|
||||
receipt.action = action;
|
||||
receipt.gid = gid;
|
||||
receipt.name = name;
|
||||
receipt.result = result;
|
||||
return receipt;
|
||||
}
|
||||
|
||||
CommandReceipt copy() {
|
||||
CommandReceipt clone = new CommandReceipt();
|
||||
clone.operationId = operationId;
|
||||
clone.action = action;
|
||||
clone.gid = gid;
|
||||
clone.name = name;
|
||||
clone.result = result;
|
||||
return clone;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 一份不可变的报告快照。
|
||||
*
|
||||
* <p>任务字段在创建快照时就拷贝,之后任务继续变化也不会改写已发出的报告内容——否则
|
||||
* 重发时会带出与首次不同的数据,主站的幂等落库会以哪一份为准变得不可确定。
|
||||
*/
|
||||
public static class ReportSnapshot {
|
||||
public String reportId;
|
||||
public long revision;
|
||||
/** 快照内容用日志结构承载,每个条目自带 revision,供 ACK 时判断是否为更新版本。 */
|
||||
public List<JournalTask> tasks = new ArrayList<>();
|
||||
public Map<Integer, String> operationIds = new LinkedHashMap<>();
|
||||
|
||||
public ReportSnapshot() {
|
||||
}
|
||||
|
||||
/** 从日志任务构造快照;任务字段已被复制过,之后的变化不会改写本快照。 */
|
||||
public static ReportSnapshot of(String reportId, long revision,
|
||||
List<JournalTask> tasks, Map<Integer, String> operationIds) {
|
||||
ReportSnapshot snapshot = new ReportSnapshot();
|
||||
snapshot.reportId = reportId;
|
||||
snapshot.revision = revision;
|
||||
for (JournalTask task : tasks) {
|
||||
JournalTask copy = new JournalTask();
|
||||
copy.gid = task.gid;
|
||||
copy.name = task.name;
|
||||
copy.status = task.status;
|
||||
copy.proceeding = task.proceeding;
|
||||
copy.operationId = task.operationId;
|
||||
copy.revision = task.revision;
|
||||
snapshot.tasks.add(copy);
|
||||
}
|
||||
snapshot.operationIds = new LinkedHashMap<>(operationIds);
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
/** 转成线路上的协议对象数组。 */
|
||||
public GalleryTask[] toGalleryTasks() {
|
||||
GalleryTask[] array = new GalleryTask[tasks.size()];
|
||||
for (int i = 0; i < tasks.size(); i++)
|
||||
array[i] = tasks.get(i).toGalleryTask();
|
||||
return array;
|
||||
}
|
||||
|
||||
ReportSnapshot copy() {
|
||||
ReportSnapshot clone = new ReportSnapshot();
|
||||
clone.reportId = reportId;
|
||||
clone.revision = revision;
|
||||
for (JournalTask task : tasks)
|
||||
clone.tasks.add(task.copy());
|
||||
clone.operationIds = new LinkedHashMap<>(operationIds);
|
||||
return clone;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Message.Main.ProtocolHelloMessage;
|
||||
import lion.Message.Main.TaskReportMessage;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Collection;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
/**
|
||||
* 报告生命周期:构建不可变快照、先落盘后发送、按 ACK 精确回收。
|
||||
*
|
||||
* <p>它保证协议 v2 的两条关键性质:
|
||||
* <ul>
|
||||
* <li>快照在写通道之前就已落盘,写失败或进程崩溃都不会让一份「已上报」的任务丢失;</li>
|
||||
* <li>只有收到匹配版本的成功 ACK 才移除任务,且只移除停在同一代的任务记录——
|
||||
* 更新的任务与 revision 永不误删。</li>
|
||||
* </ul>
|
||||
*
|
||||
* <p>能力协商结果由 {@link #negotiate} 决定:在收到 version2 hello 之前一律走旧协议路径,
|
||||
* 旧主站不会看到任何它不认识的消息。
|
||||
*/
|
||||
@Slf4j
|
||||
public class TaskReporter {
|
||||
|
||||
private final TaskJournal journal;
|
||||
|
||||
private final Supplier<String> reportIdSupplier;
|
||||
|
||||
private volatile boolean reliableV2;
|
||||
|
||||
public TaskReporter(TaskJournal journal) {
|
||||
this(journal, () -> UUID.randomUUID().toString());
|
||||
}
|
||||
|
||||
TaskReporter(TaskJournal journal, Supplier<String> reportIdSupplier) {
|
||||
this.journal = journal;
|
||||
this.reportIdSupplier = reportIdSupplier;
|
||||
}
|
||||
|
||||
/**
|
||||
* 处理对端 hello。
|
||||
*
|
||||
* <p>只有声明了 version2 + reliable-task-v2 才升级;其余情况保持旧协议,
|
||||
* 未知/过低版本不得导致连接中断。
|
||||
*/
|
||||
public void negotiate(ProtocolHelloMessage hello) {
|
||||
if (hello != null && hello.supportsReliableTask()) {
|
||||
if (!reliableV2)
|
||||
log.info("主站已协商可靠任务协议 v2");
|
||||
reliableV2 = true;
|
||||
} else {
|
||||
log.info("对端未声明可靠任务协议,继续使用旧上报路径");
|
||||
}
|
||||
}
|
||||
|
||||
public boolean reliableV2() {
|
||||
return reliableV2;
|
||||
}
|
||||
|
||||
/** 连接断开后复位协商结果;新连接必须重新证明支持 v2。 */
|
||||
public void resetNegotiation() {
|
||||
reliableV2 = false;
|
||||
}
|
||||
|
||||
public boolean hasPendingReports() {
|
||||
return !journal.pendingReports().isEmpty();
|
||||
}
|
||||
|
||||
public Collection<TaskJournal.ReportSnapshot> pendingReports() {
|
||||
return List.copyOf(journal.pendingReports().values());
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建并持久化一份快照。
|
||||
*
|
||||
* <p>调用方必须在本方法成功返回之后才写通道;返回即代表内容已落盘可重发。
|
||||
*/
|
||||
public TaskJournal.ReportSnapshot persistSnapshot(List<TaskJournal.JournalTask> tasks,
|
||||
Map<Integer, String> operationIds) throws IOException {
|
||||
long revision = journal.nextRevision();
|
||||
TaskJournal.ReportSnapshot snapshot = TaskJournal.ReportSnapshot.of(
|
||||
reportIdSupplier.get(), revision, tasks, operationIds);
|
||||
journal.putPendingReport(snapshot);
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
/** 把快照转成线路消息;重发复用同一 reportId 与 revision。 */
|
||||
public TaskReportMessage toMessage(TaskJournal.ReportSnapshot snapshot) {
|
||||
TaskReportMessage message = new TaskReportMessage();
|
||||
message.setReportId(snapshot.reportId);
|
||||
message.setRevision(snapshot.revision);
|
||||
message.setGalleryTasks(snapshot.toGalleryTasks());
|
||||
message.setOperationIds(snapshot.operationIds);
|
||||
return message;
|
||||
}
|
||||
|
||||
/**
|
||||
* 结清一次成功的 ACK。
|
||||
*
|
||||
* <p>它只做两件事:移除该报告的待发快照,并回传快照内容供调用方判断哪些任务可以真正
|
||||
* 从队列移除。判断「能否移除」必须由持有运行期队列的调用方完成——ACK 只说明主站已收到
|
||||
* 报告,并不说明任务已经完成。
|
||||
*
|
||||
* @return 被确认的快照;报告未知、失败或版本已被更新时返回 {@code null}
|
||||
*/
|
||||
public TaskJournal.ReportSnapshot settleAck(String reportId, boolean success) throws IOException {
|
||||
TaskJournal.ReportSnapshot snapshot = journal.pendingReport(reportId);
|
||||
if (snapshot == null) {
|
||||
log.debug("收到未知或重复的报告 ACK,忽略: reportId={}", shortId(reportId));
|
||||
return null;
|
||||
}
|
||||
if (!success) {
|
||||
log.warn("主站未提交报告,保留快照等待重试: reportId={}, revision={}", shortId(reportId), snapshot.revision);
|
||||
return null;
|
||||
}
|
||||
journal.removePendingReport(reportId, snapshot.revision);
|
||||
log.info("报告已确认: reportId={}, revision={}", shortId(reportId), snapshot.revision);
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
private static String shortId(String id) {
|
||||
return id == null ? null : id.substring(0, Math.min(8, id.length()));
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,17 @@
|
||||
package lion;
|
||||
|
||||
import io.netty.util.concurrent.Promise;
|
||||
import lion.Config.Config;
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.Message.*;
|
||||
import lion.Message.Main.*;
|
||||
import lion.Service.DeleteService;
|
||||
import lion.Service.SubscriptionSnapshotStore;
|
||||
import lion.Service.DownloadCheckService;
|
||||
import lion.Service.PrimaryChannelTracker;
|
||||
import lion.Service.RekickPolicy;
|
||||
import lion.Service.ReliableTaskService;
|
||||
import lion.Service.TaskJournal;
|
||||
import lion.Service.NodeErrorCode;
|
||||
import io.netty.bootstrap.ServerBootstrap;
|
||||
import io.netty.channel.*;
|
||||
import io.netty.channel.nio.NioEventLoopGroup;
|
||||
@@ -14,22 +20,38 @@ import io.netty.channel.socket.nio.NioSocketChannel;
|
||||
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.Socket;
|
||||
import java.nio.file.Path;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
|
||||
@Slf4j
|
||||
public class storageNode {
|
||||
|
||||
ChannelFuture channelFuture;
|
||||
// 主站通道引用由 PrimaryChannelTracker 统一管理,避免重复认证把可用通道误清空。
|
||||
final PrimaryChannelTracker primaryChannel = new PrimaryChannelTracker();
|
||||
|
||||
Channel server;
|
||||
// 有待上报任务却无可用通道时,主动重新唤起主站,避免干等下一个探测周期。
|
||||
final RekickPolicy rekickPolicy = new RekickPolicy();
|
||||
|
||||
Channel node;
|
||||
// 重新唤起主站是阻塞式端口探测(最多 21×3 秒),不得占用 5 秒调度线程,
|
||||
// 否则下载进度扫描与压缩都会被拖住;同一时刻只允许一次探测。
|
||||
final java.util.concurrent.atomic.AtomicBoolean rekicking = new java.util.concurrent.atomic.AtomicBoolean();
|
||||
|
||||
// 专门用于阻塞式重连探测;不能复用单线程的 checkThreadPool,
|
||||
// 在同一个线程池里提交阻塞任务会把调度线程自己堵住。
|
||||
final ExecutorService rekickExecutor = Executors.newSingleThreadExecutor(r -> {
|
||||
Thread thread = new Thread(r, "storage-node-rekick");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
|
||||
volatile Channel node;
|
||||
|
||||
DownloadCheckService downloadCheckService;
|
||||
|
||||
@@ -39,24 +61,82 @@ public class storageNode {
|
||||
|
||||
ScheduledExecutorService checkThreadPool;
|
||||
|
||||
HashMap<Integer, Promise<AbstractMessage>> promises;
|
||||
|
||||
int counter;
|
||||
|
||||
ReentrantLock lock;
|
||||
|
||||
final SubscriptionSnapshotStore subscriptionSnapshotStore;
|
||||
|
||||
final ExecutorService subscriptionApplyExecutor;
|
||||
|
||||
/**
|
||||
* 可靠任务协议的执行器。
|
||||
*
|
||||
* <p>命令处理与日志落盘都是阻塞式磁盘 I/O,绝不能放在 Netty 事件循环上;单线程还顺带
|
||||
* 保证同一 gid 的「命令 → 回执 → 上报」顺序确定。
|
||||
*/
|
||||
final ExecutorService reliableExecutor;
|
||||
|
||||
/** 可靠任务协议的状态编排;为 null 时节点只跑旧协议。 */
|
||||
ReliableTaskService reliableTaskService;
|
||||
|
||||
/**
|
||||
* 每份待 ACK 报告最近一次写入的通道与时刻。
|
||||
*
|
||||
* <p>判断重发同时看两件事:通道是否变化、是否超过重发间隔。只看通道的话,TCP 仍连着、
|
||||
* 但主站没落库或 ACK 在链路上丢失时通道不会变,报告会永远停在待确认;只看时间又会在
|
||||
* 重连后仍等满一个间隔。写入失败时该记录会被清除,下一轮立即重试。
|
||||
*/
|
||||
final Map<String, ReportSend> reportSends = new ConcurrentHashMap<>();
|
||||
|
||||
/** 一份报告的最近发送目标与时刻。 */
|
||||
record ReportSend(Channel channel, long sentAt) {}
|
||||
|
||||
/** 同一份待确认报告的重发间隔上限;保证 ACK 丢失后能在有限时间内自愈。 */
|
||||
static final long REPORT_RETRY_INTERVAL_MILLIS = 5_000L;
|
||||
|
||||
/** 报告重发间隔,可在测试中调小以便断言同通道重试;非 final 契约与字段一致。 */
|
||||
long reportRetryIntervalMillis = REPORT_RETRY_INTERVAL_MILLIS;
|
||||
|
||||
public static String storagePath = "/root/gallery/gallery/";
|
||||
|
||||
public storageNode(){
|
||||
queue = new HashMap<>();
|
||||
public storageNode(SubscriptionSnapshotStore subscriptionSnapshotStore){
|
||||
this(subscriptionSnapshotStore, null, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* 完整构造。
|
||||
*
|
||||
* <p>{@code startNetwork=false} 专供测试:不绑定端口、不主动订阅唤醒主站、不启动调度器,
|
||||
* 因此可以在无网络、无真实主站的条件下直接驱动 {@link #mainThread()} 与消息处理器。
|
||||
* 这避免了历史测试会去连接真实主站端口、或与生产争用固定端口的问题。
|
||||
*
|
||||
* @param journalDirectory 任务日志目录;为 null 时该节点启用旧协议路径
|
||||
*/
|
||||
storageNode(SubscriptionSnapshotStore subscriptionSnapshotStore, Path journalDirectory, boolean startNetwork){
|
||||
this.subscriptionSnapshotStore = subscriptionSnapshotStore;
|
||||
this.subscriptionApplyExecutor = Executors.newSingleThreadExecutor(r -> {
|
||||
Thread thread = new Thread(r, "subscription-snapshot-apply");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
this.reliableExecutor = Executors.newSingleThreadExecutor(r -> {
|
||||
Thread thread = new Thread(r, "storage-node-reliable");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
queue = new ConcurrentHashMap<>();
|
||||
tempQueue = new HashMap<>();
|
||||
lock = new ReentrantLock();
|
||||
counter = 0;
|
||||
promises = new HashMap<>();
|
||||
|
||||
if (!startNetwork) {
|
||||
downloadCheckService = new DownloadCheckService(queue, false);
|
||||
if (journalDirectory != null)
|
||||
reliableTaskService = restoreReliableService(journalDirectory);
|
||||
return;
|
||||
}
|
||||
|
||||
int real_port = CustomUtil._findIdlePort(26321);
|
||||
|
||||
channelFuture = new ServerBootstrap()
|
||||
ChannelFuture bindFuture = new ServerBootstrap()
|
||||
.channel(NioServerSocketChannel.class)
|
||||
.group(new NioEventLoopGroup())
|
||||
.childHandler(new ChannelInitializer<NioSocketChannel>() {
|
||||
@@ -67,8 +147,75 @@ public class storageNode {
|
||||
channel.pipeline().addLast(new MyChannelInboundHandlerAdapter(tempQueue));
|
||||
}
|
||||
}).bind(real_port);
|
||||
// 必须等绑定结果:端口被别人抢到(探测与绑定之间的 TOCTOU)时,若不检查,
|
||||
// 节点会「启动成功」却根本没在监听,主站怎么连都连不上。
|
||||
try {
|
||||
bindFuture.sync();
|
||||
log.info("listening on port {}", real_port);
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("绑定端口 " + real_port + " 被中断", e);
|
||||
} catch (Exception e) {
|
||||
throw new IllegalStateException("绑定端口 " + real_port + " 失败", e);
|
||||
}
|
||||
|
||||
wakeMainServer();
|
||||
downloadCheckService = new DownloadCheckService(queue);
|
||||
if (journalDirectory != null)
|
||||
reliableTaskService = restoreReliableService(journalDirectory);
|
||||
checkThreadPool = Executors.newScheduledThreadPool(1);
|
||||
checkThreadPool.scheduleAtFixedRate(this::mainThread, 5, 5, TimeUnit.SECONDS);
|
||||
}
|
||||
|
||||
/**
|
||||
* 建立可靠任务服务并恢复上次运行的任务与待 ACK 报告。
|
||||
*
|
||||
* <p>恢复失败(例如日志损坏)直接抛出并阻止启动:带着一份空日志继续跑,会把主站上已
|
||||
* 完成的任务重新当成未完成,甚至重放已执行的删除副作用。宁可拒绝启动,由运维人工处理。
|
||||
*/
|
||||
private ReliableTaskService restoreReliableService(Path journalDirectory){
|
||||
ReliableTaskService service = new ReliableTaskService(journalDirectory, queue, downloadCheckService, Path.of(storagePath));
|
||||
try {
|
||||
service.restore();
|
||||
} catch (java.io.IOException e) {
|
||||
throw new IllegalStateException("恢复任务日志失败,拒绝以不一致状态启动: " + journalDirectory, e);
|
||||
}
|
||||
return service;
|
||||
}
|
||||
|
||||
/**
|
||||
* 在独立线程里重新唤起主站。
|
||||
*
|
||||
* <p>{@link #wakeMainServer()} 是阻塞的端口探测,最坏可占 21×3 秒;直接在
|
||||
* {@link #mainThread()} 里调用会让下载扫描与压缩停摆,因此投递到独立线程执行。
|
||||
* 该执行器与 5 秒调度器分离,避免阻塞任务把调度线程自己堵住;
|
||||
* 同一时刻只允许一次探测,防止主站长时间离线时堆积。
|
||||
*/
|
||||
void rekickMainServerAsync(){
|
||||
if (!rekicking.compareAndSet(false, true))
|
||||
return;
|
||||
try {
|
||||
rekickExecutor.execute(() -> {
|
||||
try {
|
||||
wakeMainServer();
|
||||
} finally {
|
||||
rekicking.set(false);
|
||||
}
|
||||
});
|
||||
} catch (Exception e) {
|
||||
rekicking.set(false);
|
||||
log.warn("安排主站重连失败:{}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 按既定顺序敲主站监听端口,请它回来建立数据通道。
|
||||
*
|
||||
* <p>启动时调用一次,另在「有待上报任务但无可用通道」时按 {@link RekickPolicy}
|
||||
* 限流重试。主站侧的可用性探测只能覆盖它认为连接仍在的情况;一旦双方都判定
|
||||
* 对方不在,只有本方法能重新建立通道,否则任务状态会一直卡在未完成。
|
||||
*/
|
||||
void wakeMainServer(){
|
||||
int i;
|
||||
for(i=0; i<=20; i++) {
|
||||
try (Socket socket = new Socket()) {
|
||||
@@ -85,21 +232,37 @@ public class storageNode {
|
||||
if (i==20) {
|
||||
log.info("server connect failed");
|
||||
}
|
||||
downloadCheckService = new DownloadCheckService(queue, promises);
|
||||
checkThreadPool = Executors.newScheduledThreadPool(1);
|
||||
checkThreadPool.scheduleAtFixedRate(this::mainThread, 5, 5, TimeUnit.SECONDS);
|
||||
}
|
||||
|
||||
public void mainThread(){
|
||||
try {
|
||||
lock.lock();
|
||||
try {
|
||||
if(!tempQueue.isEmpty()){
|
||||
queue.putAll(tempQueue);
|
||||
tempQueue.clear();
|
||||
}
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
|
||||
// 有暂存内容却无可用通道:告警,并按限流主动重新唤起主站。
|
||||
// 这里只告警与重连,不提前返回——本地下载与压缩必须继续推进,
|
||||
// 否则主站恢复前任务会在节点侧整体停摆。
|
||||
// 该判断在 downloadCheck 之前:正在下载、进度无变化的任务稍后会走
|
||||
// 「无需上报」的提前返回路径,若只在回报分支里判定,通道失联后
|
||||
// 这类任务既不会被发现也不会触发重连,就复现了历史上的静默卡死。
|
||||
if (!queue.isEmpty() && !primaryChannel.usable()) {
|
||||
log.warn("主站通道不可用,{} 个任务状态暂存待上报", queue.size());
|
||||
if (rekickPolicy.shouldRekick(true, false, System.currentTimeMillis()))
|
||||
rekickMainServerAsync();
|
||||
}
|
||||
|
||||
//检查,当任务状态发生变化即方法返回true时,再更新,否则return
|
||||
if (!downloadCheckService.downloadCheck()) {
|
||||
// 即便本轮没有状态变化,也必须先重试待 ACK 报告:连接可能刚刚恢复,
|
||||
// 而重发不能依赖「有任务变化」这个条件,否则 ACK 丢失后会一直卡住。
|
||||
retryPendingReportsIfNegotiated();
|
||||
boolean isSkip = true;
|
||||
//返回false之后,还要额外检查是否有压缩完成的任务
|
||||
if(!queue.isEmpty())
|
||||
@@ -110,73 +273,325 @@ public class storageNode {
|
||||
}
|
||||
if(isSkip) {
|
||||
counter++;
|
||||
if (server != null && server.isActive() && counter > 10) {
|
||||
server.writeAndFlush(new MaintainMessage());
|
||||
if (primaryChannel.usable() && counter > 10) {
|
||||
primaryChannel.current().writeAndFlush(new MaintainMessage());
|
||||
counter = 0;
|
||||
}
|
||||
return;
|
||||
}
|
||||
}
|
||||
//主站未连接时不上报,队列保留、等主站重连时重放(无法写通道时上面的分支已兜住)。
|
||||
Channel target = primaryChannel.current();
|
||||
if (target == null)
|
||||
return;
|
||||
//没有待上报任务时不必发送空数组(此时 downloadCheck 为 false 且无压缩完成任务)。
|
||||
if (queue.isEmpty())
|
||||
return;
|
||||
// 已协商 v2:走可靠报告路径(快照先落盘、任务留待 ACK 后才清理),
|
||||
// 旧路径的「发完即出队」在没有确认的链路上会丢任务。
|
||||
if (reliableTaskService != null && reliableTaskService.reporter().reliableV2()) {
|
||||
flushReport();
|
||||
return;
|
||||
}
|
||||
//发送
|
||||
//上锁后再发送,避免出现发送完之后再下载完成
|
||||
lock.lock();
|
||||
try {
|
||||
DownloadStatusMessage downloadStatusMessage = new DownloadStatusMessage();
|
||||
downloadStatusMessage.setGalleryTasks(queue.values().toArray(GalleryTask[]::new));
|
||||
server.writeAndFlush(downloadStatusMessage);
|
||||
|
||||
queue.entrySet().removeIf(entry -> entry.getValue().is_compress_complete());
|
||||
log.info("任务状态发送完成");
|
||||
|
||||
// 旧协议没有 ACK,只能在写成功后才允许移除:写失败(对方已断开、缓冲区报错等)
|
||||
// 时如果照样出队,完成状态会永久丢失且再也不会重发。
|
||||
java.util.Set<Integer> completedGids = new java.util.HashSet<>();
|
||||
for (GalleryTask galleryTask : queue.values())
|
||||
if (galleryTask.is_compress_complete())
|
||||
completedGids.add(galleryTask.getGid());
|
||||
|
||||
ChannelFuture future = target.writeAndFlush(downloadStatusMessage);
|
||||
future.addListener(result -> {
|
||||
if (!result.isSuccess()) {
|
||||
log.warn("任务状态写入失败,保留任务待重发: {}", result.cause() == null ? "unknown" : result.cause().getMessage());
|
||||
return;
|
||||
}
|
||||
lock.lock();
|
||||
try {
|
||||
queue.entrySet().removeIf(entry -> completedGids.contains(entry.getKey())
|
||||
&& entry.getValue().is_compress_complete());
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
});
|
||||
log.info("任务状态发送完成");
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
}catch (Exception e){
|
||||
log.error("发送任务状态时发生异常:{}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
int counter;
|
||||
|
||||
/**
|
||||
* 仅重试已登记但尚未确认的报告;不做新扫描,也不生成新快照。
|
||||
*
|
||||
* <p>用于「本轮无状态变化」的路径:任务不再变化并不意味着报告已被确认,连接恢复后
|
||||
* 仍必须把旧快照重发出去。
|
||||
*/
|
||||
private void retryPendingReportsIfNegotiated(){
|
||||
if (reliableTaskService == null || !reliableTaskService.reporter().reliableV2())
|
||||
return;
|
||||
if (reliableTaskService.pendingReports().isEmpty())
|
||||
return;
|
||||
Channel target = primaryChannel.current();
|
||||
if (target == null)
|
||||
return;
|
||||
long now = System.currentTimeMillis();
|
||||
for (TaskJournal.ReportSnapshot pending : reliableTaskService.pendingReports()) {
|
||||
ReportSend previous = reportSends.get(pending.reportId);
|
||||
boolean sameChannel = previous != null && previous.channel() == target;
|
||||
boolean fresh = previous != null && now - previous.sentAt() < reportRetryIntervalMillis;
|
||||
if (sameChannel && fresh)
|
||||
continue;
|
||||
sendReport(target, pending);
|
||||
log.info("重试待确认报告: reportId={}, revision={}", pending.reportId, pending.revision);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 该消息是否只有主站会发。
|
||||
*
|
||||
* <p>白名单而非黑名单:只有这些类型能证明对端是主站并据此认领通道引用。
|
||||
* 名单严格等于主站在 {@code RemoteService} 里实际会发出的类型;
|
||||
* {@code ResponseMessage}、{@code DownloadStatusMessage}、{@code MaintainMessage}
|
||||
* 都是本节点自己的出站类型,备机({@code lionwebsiteside})也会发身份消息,
|
||||
* 都不能用来认领主站引用。
|
||||
*/
|
||||
static boolean isPrimaryEvidence(AbstractMessage message) {
|
||||
if (message.messageType == AbstractMessage.IDENTITY_MESSAGE) {
|
||||
// 备机也用身份消息自我介绍,只有 identity 为 lionwebsite 的才是主站。
|
||||
IdentityMessage identity = (IdentityMessage) message;
|
||||
return "lionwebsite".equals(identity.getIdentity());
|
||||
}
|
||||
// v2 能力协商只由主站发起,同样构成主站证据;即便对端版本过低也先认领通道。
|
||||
if (message.messageType == AbstractMessage.PROTOCOL_HELLO_MESSAGE)
|
||||
return true;
|
||||
return message.messageType == AbstractMessage.DOWNLOAD_POST_MESSAGE
|
||||
|| message.messageType == AbstractMessage.DELETE_GALLERY_MESSAGE
|
||||
|| message.messageType == AbstractMessage.AVAILABLE_CHECK_MESSAGE
|
||||
|| message.messageType == AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE;
|
||||
}
|
||||
|
||||
/**
|
||||
* 在可靠执行器上处理命令,并在完成后立即把结果回给主站。
|
||||
*
|
||||
* <p>结果必须回到当初收到命令的那条 channel:主站按 messageId 与通道配对等待响应,
|
||||
* 若改写到「当前主站通道」上,命令与响应可能落在不同连接而永远配不上。
|
||||
*/
|
||||
private void handleTaskCommandAsync(ChannelHandlerContext ctx, TaskCommandMessage command){
|
||||
reliableExecutor.execute(() -> {
|
||||
byte result;
|
||||
try {
|
||||
result = reliableTaskService.handleCommand(command);
|
||||
} catch (Exception e) {
|
||||
log.error("处理可靠任务命令失败: operationId={}", command.getOperationId(), e);
|
||||
result = NodeErrorCode.IO_ERROR;
|
||||
}
|
||||
// 命令成功后立刻尝试上报一份新状态,让主站尽快看到结果。
|
||||
if (NodeErrorCode.isTerminal(result))
|
||||
flushReportAsync();
|
||||
ctx.writeAndFlush(new ResponseMessage(command.messageId, result));
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 生成并发送一份可靠报告(若已协商 v2)。
|
||||
*
|
||||
* <p>顺序至关重要:先落盘快照,再写通道。写失败时快照仍留在日志里,重连后重发;
|
||||
* 因此任何一次发送失败都不会让任务从节点侧消失。
|
||||
*/
|
||||
void flushReportAsync(){
|
||||
reliableExecutor.execute(this::flushReport);
|
||||
}
|
||||
|
||||
/** 同步版本,供调度线程与测试直接调用。 */
|
||||
void flushReport(){
|
||||
if (reliableTaskService == null || !reliableTaskService.reporter().reliableV2())
|
||||
return;
|
||||
try {
|
||||
// 先把扫描出的最新进度对齐日志,保证报告内容与磁盘事实一致。
|
||||
reliableTaskService.refreshJournalFromQueue();
|
||||
Channel target = primaryChannel.current();
|
||||
if (target == null)
|
||||
return;
|
||||
|
||||
// 断线后重发尚未 ACK 的快照,且只在「换了通道」时发一次,避免每轮调度重复轰炸。
|
||||
for (TaskJournal.ReportSnapshot pending : reliableTaskService.pendingReports()) {
|
||||
ReportSend previous = reportSends.get(pending.reportId);
|
||||
// 未发过、通道已变更、或超过重发间隔:都需要重发(reportId 与 revision 保持不变)。
|
||||
boolean sameChannel = previous != null && previous.channel() == target;
|
||||
boolean fresh = previous != null
|
||||
&& System.currentTimeMillis() - previous.sentAt() < reportRetryIntervalMillis;
|
||||
if (sameChannel && fresh)
|
||||
continue;
|
||||
sendReport(target, pending);
|
||||
log.info("重发待确认报告: reportId={}, revision={}", pending.reportId, pending.revision);
|
||||
}
|
||||
|
||||
Optional<TaskJournal.ReportSnapshot> snapshot = reliableTaskService.buildReport();
|
||||
if (snapshot.isEmpty())
|
||||
return;
|
||||
sendReport(target, snapshot.get());
|
||||
log.info("可靠报告已发送: reportId={}, revision={}",
|
||||
snapshot.get().reportId, snapshot.get().revision);
|
||||
} catch (Exception e) {
|
||||
log.warn("发送可靠报告失败,快照保留待重发: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 写一份报告并登记发送时刻;写入失败时清除登记,让下一轮立即重试。
|
||||
*
|
||||
* <p>只有真正写成功才记时刻:若先记后写,一次写失败会被当成「刚发过」而白白等一个
|
||||
* 重发周期。监听器在写失败时也会清除登记,覆盖「已入队但最终写失败」的情况。
|
||||
*/
|
||||
private void sendReport(Channel target, TaskJournal.ReportSnapshot snapshot){
|
||||
String reportId = snapshot.reportId;
|
||||
ChannelFuture future = target.writeAndFlush(reliableTaskService.reporter().toMessage(snapshot));
|
||||
reportSends.put(reportId, new ReportSend(target, System.currentTimeMillis()));
|
||||
future.addListener(result -> {
|
||||
if (!result.isSuccess()) {
|
||||
reportSends.remove(reportId);
|
||||
log.warn("报告写入通道失败,下一轮重发: reportId={}", reportId);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
class MyChannelInboundHandlerAdapter extends ChannelInboundHandlerAdapter{
|
||||
Map<Integer, GalleryTask> queue;
|
||||
|
||||
/** 本通道是否充当过主站通道。按连接记录,避免断开时引用已回退到别的通道而判断失准。 */
|
||||
boolean claimedPrimary;
|
||||
|
||||
public MyChannelInboundHandlerAdapter(Map<Integer, GalleryTask> queue) {
|
||||
this.queue = queue;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void channelRead(ChannelHandlerContext ctx, Object msg) {
|
||||
if (msg instanceof SubscriptionSnapshotMessage snapshot)
|
||||
log.info("收到订阅快照 revision={}", shortRevision(snapshot.getRevision()));
|
||||
else
|
||||
log.info(String.valueOf(msg));
|
||||
AbstractMessage abstractMessage = (AbstractMessage) msg;
|
||||
|
||||
// 只有主站会发的消息类型(身份/任务下发/删除/探活/订阅快照)同时承担两件事:
|
||||
// 证明「主站仍在」,以及恢复通道引用。并发认证时引用可能先指向 X 再被 Y 覆盖,
|
||||
// Y 断开后引用被清空,而仍可用的 X 继续发送探活等消息;收到这些消息即证明
|
||||
// X 就是主站,必须无条件认领——引用被清空时 current() 为 null,
|
||||
// 任何以其为前提的判断都不成立,自愈也就无从触发。
|
||||
if (isPrimaryEvidence(abstractMessage)) {
|
||||
claimedPrimary = true;
|
||||
primaryChannel.onPrimaryMessage(ctx.channel());
|
||||
subscriptionSnapshotStore.markPrimaryContact();
|
||||
}
|
||||
|
||||
switch (abstractMessage.messageType){
|
||||
case AbstractMessage.IDENTITY_MESSAGE -> {
|
||||
IdentityMessage identityMessage = (IdentityMessage) abstractMessage;
|
||||
if(identityMessage.getIdentity().equals("lionwebsite")) {
|
||||
server = ctx.channel();
|
||||
// 引用已在上面按消息类型认领,这里只保留「上线」日志语义。
|
||||
log.info("server 上线");
|
||||
} else if(identityMessage.getIdentity().equals("lionwebsiteside")){
|
||||
node = ctx.channel();
|
||||
log.info("node上线");
|
||||
downloadCheckService.setNode(node);
|
||||
}
|
||||
}
|
||||
case AbstractMessage.DOWNLOAD_POST_MESSAGE -> {
|
||||
DownloadPostMessage dpm = (DownloadPostMessage) abstractMessage;
|
||||
lock.lock();
|
||||
//添加到队列方法返回真说明该任务已下载完成,直接发送下载进度
|
||||
if(downloadCheckService.addToQueue(dpm.getGalleryTask())){
|
||||
try {
|
||||
//每次收到任务都重新检查并立即回传当前状态,供主站的单任务重试接口使用
|
||||
GalleryTask currentTask = downloadCheckService.addToQueue(dpm.getGalleryTask());
|
||||
DownloadStatusMessage downloadStatusMessage = new DownloadStatusMessage();
|
||||
downloadStatusMessage.setGalleryTasks(new GalleryTask[]{dpm.getGalleryTask()});
|
||||
server.writeAndFlush(downloadStatusMessage);
|
||||
}
|
||||
downloadStatusMessage.setGalleryTasks(new GalleryTask[]{currentTask});
|
||||
Channel target = primaryChannel.current();
|
||||
if (target != null && target.isActive())
|
||||
target.writeAndFlush(downloadStatusMessage);
|
||||
log.info(String.valueOf(queue));
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
ctx.writeAndFlush(new ResponseMessage(dpm.messageId, (byte) 0));
|
||||
}
|
||||
case AbstractMessage.DELETE_GALLERY_MESSAGE -> {
|
||||
DeleteGalleryMessage deleteGalleryMessage = (DeleteGalleryMessage) abstractMessage;
|
||||
byte result = DeleteService.deleteAll(storagePath + deleteGalleryMessage.getGalleryName());
|
||||
String galleryName = deleteGalleryMessage.getGalleryName();
|
||||
lock.lock();
|
||||
try {
|
||||
// 删除任务时也要从待下载队列移除,避免继续向主站上报已经删除的任务状态。
|
||||
storageNode.this.queue.entrySet().removeIf(entry -> galleryName.equals(entry.getValue().getName()));
|
||||
queue.entrySet().removeIf(entry -> galleryName.equals(entry.getValue().getName()));
|
||||
} finally {
|
||||
lock.unlock();
|
||||
}
|
||||
byte result = DeleteService.deleteWithin(storagePath, galleryName);
|
||||
ResponseMessage responseMessage = new ResponseMessage(deleteGalleryMessage.messageId, result);
|
||||
ctx.writeAndFlush(responseMessage);
|
||||
}
|
||||
case AbstractMessage.AVAILABLE_CHECK_MESSAGE -> {
|
||||
AvailableCheckMessage acm = (AvailableCheckMessage) abstractMessage;
|
||||
ResponseMessage responseMessage = new ResponseMessage(acm.messageId, (byte)0);
|
||||
ctx.writeAndFlush(responseMessage);
|
||||
}
|
||||
case AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE -> {
|
||||
SubscriptionSnapshotMessage snapshotMessage = (SubscriptionSnapshotMessage) abstractMessage;
|
||||
if (!Config.subscriptionSyncEnabled || !ctx.channel().equals(primaryChannel.current())) {
|
||||
ctx.writeAndFlush(new ResponseMessage(snapshotMessage.messageId, SubscriptionSnapshotStore.APPLY_INVALID));
|
||||
return;
|
||||
}
|
||||
subscriptionApplyExecutor.execute(() -> {
|
||||
SubscriptionSnapshotStore.ApplyResult result = subscriptionSnapshotStore.apply(snapshotMessage);
|
||||
ctx.writeAndFlush(new ResponseMessage(snapshotMessage.messageId, result.code()));
|
||||
log.info("订阅快照处理完成 revision={} result={}", shortRevision(snapshotMessage.getRevision()), result.code());
|
||||
});
|
||||
}
|
||||
case AbstractMessage.PROTOCOL_HELLO_MESSAGE -> {
|
||||
ProtocolHelloMessage helloMessage = (ProtocolHelloMessage) abstractMessage;
|
||||
if (reliableTaskService == null) {
|
||||
// 未启用可靠协议(例如测试构造或旧部署):按未知类型安静忽略,
|
||||
// 既不回包也不断开,避免把新主站误判为协议错误。
|
||||
log.info("未启用可靠任务协议,忽略 hello");
|
||||
return;
|
||||
}
|
||||
reliableTaskService.reporter().negotiate(helloMessage);
|
||||
// 节点只在确认对端支持 v2 后回 hello;旧主站因此永远收不到 type10。
|
||||
if (reliableTaskService.reporter().reliableV2())
|
||||
ctx.writeAndFlush(ProtocolHelloMessage.v2());
|
||||
}
|
||||
case AbstractMessage.TASK_COMMAND_MESSAGE -> {
|
||||
if (reliableTaskService == null) {
|
||||
log.warn("未启用可靠任务协议,拒绝任务命令");
|
||||
return;
|
||||
}
|
||||
handleTaskCommandAsync(ctx, (TaskCommandMessage) abstractMessage);
|
||||
}
|
||||
case AbstractMessage.TASK_REPORT_ACK_MESSAGE -> {
|
||||
if (reliableTaskService == null)
|
||||
return;
|
||||
TaskReportAckMessage ack = (TaskReportAckMessage) abstractMessage;
|
||||
reliableExecutor.execute(() -> {
|
||||
try {
|
||||
Set<Integer> cleared = reliableTaskService.applyAck(ack.getReportId(), ack.getResult());
|
||||
// 该报告的发送登记已无意义(无论是否清理了任务),清掉避免 map 无限增长。
|
||||
if (ack.isSuccess())
|
||||
reportSends.remove(ack.getReportId());
|
||||
if (!cleared.isEmpty())
|
||||
log.info("主站已确认报告,清理任务: {}", cleared);
|
||||
} catch (Exception e) {
|
||||
log.error("处理报告 ACK 失败: reportId={}", ack.getReportId(), e);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
//
|
||||
// //修复预览
|
||||
@@ -186,15 +601,41 @@ public class storageNode {
|
||||
|
||||
@Override
|
||||
public void channelUnregistered(ChannelHandlerContext ctx) {
|
||||
if(ctx.channel().equals(server)) {
|
||||
if(claimedPrimary) {
|
||||
log.info("server 下线");
|
||||
server = null;
|
||||
// 连接断开后 v2 能力不再成立:新连接必须重新协商,否则可能向旧主站
|
||||
// 发送它不认识的 type12/13。已登记的报告发送目标也一并作废。
|
||||
// 仅当已无其它可用主站通道时才复位:并发认证下可能仍有另一条已协商 v2 的通道,
|
||||
// 此时回退到旧协议会白白放弃已确认的可靠上报能力。
|
||||
boolean noSurvivingPrimary = primaryChannel.current() == null;
|
||||
if (noSurvivingPrimary && reliableTaskService != null)
|
||||
reliableTaskService.reporter().resetNegotiation();
|
||||
reportSends.clear();
|
||||
} else if(ctx.channel().equals(node)){
|
||||
log.info("node 下线");
|
||||
node = null;
|
||||
downloadCheckService.setNode(null);
|
||||
}
|
||||
primaryChannel.unregister(ctx.channel());
|
||||
}
|
||||
}
|
||||
|
||||
private static String shortRevision(String revision) {
|
||||
return revision == null ? null : revision.substring(0, Math.min(12, revision.length()));
|
||||
}
|
||||
|
||||
/**
|
||||
* 关闭节点持有的执行器与调度器。
|
||||
*
|
||||
* <p>线程都是守护线程,生产进程退出不依赖本方法;但测试与热重启需要显式释放,
|
||||
* 否则会残留仍在调度或写日志的线程。
|
||||
*/
|
||||
public void shutdown(){
|
||||
if (checkThreadPool != null)
|
||||
checkThreadPool.shutdownNow();
|
||||
rekickExecutor.shutdownNow();
|
||||
subscriptionApplyExecutor.shutdownNow();
|
||||
reliableExecutor.shutdownNow();
|
||||
if (downloadCheckService != null)
|
||||
downloadCheckService.shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,2 +1,17 @@
|
||||
DouNaiV2ray=https://aaaa.gay/link/X7zEqkIx5gtIGugO?client=v2
|
||||
DouNaiClash=https://aaaa.gay/link/X7zEqkIx5gtIGugO?client=clashmeta
|
||||
# 订阅快照由主站通过 Netty 长连接推送;不要在此保存上游订阅地址或 Key。
|
||||
SubscriptionSyncEnabled=false
|
||||
SubscriptionSyncSecret=
|
||||
SubscriptionDataDir=/root/gallery/storageNode/sub
|
||||
SubscriptionMaxStaleSeconds=604800
|
||||
SubscriptionMaxPayloadBytes=52428800
|
||||
SubscriptionHttpPort=8889
|
||||
SubscriptionHttpWorkers=4
|
||||
SubscriptionSocketTimeoutMs=10000
|
||||
# 可靠任务协议的任务日志目录(独立于订阅数据目录,默认不接触订阅/测试生产目录)
|
||||
TaskJournalDir=/root/gallery/storageNode/task-journal
|
||||
|
||||
# 主站优先网关(与订阅服务共用8889;空值关闭,保留快照直读)
|
||||
# 生产设置 https://lionwebsite.xyz,仅允许固定origin,不含路径/认证信息。
|
||||
SubscriptionPrimaryUrl=
|
||||
# 离线成功GET的耐久队列,独立于snapshot/task-journal,权限700/600。
|
||||
SubscriptionEventDir=/root/gallery/storageNode/subscription-events
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<configuration>
|
||||
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>run.out</file>
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
|
||||
<fileNamePattern>run.%i.out</fileNamePattern>
|
||||
<minIndex>1</minIndex>
|
||||
<maxIndex>10</maxIndex>
|
||||
</rollingPolicy>
|
||||
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
|
||||
<maxFileSize>10MB</maxFileSize>
|
||||
</triggeringPolicy>
|
||||
<encoder>
|
||||
<pattern>%d{yyyy-MM-dd HH:mm:ss} [%level] %msg%n</pattern>
|
||||
</encoder>
|
||||
</appender>
|
||||
|
||||
<root level="info">
|
||||
<appender-ref ref="FILE"/>
|
||||
</root>
|
||||
</configuration>
|
||||
@@ -53,6 +53,132 @@
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.AvailableCheckMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.SubscriptionSnapshotMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.SubscriptionSnapshotPayload",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.SubscriptionAccountSnapshot",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.SubscriptionBindingSnapshot",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Service.SubscriptionSnapshotStore$Status",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.ProtocolHelloMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.TaskCommandMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.TaskReportMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.Main.TaskReportAckMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Service.TaskJournal$State",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Service.TaskJournal$JournalTask",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Service.TaskJournal$CommandReceipt",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Service.TaskJournal$ReportSnapshot",
|
||||
"allDeclaredConstructors" : true,
|
||||
"allPublicConstructors" : true,
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.AbstractMessage",
|
||||
"allDeclaredConstructors" : true,
|
||||
@@ -70,5 +196,15 @@
|
||||
"allPublicMethods" : true,
|
||||
"allDeclaredFields" : true,
|
||||
"allPublicFields" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.storageNode$MyChannelInboundHandlerAdapter",
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true
|
||||
},
|
||||
{
|
||||
"name": "lion.Message.MessageCodec",
|
||||
"allDeclaredMethods" : true,
|
||||
"allPublicMethods" : true
|
||||
}
|
||||
]
|
||||
@@ -1,9 +0,0 @@
|
||||
|
||||
org.slf4j.simpleLogger.defaultLogLevel=info
|
||||
|
||||
org.slf4j.simpleLogger.showDateTime=true
|
||||
|
||||
org.slf4j.simpleLogger.dateTimeFormat=yyyy-MM-dd HH:mm:ss
|
||||
|
||||
org.slf4j.simpleLogger.logFile=run.out
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package lion;
|
||||
|
||||
import lion.Config.Config;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 配置加载。
|
||||
*
|
||||
* <p>生产配置来自 {@code /root/gallery/storageNode/config.properties},任何一项读错都会改变
|
||||
* 订阅同步与分发端口的行为。这里对默认值填充、显式覆盖、文件缺失回退与「启用同步却
|
||||
* 没配密钥就拒绝启动」逐条断言。
|
||||
*/
|
||||
class ConfigTest {
|
||||
|
||||
@Test
|
||||
void fillsBuiltInDefaultsWhenKeysAreAbsent(@TempDir Path root) throws Exception {
|
||||
Path file = Files.writeString(root.resolve("config.properties"), "");
|
||||
|
||||
Config.loadConfig(file.toString());
|
||||
|
||||
assertFalse(Config.subscriptionSyncEnabled);
|
||||
assertEquals("/root/gallery/storageNode/sub", Config.subscriptionDataDir);
|
||||
assertEquals(604800L, Config.subscriptionMaxStaleSeconds);
|
||||
assertEquals(52428800, Config.subscriptionMaxPayloadBytes);
|
||||
assertEquals(8889, Config.subscriptionHttpPort);
|
||||
assertEquals(4, Config.subscriptionHttpWorkers);
|
||||
assertEquals(10000, Config.subscriptionSocketTimeoutMs);
|
||||
assertEquals("", Config.subscriptionPrimaryUrl);
|
||||
assertEquals("/root/gallery/storageNode/subscription-events", Config.subscriptionEventDir);
|
||||
}
|
||||
|
||||
@Test
|
||||
void readsExplicitValues(@TempDir Path root) throws Exception {
|
||||
Path file = Files.writeString(root.resolve("config.properties"), String.join("\n",
|
||||
"SubscriptionSyncEnabled=true",
|
||||
"SubscriptionSyncSecret=file-secret",
|
||||
"SubscriptionDataDir=/tmp/sub",
|
||||
"SubscriptionMaxStaleSeconds=120",
|
||||
"SubscriptionMaxPayloadBytes=4096",
|
||||
"SubscriptionHttpPort=9999",
|
||||
"SubscriptionHttpWorkers=2",
|
||||
"SubscriptionSocketTimeoutMs=500"));
|
||||
|
||||
Config.loadConfig(file.toString());
|
||||
|
||||
assertTrue(Config.subscriptionSyncEnabled);
|
||||
assertEquals("file-secret", Config.subscriptionSyncSecret);
|
||||
assertEquals("/tmp/sub", Config.subscriptionDataDir);
|
||||
assertEquals(120L, Config.subscriptionMaxStaleSeconds);
|
||||
assertEquals(4096, Config.subscriptionMaxPayloadBytes);
|
||||
assertEquals(9999, Config.subscriptionHttpPort);
|
||||
assertEquals(2, Config.subscriptionHttpWorkers);
|
||||
assertEquals(500, Config.subscriptionSocketTimeoutMs);
|
||||
}
|
||||
|
||||
/** 配置文件缺失不是致命错误,但要落到默认值而不是保留上一次的加载结果。 */
|
||||
@Test
|
||||
void missingFileFallsBackToDefaults(@TempDir Path root) {
|
||||
Config.subscriptionHttpPort = 12345;
|
||||
|
||||
Config.loadConfig(root.resolve("absent.properties").toString());
|
||||
|
||||
assertEquals(8889, Config.subscriptionHttpPort, "缺失文件应回到默认值");
|
||||
}
|
||||
|
||||
/**
|
||||
* 管理员口令默认必须为空(即关闭 8888 的管理员直取能力),
|
||||
* 且不再有写死在源码里的默认口令。
|
||||
*/
|
||||
@Test
|
||||
void adminDownloadCodeDefaultsToDisabled(@TempDir Path root) throws Exception {
|
||||
Path file = Files.writeString(root.resolve("config.properties"), "");
|
||||
|
||||
Config.loadConfig(file.toString());
|
||||
|
||||
assertEquals("", Config.adminDownloadCode, "未配置口令时管理员能力必须关闭");
|
||||
assertEquals("/root/gallery/gallery", Config.adminDownloadRoot, "默认根目录应与下载目录一致");
|
||||
}
|
||||
|
||||
/** 口令与根目录都应能从配置文件读入。 */
|
||||
@Test
|
||||
void readsAdminDownloadSettings(@TempDir Path root) throws Exception {
|
||||
Path file = Files.writeString(root.resolve("config.properties"), String.join("\n",
|
||||
"AdminDownloadCode=alone",
|
||||
"AdminDownloadRoot=/root/gallery"));
|
||||
|
||||
Config.loadConfig(file.toString());
|
||||
|
||||
assertEquals("alone", Config.adminDownloadCode);
|
||||
assertEquals("/root/gallery", Config.adminDownloadRoot);
|
||||
}
|
||||
|
||||
@Test
|
||||
void gatewayConfigurationRequiresSecretAndHasIndependentEventDirectory(@TempDir Path root) throws Exception {
|
||||
Path file = Files.writeString(root.resolve("gateway.properties"),
|
||||
"SubscriptionPrimaryUrl=https://lionwebsite.xyz\nSubscriptionEventDir=/tmp/events\n");
|
||||
assertThrows(IllegalStateException.class, () -> Config.loadConfig(file.toString()));
|
||||
Files.writeString(file, "SubscriptionPrimaryUrl=https://lionwebsite.xyz\n"
|
||||
+ "SubscriptionSyncSecret=gateway-secret\nSubscriptionEventDir=/tmp/events\n");
|
||||
Config.loadConfig(file.toString());
|
||||
assertEquals("https://lionwebsite.xyz", Config.subscriptionPrimaryUrl);
|
||||
assertEquals("/tmp/events", Config.subscriptionEventDir);
|
||||
}
|
||||
|
||||
/**
|
||||
* 启用订阅同步却没有密钥时必须拒绝启动。
|
||||
* 否则备机会以空密钥运行,签名校验形同虚设。
|
||||
*/
|
||||
@Test
|
||||
void enabledSyncWithoutSecretIsRejected(@TempDir Path root) throws Exception {
|
||||
Path file = Files.writeString(root.resolve("config.properties"), "SubscriptionSyncEnabled=true");
|
||||
|
||||
IllegalStateException failure = assertThrows(IllegalStateException.class,
|
||||
() -> Config.loadConfig(file.toString()));
|
||||
assertTrue(failure.getMessage().contains("SUBSCRIPTION_SYNC_SECRET"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package lion;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.net.ServerSocket;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/**
|
||||
* CustomUtil 保留的工具方法:空闲端口探测。
|
||||
*
|
||||
* <p>原先针对 {@code getRequestHeader} 的用例已随 Range 解析迁移到
|
||||
* {@code lion.Http.ByteRangeTest},那里覆盖普通、后缀、开放结尾与非法区间。
|
||||
*/
|
||||
class CustomUtilTest {
|
||||
|
||||
/** 从指定端口开始探测,且返回的端口确实可以立刻绑定。 */
|
||||
@Test
|
||||
void returnsBindableIdlePort() throws Exception {
|
||||
int start = freePort();
|
||||
int found = CustomUtil._findIdlePort(start);
|
||||
|
||||
assertTrue(found >= start, "应在起始端口或其之后找到空闲端口");
|
||||
try (ServerSocket socket = new ServerSocket(found)) {
|
||||
assertEquals(found, socket.getLocalPort(), "返回的端口必须可以绑定");
|
||||
}
|
||||
}
|
||||
|
||||
/** 起始端口被占用时不得返回它,必须继续往后找。 */
|
||||
@Test
|
||||
void skipsOccupiedPort() throws Exception {
|
||||
try (ServerSocket occupied = new ServerSocket(0)) {
|
||||
int busy = occupied.getLocalPort();
|
||||
|
||||
int found = CustomUtil._findIdlePort(busy);
|
||||
|
||||
assertTrue(found > busy, "被占用的端口必须跳过");
|
||||
}
|
||||
}
|
||||
|
||||
private static int freePort() throws Exception {
|
||||
try (ServerSocket socket = new ServerSocket(0)) {
|
||||
return socket.getLocalPort();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,298 @@
|
||||
package lion.Externel;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import lion.CustomUtil;
|
||||
import lion.Message.Main.SubscriptionAccountSnapshot;
|
||||
import lion.Message.Main.SubscriptionBindingSnapshot;
|
||||
import lion.Message.Main.SubscriptionSnapshotMessage;
|
||||
import lion.Message.Main.SubscriptionSnapshotPayload;
|
||||
import lion.Service.SubscriptionSnapshotStore;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.ServerSocket;
|
||||
import java.net.Socket;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.Arrays;
|
||||
import java.util.Base64;
|
||||
import java.util.HashMap;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 备机订阅分发 HTTP 服务(端口 8889)。
|
||||
*
|
||||
* <p>这是订阅用户的唯一入口,此前完全没有测试。这里起真实端口,按线路字节校验
|
||||
* 状态码、正文与响应头:按公开 Key 命中、两种订阅格式、Range 断点续传、HEAD、
|
||||
* 未知 Key、无快照时的 503、健康检查不泄露、方法不被允许以及不可满足的 Range。
|
||||
*/
|
||||
class BackupSubServerTest {
|
||||
|
||||
private static final String SECRET = "backup-sub-test-secret";
|
||||
private static final String PUBLIC_KEY = "public-key-abcdef";
|
||||
private static final String V2_CONTENT = "v2ray subscription body";
|
||||
private static final String CLASH_CONTENT = "clash subscription body";
|
||||
|
||||
@TempDir
|
||||
static Path tempDir;
|
||||
|
||||
static int readyPort;
|
||||
static int emptyPort;
|
||||
static BackupSubServer readyServer;
|
||||
static BackupSubServer emptyServer;
|
||||
|
||||
@BeforeAll
|
||||
static void startServers() throws Exception {
|
||||
// 有有效快照的备机:正常分发。
|
||||
SubscriptionSnapshotStore ready = store(tempDir.resolve("ready"));
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, ready.apply(snapshot()).code());
|
||||
readyServer = new BackupSubServer(ready, 0, 2);
|
||||
readyPort = readyServer.startServer().port();
|
||||
|
||||
// 从未收到过快照的备机:必须回 503,而不是把请求静默挂起。
|
||||
emptyServer = new BackupSubServer(store(tempDir.resolve("empty")), 0, 2);
|
||||
emptyPort = emptyServer.startServer().port();
|
||||
}
|
||||
|
||||
/** 命中公开 Key:v2 与 clash 各自返回对应正文。 */
|
||||
@Test
|
||||
void servesBothSubscriptionFormatsByPublicKey() throws Exception {
|
||||
Response v2 = get(readyPort, "/sub/v2/" + PUBLIC_KEY);
|
||||
assertEquals(200, v2.status());
|
||||
assertEquals(V2_CONTENT, v2.bodyAsString());
|
||||
assertTrue(v2.header("content-type").startsWith("text/plain"));
|
||||
|
||||
Response clash = get(readyPort, "/sub/cat/" + PUBLIC_KEY);
|
||||
assertEquals(200, clash.status());
|
||||
assertEquals(CLASH_CONTENT, clash.bodyAsString());
|
||||
assertTrue(clash.header("content-type").startsWith("text/yaml"));
|
||||
}
|
||||
|
||||
/** Range 请求必须按断点续传返回 206,并给出正确的 Content-Range。 */
|
||||
@Test
|
||||
void supportsPartialContentForResumedDownloads() throws Exception {
|
||||
Response response = request(readyPort, "GET", "/sub/v2/" + PUBLIC_KEY, "Range: bytes=0-3");
|
||||
|
||||
assertEquals(206, response.status());
|
||||
assertEquals("v2ra", response.bodyAsString());
|
||||
assertEquals("bytes 0-3/" + V2_CONTENT.length(), response.header("content-range"));
|
||||
assertEquals("4", response.header("content-length"));
|
||||
}
|
||||
|
||||
/** HEAD 必须有正确的 Content-Length,但不回正文。 */
|
||||
@Test
|
||||
void headReturnsHeadersWithoutBody() throws Exception {
|
||||
Response response = request(readyPort, "HEAD", "/sub/v2/" + PUBLIC_KEY);
|
||||
|
||||
assertEquals(200, response.status());
|
||||
assertEquals(String.valueOf(V2_CONTENT.length()), response.header("content-length"));
|
||||
assertEquals(0, response.body().length, "HEAD 不得返回正文");
|
||||
}
|
||||
|
||||
/** 未知 Key 回 404。 */
|
||||
@Test
|
||||
void unknownKeyIsNotFound() throws Exception {
|
||||
assertEquals(404, get(readyPort, "/sub/v2/unknown-public-key").status());
|
||||
}
|
||||
|
||||
/** Key 过短属于非法请求,直接 404 而不是去查表。 */
|
||||
@Test
|
||||
void tooShortKeyIsRejected() throws Exception {
|
||||
assertEquals(404, get(readyPort, "/sub/v2/abc").status());
|
||||
}
|
||||
|
||||
/** 没有有效快照时必须回 503,让客户端重试,而不是假装有内容。 */
|
||||
@Test
|
||||
void returnsServiceUnavailableWhenNoSnapshotExists() throws Exception {
|
||||
assertEquals(503, get(emptyPort, "/sub/v2/" + PUBLIC_KEY).status());
|
||||
}
|
||||
|
||||
/** 健康检查只报快照状态,不得包含 Key 或订阅正文。 */
|
||||
@Test
|
||||
void healthEndpointReportsStateWithoutLeakingKeyOrContent() throws Exception {
|
||||
Response response = get(readyPort, "/health/subscription");
|
||||
|
||||
assertEquals(200, response.status());
|
||||
assertTrue(response.header("content-type").startsWith("application/json"));
|
||||
String body = response.bodyAsString();
|
||||
assertTrue(body.contains("ready"), "有快照且新鲜时应报告 ready");
|
||||
assertFalse(body.contains(PUBLIC_KEY), "健康检查不得泄露公开 Key");
|
||||
assertFalse(body.contains(V2_CONTENT), "健康检查不得泄露订阅正文");
|
||||
}
|
||||
|
||||
/** 非 GET/HEAD 方法回 405。 */
|
||||
@Test
|
||||
void rejectsUnsupportedMethods() throws Exception {
|
||||
assertEquals(405, request(readyPort, "POST", "/sub/v2/" + PUBLIC_KEY).status());
|
||||
}
|
||||
|
||||
/** 起点越界的 Range 回 416,而不是返回空 200。 */
|
||||
@Test
|
||||
void unsatisfiableRangeIsRejected() throws Exception {
|
||||
assertEquals(416, request(readyPort, "GET", "/sub/v2/" + PUBLIC_KEY, "Range: bytes=999-1000").status());
|
||||
}
|
||||
|
||||
/** 后缀区间 bytes=-4 取正文末尾(旧实现不支持后缀写法)。 */
|
||||
@Test
|
||||
void suffixRangeReturnsTrailingBytes() throws Exception {
|
||||
String expected = V2_CONTENT.substring(V2_CONTENT.length() - 4);
|
||||
Response response = request(readyPort, "GET", "/sub/v2/" + PUBLIC_KEY, "Range: bytes=-4");
|
||||
|
||||
assertEquals(206, response.status());
|
||||
assertEquals(expected, response.bodyAsString());
|
||||
assertEquals("bytes " + (V2_CONTENT.length() - 4) + "-" + (V2_CONTENT.length() - 1) + "/" + V2_CONTENT.length(),
|
||||
response.header("content-range"));
|
||||
}
|
||||
|
||||
/** 开放结尾 bytes=4- 一直取到正文末尾。 */
|
||||
@Test
|
||||
void openEndedRangeGoesToEnd() throws Exception {
|
||||
Response response = request(readyPort, "GET", "/sub/v2/" + PUBLIC_KEY, "Range: bytes=4-");
|
||||
|
||||
assertEquals(206, response.status());
|
||||
assertEquals(V2_CONTENT.substring(4), response.bodyAsString());
|
||||
}
|
||||
|
||||
/** 关闭服务后端口必须释放,且可重复关闭。 */
|
||||
@Test
|
||||
void serverShutdownReleasesPort(@TempDir Path temp) throws Exception {
|
||||
SubscriptionSnapshotStore store = store(temp.resolve("shutdown"));
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(snapshot()).code());
|
||||
|
||||
BackupSubServer server = new BackupSubServer(store, 0, 2);
|
||||
int port = server.startServer().port();
|
||||
|
||||
// 先确认真的在服务。
|
||||
assertEquals(200, get(port, "/sub/v2/" + PUBLIC_KEY).status());
|
||||
|
||||
server.server().close();
|
||||
server.server().close(); // 重复关闭不得抛异常
|
||||
|
||||
try (ServerSocket rebind = new ServerSocket(port)) {
|
||||
assertEquals(port, rebind.getLocalPort(), "关闭后端口必须可重新绑定");
|
||||
}
|
||||
}
|
||||
|
||||
// ---- 测试脚手架 ----
|
||||
|
||||
private static SubscriptionSnapshotStore store(Path root) {
|
||||
return new SubscriptionSnapshotStore(root, SECRET, 3600, 1024 * 1024);
|
||||
}
|
||||
|
||||
private static Response get(int port, String path) throws IOException {
|
||||
return request(port, "GET", path);
|
||||
}
|
||||
|
||||
private static Response request(int port, String method, String path, String... headers) throws IOException {
|
||||
try (Socket socket = new Socket("127.0.0.1", port)) {
|
||||
socket.setSoTimeout(5_000);
|
||||
StringBuilder request = new StringBuilder()
|
||||
.append(method).append(' ').append(path).append(" HTTP/1.1\r\n")
|
||||
.append("Host: 127.0.0.1\r\n");
|
||||
for (String header : headers)
|
||||
request.append(header).append("\r\n");
|
||||
request.append("Connection: close\r\n\r\n");
|
||||
socket.getOutputStream().write(request.toString().getBytes(StandardCharsets.US_ASCII));
|
||||
socket.getOutputStream().flush();
|
||||
|
||||
byte[] raw = socket.getInputStream().readAllBytes();
|
||||
int split = indexOfHeaderEnd(raw);
|
||||
assertTrue(split > 0, "响应必须含完整的头部结束标记");
|
||||
String head = new String(raw, 0, split, StandardCharsets.ISO_8859_1);
|
||||
byte[] body = Arrays.copyOfRange(raw, split + 4, raw.length);
|
||||
|
||||
String[] lines = head.split("\r\n");
|
||||
int status = Integer.parseInt(lines[0].split(" ")[1]);
|
||||
Map<String, String> parsed = new HashMap<>();
|
||||
for (int i = 1; i < lines.length; i++) {
|
||||
int colon = lines[i].indexOf(':');
|
||||
if (colon > 0)
|
||||
parsed.put(lines[i].substring(0, colon).trim().toLowerCase(Locale.ROOT),
|
||||
lines[i].substring(colon + 1).trim());
|
||||
}
|
||||
return new Response(status, parsed, body);
|
||||
}
|
||||
}
|
||||
|
||||
private static int indexOfHeaderEnd(byte[] raw) {
|
||||
for (int i = 0; i + 3 < raw.length; i++)
|
||||
if (raw[i] == '\r' && raw[i + 1] == '\n' && raw[i + 2] == '\r' && raw[i + 3] == '\n')
|
||||
return i;
|
||||
return -1;
|
||||
}
|
||||
|
||||
private record Response(int status, Map<String, String> headers, byte[] body) {
|
||||
String header(String name) {
|
||||
return headers.get(name);
|
||||
}
|
||||
|
||||
String bodyAsString() {
|
||||
return new String(body, StandardCharsets.UTF_8);
|
||||
}
|
||||
}
|
||||
|
||||
private static SubscriptionSnapshotMessage snapshot() throws Exception {
|
||||
ObjectMapper mapper = CustomUtil.objectMapper;
|
||||
byte[] v2 = V2_CONTENT.getBytes(StandardCharsets.UTF_8);
|
||||
byte[] clash = CLASH_CONTENT.getBytes(StandardCharsets.UTF_8);
|
||||
|
||||
SubscriptionAccountSnapshot account = new SubscriptionAccountSnapshot();
|
||||
account.setAccountId(1);
|
||||
account.setEnabled(true);
|
||||
account.setV2ContentBase64(Base64.getEncoder().encodeToString(v2));
|
||||
account.setV2Sha256(sha256(v2));
|
||||
account.setClashContentBase64(Base64.getEncoder().encodeToString(clash));
|
||||
account.setClashSha256(sha256(clash));
|
||||
|
||||
SubscriptionBindingSnapshot binding = new SubscriptionBindingSnapshot();
|
||||
binding.setPublicKeySha256(sha256(PUBLIC_KEY.getBytes(StandardCharsets.UTF_8)));
|
||||
binding.setAccountId(1);
|
||||
|
||||
SubscriptionSnapshotPayload payload = new SubscriptionSnapshotPayload();
|
||||
payload.setSchemaVersion(1);
|
||||
payload.setAccounts(List.of(account));
|
||||
payload.setBindings(List.of(binding));
|
||||
byte[] json = mapper.writeValueAsBytes(payload);
|
||||
byte[] compressed = gzip(json);
|
||||
|
||||
SubscriptionSnapshotMessage message = new SubscriptionSnapshotMessage();
|
||||
message.setSchemaVersion(1);
|
||||
message.setRevision(sha256(json));
|
||||
message.setGeneratedAt(System.currentTimeMillis());
|
||||
message.setPayloadBase64(Base64.getEncoder().encodeToString(compressed));
|
||||
message.setPayloadSha256(sha256(compressed));
|
||||
String input = "1\n" + message.getRevision() + "\n" + message.getGeneratedAt() + "\n" + message.getPayloadSha256();
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
message.setSignature(hex(mac.doFinal(input.getBytes(StandardCharsets.UTF_8))));
|
||||
return message;
|
||||
}
|
||||
|
||||
private static byte[] gzip(byte[] bytes) throws Exception {
|
||||
ByteArrayOutputStream output = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream gzip = new GZIPOutputStream(output)) {
|
||||
gzip.write(bytes);
|
||||
}
|
||||
return output.toByteArray();
|
||||
}
|
||||
|
||||
private static String sha256(byte[] bytes) throws Exception {
|
||||
return hex(MessageDigest.getInstance("SHA-256").digest(bytes));
|
||||
}
|
||||
|
||||
private static String hex(byte[] bytes) {
|
||||
return HexFormat.of().formatHex(bytes);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package lion.Http;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/** Range 头解析的边界矩阵:普通、后缀、开放结尾、非法与不可满足。 */
|
||||
class ByteRangeTest {
|
||||
|
||||
@Test
|
||||
void absentOrUnsupportedHeaderMeansFullEntity() {
|
||||
assertEquals(ByteRange.Kind.FULL, ByteRange.parse(null, 10).kind());
|
||||
assertEquals(ByteRange.Kind.FULL, ByteRange.parse("items=0-3", 10).kind());
|
||||
assertEquals(ByteRange.Kind.FULL, ByteRange.parse("bytes=", 10).kind());
|
||||
assertEquals(ByteRange.Kind.FULL, ByteRange.parse("bytes=abc-def", 10).kind());
|
||||
}
|
||||
|
||||
@Test
|
||||
void plainRangeIsPartial() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=2-5", 10);
|
||||
assertEquals(ByteRange.Kind.PARTIAL, range.kind());
|
||||
assertEquals(2, range.start());
|
||||
assertEquals(5, range.endInclusive());
|
||||
assertEquals(4, range.length());
|
||||
assertEquals("bytes 2-5/10", range.contentRange(10));
|
||||
}
|
||||
|
||||
@Test
|
||||
void endBeyondFileIsClamped() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=8-999", 10);
|
||||
assertEquals(8, range.start());
|
||||
assertEquals(9, range.endInclusive());
|
||||
}
|
||||
|
||||
@Test
|
||||
void openEndedRangeCoversToFileEnd() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=7-", 10);
|
||||
assertTrue(range.partial());
|
||||
assertEquals(7, range.start());
|
||||
assertEquals(9, range.endInclusive());
|
||||
}
|
||||
|
||||
@Test
|
||||
void suffixRangeTakesLastBytes() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=-3", 10);
|
||||
assertTrue(range.partial());
|
||||
assertEquals(7, range.start());
|
||||
assertEquals(9, range.endInclusive());
|
||||
}
|
||||
|
||||
/** 后缀长度超过实体本身时返回整个实体,而不是负数起点。 */
|
||||
@Test
|
||||
void suffixLongerThanFileCoversWholeEntity() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=-100", 10);
|
||||
assertEquals(0, range.start());
|
||||
assertEquals(9, range.endInclusive());
|
||||
}
|
||||
|
||||
/** 后缀长度为 0 是合法语法但不可满足。 */
|
||||
@Test
|
||||
void zeroLengthSuffixIsUnsatisfiable() {
|
||||
assertTrue(ByteRange.parse("bytes=-0", 10).unsatisfiable());
|
||||
}
|
||||
|
||||
@Test
|
||||
void startBeyondFileIsUnsatisfiable() {
|
||||
assertTrue(ByteRange.parse("bytes=10-", 10).unsatisfiable());
|
||||
assertTrue(ByteRange.parse("bytes=999-1000", 10).unsatisfiable());
|
||||
}
|
||||
|
||||
/** 起点晚于终点属于不可满足。 */
|
||||
@Test
|
||||
void reversedRangeIsUnsatisfiable() {
|
||||
assertTrue(ByteRange.parse("bytes=6-2", 10).unsatisfiable());
|
||||
}
|
||||
|
||||
/** 空文件上的任何区间都不可满足。 */
|
||||
@Test
|
||||
void emptyFileHasNoSatisfiableRange() {
|
||||
assertTrue(ByteRange.parse("bytes=0-0", 0).unsatisfiable());
|
||||
assertTrue(ByteRange.parse("bytes=-1", 0).unsatisfiable());
|
||||
}
|
||||
|
||||
/** 多区间只取第一段,与旧实现一致。 */
|
||||
@Test
|
||||
void multipleRangesUseFirstSegmentOnly() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=1-2,5-6", 10);
|
||||
assertEquals(1, range.start());
|
||||
assertEquals(2, range.endInclusive());
|
||||
}
|
||||
|
||||
/** 单位大小写不敏感。 */
|
||||
@Test
|
||||
void rangeUnitIsCaseInsensitive() {
|
||||
assertTrue(ByteRange.parse("Bytes=1-2", 10).partial());
|
||||
}
|
||||
|
||||
/** 越界的 Long 值不得抛异常,退回完整实体。 */
|
||||
@Test
|
||||
void overflowingNumbersFallBackToFullEntity() {
|
||||
ByteRange.Resolution range = ByteRange.parse("bytes=99999999999999999999-", 10);
|
||||
assertEquals(ByteRange.Kind.FULL, range.kind());
|
||||
assertFalse(range.partial());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package lion.Http;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/** gid → 归档索引:命中、重扫发现新归档、失效与标记提取。 */
|
||||
class GalleryArchiveIndexTest {
|
||||
|
||||
@Test
|
||||
void scansDirectoryOnceThenServesFromIndex(@TempDir Path root) throws IOException {
|
||||
writeArchive(root, "Title [123]", "zip");
|
||||
AtomicInteger scans = new AtomicInteger();
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(10_000L, System::currentTimeMillis,
|
||||
counting(root, scans));
|
||||
|
||||
assertNotNull(index.find("123"));
|
||||
assertNotNull(index.find("123"));
|
||||
assertEquals(1, scans.get(), "命中不应重复扫描目录");
|
||||
}
|
||||
|
||||
/** 归档在索引建立后出现,超过重扫间隔必须能被发现。 */
|
||||
@Test
|
||||
void newlyAddedArchiveBecomesVisibleAfterRescanWindow(@TempDir Path root) throws IOException {
|
||||
AtomicLong now = new AtomicLong(1_000L);
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(5_000L, now::get,
|
||||
() -> root.toFile().listFiles(File::isDirectory));
|
||||
|
||||
assertNull(index.find("555"));
|
||||
|
||||
writeArchive(root, "Later [555]", "zip");
|
||||
now.addAndGet(5_000L);
|
||||
|
||||
assertNotNull(index.find("555"), "超过重扫间隔后新归档必须可见");
|
||||
}
|
||||
|
||||
/** 未超过重扫间隔时不得反复扫描同一批无意义 gid。 */
|
||||
@Test
|
||||
void repeatedMissesWithinWindowDoNotRescan(@TempDir Path root) throws IOException {
|
||||
writeArchive(root, "Title [123]", "zip");
|
||||
AtomicInteger scans = new AtomicInteger();
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(10_000L, System::currentTimeMillis,
|
||||
counting(root, scans));
|
||||
|
||||
assertNull(index.find("999"));
|
||||
assertNull(index.find("998"));
|
||||
assertEquals(1, scans.get(), "间隔内不应重复扫描");
|
||||
}
|
||||
|
||||
@Test
|
||||
void invalidateForcesFreshScan(@TempDir Path root) throws IOException {
|
||||
AtomicInteger scans = new AtomicInteger();
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(10_000L, System::currentTimeMillis,
|
||||
counting(root, scans));
|
||||
assertNull(index.find("42"));
|
||||
|
||||
writeArchive(root, "Fresh [42]", "zip");
|
||||
index.invalidate();
|
||||
|
||||
assertNotNull(index.find("42"), "invalidate 后必须重新扫描");
|
||||
assertEquals(2, scans.get());
|
||||
}
|
||||
|
||||
/** 索引只收录归档真实存在的目录。 */
|
||||
@Test
|
||||
void directoryWithoutArchiveIsNotIndexed(@TempDir Path root) throws IOException {
|
||||
Files.createDirectories(root.resolve("No archive [900]"));
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(10_000L, System::currentTimeMillis,
|
||||
() -> root.toFile().listFiles(File::isDirectory));
|
||||
|
||||
assertNull(index.find("900"));
|
||||
assertEquals(0, index.size());
|
||||
}
|
||||
|
||||
/** 非数字、空白与 null 的 gid 直接返回 null,不触碰磁盘。 */
|
||||
@Test
|
||||
void invalidGidNeverHitsDisk(@TempDir Path root) {
|
||||
AtomicInteger scans = new AtomicInteger();
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(10_000L, System::currentTimeMillis,
|
||||
counting(root, scans));
|
||||
|
||||
assertNull(index.find(null));
|
||||
assertNull(index.find(""));
|
||||
assertNull(index.find(" "));
|
||||
assertNull(index.find("abc"));
|
||||
assertEquals(0, scans.get());
|
||||
}
|
||||
|
||||
/** 目录名里的括号数字按「标记后紧跟 ] 或 -」提取,忽略不合规的方括号。 */
|
||||
@Test
|
||||
void extractsOnlyWellFormedGidMarkers() {
|
||||
assertEquals(List.of(123), GalleryArchiveIndex.extractGids("Title [123]"));
|
||||
assertEquals(List.of(123), GalleryArchiveIndex.extractGids("Title [123]-extra"));
|
||||
assertEquals(List.of(7, 42), GalleryArchiveIndex.extractGids("A [7] B [42]"));
|
||||
assertTrue(GalleryArchiveIndex.extractGids("Title [abc]").isEmpty());
|
||||
assertTrue(GalleryArchiveIndex.extractGids("Title [123").isEmpty(), "缺少闭合符不算标记");
|
||||
assertTrue(GalleryArchiveIndex.extractGids("Title 123]").isEmpty());
|
||||
assertTrue(GalleryArchiveIndex.extractGids("").isEmpty());
|
||||
assertTrue(GalleryArchiveIndex.extractGids(null).isEmpty());
|
||||
}
|
||||
|
||||
/** 同一个 gid 在多个目录出现时,命中其中之一(与逐请求扫描同类)。 */
|
||||
@Test
|
||||
void duplicateGidResolvesToAnExistingArchive(@TempDir Path root) throws IOException {
|
||||
writeArchive(root, "A [77]", "first");
|
||||
writeArchive(root, "B [77]", "second");
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(10_000L, System::currentTimeMillis,
|
||||
() -> root.toFile().listFiles(File::isDirectory));
|
||||
|
||||
File found = index.find("77");
|
||||
assertNotNull(found);
|
||||
assertTrue(found.isFile());
|
||||
assertTrue(found.getName().endsWith(".zip"));
|
||||
}
|
||||
|
||||
private static java.util.function.Supplier<File[]> counting(Path root, AtomicInteger scans) {
|
||||
return () -> {
|
||||
scans.incrementAndGet();
|
||||
return root.toFile().listFiles(File::isDirectory);
|
||||
};
|
||||
}
|
||||
|
||||
private static void writeArchive(Path root, String galleryName, String content) throws IOException {
|
||||
Path directory = Files.createDirectories(root.resolve(galleryName));
|
||||
Files.write(directory.resolve(galleryName + ".zip"), content.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package lion.Http;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.ServerSocket;
|
||||
import java.net.Socket;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Arrays;
|
||||
import java.util.concurrent.ArrayBlockingQueue;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* HTTP 外壳本身:有界执行器与关闭生命周期。
|
||||
*
|
||||
* <p>处理器只返回固定字节,不需要文件系统;端口用 0 由系统分配。
|
||||
*/
|
||||
class HttpFileServerTest {
|
||||
|
||||
/** 执行器必须是有界队列 + CallerRuns,不能是无上限堆积的队列。 */
|
||||
@Test
|
||||
void executorUsesBoundedQueueWithCallerRunsBackpressure(@TempDir Path ignored) throws Exception {
|
||||
HttpFileServer server = HttpFileServer.create(0, "/",
|
||||
request -> HttpResponseModel.text(200, "text/plain", "ok"), 2, 3);
|
||||
try {
|
||||
ThreadPoolExecutor pool = assertInstanceOf(ThreadPoolExecutor.class, server.executor());
|
||||
var queue = assertInstanceOf(ArrayBlockingQueue.class, pool.getQueue());
|
||||
assertEquals(3, queue.remainingCapacity(), "队列深度必须是显式上限");
|
||||
assertEquals(2, pool.getMaximumPoolSize());
|
||||
assertInstanceOf(ThreadPoolExecutor.CallerRunsPolicy.class, pool.getRejectedExecutionHandler());
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 关闭后端口必须立即释放,且可重复关闭。 */
|
||||
@Test
|
||||
void closeReleasesPortAndIsIdempotent(@TempDir Path ignored) throws Exception {
|
||||
HttpFileServer server = HttpFileServer.create(0, "/",
|
||||
request -> HttpResponseModel.text(200, "text/plain", "ok"), 2);
|
||||
int port = server.port();
|
||||
server.start();
|
||||
|
||||
assertEquals(200, status(port, "GET", "/"));
|
||||
|
||||
server.close();
|
||||
server.close(); // 重复关闭不得抛异常
|
||||
|
||||
try (ServerSocket rebind = new ServerSocket(port)) {
|
||||
assertEquals(port, rebind.getLocalPort(), "关闭后端口必须可重新绑定");
|
||||
}
|
||||
}
|
||||
|
||||
/** 未启动就关闭不得抛异常。 */
|
||||
@Test
|
||||
void closeBeforeStartIsSafe(@TempDir Path ignored) throws Exception {
|
||||
HttpFileServer server = HttpFileServer.create(0, "/",
|
||||
request -> HttpResponseModel.text(200, "text/plain", "ok"), 1);
|
||||
assertDoesNotThrow(server::close);
|
||||
}
|
||||
|
||||
/** 处理器抛异常时必须回 500,且外壳继续可用。 */
|
||||
@Test
|
||||
void handlerFailureBecomesInternalServerError(@TempDir Path ignored) throws Exception {
|
||||
HttpFileServer server = HttpFileServer.create(0, "/", request -> {
|
||||
throw new IllegalStateException("boom");
|
||||
}, 2);
|
||||
try {
|
||||
server.start();
|
||||
assertEquals(500, status(server.port(), "GET", "/"));
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 请求模型暴露的路径保留字面量 +,查询值按表单语义把 + 解码为空格。 */
|
||||
@Test
|
||||
void requestModelDecodesPathAndQueryConsistently(@TempDir Path ignored) throws Exception {
|
||||
HttpFileServer server = HttpFileServer.create(0, "/",
|
||||
request -> HttpResponseModel.text(200, "text/plain",
|
||||
(request.decodedPath() == null ? "" : request.decodedPath())
|
||||
+ "|" + request.parameter("AuthCode")),
|
||||
2);
|
||||
try {
|
||||
server.start();
|
||||
assertEquals("/plus+name.zip|a b", body(server.port(), "GET", "/plus+name.zip?AuthCode=a+b"));
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
}
|
||||
|
||||
private static int status(int port, String method, String path) throws IOException {
|
||||
return (int) parse(port, method, path)[0];
|
||||
}
|
||||
|
||||
private static String body(int port, String method, String path) throws IOException {
|
||||
return new String((byte[]) parse(port, method, path)[1], StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
/** 返回 [status, bodyBytes]。 */
|
||||
private static Object[] parse(int port, String method, String path) throws IOException {
|
||||
try (Socket socket = new Socket("127.0.0.1", port)) {
|
||||
socket.setSoTimeout(5_000);
|
||||
String request = method + " " + path + " HTTP/1.1" + CRLF
|
||||
+ "Host: 127.0.0.1" + CRLF + "Connection: close" + CRLF + CRLF;
|
||||
socket.getOutputStream().write(request.getBytes(StandardCharsets.ISO_8859_1));
|
||||
socket.getOutputStream().flush();
|
||||
|
||||
byte[] raw = socket.getInputStream().readAllBytes();
|
||||
int split = indexOfHeaderEnd(raw);
|
||||
assertTrue(split > 0, "响应必须含完整的头部结束标记");
|
||||
String head = new String(raw, 0, split, StandardCharsets.ISO_8859_1);
|
||||
int code = Integer.parseInt(head.split(CRLF)[0].split(" ")[1]);
|
||||
return new Object[]{code, Arrays.copyOfRange(raw, split + 4, raw.length)};
|
||||
}
|
||||
}
|
||||
|
||||
private static int indexOfHeaderEnd(byte[] raw) {
|
||||
for (int i = 0; i + 3 < raw.length; i++)
|
||||
if (raw[i] == 13 && raw[i + 1] == 10 && raw[i + 2] == 13 && raw[i + 3] == 10)
|
||||
return i;
|
||||
return -1;
|
||||
}
|
||||
|
||||
private static final String CRLF = "\r\n";
|
||||
}
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package lion.Http;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.URLDecoder;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
class UnicodeDownloadTest {
|
||||
@Test void asciiNamesKeepTheExistingHeader() {
|
||||
assertEquals("attachment; filename=\"archive + name.zip\"",
|
||||
FileResponseFactory.contentDisposition("archive + name.zip"));
|
||||
}
|
||||
|
||||
@Test void unicodeNamesHaveAsciiHeadersAndLosslessUtf8Filename() {
|
||||
for (String filename : new String[]{"中文 日本語.zip", "café + 100%*.zip", "附件 📚.zip"}) {
|
||||
String header = FileResponseFactory.contentDisposition(filename);
|
||||
assertTrue(header.chars().allMatch(c -> c >= 32 && c < 127));
|
||||
assertEquals(filename, decodeFilename(header));
|
||||
assertFalse(header.contains("+100"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test void allControlsAndQuotedFilenameEscapesAreRemoved() {
|
||||
String header = FileResponseFactory.contentDisposition("标题\r\n\u0000\u007f\u0085\"\\.zip");
|
||||
assertTrue(header.chars().allMatch(c -> c >= 32 && c < 127));
|
||||
assertEquals("标题_______.zip", decodeFilename(header));
|
||||
}
|
||||
|
||||
@Test void realHttpServerDownloadsUnicodeFileAndSupportsHeadAndResume(@TempDir Path directory) throws Exception {
|
||||
String filename = "[中文 (日本語)] 下载 + 100% 📚.zip";
|
||||
Path file = directory.resolve(filename);
|
||||
byte[] content = new byte[128 * 1024 + 73];
|
||||
new java.util.Random(73).nextBytes(content);
|
||||
Files.write(file, content);
|
||||
try (HttpFileServer server = HttpFileServer.create(0, "/", request ->
|
||||
FileResponseFactory.forFile(request.rangeHeader(), file.toFile(), filename), 2);
|
||||
HttpClient client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build()) {
|
||||
server.start();
|
||||
URI uri = URI.create("http://127.0.0.1:" + server.port() + "/archive.zip");
|
||||
var full = client.send(HttpRequest.newBuilder(uri).GET().build(), HttpResponse.BodyHandlers.ofByteArray());
|
||||
assertEquals(200, full.statusCode());
|
||||
assertArrayEquals(content, full.body());
|
||||
assertEquals(filename, decodeFilename(full.headers().firstValue("Content-Disposition").orElseThrow()));
|
||||
var head = client.send(HttpRequest.newBuilder(uri).method("HEAD", HttpRequest.BodyPublishers.noBody()).build(),
|
||||
HttpResponse.BodyHandlers.ofByteArray());
|
||||
assertEquals(200, head.statusCode());
|
||||
assertEquals(Integer.toString(content.length), head.headers().firstValue("Content-Length").orElseThrow());
|
||||
assertEquals(0, head.body().length);
|
||||
var range = client.send(HttpRequest.newBuilder(uri).header("Range", "bytes=4096-8191").GET().build(),
|
||||
HttpResponse.BodyHandlers.ofByteArray());
|
||||
assertEquals(206, range.statusCode());
|
||||
assertArrayEquals(Arrays.copyOfRange(content, 4096, 8192), range.body());
|
||||
assertEquals("bytes 4096-8191/" + content.length, range.headers().firstValue("Content-Range").orElseThrow());
|
||||
assertEquals(filename, decodeFilename(range.headers().firstValue("Content-Disposition").orElseThrow()));
|
||||
}
|
||||
}
|
||||
|
||||
private static String decodeFilename(String header) {
|
||||
return URLDecoder.decode(header.substring(header.indexOf("filename*=UTF-8''") + "filename*=UTF-8''".length()),
|
||||
StandardCharsets.UTF_8);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
package lion;
|
||||
|
||||
import io.netty.buffer.ByteBuf;
|
||||
import io.netty.buffer.Unpooled;
|
||||
import io.netty.channel.embedded.EmbeddedChannel;
|
||||
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.Message.AbstractMessage;
|
||||
import lion.Message.MessageCodec;
|
||||
import lion.Message.Main.AvailableCheckMessage;
|
||||
import lion.Message.Main.DeleteGalleryMessage;
|
||||
import lion.Message.Main.DownloadPostMessage;
|
||||
import lion.Message.Main.DownloadStatusMessage;
|
||||
import lion.Message.Main.IdentityMessage;
|
||||
import lion.Message.Main.MaintainMessage;
|
||||
import lion.Message.Main.ProtocolHelloMessage;
|
||||
import lion.Message.Main.ResponseMessage;
|
||||
import lion.Message.Main.SubscriptionSnapshotMessage;
|
||||
import lion.Message.Main.TaskCommandMessage;
|
||||
import lion.Message.Main.TaskReportAckMessage;
|
||||
import lion.Message.Main.TaskReportMessage;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 自定义 TCP 协议 {@code [1 字节类型][4 字节长度][JSON]} 的编解码。
|
||||
*
|
||||
* <p>这是节点与主站之间唯一的线路格式:类型字节决定反序列化成哪个消息类,
|
||||
* 长度前缀决定帧边界。两边任何一处不一致都是掉线或错消息,而此前完全无覆盖。
|
||||
*/
|
||||
class MessageCodecTest {
|
||||
|
||||
/** 每种消息类型都要能按自身类型字节往返,且解出的类型正确。 */
|
||||
@Test
|
||||
void roundTripsEveryMessageType() {
|
||||
assertRoundTrip(new ResponseMessage(7, (byte) 3), AbstractMessage.RESPONSE_MESSAGE);
|
||||
assertRoundTrip(downloadPost(), AbstractMessage.DOWNLOAD_POST_MESSAGE);
|
||||
assertRoundTrip(downloadStatus(), AbstractMessage.DOWNLOAD_STATUS_MESSAGE);
|
||||
assertRoundTrip(deleteGallery(), AbstractMessage.DELETE_GALLERY_MESSAGE);
|
||||
assertRoundTrip(identity(), AbstractMessage.IDENTITY_MESSAGE);
|
||||
assertRoundTrip(new MaintainMessage(), AbstractMessage.MAINTAIN_MESSAGE);
|
||||
assertRoundTrip(new AvailableCheckMessage(), AbstractMessage.AVAILABLE_CHECK_MESSAGE);
|
||||
assertRoundTrip(snapshot(), AbstractMessage.SUBSCRIPTION_SNAPSHOT_MESSAGE);
|
||||
assertRoundTrip(ProtocolHelloMessage.v2(), AbstractMessage.PROTOCOL_HELLO_MESSAGE);
|
||||
assertRoundTrip(taskCommand(), AbstractMessage.TASK_COMMAND_MESSAGE);
|
||||
assertRoundTrip(taskReport(), AbstractMessage.TASK_REPORT_MESSAGE);
|
||||
assertRoundTrip(ack(), AbstractMessage.TASK_REPORT_ACK_MESSAGE);
|
||||
}
|
||||
|
||||
/**
|
||||
* 新增消息的线上字段集必须精确可控。
|
||||
*
|
||||
* <p>主站使用更严格的 Jackson 并默认拒绝未知字段,任何多出来的属性都会让对端反序列化失败。
|
||||
* Jackson 2 会把 {@code isXxx()} 当成布尔属性序列化,因此 {@code TaskCommandMessage} 的
|
||||
* {@code isDownload()/isDelete()} 与 {@code TaskReportAckMessage.isSuccess()} 必须带
|
||||
* {@code @JsonIgnore};本用例把实际字段集钉死,防止它们悄悄回到线路上。
|
||||
*/
|
||||
@Test
|
||||
void newMessageWireFieldsAreExactlyAsContract() throws Exception {
|
||||
assertEquals(java.util.Set.of("messageType", "messageId", "operationId", "action", "gid", "name"),
|
||||
fieldsOf(taskCommand()), "任务命令不得额外序列化 download/delete 等派生属性");
|
||||
assertEquals(java.util.Set.of("messageType", "messageId", "reportId", "result"),
|
||||
fieldsOf(ack()), "报告确认不得额外序列化 success");
|
||||
assertEquals(java.util.Set.of("messageType", "messageId", "protocolVersion", "features"),
|
||||
fieldsOf(ProtocolHelloMessage.v2()), "hello 不得额外序列化 supportsReliableTask");
|
||||
assertEquals(java.util.Set.of("messageType", "messageId", "reportId", "revision", "galleryTasks", "operationIds"),
|
||||
fieldsOf(taskReport()));
|
||||
// 旧类型绝不能因为基类改动而多出字段:旧主站的 Jackson 会拒绝未知字段。
|
||||
assertEquals(java.util.Set.of("messageType", "messageId", "galleryTasks"),
|
||||
fieldsOf(downloadStatus()), "旧状态消息的字段集不得变化");
|
||||
assertEquals(java.util.Set.of("messageType", "messageId", "identity"),
|
||||
fieldsOf(identity()), "旧身份消息的字段集不得变化");
|
||||
}
|
||||
|
||||
/** 序列化一份消息并取回其顶层 JSON 字段名集合。 */
|
||||
private static java.util.Set<String> fieldsOf(AbstractMessage message) throws Exception {
|
||||
String json = lion.CustomUtil.objectMapper.writeValueAsString(message);
|
||||
com.fasterxml.jackson.databind.JsonNode node = lion.CustomUtil.objectMapper.readTree(json);
|
||||
java.util.Set<String> names = new java.util.LinkedHashSet<>();
|
||||
node.fieldNames().forEachRemaining(names::add);
|
||||
return names;
|
||||
}
|
||||
|
||||
private static TaskCommandMessage taskCommand() {
|
||||
TaskCommandMessage message = new TaskCommandMessage();
|
||||
message.messageId = 21;
|
||||
message.setOperationId("op-1");
|
||||
message.setAction(TaskCommandMessage.ACTION_DOWNLOAD);
|
||||
message.setGid(4242);
|
||||
message.setName("sample [4242]");
|
||||
return message;
|
||||
}
|
||||
|
||||
private static TaskReportMessage taskReport() {
|
||||
TaskReportMessage message = new TaskReportMessage();
|
||||
message.messageId = 22;
|
||||
message.setReportId("report-1");
|
||||
message.setRevision(5);
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(4242);
|
||||
task.setName("sample [4242]");
|
||||
task.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
message.setGalleryTasks(new GalleryTask[]{task});
|
||||
message.setOperationIds(java.util.Map.of(4242, "op-1"));
|
||||
return message;
|
||||
}
|
||||
|
||||
private static TaskReportAckMessage ack() {
|
||||
TaskReportAckMessage message = new TaskReportAckMessage();
|
||||
message.messageId = 23;
|
||||
message.setReportId("report-1");
|
||||
message.setResult(TaskReportAckMessage.RESULT_SUCCESS);
|
||||
return message;
|
||||
}
|
||||
|
||||
/** 帧头就是「类型字节 + 大端 4 字节长度」,长度必须等于 JSON 负载的字节数。 */
|
||||
@Test
|
||||
void writesTypeByteAndBigEndianLengthPrefix() {
|
||||
ByteBuf frame = encode(new ResponseMessage(7, (byte) 3));
|
||||
try {
|
||||
assertEquals(AbstractMessage.RESPONSE_MESSAGE, frame.readByte(), "首字节必须是消息类型");
|
||||
int length = frame.readInt();
|
||||
int jsonLength = frame.readableBytes();
|
||||
assertEquals(jsonLength, length, "长度前缀必须等于 JSON 负载字节数");
|
||||
assertTrue(jsonLength > 0);
|
||||
} finally {
|
||||
frame.release();
|
||||
}
|
||||
}
|
||||
|
||||
/** 未知类型字节必须被安静丢弃(不产出消息),而不是抛出或产出错误类型的消息。 */
|
||||
@Test
|
||||
void unknownTypeByteIsDroppedWithoutThrowing() {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(new MessageCodec());
|
||||
try {
|
||||
byte[] payload = "{}".getBytes(StandardCharsets.UTF_8);
|
||||
ByteBuf frame = Unpooled.buffer();
|
||||
frame.writeByte(99);
|
||||
frame.writeInt(payload.length);
|
||||
frame.writeBytes(payload);
|
||||
|
||||
channel.writeInbound(frame);
|
||||
|
||||
assertNull(channel.readInbound(), "未知类型不应产出任何消息");
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 生产管线是「先分帧、再解码」:{@link LengthFieldBasedFrameDecoder} 负责等齐整帧,
|
||||
* 紧跟其后的 {@link MessageCodec} 才假设长度前缀完整。半个帧到达时必须按兵不动、
|
||||
* 等剩余字节,而不能抛异常或产出半条消息。
|
||||
*/
|
||||
@Test
|
||||
void incompleteFrameIsHeldByFrameDecoderUntilComplete() {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(
|
||||
new LengthFieldBasedFrameDecoder(100_000_000, 1, 4), new MessageCodec());
|
||||
try {
|
||||
AvailableCheckMessage probe = new AvailableCheckMessage();
|
||||
probe.messageId = 11;
|
||||
|
||||
ByteBuf frame = encode(probe);
|
||||
byte[] bytes = new byte[frame.readableBytes()];
|
||||
frame.getBytes(0, bytes);
|
||||
frame.release();
|
||||
|
||||
int half = bytes.length / 2;
|
||||
ByteBuf firstHalf = Unpooled.copiedBuffer(bytes, 0, half);
|
||||
ByteBuf secondHalf = Unpooled.copiedBuffer(bytes, half, bytes.length - half);
|
||||
|
||||
assertFalse(channel.writeInbound(firstHalf), "半个帧不应产出消息");
|
||||
assertNull(channel.readInbound());
|
||||
|
||||
channel.writeInbound(secondHalf);
|
||||
AbstractMessage decoded = channel.readInbound();
|
||||
assertInstanceOf(AvailableCheckMessage.class, decoded);
|
||||
assertEquals(11, decoded.messageId);
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 长度前缀来自线路,必须设上限;否则一个报文就能让对端按声明的长度申请内存。 */
|
||||
@Test
|
||||
void rejectsAbsurdDeclaredLength() {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(new MessageCodec());
|
||||
try {
|
||||
ByteBuf frame = Unpooled.buffer();
|
||||
frame.writeByte(AbstractMessage.RESPONSE_MESSAGE);
|
||||
frame.writeInt(Integer.MAX_VALUE);
|
||||
|
||||
assertThrows(Exception.class, () -> channel.writeInbound(frame),
|
||||
"超出上限的帧长度必须被拒绝,而不是按声明分配字节数组");
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 序列化失败时绝不能写出半帧。
|
||||
*
|
||||
* <p>原实现先写类型字节、再序列化,失败时只记日志,于是留下「有类型字节、没有长度和
|
||||
* JSON」的残帧,对端的分帧器会永远等长度前缀。现在必须先序列化成功再写帧头。
|
||||
*/
|
||||
@Test
|
||||
void serializationFailureDoesNotEmitHalfFrame() {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(new MessageCodec());
|
||||
try {
|
||||
assertThrows(Exception.class, () -> channel.writeOutbound(new UnserializableMessage()),
|
||||
"序列化失败必须抛错,而不是安静地留下半帧");
|
||||
assertNull(channel.readOutbound(), "失败时不得产出任何字节");
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 故意让 Jackson 在序列化属性时抛异常。 */
|
||||
static class UnserializableMessage extends AbstractMessage {
|
||||
UnserializableMessage() {
|
||||
messageType = AbstractMessage.RESPONSE_MESSAGE;
|
||||
}
|
||||
|
||||
@SuppressWarnings("unused")
|
||||
public String getExplode() {
|
||||
throw new IllegalStateException("boom");
|
||||
}
|
||||
}
|
||||
|
||||
private static void assertRoundTrip(AbstractMessage message, byte expectedType) {
|
||||
ByteBuf frame = encode(message);
|
||||
assertEquals(expectedType, frame.getByte(0), "线路上的类型字节必须与消息类匹配");
|
||||
|
||||
EmbeddedChannel channel = new EmbeddedChannel(
|
||||
new LengthFieldBasedFrameDecoder(100_000_000, 1, 4), new MessageCodec());
|
||||
try {
|
||||
channel.writeInbound(frame);
|
||||
AbstractMessage decoded = channel.readInbound();
|
||||
assertNotNull(decoded, "整帧必须被解码出来");
|
||||
assertEquals(message.getClass(), decoded.getClass(), "解码类型必须与编码类型一致");
|
||||
assertEquals(expectedType, decoded.messageType);
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
private static ByteBuf encode(AbstractMessage message) {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(new MessageCodec());
|
||||
try {
|
||||
channel.writeOutbound(message);
|
||||
ByteBuf out = channel.readOutbound();
|
||||
assertNotNull(out, "编码后必须产出字节");
|
||||
return out;
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
private static DownloadPostMessage downloadPost() {
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(4242);
|
||||
task.setName("sample [4242]");
|
||||
task.setStatus(GalleryTask.DOWNLOADING);
|
||||
DownloadPostMessage message = new DownloadPostMessage();
|
||||
message.messageId = 1;
|
||||
message.setGalleryTask(task);
|
||||
return message;
|
||||
}
|
||||
|
||||
private static DownloadStatusMessage downloadStatus() {
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(4242);
|
||||
task.setName("sample [4242]");
|
||||
task.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
DownloadStatusMessage message = new DownloadStatusMessage();
|
||||
message.setGalleryTasks(new GalleryTask[]{task});
|
||||
return message;
|
||||
}
|
||||
|
||||
private static DeleteGalleryMessage deleteGallery() {
|
||||
DeleteGalleryMessage message = new DeleteGalleryMessage();
|
||||
message.messageId = 5;
|
||||
message.setGalleryName("sample [4242]");
|
||||
return message;
|
||||
}
|
||||
|
||||
private static IdentityMessage identity() {
|
||||
IdentityMessage message = new IdentityMessage();
|
||||
message.setIdentity("lionwebsite");
|
||||
return message;
|
||||
}
|
||||
|
||||
private static SubscriptionSnapshotMessage snapshot() {
|
||||
SubscriptionSnapshotMessage message = new SubscriptionSnapshotMessage();
|
||||
message.setSchemaVersion(1);
|
||||
message.setRevision("a".repeat(64));
|
||||
message.setGeneratedAt(1_700_000_000_000L);
|
||||
message.setPayloadBase64("ZmFrZQ==");
|
||||
message.setPayloadSha256("b".repeat(64));
|
||||
message.setSignature("c".repeat(64));
|
||||
return message;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,461 @@
|
||||
package lion;
|
||||
|
||||
import lion.Http.GalleryArchiveIndex;
|
||||
import lion.Http.GalleryDownloadHandler;
|
||||
import lion.Http.HttpFileServer;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.ServerSocket;
|
||||
import java.net.Socket;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashMap;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 8888 下载服务:请求行解析、管理员口令/目录约束,以及真实 HTTP 端口上的
|
||||
* GET/HEAD 与 Range 语义。
|
||||
*
|
||||
* <p>集成用例起在临时端口、只使用 {@link TempDir} 里的临时目录,不读取生产配置、
|
||||
* 不启动 {@code main}。
|
||||
*/
|
||||
class MultiThreadedHTTPServerTest {
|
||||
|
||||
// ---- 请求行解析(公开兼容入口) ----
|
||||
|
||||
@Test
|
||||
void parsesPathAndQueryParameters() {
|
||||
Map<String, String> params = MultiThreadedHTTPServer.parseRequestLine(
|
||||
"/download?AuthCode=abc&gid=12345");
|
||||
|
||||
assertEquals("/download", params.get("path"));
|
||||
assertEquals("abc", params.get("AuthCode"));
|
||||
assertEquals("12345", params.get("gid"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void parsesPathWithoutQueryString() {
|
||||
Map<String, String> params = MultiThreadedHTTPServer.parseRequestLine("/archive/file.zip");
|
||||
assertEquals("/archive/file.zip", params.get("path"));
|
||||
assertNull(params.get("gid"));
|
||||
}
|
||||
|
||||
/** 回归:值内的 '=' 不得被截断(base64 padding)。 */
|
||||
@Test
|
||||
void keepsEqualsSignInsideParameterValue() {
|
||||
Map<String, String> params = MultiThreadedHTTPServer.parseRequestLine(
|
||||
"/download?AuthCode=alone&token=YWJjZA==");
|
||||
|
||||
assertEquals("YWJjZA==", params.get("token"), "值内的 '=' 不得被截断");
|
||||
assertEquals("alone", params.get("AuthCode"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void decodesPercentEncodedValues() {
|
||||
Map<String, String> params = MultiThreadedHTTPServer.parseRequestLine(
|
||||
"/download?AuthCode=alone&name=a%20b");
|
||||
assertEquals("a b", params.get("name"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void returnsNullForNullRequestLine() {
|
||||
assertNull(MultiThreadedHTTPServer.parseRequestLine(null));
|
||||
}
|
||||
|
||||
/** 路径里的 {@code +} 是字面量,不得被当成空格(query 值才按 + → 空格)。 */
|
||||
@Test
|
||||
void keepsPlusLiteralInPathButDecodesItInQuery() {
|
||||
Map<String, String> params = MultiThreadedHTTPServer.parseRequestLine("/plus+name.zip?AuthCode=a+b");
|
||||
|
||||
assertEquals("/plus+name.zip", params.get("path"), "路径里的 '+' 必须保留");
|
||||
assertEquals("a b", params.get("AuthCode"), "查询值里的 '+' 应按表单语义解码为空格");
|
||||
}
|
||||
|
||||
/** 路径解码保留字面量 '+';查询解码按标准 + → 空格。 */
|
||||
@Test
|
||||
void pathDecoderPreservesPlusWhileQueryDecoderDoesNot() {
|
||||
assertEquals("/a+b.zip", lion.Http.HttpFileServer.decodePath("/a+b.zip"));
|
||||
assertEquals("/a+b.zip", lion.Http.HttpFileServer.decodePath("/a%2Bb.zip"),
|
||||
"百分号编码的 '+' 解码后同样是 '+'");
|
||||
assertEquals("a b", lion.Http.HttpFileServer.decodeQuery("a+b"));
|
||||
}
|
||||
|
||||
// ---- 管理员直取文件的口令与目录约束 ----
|
||||
|
||||
@Test
|
||||
void adminCapabilityIsOffUntilCodeConfigured() {
|
||||
lion.Config.Config.adminDownloadCode = "";
|
||||
|
||||
assertFalse(MultiThreadedHTTPServer.isAdminRequest("alone"));
|
||||
assertFalse(MultiThreadedHTTPServer.isAdminRequest(""));
|
||||
}
|
||||
|
||||
@Test
|
||||
void onlyConfiguredAdminCodeIsAccepted() {
|
||||
lion.Config.Config.adminDownloadCode = "s3cret-code";
|
||||
try {
|
||||
assertTrue(MultiThreadedHTTPServer.isAdminRequest("s3cret-code"));
|
||||
assertFalse(MultiThreadedHTTPServer.isAdminRequest("alone"), "写死的旧值不应再有效");
|
||||
assertFalse(MultiThreadedHTTPServer.isAdminRequest("s3cret"));
|
||||
assertFalse(MultiThreadedHTTPServer.isAdminRequest(null));
|
||||
} finally {
|
||||
lion.Config.Config.adminDownloadCode = "";
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void adminDownloadIsConfinedToConfiguredRoot(@TempDir Path root) throws Exception {
|
||||
Path galleryRoot = Files.createDirectories(root.resolve("gallery"));
|
||||
Path archive = Files.write(galleryRoot.resolve("sample.zip"), "zip".getBytes(StandardCharsets.UTF_8));
|
||||
Path outside = Files.write(root.resolve("secret.txt"), "top secret".getBytes(StandardCharsets.UTF_8));
|
||||
lion.Config.Config.adminDownloadRoot = galleryRoot.toString();
|
||||
try {
|
||||
assertEquals(archive.toRealPath().toFile(), MultiThreadedHTTPServer.resolveAdminFile(archive.toString()));
|
||||
assertEquals(archive.toRealPath().toFile(), MultiThreadedHTTPServer.resolveAdminFile("sample.zip"));
|
||||
|
||||
assertNull(MultiThreadedHTTPServer.resolveAdminFile("../secret.txt"), "../ 越界必须被拒");
|
||||
assertNull(MultiThreadedHTTPServer.resolveAdminFile(outside.toString()), "根目录之外的绝对路径必须被拒");
|
||||
assertNull(MultiThreadedHTTPServer.resolveAdminFile("/etc/passwd"), "任意系统文件必须被拒");
|
||||
assertNull(MultiThreadedHTTPServer.resolveAdminFile(galleryRoot.toString()), "目录本身不是可下载文件");
|
||||
} finally {
|
||||
lion.Config.Config.adminDownloadRoot = "/root/gallery/gallery";
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void adminDownloadRejectsSymlinkEscape(@TempDir Path root) throws Exception {
|
||||
Path galleryRoot = Files.createDirectories(root.resolve("gallery"));
|
||||
Path outside = Files.write(root.resolve("secret.txt"), "top secret".getBytes(StandardCharsets.UTF_8));
|
||||
Path link = galleryRoot.resolve("link.zip");
|
||||
try {
|
||||
Files.createSymbolicLink(link, outside);
|
||||
} catch (UnsupportedOperationException | IOException e) {
|
||||
return; // 平台不支持符号链接时跳过
|
||||
}
|
||||
lion.Config.Config.adminDownloadRoot = galleryRoot.toString();
|
||||
try {
|
||||
assertNull(MultiThreadedHTTPServer.resolveAdminFile("link.zip"), "指向根目录外的符号链接必须被拒");
|
||||
} finally {
|
||||
lion.Config.Config.adminDownloadRoot = "/root/gallery/gallery";
|
||||
}
|
||||
}
|
||||
|
||||
// ---- 真实 HTTP 集成(临时端口 + 临时目录) ----
|
||||
|
||||
/**
|
||||
* 管理员直取:路径里含字面量 '+' 的文件名必须能按实际路径打开。
|
||||
*
|
||||
* <p>回归:URLDecoder 会把 '+' 解成空格,导致 {@code plus+name.zip} 被解析为
|
||||
* {@code plus name.zip}(两种写法都找不到文件)。
|
||||
*/
|
||||
@Test
|
||||
void adminDownloadServesFileWithLiteralPlusInName(@TempDir Path root) throws Exception {
|
||||
Path galleryRoot = Files.createDirectories(root.resolve("gallery"));
|
||||
byte[] content = "plus-payload".getBytes(StandardCharsets.UTF_8);
|
||||
Path archive = Files.write(galleryRoot.resolve("plus+name.zip"), content);
|
||||
String previousRoot = lion.Config.Config.adminDownloadRoot;
|
||||
Fixture fixture = Fixture.start(new GalleryDownloadHandler(
|
||||
new GalleryArchiveIndex(galleryRoot.toFile(), 1_000L)), "plus-code", galleryRoot.toString());
|
||||
try {
|
||||
// 管理员通道按真实路径直取,因此用绝对路径请求。
|
||||
// 关键点:路径里保持「字面量 +」(不编码成 %2B)——若用 URLDecoder 解码路径,
|
||||
// '+' 会被解成空格,从而 404。
|
||||
String requestPath = archive.toRealPath().toString();
|
||||
Response response = fixture.get(requestPath + "?AuthCode=plus-code");
|
||||
assertEquals(200, response.status(), "字面量 '+' 文件应可下载");
|
||||
assertArrayEquals(content, response.body());
|
||||
} finally {
|
||||
fixture.close();
|
||||
lion.Config.Config.adminDownloadCode = "";
|
||||
lion.Config.Config.adminDownloadRoot = previousRoot;
|
||||
}
|
||||
}
|
||||
|
||||
/** 完整下载必须回 200 且 Content-Length 等于文件长度(旧实现恒为 206)。 */
|
||||
@Test
|
||||
void fullDownloadReturnsOkWithExactLength(@TempDir Path root) throws Exception {
|
||||
byte[] payload = "0123456789".getBytes(StandardCharsets.UTF_8);
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", payload);
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123");
|
||||
assertEquals(200, response.status());
|
||||
assertEquals("10", response.header("content-length"));
|
||||
assertArrayEquals(payload, response.body());
|
||||
assertNull(response.header("content-range"), "完整下载不应带 Content-Range");
|
||||
assertEquals("bytes", response.header("accept-ranges"));
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 普通区间 Range:bytes=1-3。 */
|
||||
@Test
|
||||
void plainRangeReturnsPartialContent(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123", "bytes=1-3");
|
||||
assertEquals(206, response.status());
|
||||
assertEquals("bytes 1-3/10", response.header("content-range"));
|
||||
assertEquals("3", response.header("content-length"));
|
||||
assertEquals("123", response.bodyAsString());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 后缀区间 bytes=-4 取最后 4 字节(旧实现不支持)。 */
|
||||
@Test
|
||||
void suffixRangeReturnsLastBytes(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123", "bytes=-4");
|
||||
assertEquals(206, response.status());
|
||||
assertEquals("bytes 6-9/10", response.header("content-range"));
|
||||
assertEquals("6789", response.bodyAsString());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 开放结尾 bytes=2- 取到文件末尾。 */
|
||||
@Test
|
||||
void openEndedRangeGoesToFileEnd(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123", "bytes=2-");
|
||||
assertEquals(206, response.status());
|
||||
assertEquals("bytes 2-9/10", response.header("content-range"));
|
||||
assertEquals("23456789", response.bodyAsString());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 超出末端的区间必须夹到文件末尾。 */
|
||||
@Test
|
||||
void oversizedRangeIsClampedToFileEnd(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123", "bytes=4-999999");
|
||||
assertEquals(206, response.status());
|
||||
assertEquals("bytes 4-9/10", response.header("content-range"));
|
||||
assertEquals("6", response.header("content-length"));
|
||||
assertEquals("456789", response.bodyAsString());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 起点越界回 416。 */
|
||||
@Test
|
||||
void rangeStartingBeyondFileIsNotSatisfiable(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
assertEquals(416, fixture.get("/file?AuthCode=x&gid=123", "bytes=999-1000").status());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 语法无法解析的 Range 按 RFC 忽略,退回完整 200。 */
|
||||
@Test
|
||||
void malformedRangeFallsBackToFullEntity(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123", "bytes=abc-def");
|
||||
assertEquals(200, response.status());
|
||||
assertEquals("10", response.header("content-length"));
|
||||
assertEquals("0123456789", response.bodyAsString());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** HEAD 必须给出 Content-Length,但不回正文。 */
|
||||
@Test
|
||||
void headReturnsHeadersWithoutBody(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.request("HEAD", "/file?AuthCode=x&gid=123");
|
||||
assertEquals(200, response.status());
|
||||
assertEquals("10", response.header("content-length"));
|
||||
assertEquals(0, response.body().length, "HEAD 不得返回正文");
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** HEAD 带 Range 时同样给出区间长度且不回正文。 */
|
||||
@Test
|
||||
void headWithRangeReportsPartialLength(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "0123456789".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.request("HEAD", "/file?AuthCode=x&gid=123", "bytes=2-4");
|
||||
assertEquals(206, response.status());
|
||||
assertEquals("3", response.header("content-length"));
|
||||
assertEquals("bytes 2-4/10", response.header("content-range"));
|
||||
assertEquals(0, response.body().length);
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 归档响应带 Content-Disposition,且文件名中的引号已被清理。 */
|
||||
@Test
|
||||
void archiveResponseCarriesSanitizedDisposition(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title \"quoted\" [123]", "abc".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
Response response = fixture.get("/file?AuthCode=x&gid=123");
|
||||
assertEquals(200, response.status());
|
||||
String disposition = response.header("content-disposition");
|
||||
assertNotNull(disposition);
|
||||
assertTrue(disposition.startsWith("attachment; filename=\""));
|
||||
assertFalse(disposition.contains("\"quoted\""), "文件名里的引号必须被清理:" + disposition);
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 不存在的 gid 回 404。 */
|
||||
@Test
|
||||
void unknownGidIsNotFound(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "abc".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
assertEquals(404, fixture.get("/file?AuthCode=x&gid=999").status());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 缺少 AuthCode 参数一律 403。 */
|
||||
@Test
|
||||
void missingAuthCodeIsForbidden(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "abc".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
assertEquals(403, fixture.get("/file?gid=123").status());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 非 GET/HEAD 方法回 501(保持旧行为)。 */
|
||||
@Test
|
||||
void unsupportedMethodIsNotImplemented(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "abc".getBytes(StandardCharsets.UTF_8));
|
||||
try {
|
||||
assertEquals(501, fixture.request("POST", "/file?AuthCode=x&gid=123").status());
|
||||
} finally {
|
||||
fixture.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 关闭服务后端口必须立即释放,且可重复关闭。 */
|
||||
@Test
|
||||
void closeReleasesPortAndIsIdempotent(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = Fixture.start(root, "Title [123]", "abc".getBytes(StandardCharsets.UTF_8));
|
||||
int port = fixture.serverPort;
|
||||
|
||||
fixture.close();
|
||||
fixture.close(); // 重复关闭不得抛异常
|
||||
|
||||
try (ServerSocket rebind = new ServerSocket(port)) {
|
||||
assertEquals(port, rebind.getLocalPort(), "关闭后端口必须可重新绑定");
|
||||
}
|
||||
}
|
||||
|
||||
// ---- 测试脚手架 ----
|
||||
|
||||
/** 起一个真实端口的下载服务,归档放在临时画廊根目录下。 */
|
||||
private static final class Fixture {
|
||||
private final HttpFileServer server;
|
||||
private final int serverPort;
|
||||
|
||||
private Fixture(HttpFileServer server, int serverPort) {
|
||||
this.server = server;
|
||||
this.serverPort = serverPort;
|
||||
}
|
||||
|
||||
static Fixture start(Path galleryRoot, String galleryName, byte[] archiveContent) throws Exception {
|
||||
Path directory = Files.createDirectories(galleryRoot.resolve(galleryName));
|
||||
Files.write(directory.resolve(galleryName + ".zip"), archiveContent);
|
||||
|
||||
// 管理员能力关闭:测试走按 gid 查归档的正常通道。
|
||||
GalleryArchiveIndex index = new GalleryArchiveIndex(galleryRoot.toFile(), 1_000L);
|
||||
return start(new GalleryDownloadHandler(index), null, null);
|
||||
}
|
||||
|
||||
/** 起服务并临时改写管理员配置,测试结束后由调用方恢复。 */
|
||||
static Fixture start(GalleryDownloadHandler handler, String adminCode, String adminRoot) throws Exception {
|
||||
lion.Config.Config.adminDownloadCode = adminCode == null ? "" : adminCode;
|
||||
if (adminRoot != null)
|
||||
lion.Config.Config.adminDownloadRoot = adminRoot;
|
||||
HttpFileServer server = HttpFileServer.create(0, "/", handler, 4);
|
||||
server.start();
|
||||
return new Fixture(server, server.port());
|
||||
}
|
||||
|
||||
void close() {
|
||||
server.close();
|
||||
}
|
||||
|
||||
Response get(String path, String... headers) throws IOException {
|
||||
return request("GET", path, headers);
|
||||
}
|
||||
|
||||
Response request(String method, String path, String... headers) throws IOException {
|
||||
try (Socket socket = new Socket("127.0.0.1", serverPort)) {
|
||||
socket.setSoTimeout(5_000);
|
||||
StringBuilder request = new StringBuilder()
|
||||
.append(method).append(' ').append(path).append(" HTTP/1.1\r\n")
|
||||
.append("Host: 127.0.0.1\r\n");
|
||||
for (String header : headers)
|
||||
request.append("Range: ").append(header).append("\r\n");
|
||||
request.append("Connection: close\r\n\r\n");
|
||||
socket.getOutputStream().write(request.toString().getBytes(StandardCharsets.ISO_8859_1));
|
||||
socket.getOutputStream().flush();
|
||||
|
||||
byte[] raw = socket.getInputStream().readAllBytes();
|
||||
int split = indexOfHeaderEnd(raw);
|
||||
assertTrue(split > 0, "响应必须含完整的头部结束标记");
|
||||
String head = new String(raw, 0, split, StandardCharsets.ISO_8859_1);
|
||||
byte[] body = Arrays.copyOfRange(raw, split + 4, raw.length);
|
||||
|
||||
String[] lines = head.split("\r\n");
|
||||
int status = Integer.parseInt(lines[0].split(" ")[1]);
|
||||
Map<String, String> parsed = new HashMap<>();
|
||||
for (int i = 1; i < lines.length; i++) {
|
||||
int colon = lines[i].indexOf(':');
|
||||
if (colon > 0)
|
||||
parsed.put(lines[i].substring(0, colon).trim().toLowerCase(Locale.ROOT),
|
||||
lines[i].substring(colon + 1).trim());
|
||||
}
|
||||
return new Response(status, parsed, body);
|
||||
}
|
||||
}
|
||||
|
||||
private static int indexOfHeaderEnd(byte[] raw) {
|
||||
for (int i = 0; i + 3 < raw.length; i++)
|
||||
if (raw[i] == '\r' && raw[i + 1] == '\n' && raw[i + 2] == '\r' && raw[i + 3] == '\n')
|
||||
return i;
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
private record Response(int status, Map<String, String> headers, byte[] body) {
|
||||
String header(String name) {
|
||||
return headers.get(name);
|
||||
}
|
||||
|
||||
String bodyAsString() {
|
||||
return new String(body, StandardCharsets.UTF_8);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package lion;
|
||||
|
||||
import io.netty.channel.embedded.EmbeddedChannel;
|
||||
import lion.Message.AbstractMessage;
|
||||
import lion.Message.Main.AvailableCheckMessage;
|
||||
import lion.Message.Main.IdentityMessage;
|
||||
import lion.Service.PrimaryChannelTracker;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 复现并锁定线上事故的通道恢复语义。
|
||||
*
|
||||
* <p>事故经过:主站重连期间节点先后认证了多条通道,引用被最后一条覆盖;那条通道断开后
|
||||
* 引用被清空,而更早建立、仍然可用的通道继续发送可用性探测。节点因引用为空不再上报
|
||||
* 任何任务状态,主站又能在旧通道上收到探活响应,双方都判定连接正常,未完成任务
|
||||
* 便永久停在「已提交」,直到人工触发重连。
|
||||
*/
|
||||
class PrimaryChannelRecoveryTest {
|
||||
|
||||
@Test
|
||||
void availableCheckOnSurvivingChannelRestoresReporting() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel survivor = new EmbeddedChannel();
|
||||
EmbeddedChannel overwriter = new EmbeddedChannel();
|
||||
try {
|
||||
// 主站两次上线:引用最终落在后建立的 overwriter 上。
|
||||
tracker.onPrimaryMessage(survivor);
|
||||
tracker.onPrimaryMessage(overwriter);
|
||||
|
||||
// overwriter 断开,引用被清空——事故的起点。
|
||||
tracker.unregister(overwriter);
|
||||
|
||||
// 仍然可用的 survivor 发来探活消息:引用应当立即恢复。
|
||||
AvailableCheckMessage probe = new AvailableCheckMessage();
|
||||
assertEquals(AbstractMessage.AVAILABLE_CHECK_MESSAGE, probe.messageType);
|
||||
tracker.onPrimaryMessage(survivor);
|
||||
|
||||
assertSame(survivor, tracker.current(), "探活消息应恢复引用");
|
||||
assertTrue(tracker.usable(), "恢复后必须能继续上报任务状态");
|
||||
} finally {
|
||||
survivor.finishAndReleaseAll();
|
||||
overwriter.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 首选通道断开但仍有其它已认证通道时,不得出现「无通道可用」的空窗。 */
|
||||
@Test
|
||||
void referenceNeverDropsWhileAnotherAuthenticatedChannelLives() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel first = new EmbeddedChannel();
|
||||
EmbeddedChannel second = new EmbeddedChannel();
|
||||
try {
|
||||
tracker.onPrimaryMessage(first);
|
||||
tracker.onPrimaryMessage(second);
|
||||
|
||||
tracker.unregister(second);
|
||||
|
||||
assertTrue(tracker.usable(), "仍有已认证通道时必须可用");
|
||||
assertSame(first, tracker.current());
|
||||
} finally {
|
||||
first.finishAndReleaseAll();
|
||||
second.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 备机的身份消息不得夺走主站引用。
|
||||
*
|
||||
* <p>备机在启动时同样会自我介绍,若按消息类型无条件认领,备机通道会被误当成主站,
|
||||
* 任务状态将上报到错误的连接上。
|
||||
*/
|
||||
@Test
|
||||
void sidecarIdentityDoesNotStealPrimaryReference() {
|
||||
assertTrue(storageNode.isPrimaryEvidence(identity("lionwebsite")), "主站身份应被认作证据");
|
||||
assertFalse(storageNode.isPrimaryEvidence(identity("lionwebsiteside")),
|
||||
"备机身份不是主站证据,否则任务状态会上报到错误的连接");
|
||||
assertFalse(storageNode.isPrimaryEvidence(new lion.Message.Main.ResponseMessage()),
|
||||
"响应消息是节点自己的出站类型,不能用来认领主站引用");
|
||||
}
|
||||
|
||||
private static IdentityMessage identity(String value) {
|
||||
IdentityMessage message = new IdentityMessage();
|
||||
message.setIdentity(value);
|
||||
return message;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package lion;
|
||||
import lion.Message.*;
|
||||
import lion.Message.Main.*;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import java.util.Set;
|
||||
import java.util.HashSet;
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/** Both ends assert this exact wire shape; helper getters must not leak onto the wire. */
|
||||
class ReliableWireContractTest {
|
||||
private final ObjectMapper mapper = new ObjectMapper();
|
||||
private void fields(Object message, Set<String> allowed) throws Exception {
|
||||
var tree = mapper.readTree(mapper.writeValueAsString(message));
|
||||
Set<String> actual = new HashSet<>();
|
||||
tree.fieldNames().forEachRemaining(actual::add);
|
||||
assertEquals(allowed, actual);
|
||||
}
|
||||
@Test void commandWireShapeAndGoldenInput() throws Exception {
|
||||
var command = new TaskCommandMessage();
|
||||
command.setMessageId(71); command.setOperationId("op-download");
|
||||
command.setAction("DOWNLOAD"); command.setGid(17); command.setName("中文画廊");
|
||||
fields(command, Set.of("messageType","messageId","operationId","action","gid","name"));
|
||||
var input = mapper.readValue("{\"messageType\":11,\"messageId\":71,\"operationId\":\"op-download\",\"action\":\"DOWNLOAD\",\"gid\":17,\"name\":\"中文画廊\"}", TaskCommandMessage.class);
|
||||
assertEquals(command.getOperationId(), input.getOperationId());
|
||||
assertEquals(command.getGid(), input.getGid());
|
||||
}
|
||||
@Test void reportWireShapeAndGoldenInput() throws Exception {
|
||||
var report = mapper.readValue("{\"messageType\":12,\"messageId\":72,\"reportId\":\"report-1\",\"revision\":9007199254,\"galleryTasks\":[{\"gid\":17,\"name\":\"中文画廊\",\"status\":4,\"proceeding\":20}],\"operationIds\":{\"17\":\"op-download\"}}", TaskReportMessage.class);
|
||||
fields(report, Set.of("messageType","messageId","reportId","revision","galleryTasks","operationIds"));
|
||||
assertEquals(9007199254L, report.getRevision());
|
||||
assertEquals("op-download", report.getOperationIds().get(17));
|
||||
assertEquals(20, report.getGalleryTasks()[0].getProceeding());
|
||||
}
|
||||
@Test void ackWireShapeAndGoldenInput() throws Exception {
|
||||
var ack = mapper.readValue("{\"messageType\":13,\"messageId\":73,\"reportId\":\"report-1\",\"result\":0}", TaskReportAckMessage.class);
|
||||
fields(ack, Set.of("messageType","messageId","reportId","result"));
|
||||
assertEquals(0, ack.getResult());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.ErrorCode.ErrorCode;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/** 画廊目录删除:成功归零,失败按 ErrorCode 区分「路径不存在」与其它 IO 错误。 */
|
||||
class DeleteServiceTest {
|
||||
|
||||
@Test
|
||||
void deletesExistingDirectoryAndReturnsSuccess(@TempDir Path root) throws Exception {
|
||||
Path gallery = Files.createDirectories(root.resolve("sample [123]"));
|
||||
Files.writeString(gallery.resolve("1.jpg"), "image bytes");
|
||||
|
||||
assertEquals(0, DeleteService.deleteAll(gallery.toString()));
|
||||
assertFalse(Files.exists(gallery), "目录应被整棵删除");
|
||||
}
|
||||
|
||||
/** 路径不存在时必须回 FILE_NOT_FOUND,让主站能区分「本来就没有」。 */
|
||||
@Test
|
||||
void missingPathReportsFileNotFound(@TempDir Path root) {
|
||||
assertEquals(ErrorCode.FILE_NOT_FOUND, DeleteService.deleteAll(root.resolve("absent").toString()));
|
||||
}
|
||||
|
||||
/** 目标是普通文件(而非目录)时同样按 FILE_NOT_FOUND 处理,不得误删文件。 */
|
||||
@Test
|
||||
void plainFileIsNotTreatedAsDirectory(@TempDir Path root) throws Exception {
|
||||
Path file = root.resolve("not-a-directory.txt");
|
||||
Files.writeString(file, "keep me");
|
||||
|
||||
assertEquals(ErrorCode.FILE_NOT_FOUND, DeleteService.deleteAll(file.toString()));
|
||||
assertTrue(Files.exists(file), "普通文件不应被删除");
|
||||
}
|
||||
|
||||
/** 正常的子目录名应在根目录内被删除。 */
|
||||
@Test
|
||||
void deleteWithinRemovesChildOfRoot(@TempDir Path root) throws Exception {
|
||||
Path gallery = Files.createDirectories(root.resolve("sample [123]"));
|
||||
|
||||
assertEquals(0, DeleteService.deleteWithin(root.toString(), "sample [123]"));
|
||||
assertFalse(Files.exists(gallery));
|
||||
}
|
||||
|
||||
/**
|
||||
* 画廊名来自主站下发的消息,带 {@code ../} 时必须被拒绝。
|
||||
* 否则路径会被规范化到根目录之外,删掉节点上的任意目录。
|
||||
*/
|
||||
@Test
|
||||
void deleteWithinRejectsPathTraversal(@TempDir Path root) throws Exception {
|
||||
Path outside = Files.createDirectories(root.resolve("important"));
|
||||
Path galleryRoot = Files.createDirectories(root.resolve("gallery"));
|
||||
|
||||
byte result = DeleteService.deleteWithin(galleryRoot.toString(), "../important");
|
||||
|
||||
assertEquals(ErrorCode.FILE_NOT_FOUND, result, "越界路径必须按找不到处理");
|
||||
assertTrue(Files.exists(outside), "根目录之外的内容绝不能被删除");
|
||||
}
|
||||
|
||||
/** 绝对路径同样不得越过根目录。 */
|
||||
@Test
|
||||
void deleteWithinRejectsAbsolutePath(@TempDir Path root) throws Exception {
|
||||
Path outside = Files.createDirectories(root.resolve("important"));
|
||||
Path galleryRoot = Files.createDirectories(root.resolve("gallery"));
|
||||
|
||||
assertEquals(ErrorCode.FILE_NOT_FOUND, DeleteService.deleteWithin(galleryRoot.toString(), outside.toString()));
|
||||
assertTrue(Files.exists(outside));
|
||||
}
|
||||
|
||||
/** 空名会导致规范化结果等于根目录本身,必须拒绝。 */
|
||||
@Test
|
||||
void deleteWithinRejectsEmptyName(@TempDir Path root) throws Exception {
|
||||
Path galleryRoot = Files.createDirectories(root.resolve("gallery"));
|
||||
|
||||
assertEquals(ErrorCode.FILE_NOT_FOUND, DeleteService.deleteWithin(galleryRoot.toString(), " "));
|
||||
assertTrue(Files.exists(galleryRoot), "根目录本身不得被删除");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.Message.Main.TaskCommandMessage;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 下载扫描与删除的互斥(问题 B)。
|
||||
*
|
||||
* <p>缺陷:扫描线程在「把任务标成 COMPRESSING」与「排进压缩队列」之间可以被抢占;删除此刻看到
|
||||
* active 集合与压缩队列里都没有该 gid,判定为「不忙」并把画廊删掉;扫描随后仍把删除前选中的旧
|
||||
* 对象排进压缩队列,压缩线程又把归档写出来,已删除的画廊被复活并上报。
|
||||
*
|
||||
* <p>仅检查状态位({@code isCompressing})无法修复:窗口正是「状态已改、还没入队」的那段。必须让
|
||||
* 「查忙 → 删磁盘 → 清队列」与「选任务 → 标状态 → 入队」落在同一把服务监视器下原子执行。
|
||||
*
|
||||
* <p>所有交错点用 CountDownLatch / 线程状态确定性控制,不用 sleep 概率等待。
|
||||
*/
|
||||
class DownloadCheckDeleteRaceTest {
|
||||
|
||||
/**
|
||||
* 端到端交错:扫描已选中任务、正处在「标 COMPRESSING 到入队」的窗口内时,删除不能同时成功。
|
||||
*
|
||||
* <p>用可控任务在 {@code setStatus(COMPRESSING)} 处暂停——那正是缺陷窗口的位置,且该调用在
|
||||
* 缺陷版本与修复版本中同样存在。暂停时删除线程必须阻塞在服务监视器上,直到扫描完成入队;
|
||||
* 随后删除看到「正在压缩」,回可重试的忙,画廊与归档都不会被破坏。
|
||||
*
|
||||
* <p>缺陷版本里删除不被监视器阻挡,会在此窗口内直接完成,从而与「扫描已入队压缩」同时成功——
|
||||
* 该断言正是对缺陷的判别。
|
||||
*/
|
||||
@Test
|
||||
void deleteCannotCompleteWhileScanIsBetweenCompressingMarkAndEnqueue(@TempDir Path root) throws Exception {
|
||||
Path storage = Files.createDirectories(root.resolve("gallery"));
|
||||
Path downloads = Files.createDirectories(root.resolve("downloads"));
|
||||
Path source = Files.createDirectories(downloads.resolve("sample [90003]"));
|
||||
Files.writeString(source.resolve("1.jpg"), "a");
|
||||
Files.writeString(source.resolve("galleryinfo.txt"), "meta");
|
||||
// 删除目标:gid 命中的画廊目录(压缩归档还未生成)。
|
||||
Path gallery = Files.createDirectories(storage.resolve("sample [90003]"));
|
||||
|
||||
Map<Integer, GalleryTask> queue = new ConcurrentHashMap<>();
|
||||
DownloadCheckService service = new DownloadCheckService(queue, false);
|
||||
service.storagePath = storage.toString();
|
||||
service.downloadPath = downloads.toString();
|
||||
ReliableTaskService reliable = new ReliableTaskService(
|
||||
root.resolve("journal"), queue, service, storage);
|
||||
|
||||
PausingTask task = new PausingTask();
|
||||
task.setGid(90003);
|
||||
task.setName("sample [90003]");
|
||||
task.setPath(source.toString());
|
||||
task.setStatus(GalleryTask.DOWNLOAD_COMPLETE);
|
||||
queue.put(90003, task);
|
||||
task.monitor = service;
|
||||
task.arm();
|
||||
|
||||
Thread scan = new Thread(service::downloadCheck, "scan");
|
||||
scan.start();
|
||||
assertTrue(task.entered.await(5, TimeUnit.SECONDS), "扫描应到达压缩入队前的标记点");
|
||||
assertTrue(task.heldMonitor, "标记点必须位于服务监视器内");
|
||||
assertTrue(Files.exists(gallery), "此时画廊目录应仍然存在");
|
||||
|
||||
byte[] deleteResult = new byte[1];
|
||||
Thread delete = new Thread(() -> {
|
||||
try {
|
||||
deleteResult[0] = reliable.handleCommand(
|
||||
deleteCommand("op-race-e2e", 90003, "sample [90003]"));
|
||||
} catch (Exception e) {
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}, "delete");
|
||||
delete.start();
|
||||
awaitBlocked(delete);
|
||||
assertTrue(Files.exists(gallery), "扫描持锁期间删除不得移除画廊目录");
|
||||
|
||||
task.resume.countDown();
|
||||
scan.join(5000);
|
||||
delete.join(5000);
|
||||
|
||||
assertFalse(scan.isAlive());
|
||||
assertFalse(delete.isAlive());
|
||||
assertEquals(NodeErrorCode.BUSY_COMPRESSING, deleteResult[0],
|
||||
"扫描已进入压缩调度时删除必须回可重试的忙,两者不能同时成功");
|
||||
assertTrue(service.isCompressing(90003), "任务应仍处于压缩调度中");
|
||||
|
||||
service.compress();
|
||||
assertTrue(DownloadCheckService.isValidArchive(
|
||||
storage.resolve("sample [90003]/sample [90003].zip").toFile()),
|
||||
"压缩应发布有效归档");
|
||||
assertEquals(GalleryTask.COMPRESS_COMPLETE, task.getStatus());
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除已经完成时,随后的扫描不得把已删除任务重新压缩,画廊也不得被复活。
|
||||
*
|
||||
* <p>覆盖「DELETE 先落地」的时序:入队按 gid 校验队列中的当前对象,避免把删除前遍历到的
|
||||
* 旧对象重新排进压缩队列。
|
||||
*/
|
||||
@Test
|
||||
void scanAfterCompletedDeleteDoesNotCompressOrResurrectGallery(@TempDir Path root) throws Exception {
|
||||
Fixture fixture = fixture(root, 90004, "sample [90004]");
|
||||
Files.createDirectories(root.resolve("gallery/sample [90004]"));
|
||||
|
||||
assertEquals(NodeErrorCode.SUCCESS,
|
||||
fixture.reliable.handleCommand(deleteCommand("op-e2e-del", 90004, "sample [90004]")));
|
||||
assertTrue(fixture.queue.isEmpty(), "删除成功后任务应从队列移除");
|
||||
|
||||
fixture.service.downloadCheck();
|
||||
assertFalse(fixture.service.isCompressing(90004), "已删除任务不得再被排进压缩");
|
||||
fixture.service.compress();
|
||||
assertFalse(Files.exists(root.resolve("gallery/sample [90004]")),
|
||||
"已删除的画廊不得被压缩复活");
|
||||
}
|
||||
|
||||
// ---- 测试装配 ----
|
||||
|
||||
private record Fixture(Map<Integer, GalleryTask> queue, DownloadCheckService service,
|
||||
ReliableTaskService reliable, GalleryTask task, int gid) {}
|
||||
|
||||
private static Fixture fixture(Path root, int gid, String name) throws Exception {
|
||||
Map<Integer, GalleryTask> queue = new ConcurrentHashMap<>();
|
||||
Path storage = Files.createDirectories(root.resolve("gallery"));
|
||||
Path downloads = Files.createDirectories(root.resolve("downloads"));
|
||||
Path source = Files.createDirectories(downloads.resolve(name));
|
||||
Files.writeString(source.resolve("1.jpg"), "a");
|
||||
Files.writeString(source.resolve("galleryinfo.txt"), "meta");
|
||||
|
||||
DownloadCheckService service = new DownloadCheckService(queue, false);
|
||||
service.storagePath = storage.toString();
|
||||
service.downloadPath = downloads.toString();
|
||||
ReliableTaskService reliable = new ReliableTaskService(
|
||||
root.resolve("journal"), queue, service, storage);
|
||||
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(gid);
|
||||
task.setName(name);
|
||||
task.setPath(source.toString());
|
||||
task.setStatus(GalleryTask.DOWNLOAD_COMPLETE);
|
||||
queue.put(gid, task);
|
||||
return new Fixture(queue, service, reliable, task, gid);
|
||||
}
|
||||
|
||||
private static TaskCommandMessage deleteCommand(String operationId, int gid, String name) {
|
||||
TaskCommandMessage command = new TaskCommandMessage();
|
||||
command.setOperationId(operationId);
|
||||
command.setAction(TaskCommandMessage.ACTION_DELETE);
|
||||
command.setGid(gid);
|
||||
command.setName(name);
|
||||
return command;
|
||||
}
|
||||
|
||||
/** 有界等待线程真正阻塞在监视器上;超时即视为「未按预期互斥」而失败。 */
|
||||
private static void awaitBlocked(Thread thread) {
|
||||
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
|
||||
while (thread.getState() != Thread.State.BLOCKED) {
|
||||
if (System.nanoTime() > deadline)
|
||||
fail("线程未在预期时间内阻塞在服务监视器上,当前状态=" + thread.getState());
|
||||
Thread.onSpinWait();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 可在「标记 COMPRESSING」处暂停的任务。
|
||||
*
|
||||
* <p>该行存在于缺陷版本与修复版本中,正是「状态已改、尚未入队」的缺陷窗口。
|
||||
*/
|
||||
static final class PausingTask extends GalleryTask {
|
||||
final CountDownLatch entered = new CountDownLatch(1);
|
||||
final CountDownLatch resume = new CountDownLatch(1);
|
||||
volatile boolean armed;
|
||||
volatile boolean heldMonitor;
|
||||
|
||||
void arm() {
|
||||
armed = true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setStatus(byte status) {
|
||||
if (armed && status == COMPRESSING) {
|
||||
armed = false;
|
||||
heldMonitor = Thread.holdsLock(this.monitor);
|
||||
entered.countDown();
|
||||
try {
|
||||
if (!resume.await(5, TimeUnit.SECONDS))
|
||||
throw new IllegalStateException("测试未释放扫描暂停点");
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
super.setStatus(status);
|
||||
}
|
||||
|
||||
/** 服务实例;由测试设置,用于断言暂停点确实位于服务监视器内。 */
|
||||
volatile Object monitor;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.zip.ZipEntry;
|
||||
import java.util.zip.ZipOutputStream;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 入队时按 gid 定位磁盘内容。
|
||||
*
|
||||
* <p>gid 稳定、目录名不稳定(下载器会追加分辨率等后缀),因此匹配不能只看名字相等,
|
||||
* 而要按 {@code [gid]} / {@code [gid-...]} 标记判断;同时绝不能把 {@code [1234]}
|
||||
* 误判成 gid 123,否则会把下载链接指向错误的画廊。
|
||||
*/
|
||||
class DownloadCheckServiceGidMatchingTest {
|
||||
|
||||
/** 目录名带后缀时也要能按 gid 命中已有归档,并判为压缩完成。 */
|
||||
@Test
|
||||
void matchesStoredArchiveWhenDirectoryNameHasSuffix(@TempDir Path root) throws Exception {
|
||||
DownloadCheckService service = service(root);
|
||||
Path directory = Files.createDirectories(
|
||||
Path.of(service.storagePath, "sample [123-456]"));
|
||||
writeValidZip(directory.resolve("sample [123-456].zip"));
|
||||
|
||||
GalleryTask result = service.addToQueue(task(123));
|
||||
|
||||
assertTrue(result.is_compress_complete(), "gid 命中已有归档应判为已完成");
|
||||
assertEquals("sample [123-456]", result.getName(), "名称应取实际目录名");
|
||||
}
|
||||
|
||||
/** 尚未出现 galleryinfo.txt:判为下载中,并统计已下载页数。 */
|
||||
@Test
|
||||
void marksTaskDownloadingWhileGalleryInfoMissing(@TempDir Path root) throws Exception {
|
||||
DownloadCheckService service = service(root);
|
||||
Path directory = Files.createDirectories(Path.of(service.downloadPath, "sample [123-456]"));
|
||||
Files.writeString(directory.resolve("1.jpg"), "x");
|
||||
Files.writeString(directory.resolve("2.jpg"), "y");
|
||||
|
||||
GalleryTask result = service.addToQueue(task(123));
|
||||
|
||||
assertEquals(GalleryTask.DOWNLOADING, result.getStatus());
|
||||
assertEquals(2, result.getProceeding(), "应统计除 galleryinfo.txt 外的页数");
|
||||
assertEquals("sample [123-456]", result.getName());
|
||||
}
|
||||
|
||||
/** 出现 galleryinfo.txt:判为下载完成,并记录源目录用于后续压缩。 */
|
||||
@Test
|
||||
void marksTaskCompleteWhenGalleryInfoPresent(@TempDir Path root) throws Exception {
|
||||
DownloadCheckService service = service(root);
|
||||
Path directory = Files.createDirectories(Path.of(service.downloadPath, "sample [123]"));
|
||||
Files.writeString(directory.resolve("1.jpg"), "x");
|
||||
Files.writeString(directory.resolve("galleryinfo.txt"), "meta");
|
||||
|
||||
GalleryTask result = service.addToQueue(task(123));
|
||||
|
||||
assertEquals(GalleryTask.DOWNLOAD_COMPLETE, result.getStatus());
|
||||
assertEquals(directory.toString(), result.getPath());
|
||||
}
|
||||
|
||||
/** gid 前缀相同但实际不同的目录不得被匹配。 */
|
||||
@Test
|
||||
void doesNotMatchLongerGidSharingTheSamePrefix(@TempDir Path root) throws Exception {
|
||||
DownloadCheckService service = service(root);
|
||||
Files.createDirectories(Path.of(service.downloadPath, "sample [1234]"));
|
||||
|
||||
GalleryTask result = service.addToQueue(task(123));
|
||||
|
||||
assertEquals(0, result.getStatus(), "[1234] 不得被当成 gid 123 的目录");
|
||||
assertNull(result.getName(), "未命中时不应改写任务名称");
|
||||
}
|
||||
|
||||
/** 归档校验:空文件与不存在的路径都不算有效归档。 */
|
||||
@Test
|
||||
void rejectsEmptyAndMissingArchives(@TempDir Path root) throws Exception {
|
||||
Path empty = Files.createFile(root.resolve("empty.zip"));
|
||||
assertFalse(DownloadCheckService.isValidArchive(empty.toFile()));
|
||||
assertFalse(DownloadCheckService.isValidArchive(root.resolve("missing.zip").toFile()));
|
||||
|
||||
Path valid = root.resolve("valid.zip");
|
||||
writeValidZip(valid);
|
||||
assertTrue(DownloadCheckService.isValidArchive(valid.toFile()));
|
||||
}
|
||||
|
||||
private static DownloadCheckService service(Path root) throws Exception {
|
||||
DownloadCheckService service = new DownloadCheckService(new ConcurrentHashMap<>(), false);
|
||||
service.downloadPath = Files.createDirectories(root.resolve("download")).toString();
|
||||
service.storagePath = Files.createDirectories(root.resolve("gallery")).toString();
|
||||
return service;
|
||||
}
|
||||
|
||||
private static GalleryTask task(int gid) {
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(gid);
|
||||
return task;
|
||||
}
|
||||
|
||||
private static void writeValidZip(Path zip) throws Exception {
|
||||
Files.createDirectories(zip.getParent());
|
||||
try (ZipOutputStream out = new ZipOutputStream(Files.newOutputStream(zip))) {
|
||||
out.putNextEntry(new ZipEntry("1.jpg"));
|
||||
out.write("image bytes".getBytes(StandardCharsets.UTF_8));
|
||||
out.closeEntry();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import java.nio.file.*;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
class DownloadCheckServiceTest {
|
||||
@Test
|
||||
void failedCompressionRetainsSourceAndCanRetry(@TempDir Path root) throws Exception {
|
||||
var queue = new ConcurrentHashMap<Integer, GalleryTask>();
|
||||
var service = new DownloadCheckService(queue, false);
|
||||
Path downloads = Files.createDirectory(root.resolve("downloads"));
|
||||
Path source = Files.createDirectory(downloads.resolve("sample [123]"));
|
||||
Files.writeString(source.resolve("galleryinfo.txt"), "metadata");
|
||||
Files.writeString(source.resolve("1.jpg"), "image bytes");
|
||||
Path storage = root.resolve("storage");
|
||||
Files.writeString(storage, "block mkdir");
|
||||
service.downloadPath = downloads.toString();
|
||||
service.storagePath = storage.toString();
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(123);
|
||||
service.addToQueue(task);
|
||||
service.downloadCheck();
|
||||
service.compress();
|
||||
assertEquals(GalleryTask.DOWNLOAD_COMPLETE, task.getStatus());
|
||||
assertTrue(Files.exists(source.resolve("1.jpg")));
|
||||
Files.delete(storage);
|
||||
Files.createDirectory(storage);
|
||||
service.addToQueue(task); // manual retry removes the backoff
|
||||
service.downloadCheck();
|
||||
service.compress();
|
||||
assertEquals(GalleryTask.COMPRESS_COMPLETE, task.getStatus());
|
||||
assertTrue(DownloadCheckService.isValidArchive(storage.resolve("sample [123]/sample [123].zip").toFile()));
|
||||
assertFalse(Files.exists(source));
|
||||
}
|
||||
|
||||
@Test
|
||||
void corruptArchiveIsNotTreatedAsCompleted(@TempDir Path root) throws Exception {
|
||||
Path stored = Files.createDirectories(root.resolve("stored/sample [123]"));
|
||||
Files.writeString(stored.resolve("sample [123].zip"), "partial zip");
|
||||
var service = new DownloadCheckService(new ConcurrentHashMap<>(), false);
|
||||
service.storagePath = root.resolve("stored").toString();
|
||||
service.downloadPath = root.resolve("downloads").toString();
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(123);
|
||||
assertFalse(service.addToQueue(task).is_compress_complete());
|
||||
}
|
||||
|
||||
/**
|
||||
* 已有有效归档时必须判为完成,且重复添加(命中校验缓存)行为保持一致。
|
||||
* 校验缓存按「路径 + 大小」复用,目的是避免每次 addToQueue 都把整包读一遍,
|
||||
* 但它绝不能让有效归档被误判为未完成。
|
||||
*/
|
||||
@Test
|
||||
void validArchiveIsRecognizedAcrossRepeatedQueueAdds(@TempDir Path root) throws Exception {
|
||||
Path stored = Files.createDirectories(root.resolve("stored/sample [123]"));
|
||||
Path zip = stored.resolve("sample [123].zip");
|
||||
try (var out = new java.util.zip.ZipOutputStream(Files.newOutputStream(zip))) {
|
||||
out.putNextEntry(new java.util.zip.ZipEntry("1.jpg"));
|
||||
out.write("image bytes".getBytes(java.nio.charset.StandardCharsets.UTF_8));
|
||||
out.closeEntry();
|
||||
}
|
||||
var service = new DownloadCheckService(new ConcurrentHashMap<>(), false);
|
||||
service.storagePath = root.resolve("stored").toString();
|
||||
service.downloadPath = root.resolve("downloads").toString();
|
||||
|
||||
GalleryTask first = new GalleryTask();
|
||||
first.setGid(123);
|
||||
assertTrue(service.addToQueue(first).is_compress_complete(), "存在有效归档时应判为完成");
|
||||
|
||||
GalleryTask second = new GalleryTask();
|
||||
second.setGid(123);
|
||||
assertTrue(service.addToQueue(second).is_compress_complete(), "第二次应命中校验缓存且结论一致");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
package lion.Service;
|
||||
|
||||
import io.netty.channel.Channel;
|
||||
import io.netty.channel.embedded.EmbeddedChannel;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
class PrimaryChannelTrackerTest {
|
||||
|
||||
/**
|
||||
* 未认证通道退出时不得清空引用。
|
||||
*
|
||||
* <p>这正是线上事故的成因:并发认证让引用先指向 X 再被 Y 覆盖,
|
||||
* Y 退出时把引用清空,而仍在工作的 X 从此不再被用于上报任务状态。
|
||||
*/
|
||||
@Test
|
||||
void unregisterOfNonPrimaryChannelKeepsReference() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel primary = new EmbeddedChannel();
|
||||
EmbeddedChannel other = new EmbeddedChannel();
|
||||
try {
|
||||
tracker.onPrimaryMessage(primary);
|
||||
tracker.onPrimaryMessage(other);
|
||||
tracker.onPrimaryMessage(primary); // 引用回到仍在工作的那条通道
|
||||
|
||||
tracker.unregister(other);
|
||||
|
||||
assertSame(primary, tracker.current());
|
||||
assertTrue(tracker.usable());
|
||||
} finally {
|
||||
primary.finishAndReleaseAll();
|
||||
other.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 首选通道退出时立即回退到其它仍可用的已认证通道。
|
||||
*
|
||||
* <p>事故中节点引用的那条通道退出后引用被清空,而另一条已认证通道仍然活跃,
|
||||
* 任务状态就此无人上报。回退保证「有已认证通道在,就一定能上报」。
|
||||
*/
|
||||
@Test
|
||||
void unregisterOfPrimaryFallsBackToSurvivingChannel() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel survivor = new EmbeddedChannel();
|
||||
EmbeddedChannel transientChannel = new EmbeddedChannel();
|
||||
try {
|
||||
tracker.onPrimaryMessage(survivor);
|
||||
tracker.onPrimaryMessage(transientChannel);
|
||||
tracker.unregister(transientChannel);
|
||||
|
||||
assertSame(survivor, tracker.current(), "应回退到仍可用的已认证通道");
|
||||
assertTrue(tracker.usable());
|
||||
} finally {
|
||||
survivor.finishAndReleaseAll();
|
||||
transientChannel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 实际在收发消息的已认证通道应当接管引用,即使原引用仍处于 active 状态。 */
|
||||
@Test
|
||||
void authenticatedChannelTakesOverOnMessage() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel stalePrimary = new EmbeddedChannel();
|
||||
EmbeddedChannel live = new EmbeddedChannel();
|
||||
try {
|
||||
tracker.onPrimaryMessage(stalePrimary);
|
||||
tracker.onPrimaryMessage(live);
|
||||
tracker.onPrimaryMessage(stalePrimary); // 引用先落在 stalePrimary 上
|
||||
|
||||
tracker.onPrimaryMessage(live);
|
||||
|
||||
assertSame(live, tracker.current(), "说话的通道应成为引用");
|
||||
} finally {
|
||||
stalePrimary.finishAndReleaseAll();
|
||||
live.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 未经认证的通道不得占用引用,避免陌生连接抢走状态上报通道。 */
|
||||
@Test
|
||||
void unauthenticatedChannelCannotClaimReference() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel authenticated = new EmbeddedChannel();
|
||||
EmbeddedChannel stranger = new EmbeddedChannel();
|
||||
try {
|
||||
tracker.onPrimaryMessage(authenticated);
|
||||
|
||||
// 未认证通道不会进入已认证集合,因此不能用它认领。
|
||||
|
||||
assertSame(authenticated, tracker.current());
|
||||
} finally {
|
||||
authenticated.finishAndReleaseAll();
|
||||
stranger.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 已下线的通道即使曾被认证,也不能再占用引用。 */
|
||||
@Test
|
||||
void inactiveChannelIsNotUsable() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
EmbeddedChannel channel = new EmbeddedChannel();
|
||||
tracker.onPrimaryMessage(channel);
|
||||
channel.finishAndReleaseAll();
|
||||
|
||||
assertFalse(tracker.usable());
|
||||
}
|
||||
|
||||
@Test
|
||||
void emptyTrackerHasNoUsableChannel() {
|
||||
PrimaryChannelTracker tracker = new PrimaryChannelTracker();
|
||||
|
||||
assertNull(tracker.current());
|
||||
assertFalse(tracker.usable());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package lion.Service;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
class RekickPolicyTest {
|
||||
|
||||
@Test
|
||||
void noPendingReportsNeverRekicks() {
|
||||
RekickPolicy policy = new RekickPolicy();
|
||||
|
||||
assertFalse(policy.shouldRekick(false, false, 1_000L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void usableChannelNeverRekicks() {
|
||||
RekickPolicy policy = new RekickPolicy();
|
||||
|
||||
assertFalse(policy.shouldRekick(true, true, 1_000L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void pendingWithoutChannelRekicksThenRateLimits() {
|
||||
RekickPolicy policy = new RekickPolicy();
|
||||
long now = 1_000_000L;
|
||||
|
||||
assertTrue(policy.shouldRekick(true, false, now), "首次应允许重连");
|
||||
assertFalse(policy.shouldRekick(true, false, now + 1), "间隔内应被限流");
|
||||
assertFalse(policy.shouldRekick(true, false, now + RekickPolicy.MIN_INTERVAL_MILLIS - 1));
|
||||
assertTrue(policy.shouldRekick(true, false, now + RekickPolicy.MIN_INTERVAL_MILLIS),
|
||||
"超过最小间隔后应再次允许");
|
||||
}
|
||||
|
||||
/** 限流只抑制连击,不应把「已限流」永久锁死后续尝试。 */
|
||||
@Test
|
||||
void rateLimitDoesNotBlockForever() {
|
||||
RekickPolicy policy = new RekickPolicy();
|
||||
long now = 500_000L;
|
||||
|
||||
policy.shouldRekick(true, false, now);
|
||||
for (int i = 0; i < 5; i++)
|
||||
policy.shouldRekick(true, false, now + i * 1_000L);
|
||||
|
||||
assertTrue(policy.shouldRekick(true, false, now + 10 * RekickPolicy.MIN_INTERVAL_MILLIS));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,356 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.Message.Main.TaskCommandMessage;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 可靠任务协议在节点侧的行为:命令幂等、回执持久化、ACK 只清理终态任务、重启恢复。
|
||||
*
|
||||
* <p>全部离线运行:不绑定端口,也不外呼真实主站。
|
||||
*/
|
||||
class ReliableTaskServiceTest {
|
||||
|
||||
/** 相同 operationId 重复执行不得产生第二次副作用,且回放同一结果码。 */
|
||||
@Test
|
||||
void repeatedDownloadCommandReusesReceipt(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
TaskCommandMessage command = download("op-dl-1", 80001, "sample [80001]");
|
||||
|
||||
assertEquals(NodeErrorCode.SUCCESS, harness.service.handleCommand(command));
|
||||
GalleryTask first = harness.queue.get(80001);
|
||||
assertNotNull(first);
|
||||
|
||||
// 更换 name 再发同一 operationId:必须回放回执,不得改写已受理的任务。
|
||||
TaskCommandMessage retry = download("op-dl-1", 80001, "changed-name [80001]");
|
||||
assertEquals(NodeErrorCode.SUCCESS, harness.service.handleCommand(retry));
|
||||
assertSame(first, harness.queue.get(80001), "重复命令不得替换已受理的任务对象");
|
||||
assertEquals("sample [80001]", harness.queue.get(80001).getName());
|
||||
}
|
||||
|
||||
/** 删除成功是终态:重复删除直接回放,绝不再次执行删除副作用。 */
|
||||
@Test
|
||||
void repeatedDeleteCommandDoesNotRerunSideEffect(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
Path gallery = Files.createDirectories(harness.storagePath.resolve("sample [80002]"));
|
||||
|
||||
TaskCommandMessage delete = delete("op-del-1", 80002, "sample [80002]");
|
||||
assertEquals(NodeErrorCode.SUCCESS, harness.service.handleCommand(delete));
|
||||
assertFalse(Files.exists(gallery));
|
||||
|
||||
// 重新创建同名目录,再发同一 operationId:若回执失效会把它删掉,这里必须保留。
|
||||
Files.createDirectories(gallery);
|
||||
assertEquals(NodeErrorCode.SUCCESS, harness.service.handleCommand(delete));
|
||||
assertTrue(Files.exists(gallery), "重复删除命令不得再次执行副作用");
|
||||
}
|
||||
|
||||
/** 压缩中的目标必须回可重试的「忙」,且不得写入永久回执。 */
|
||||
@Test
|
||||
void deleteWhileCompressingIsRetryable(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
Path gallery = Files.createDirectories(harness.storagePath.resolve("sample [80003]"));
|
||||
harness.downloadCheck.activeCompressions.add(80003);
|
||||
|
||||
byte result = harness.service.handleCommand(delete("op-del-busy", 80003, "sample [80003]"));
|
||||
|
||||
assertEquals(NodeErrorCode.BUSY_COMPRESSING, result);
|
||||
assertNull(harness.service.journal().receipt("op-del-busy"), "可重试失败不得记为永久回执");
|
||||
assertTrue(Files.exists(gallery), "忙时不得动磁盘");
|
||||
|
||||
// 压缩结束后同一 operationId 重试应当成功。
|
||||
harness.downloadCheck.activeCompressions.remove(80003);
|
||||
assertEquals(NodeErrorCode.SUCCESS, harness.service.handleCommand(delete("op-del-busy", 80003, "sample [80003]")));
|
||||
assertFalse(Files.exists(gallery));
|
||||
}
|
||||
|
||||
/**
|
||||
* ACK 只清理压缩完成的终态任务。
|
||||
*
|
||||
* <p>这是根因级缺陷:若按「报告里出现的 status」无条件删除,正在下载的任务会在主站
|
||||
* 刚收到进度报告后就被节点遗忘,下载随后无人推进,任务永久停在未完成。
|
||||
*/
|
||||
@Test
|
||||
void ackKeepsInProgressTaskAndClearsOnlyCompleted(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
harness.service.handleCommand(download("op-a", 80010, "in-progress [80010]"));
|
||||
harness.service.handleCommand(download("op-b", 80011, "finished [80011]"));
|
||||
harness.queue.get(80011).setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
|
||||
TaskJournal.ReportSnapshot snapshot = harness.service.buildReport().orElseThrow();
|
||||
assertEquals(2, snapshot.tasks.size());
|
||||
|
||||
harness.service.applyAck(snapshot.reportId, NodeErrorCode.SUCCESS);
|
||||
|
||||
assertTrue(harness.queue.containsKey(80010), "下载中的任务不得因 ACK 被移除");
|
||||
assertTrue(harness.service.journal().tasks().containsKey(80010), "日志里也应保留进行中的任务");
|
||||
assertFalse(harness.queue.containsKey(80011), "压缩完成的任务应被清理");
|
||||
assertFalse(harness.service.journal().tasks().containsKey(80011));
|
||||
}
|
||||
|
||||
/** 失败或未知的 ACK 不得移除任何任务,也不得丢掉待发快照。 */
|
||||
@Test
|
||||
void rejectedAndDuplicateAcksKeepEverything(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
harness.service.handleCommand(download("op-c", 80020, "done [80020]"));
|
||||
harness.queue.get(80020).setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
TaskJournal.ReportSnapshot snapshot = harness.service.buildReport().orElseThrow();
|
||||
|
||||
assertTrue(harness.service.applyAck(snapshot.reportId, NodeErrorCode.IO_ERROR).isEmpty(),
|
||||
"未提交的 ACK 不得清理任务");
|
||||
assertTrue(harness.service.pendingReports().stream().anyMatch(r -> r.reportId.equals(snapshot.reportId)),
|
||||
"失败 ACK 必须保留快照等待重试");
|
||||
assertTrue(harness.queue.containsKey(80020));
|
||||
|
||||
// 重复 ACK:第一次清理,第二次无副作用。
|
||||
assertEquals(1, harness.service.applyAck(snapshot.reportId, NodeErrorCode.SUCCESS).size());
|
||||
assertTrue(harness.service.applyAck(snapshot.reportId, NodeErrorCode.SUCCESS).isEmpty(),
|
||||
"重复 ACK 不得重复清理或报错");
|
||||
}
|
||||
|
||||
/** 迟到的 ACK 碰上已经又前进的任务时,必须保留该任务。 */
|
||||
@Test
|
||||
void lateAckDoesNotRemoveTaskThatAdvanced(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
harness.service.handleCommand(download("op-late", 80030, "advancing [80030]"));
|
||||
TaskJournal.ReportSnapshot snapshot = harness.service.buildReport().orElseThrow();
|
||||
|
||||
// 报告发出后任务又前进到完成(尚未重新落盘),此时迟到的 ACK 只代表更早的状态。
|
||||
harness.queue.get(80030).setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
harness.queue.get(80030).setProceeding(99);
|
||||
|
||||
harness.service.applyAck(snapshot.reportId, NodeErrorCode.SUCCESS);
|
||||
|
||||
assertTrue(harness.queue.containsKey(80030), "ACK 内容与运行期状态不一致时不得删除任务");
|
||||
}
|
||||
|
||||
/** 重启后必须恢复任务与待 ACK 报告,并把中断的压缩退回可重试状态。 */
|
||||
@Test
|
||||
void restartRestoresTasksReportsAndResetsInterruptedCompression(@TempDir Path root) throws Exception {
|
||||
Harness first = new Harness(root);
|
||||
first.service.handleCommand(download("op-r", 80040, "resume [80040]"));
|
||||
TaskJournal.ReportSnapshot snapshot = first.service.buildReport().orElseThrow();
|
||||
// 中断在压缩中途:重启后应回到可重试的下载完成态。
|
||||
first.queue.get(80040).setStatus(GalleryTask.COMPRESSING);
|
||||
first.service.refreshJournalFromQueue();
|
||||
|
||||
Harness restarted = new Harness(root, true);
|
||||
|
||||
assertTrue(restarted.queue.containsKey(80040), "重启应恢复任务");
|
||||
assertEquals(GalleryTask.DOWNLOAD_COMPLETE, restarted.queue.get(80040).getStatus(),
|
||||
"中断的压缩应退回可重试状态");
|
||||
assertTrue(restarted.service.pendingReports().stream()
|
||||
.anyMatch(r -> r.reportId.equals(snapshot.reportId)),
|
||||
"重启应恢复待 ACK 报告以便重发");
|
||||
assertNotNull(restarted.service.journal().receipt("op-r"), "命令回执必须跨重启保留");
|
||||
}
|
||||
|
||||
/**
|
||||
* 压缩已原子发布 ZIP 并删除源目录、但 COMPRESS_COMPLETE 尚未同步进 journal 时崩溃。
|
||||
*
|
||||
* <p>这是问题 A 的核心回归:日志停在 COMPRESSING,磁盘上却已有有效归档、下载目录空空如也。
|
||||
* 若重启只把 COMPRESSING 降级成 DOWNLOAD_COMPLETE,扫描会因空下载目录提前返回、重投又只
|
||||
* 回放回执,任务永久等不到压缩。恢复必须按 gid 校验磁盘上的有效 ZIP 并重建为压缩完成。
|
||||
*/
|
||||
@Test
|
||||
void restartRecoversPublishedArchiveAfterCrashBeforeJournalSync(@TempDir Path root) throws Exception {
|
||||
Harness first = new Harness(root);
|
||||
first.service.handleCommand(download("op-archive", 80060, "sample [80060]"));
|
||||
// 模拟状态已推进到压缩中并落盘,随后压缩成功发布归档,但日志未再同步。
|
||||
first.queue.get(80060).setStatus(GalleryTask.COMPRESSING);
|
||||
first.service.refreshJournalFromQueue();
|
||||
TaskJournal.ReportSnapshot snapshot = first.service.buildReport().orElseThrow();
|
||||
|
||||
Path archiveDirectory = Files.createDirectories(first.storagePath.resolve("sample [80060]"));
|
||||
writeValidZip(archiveDirectory.resolve("sample [80060].zip"));
|
||||
// 源目录已被 compress() 删除,下载目录为空。
|
||||
assertFalse(Files.exists(first.downloadsPath.resolve("sample [80060]")));
|
||||
|
||||
Harness restarted = new Harness(root, true);
|
||||
|
||||
GalleryTask restored = restarted.queue.get(80060);
|
||||
assertNotNull(restored, "重启必须保留任务");
|
||||
assertEquals(GalleryTask.COMPRESS_COMPLETE, restored.getStatus(),
|
||||
"磁盘上已有有效归档时必须恢复为压缩完成,而不是停在等待压缩");
|
||||
assertEquals("sample [80060]", restored.getName(), "恢复必须重建目录名");
|
||||
assertEquals(archiveDirectory.toString(), restored.getPath(), "恢复必须重建归档路径");
|
||||
assertTrue(restarted.service.pendingReports().stream()
|
||||
.anyMatch(report -> report.reportId.equals(snapshot.reportId)),
|
||||
"恢复必须保留待 ACK 报告");
|
||||
assertEquals("op-archive", restarted.service.operationIds().get(80060),
|
||||
"恢复必须保留 operationId,不得重新提交上游");
|
||||
}
|
||||
|
||||
/** 没有有效归档、也没有源目录时:保留任务,但绝不捏造压缩完成。 */
|
||||
@Test
|
||||
void restartWithoutArchiveOrSourceKeepsTaskWithoutFabricatingCompletion(@TempDir Path root) throws Exception {
|
||||
Harness first = new Harness(root);
|
||||
first.service.handleCommand(download("op-empty", 80061, "sample [80061]"));
|
||||
first.queue.get(80061).setStatus(GalleryTask.COMPRESSING);
|
||||
first.service.refreshJournalFromQueue();
|
||||
|
||||
Harness restarted = new Harness(root, true);
|
||||
|
||||
GalleryTask restored = restarted.queue.get(80061);
|
||||
assertNotNull(restored, "无归档且无源时也必须保留任务");
|
||||
assertNotEquals(GalleryTask.COMPRESS_COMPLETE, restored.getStatus(),
|
||||
"没有有效归档时不得捏造压缩完成");
|
||||
assertNull(restored.getPath(), "没有真实源目录时不得编造路径");
|
||||
}
|
||||
|
||||
/** 压缩中途源目录仍在:恢复为可继续压缩的下载完成态,并带回 path 与进度。 */
|
||||
@Test
|
||||
void restartWithSourcePresentRebuildsDownloadCompleteWithPathAndProgress(@TempDir Path root) throws Exception {
|
||||
Harness first = new Harness(root);
|
||||
first.service.handleCommand(download("op-source", 80062, "sample [80062]"));
|
||||
first.queue.get(80062).setStatus(GalleryTask.COMPRESSING);
|
||||
first.service.refreshJournalFromQueue();
|
||||
|
||||
Path source = Files.createDirectories(first.downloadsPath.resolve("sample [80062]"));
|
||||
Files.writeString(source.resolve("1.jpg"), "a");
|
||||
Files.writeString(source.resolve("2.jpg"), "b");
|
||||
Files.writeString(source.resolve("galleryinfo.txt"), "meta");
|
||||
|
||||
Harness restarted = new Harness(root, true);
|
||||
|
||||
GalleryTask restored = restarted.queue.get(80062);
|
||||
assertEquals(GalleryTask.DOWNLOAD_COMPLETE, restored.getStatus(),
|
||||
"源目录仍在时应恢复为可重试的下载完成态");
|
||||
assertEquals(source.toString(), restored.getPath(), "恢复必须重建源目录路径");
|
||||
assertEquals(2, restored.getProceeding(), "恢复必须重建已下载页数");
|
||||
}
|
||||
|
||||
/** 同名位置上的 ZIP 已损坏:不得被判为压缩完成,任务必须保留待重试。 */
|
||||
@Test
|
||||
void restartWithCorruptArchiveDoesNotComplete(@TempDir Path root) throws Exception {
|
||||
Harness first = new Harness(root);
|
||||
first.service.handleCommand(download("op-corrupt", 80063, "sample [80063]"));
|
||||
first.queue.get(80063).setStatus(GalleryTask.COMPRESSING);
|
||||
first.service.refreshJournalFromQueue();
|
||||
|
||||
Path archiveDirectory = Files.createDirectories(first.storagePath.resolve("sample [80063]"));
|
||||
Files.writeString(archiveDirectory.resolve("sample [80063].zip"), "partial zip");
|
||||
|
||||
Harness restarted = new Harness(root, true);
|
||||
|
||||
GalleryTask restored = restarted.queue.get(80063);
|
||||
assertNotNull(restored, "损坏归档不得让任务消失");
|
||||
assertNotEquals(GalleryTask.COMPRESS_COMPLETE, restored.getStatus(),
|
||||
"损坏或空归档绝不能判为压缩完成");
|
||||
}
|
||||
|
||||
/** 命令缺 operationId 或 action 非法时回不可重试的无效命令,且不产生回执。 */
|
||||
@Test
|
||||
void invalidCommandsAreRejectedWithoutReceipt(@TempDir Path root) throws Exception {
|
||||
Harness harness = new Harness(root);
|
||||
|
||||
TaskCommandMessage noId = download(null, 80050, "x [80050]");
|
||||
assertEquals(NodeErrorCode.INVALID_COMMAND, harness.service.handleCommand(noId));
|
||||
|
||||
TaskCommandMessage unknownAction = download("op-u", 80051, "x [80051]");
|
||||
unknownAction.setAction("FROBNICATE");
|
||||
assertEquals(NodeErrorCode.INVALID_COMMAND, harness.service.handleCommand(unknownAction));
|
||||
|
||||
assertTrue(harness.service.journal().receipts().isEmpty(), "无效命令不得留下回执");
|
||||
}
|
||||
|
||||
/**
|
||||
* 重启把任务恢复成新事实后,崩溃前生成的旧待 ACK 报告不得清掉它。
|
||||
*
|
||||
* <p>恢复必须把重建结果落盘({@link ReliableTaskService#restore} 末尾的 syncTasks):
|
||||
* 它既让日志与磁盘一致(否则后续报告会带着过期的 COMPRESSING 上报),又用更高的 revision
|
||||
* 取代旧记录,使一份内容巧合匹配的旧 ACK 被 removeTaskIfRevision 的版本过滤拒绝。
|
||||
*/
|
||||
@Test
|
||||
void staleAckFromBeforeRestartDoesNotClearRecoveredTask(@TempDir Path root) throws Exception {
|
||||
Harness first = new Harness(root);
|
||||
first.service.handleCommand(download("op-stale", 80070, "sample [80070]"));
|
||||
// 崩溃前:日志停在压缩中,且已生成一份覆盖该状态的待 ACK 报告。
|
||||
first.queue.get(80070).setStatus(GalleryTask.COMPRESSING);
|
||||
first.service.refreshJournalFromQueue();
|
||||
TaskJournal.ReportSnapshot staleSnapshot = first.service.buildReport().orElseThrow();
|
||||
|
||||
Path archiveDirectory = Files.createDirectories(first.storagePath.resolve("sample [80070]"));
|
||||
writeValidZip(archiveDirectory.resolve("sample [80070].zip"));
|
||||
|
||||
Harness restarted = new Harness(root, true);
|
||||
GalleryTask recovered = restarted.queue.get(80070);
|
||||
assertEquals(GalleryTask.COMPRESS_COMPLETE, recovered.getStatus(), "重启应按磁盘事实恢复为压缩完成");
|
||||
// 恢复结果必须已经落盘:从磁盘重新加载日志仍是新状态。
|
||||
TaskJournal reloaded = new TaskJournal(root.resolve("task-journal"));
|
||||
reloaded.load();
|
||||
assertEquals(GalleryTask.COMPRESS_COMPLETE, reloaded.task(80070).status,
|
||||
"恢复后的新事实必须同步进日志");
|
||||
assertTrue(reloaded.task(80070).revision
|
||||
> staleSnapshot.tasks.stream().filter(t -> t.gid == 80070).findFirst().orElseThrow().revision,
|
||||
"恢复必须推进 revision,使旧快照的版本过滤失效");
|
||||
|
||||
restarted.service.applyAck(staleSnapshot.reportId, NodeErrorCode.SUCCESS);
|
||||
|
||||
assertTrue(restarted.queue.containsKey(80070),
|
||||
"崩溃前的旧 ACK 不得清掉已恢复的任务");
|
||||
assertTrue(restarted.service.journal().tasks().containsKey(80070),
|
||||
"日志里也必须保留已恢复的任务");
|
||||
}
|
||||
|
||||
/** 一个最小离线装配:显式目录、无调度器、无网络。 */
|
||||
static class Harness {
|
||||
final Map<Integer, GalleryTask> queue = new ConcurrentHashMap<>();
|
||||
final DownloadCheckService downloadCheck;
|
||||
final ReliableTaskService service;
|
||||
final Path storagePath;
|
||||
final Path downloadsPath;
|
||||
|
||||
Harness(Path root) throws Exception {
|
||||
this(root, false);
|
||||
}
|
||||
|
||||
Harness(Path root, boolean restore) throws Exception {
|
||||
storagePath = Files.createDirectories(root.resolve("gallery"));
|
||||
downloadsPath = Files.createDirectories(root.resolve("downloads"));
|
||||
Path journalDir = root.resolve("task-journal");
|
||||
downloadCheck = new DownloadCheckService(queue, false);
|
||||
downloadCheck.storagePath = storagePath.toString();
|
||||
downloadCheck.downloadPath = downloadsPath.toString();
|
||||
service = new ReliableTaskService(journalDir, queue, downloadCheck, storagePath);
|
||||
if (restore)
|
||||
service.restore();
|
||||
}
|
||||
}
|
||||
|
||||
/** 写一个真实的小 ZIP(含 CRC 校验,可被 isValidArchive 接受)。 */
|
||||
static void writeValidZip(Path zip) throws Exception {
|
||||
Files.createDirectories(zip.getParent());
|
||||
try (java.util.zip.ZipOutputStream out =
|
||||
new java.util.zip.ZipOutputStream(Files.newOutputStream(zip))) {
|
||||
out.putNextEntry(new java.util.zip.ZipEntry("1.jpg"));
|
||||
out.write("image bytes".getBytes(java.nio.charset.StandardCharsets.UTF_8));
|
||||
out.closeEntry();
|
||||
}
|
||||
}
|
||||
|
||||
private static TaskCommandMessage download(String operationId, int gid, String name) {
|
||||
TaskCommandMessage command = new TaskCommandMessage();
|
||||
command.setOperationId(operationId);
|
||||
command.setAction(TaskCommandMessage.ACTION_DOWNLOAD);
|
||||
command.setGid(gid);
|
||||
command.setName(name);
|
||||
return command;
|
||||
}
|
||||
|
||||
private static TaskCommandMessage delete(String operationId, int gid, String name) {
|
||||
TaskCommandMessage command = new TaskCommandMessage();
|
||||
command.setOperationId(operationId);
|
||||
command.setAction(TaskCommandMessage.ACTION_DELETE);
|
||||
command.setGid(gid);
|
||||
command.setName(name);
|
||||
return command;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,469 @@
|
||||
package lion.Service;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import lion.CustomUtil;
|
||||
import lion.Externel.BackupSubServer;
|
||||
import lion.Http.HttpRequestModel;
|
||||
import lion.Http.HttpResponseModel;
|
||||
import lion.Message.Main.SubscriptionAccountSnapshot;
|
||||
import lion.Message.Main.SubscriptionBindingSnapshot;
|
||||
import lion.Message.Main.SubscriptionSnapshotMessage;
|
||||
import lion.Message.Main.SubscriptionSnapshotPayload;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Base64;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
class SubscriptionGatewayTest {
|
||||
private static final String SECRET = "gateway-test-secret";
|
||||
private static final String KEY = "public-key-abcdef";
|
||||
private static final String LOCAL = "last synced snapshot";
|
||||
private static final HttpClient HTTP = HttpClient.newHttpClient();
|
||||
@TempDir Path root;
|
||||
|
||||
@Test
|
||||
void primaryPreferredAndSignedWithoutForwardingExternalAuth() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
HttpResponse<String> response = h.request("GET", "User-Agent", "Clash/1.2", "X-Real-IP", "198.51.100.9",
|
||||
"Authorization", "Bearer never-forward", "X-Subscription-Event", "attacker");
|
||||
assertEquals(200, response.statusCode());
|
||||
assertEquals("fresh primary", response.body());
|
||||
assertEquals(0, h.pending());
|
||||
JsonNode event = h.lastEvent.get();
|
||||
assertEquals(hash(KEY.getBytes(StandardCharsets.UTF_8)), event.path("keyHash").asText());
|
||||
assertEquals("Clash/1.2", event.path("userAgent").asText());
|
||||
assertEquals("198.51.100.9", event.path("clientIp").asText());
|
||||
assertEquals("v2", event.path("client").asText());
|
||||
assertNull(h.authorization.get());
|
||||
assertTrue(h.signatureValid);
|
||||
assertFalse(new String(h.eventBytes.get(), StandardCharsets.UTF_8).contains(KEY));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void fallbackDurablyRecordsAndUploadAcknowledgementClears() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
h.primaryStatus.set(503);
|
||||
assertEquals(LOCAL, h.request("GET", "User-Agent", "real-agent").body());
|
||||
assertEquals(1, h.pending());
|
||||
SubscriptionEventJournal recovered = new SubscriptionEventJournal(h.eventDir);
|
||||
assertEquals(1, recovered.size(), "事件必须可以从磁盘恢复");
|
||||
String eventId = recovered.oldest(1).getFirst().eventId();
|
||||
h.gateway.uploadOnce();
|
||||
assertEquals(0, h.pending());
|
||||
assertEquals(eventId, h.uploaded.get().path("events").get(0).path("eventId").asText());
|
||||
assertEquals(0, new SubscriptionEventJournal(h.eventDir).size());
|
||||
assertTrue(h.signatureValid);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void badUnknownAndMissingAcknowledgementsRetainStableIdsAcrossRestart() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
h.primaryStatus.set(503);
|
||||
h.request("GET");
|
||||
String id = new SubscriptionEventJournal(h.eventDir).oldest(1).getFirst().eventId();
|
||||
h.ackMode.set(1); // 未知ID
|
||||
h.gateway.uploadOnce();
|
||||
assertEquals(1, h.pending());
|
||||
h.ackMode.set(2); // 非法JSON
|
||||
assertThrows(Exception.class, h.gateway::uploadOnce);
|
||||
assertEquals(1, h.pending());
|
||||
h.ackMode.set(3); // 没有ack字段
|
||||
assertThrows(Exception.class, h.gateway::uploadOnce);
|
||||
assertEquals(1, h.pending());
|
||||
h.gateway.close();
|
||||
try (SubscriptionGateway restarted = new SubscriptionGateway(h.store, h.origin, SECRET, h.eventDir,
|
||||
Duration.ofMillis(500), 100, false)) {
|
||||
assertEquals(1, ((Number) restarted.status().get("pendingCount")).intValue());
|
||||
h.ackMode.set(0);
|
||||
restarted.uploadOnce();
|
||||
assertEquals(id, h.uploaded.get().path("events").get(0).path("eventId").asText());
|
||||
assertEquals(0, ((Number) restarted.status().get("pendingCount")).intValue());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void headAndNotFoundDoNotProduceSuccessfulRecordsOrFallback() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
h.primaryStatus.set(404);
|
||||
HttpResponse<String> missing = h.request("GET");
|
||||
assertEquals(404, missing.statusCode());
|
||||
assertEquals("not found", missing.body());
|
||||
assertEquals(0, h.pending());
|
||||
h.primaryStatus.set(503);
|
||||
assertEquals(200, h.request("HEAD").statusCode());
|
||||
assertEquals(0, h.pending());
|
||||
assertEquals(1, h.primaryGets.get(), "HEAD 不得被记成GET");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void primaryHeadAndRangeHeadersSurviveGateway() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
HttpResponse<String> head = h.request("HEAD");
|
||||
assertEquals("13", head.headers().firstValue("Content-Length").orElseThrow());
|
||||
assertEquals("", head.body());
|
||||
assertEquals(0, h.pending());
|
||||
assertNotNull(h.lastEvent.get(), "HEAD 必须附签名,主站才不会按普通请求记录");
|
||||
assertTrue(h.signatureValid);
|
||||
HttpResponse<String> range = h.request("GET", "Range", "bytes=0-4");
|
||||
assertEquals(206, range.statusCode());
|
||||
assertEquals("fresh", range.body());
|
||||
assertEquals("bytes 0-4/13", range.headers().firstValue("Content-Range").orElseThrow());
|
||||
assertEquals("bytes", range.headers().firstValue("Accept-Ranges").orElseThrow());
|
||||
assertEquals("bytes=0-4", h.lastRange.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void primaryRecordedBeforeTimeoutReplaysSameEventId() throws Exception {
|
||||
try (Harness h = new Harness(root, 200, 100)) {
|
||||
h.primaryDelay.set(650);
|
||||
long start = System.nanoTime();
|
||||
HttpResponse<String> response = h.request("GET");
|
||||
assertEquals(LOCAL, response.body());
|
||||
assertTrue(Duration.ofNanos(System.nanoTime() - start).toMillis() < 1500);
|
||||
String id = h.lastEvent.get().path("eventId").asText();
|
||||
assertEquals(id, new SubscriptionEventJournal(h.eventDir).oldest(1).getFirst().eventId());
|
||||
h.gateway.uploadOnce();
|
||||
assertEquals(id, h.uploaded.get().path("events").get(0).path("eventId").asText());
|
||||
assertEquals(0, h.pending());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void circuitSkipsFailedOriginThenRecoversToPrimary() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 150)) {
|
||||
h.primaryStatus.set(503);
|
||||
assertEquals(LOCAL, h.request("GET").body());
|
||||
assertEquals(LOCAL, h.request("GET").body());
|
||||
assertEquals(1, h.primaryGets.get());
|
||||
assertEquals("open", h.gateway.status().get("circuit"));
|
||||
h.primaryStatus.set(200);
|
||||
Thread.sleep(180);
|
||||
assertEquals("fresh primary", h.request("GET").body());
|
||||
assertEquals("closed", h.gateway.status().get("circuit"));
|
||||
assertEquals("fresh primary", h.request("GET").body());
|
||||
assertEquals(3, h.primaryGets.get());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void halfOpenAllowsOnlyOneProbeAndBackgroundUploadDoesNotBlockRequests() throws Exception {
|
||||
try (Harness h = new Harness(root, 1000, 100)) {
|
||||
h.primaryStatus.set(503);
|
||||
h.request("GET");
|
||||
Thread.sleep(130);
|
||||
h.primaryStatus.set(200);
|
||||
h.probeGate = new CountDownLatch(1);
|
||||
CompletableFuture<HttpResponse<String>> probe = CompletableFuture.supplyAsync(() -> {
|
||||
try { return h.request("GET"); } catch (Exception e) { throw new RuntimeException(e); }
|
||||
});
|
||||
assertTrue(h.probeEntered.await(1, TimeUnit.SECONDS));
|
||||
assertEquals(LOCAL, h.request("GET").body());
|
||||
assertEquals(2, h.primaryGets.get());
|
||||
h.probeGate.countDown();
|
||||
assertEquals("fresh primary", probe.get(2, TimeUnit.SECONDS).body());
|
||||
|
||||
h.uploadDelay.set(700);
|
||||
CompletableFuture<Void> upload = CompletableFuture.runAsync(() -> {
|
||||
try { h.gateway.uploadOnce(); } catch (Exception e) { throw new RuntimeException(e); }
|
||||
});
|
||||
assertTrue(h.uploadEntered.await(1, TimeUnit.SECONDS));
|
||||
long start = System.nanoTime();
|
||||
assertEquals("fresh primary", h.request("GET").body());
|
||||
assertTrue(Duration.ofNanos(System.nanoTime() - start).toMillis() < 500,
|
||||
"上传不得持有请求路径使用的熔断器锁");
|
||||
upload.get(2, TimeUnit.SECONDS);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void diskFailureReturns503InsteadOfUnrecordedSuccess() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
Files.delete(h.eventDir);
|
||||
Files.writeString(h.eventDir, "not a directory");
|
||||
h.primaryStatus.set(503);
|
||||
assertEquals(503, h.request("GET").statusCode());
|
||||
assertEquals(0, h.pending());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void onlyLoopbackCanSupplyRealIpAndXffIsAlwaysIgnored() {
|
||||
assertEquals("203.0.113.12", SubscriptionGateway.clientIp(model("GET", KEY, "203.0.113.12",
|
||||
Map.of("X-Real-IP", "198.51.100.8", "X-Forwarded-For", "198.51.100.7"))));
|
||||
assertEquals("198.51.100.8", SubscriptionGateway.clientIp(model("GET", KEY, "127.0.0.1",
|
||||
Map.of("X-Real-IP", "198.51.100.8", "X-Forwarded-For", "198.51.100.7"))));
|
||||
assertEquals("127.0.0.1", SubscriptionGateway.clientIp(model("GET", KEY, "127.0.0.1",
|
||||
Map.of("X-Real-IP", "attacker.example", "X-Forwarded-For", "198.51.100.7"))));
|
||||
}
|
||||
|
||||
@Test
|
||||
void encodedKeyNeverChangesOriginAndInvalidPathsNeverContactPrimary() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
String key = "safe+key?next=%23#fragment";
|
||||
assertEquals(200, h.gateway.handle(model("GET", key, "203.0.113.1", Map.of())).status());
|
||||
assertEquals("/sub/v2/" + key, h.lastPath.get());
|
||||
for (String path : List.of("/sub/v2/abc", "/sub/v2/abcdef/", "/sub/v2/abc/def", "/other/abcdef")) {
|
||||
assertEquals(404, h.gateway.handle(new HttpRequestModel("GET", path, path, Map.of(), null)).status());
|
||||
}
|
||||
assertEquals(1, h.primaryGets.get());
|
||||
assertThrows(IllegalArgumentException.class, () -> new SubscriptionGateway(h.store,
|
||||
h.origin + "/arbitrary/path", SECRET, root.resolve("bad")));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void uploadBatchRemainsUnderBodyLimitWithEscapedAgents() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 10000)) {
|
||||
h.primaryStatus.set(503);
|
||||
String agent = "\\".repeat(2048);
|
||||
for (int i = 0; i < 100; i++) {
|
||||
assertEquals(200, h.gateway.handle(model("GET", KEY, "203.0.113.9", Map.of("User-Agent", agent))).status());
|
||||
}
|
||||
assertEquals(100, h.pending());
|
||||
h.gateway.uploadOnce();
|
||||
assertTrue(h.uploadBodyBytes.get() <= 256 * 1024);
|
||||
assertTrue(h.pending() > 0, "100个大UA必须拆批");
|
||||
assertTrue(h.pending() < 100);
|
||||
for (var pending : new SubscriptionEventJournal(h.eventDir).oldest(100)) {
|
||||
assertTrue(Base64.getUrlEncoder().withoutPadding().encodeToString(pending.json()).length() <= 4096);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void userAgentNormalizationKeepsHttpHeadersValidAndWithinBudget() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
assertEquals(200, h.gateway.handle(model("GET", KEY, "203.0.113.1",
|
||||
Map.of("User-Agent", "猫😀\r\nAgent\t"))).status());
|
||||
assertEquals("??Agent", h.lastEvent.get().path("userAgent").asText());
|
||||
assertEquals(0, h.pending(), "规范后应正常访问主站而非仅fallback");
|
||||
assertEquals(200, h.gateway.handle(model("GET", KEY, "203.0.113.1",
|
||||
Map.of("User-Agent", "é".repeat(2048)))).status());
|
||||
assertTrue(h.lastEvent.get().path("userAgent").asText().getBytes(StandardCharsets.UTF_8).length <= 2048);
|
||||
assertTrue(h.eventHeaderLength.get() <= 4096);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void primaryRedirectIsNotFollowedAndFallsBack() throws Exception {
|
||||
try (Harness h = new Harness(root, 500, 100)) {
|
||||
h.primaryStatus.set(302);
|
||||
assertEquals(LOCAL, h.request("GET").body());
|
||||
assertEquals(1, h.primaryGets.get());
|
||||
assertEquals(1, h.pending());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void oversizedOrIncompletePrimaryBodiesFallBack() throws Exception {
|
||||
try (Harness h = new Harness(root, 1000, 1)) {
|
||||
h.primaryStatus.set(601); // 声明较长正文,只写一个字节
|
||||
assertEquals(LOCAL, h.request("GET").body());
|
||||
Thread.sleep(10);
|
||||
h.primaryStatus.set(602); // 正文超过16MiB
|
||||
assertEquals(LOCAL, h.request("GET").body());
|
||||
assertEquals(2, h.pending());
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpRequestModel model(String method, String key, String peer, Map<String, String> headers) {
|
||||
String path = "/sub/v2/" + key;
|
||||
return new HttpRequestModel(method, path, path, Map.of(), null, peer, headers);
|
||||
}
|
||||
|
||||
private static final class Harness implements AutoCloseable {
|
||||
final HttpServer primary;
|
||||
final java.util.concurrent.ExecutorService executor = Executors.newCachedThreadPool();
|
||||
final SubscriptionSnapshotStore store;
|
||||
final SubscriptionGateway gateway;
|
||||
final BackupSubServer backup;
|
||||
final Path eventDir;
|
||||
final String origin;
|
||||
final AtomicInteger primaryStatus = new AtomicInteger(200);
|
||||
final AtomicInteger primaryGets = new AtomicInteger();
|
||||
final AtomicInteger primaryDelay = new AtomicInteger();
|
||||
final AtomicInteger uploadDelay = new AtomicInteger();
|
||||
final AtomicInteger ackMode = new AtomicInteger();
|
||||
final AtomicInteger uploadBodyBytes = new AtomicInteger();
|
||||
final AtomicInteger eventHeaderLength = new AtomicInteger();
|
||||
final AtomicReference<JsonNode> lastEvent = new AtomicReference<>();
|
||||
final AtomicReference<JsonNode> uploaded = new AtomicReference<>();
|
||||
final AtomicReference<byte[]> eventBytes = new AtomicReference<>();
|
||||
final AtomicReference<String> authorization = new AtomicReference<>();
|
||||
final AtomicReference<String> lastRange = new AtomicReference<>();
|
||||
final AtomicReference<String> lastPath = new AtomicReference<>();
|
||||
final CountDownLatch probeEntered = new CountDownLatch(1);
|
||||
final CountDownLatch uploadEntered = new CountDownLatch(1);
|
||||
volatile CountDownLatch probeGate;
|
||||
volatile boolean signatureValid;
|
||||
|
||||
Harness(Path root, int timeoutMs, long cooldownMs) throws Exception {
|
||||
eventDir = root.resolve("events");
|
||||
store = readyStore(root.resolve("snapshot"));
|
||||
primary = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 16);
|
||||
primary.setExecutor(executor);
|
||||
primary.createContext("/", this::serve);
|
||||
primary.start();
|
||||
origin = "http://127.0.0.1:" + primary.getAddress().getPort();
|
||||
gateway = new SubscriptionGateway(store, origin, SECRET, eventDir,
|
||||
Duration.ofMillis(timeoutMs), cooldownMs, false);
|
||||
backup = new BackupSubServer(store, 0, 4, gateway);
|
||||
backup.startServer();
|
||||
}
|
||||
|
||||
void serve(HttpExchange exchange) {
|
||||
try (exchange) {
|
||||
if (exchange.getRequestURI().getPath().equals("/internal/subscription-events")) {
|
||||
byte[] bytes = exchange.getRequestBody().readAllBytes();
|
||||
uploadBodyBytes.set(bytes.length);
|
||||
signatureValid = signatureValid(exchange, bytes);
|
||||
JsonNode value = CustomUtil.objectMapper.readTree(bytes);
|
||||
uploaded.set(value);
|
||||
uploadEntered.countDown();
|
||||
Thread.sleep(uploadDelay.get());
|
||||
String body;
|
||||
if (ackMode.get() == 1) body = "{\"acknowledged\":[\"unknown-id\"]}";
|
||||
else if (ackMode.get() == 2) body = "invalid-json";
|
||||
else if (ackMode.get() == 3) body = "{}";
|
||||
else {
|
||||
List<String> ids = new ArrayList<>();
|
||||
value.path("events").forEach(event -> ids.add(event.path("eventId").asText()));
|
||||
body = CustomUtil.objectMapper.writeValueAsString(Map.of("acknowledged", ids));
|
||||
}
|
||||
respond(exchange, 200, body);
|
||||
return;
|
||||
}
|
||||
lastPath.set(exchange.getRequestURI().getPath());
|
||||
boolean head = "HEAD".equals(exchange.getRequestMethod());
|
||||
if (!head) primaryGets.incrementAndGet();
|
||||
authorization.set(exchange.getRequestHeaders().getFirst("Authorization"));
|
||||
lastRange.set(exchange.getRequestHeaders().getFirst("Range"));
|
||||
String encoded = exchange.getRequestHeaders().getFirst("X-Subscription-Event");
|
||||
if (encoded != null) {
|
||||
eventHeaderLength.set(encoded.length());
|
||||
byte[] bytes = Base64.getUrlDecoder().decode(encoded);
|
||||
eventBytes.set(bytes);
|
||||
lastEvent.set(CustomUtil.objectMapper.readTree(bytes));
|
||||
signatureValid = signatureValid(exchange, bytes);
|
||||
}
|
||||
if (probeGate != null) { probeEntered.countDown(); probeGate.await(2, TimeUnit.SECONDS); }
|
||||
Thread.sleep(primaryDelay.get());
|
||||
int status = primaryStatus.get();
|
||||
if (status == 601) {
|
||||
exchange.sendResponseHeaders(200, 100);
|
||||
exchange.getResponseBody().write('a');
|
||||
return;
|
||||
}
|
||||
if (status == 602) {
|
||||
exchange.sendResponseHeaders(200, 16 * 1024 * 1024 + 1);
|
||||
byte[] chunk = new byte[64 * 1024];
|
||||
for (int i = 0; i < 256; i++) exchange.getResponseBody().write(chunk);
|
||||
exchange.getResponseBody().write('z');
|
||||
return;
|
||||
}
|
||||
if (status == 302) exchange.getResponseHeaders().set("Location", origin + "/forbidden-redirect");
|
||||
exchange.getResponseHeaders().set("Content-Type", "text/plain");
|
||||
exchange.getResponseHeaders().set("Accept-Ranges", "bytes");
|
||||
if (head) {
|
||||
exchange.getResponseHeaders().set("Content-Length", "13");
|
||||
exchange.sendResponseHeaders(status, -1);
|
||||
} else if (status == 200 && lastRange.get() != null) {
|
||||
exchange.getResponseHeaders().set("Content-Range", "bytes 0-4/13");
|
||||
respond(exchange, 206, "fresh");
|
||||
} else respond(exchange, status, status == 404 ? "not found" : "fresh primary");
|
||||
} catch (Exception ignored) { /* 断连/短正文/超时正是测试用例 */ }
|
||||
}
|
||||
|
||||
HttpResponse<String> request(String method, String... headers) throws Exception {
|
||||
var builder = HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + backup.server().port() + "/sub/v2/" + KEY))
|
||||
.timeout(Duration.ofSeconds(5)).method(method, HttpRequest.BodyPublishers.noBody());
|
||||
if (headers.length > 0) builder.headers(headers);
|
||||
return HTTP.send(builder.build(), HttpResponse.BodyHandlers.ofString());
|
||||
}
|
||||
int pending() { return ((Number) gateway.status().get("pendingCount")).intValue(); }
|
||||
@Override public void close() { backup.close(); primary.stop(0); executor.shutdownNow(); }
|
||||
}
|
||||
|
||||
private static void respond(HttpExchange exchange, int code, String text) throws Exception {
|
||||
byte[] bytes = text.getBytes(StandardCharsets.UTF_8);
|
||||
exchange.sendResponseHeaders(code, bytes.length);
|
||||
exchange.getResponseBody().write(bytes);
|
||||
}
|
||||
private static boolean signatureValid(HttpExchange exchange, byte[] bytes) throws Exception {
|
||||
String timestamp = exchange.getRequestHeaders().getFirst("X-Subscription-Timestamp");
|
||||
String expected = signature("v1\n" + timestamp + "\n" + hash(bytes));
|
||||
return expected.equals(exchange.getRequestHeaders().getFirst("X-Subscription-Signature"));
|
||||
}
|
||||
private static String signature(String input) throws Exception {
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
return HexFormat.of().formatHex(mac.doFinal(input.getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
private static String hash(byte[] bytes) throws Exception {
|
||||
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes));
|
||||
}
|
||||
private static SubscriptionSnapshotStore readyStore(Path path) throws Exception {
|
||||
var account = new SubscriptionAccountSnapshot();
|
||||
account.setAccountId(1);
|
||||
account.setEnabled(true);
|
||||
byte[] content = LOCAL.getBytes(StandardCharsets.UTF_8);
|
||||
account.setV2ContentBase64(Base64.getEncoder().encodeToString(content));
|
||||
account.setV2Sha256(hash(content));
|
||||
account.setClashContentBase64(Base64.getEncoder().encodeToString(content));
|
||||
account.setClashSha256(hash(content));
|
||||
var binding = new SubscriptionBindingSnapshot();
|
||||
binding.setAccountId(1);
|
||||
binding.setPublicKeySha256(hash(KEY.getBytes(StandardCharsets.UTF_8)));
|
||||
var payload = new SubscriptionSnapshotPayload();
|
||||
payload.setSchemaVersion(1);
|
||||
payload.setAccounts(List.of(account));
|
||||
payload.setBindings(List.of(binding));
|
||||
byte[] json = CustomUtil.objectMapper.writeValueAsBytes(payload);
|
||||
ByteArrayOutputStream out = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream gzip = new GZIPOutputStream(out)) { gzip.write(json); }
|
||||
byte[] zipped = out.toByteArray();
|
||||
var message = new SubscriptionSnapshotMessage();
|
||||
message.setSchemaVersion(1);
|
||||
message.setRevision(hash(json));
|
||||
message.setGeneratedAt(System.currentTimeMillis());
|
||||
message.setPayloadBase64(Base64.getEncoder().encodeToString(zipped));
|
||||
message.setPayloadSha256(hash(zipped));
|
||||
message.setSignature(signature("1\n" + message.getRevision() + "\n" + message.getGeneratedAt() + "\n" + message.getPayloadSha256()));
|
||||
var store = new SubscriptionSnapshotStore(path, SECRET, 3600, 1024 * 1024);
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
return store;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
package lion.Service;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import lion.CustomUtil;
|
||||
import lion.Message.Main.SubscriptionAccountSnapshot;
|
||||
import lion.Message.Main.SubscriptionBindingSnapshot;
|
||||
import lion.Message.Main.SubscriptionSnapshotMessage;
|
||||
import lion.Message.Main.SubscriptionSnapshotPayload;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.Base64;
|
||||
import java.util.HexFormat;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
class SubscriptionSnapshotStoreTest {
|
||||
private static final String SECRET = "snapshot-test-secret";
|
||||
private final ObjectMapper mapper = CustomUtil.objectMapper;
|
||||
|
||||
@Test
|
||||
void appliesSnapshotAndServesByHashedPublicKey(@TempDir Path directory) throws Exception {
|
||||
SubscriptionSnapshotStore store = new SubscriptionSnapshotStore(directory, SECRET, 3600, 1024 * 1024);
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content", System.currentTimeMillis());
|
||||
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
SubscriptionSnapshotStore.Lookup v2 = store.lookup("v2", "public-key-1");
|
||||
SubscriptionSnapshotStore.Lookup clash = store.lookup("cat", "public-key-1");
|
||||
assertArrayEquals("v2-content".getBytes(StandardCharsets.UTF_8), v2.content());
|
||||
assertArrayEquals("clash-content".getBytes(StandardCharsets.UTF_8), clash.content());
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_OLD, store.apply(message).code());
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsTamperedPayloadAndKeepsPreviousSnapshot(@TempDir Path directory) throws Exception {
|
||||
SubscriptionSnapshotStore store = new SubscriptionSnapshotStore(directory, SECRET, 3600, 1024 * 1024);
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content", System.currentTimeMillis());
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
message.setPayloadBase64(Base64.getEncoder().encodeToString("tampered".getBytes(StandardCharsets.UTF_8)));
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_INVALID, store.apply(message).code());
|
||||
assertArrayEquals("v2-content".getBytes(StandardCharsets.UTF_8), store.lookup("v2", "public-key-1").content());
|
||||
}
|
||||
|
||||
@Test
|
||||
void loadsLastGoodSnapshotAfterRestart(@TempDir Path directory) throws Exception {
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content", System.currentTimeMillis());
|
||||
SubscriptionSnapshotStore first = new SubscriptionSnapshotStore(directory, SECRET, 3600, 1024 * 1024);
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, first.apply(message).code());
|
||||
SubscriptionSnapshotStore second = new SubscriptionSnapshotStore(directory, SECRET, 3600, 1024 * 1024);
|
||||
second.load();
|
||||
assertArrayEquals("clash-content".getBytes(StandardCharsets.UTF_8), second.lookup("cat", "public-key-1").content());
|
||||
}
|
||||
|
||||
/**
|
||||
* 回归:节点曾按「内容里的 generatedAt」判过期,而 revision 是内容寻址的——
|
||||
* 内容长期不变时主站每分钟重发同一 revision 会被判为 APPLY_OLD 直接丢弃,
|
||||
* 新鲜度永远停在首次接收那天,于是主站在线且持续同步,备机仍在第 7 天开始 503。
|
||||
* 现在过期只取决于「最近一次成功接收主站快照的时刻」。
|
||||
*/
|
||||
@Test
|
||||
void repeatedSameRevisionRefreshesFreshnessAndNeverExpires(@TempDir Path directory) throws Exception {
|
||||
// 有效期 1 秒,便于在用例内观察「不续期会过期、续期后恢复」。
|
||||
SubscriptionSnapshotStore store = new SubscriptionSnapshotStore(directory, SECRET, 1, 1024 * 1024);
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content", System.currentTimeMillis());
|
||||
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
assertEquals("ready", store.status().state());
|
||||
|
||||
// 超过有效期且期间没有任何推送 → 过期(证明判定确实生效,不是恒 ready)
|
||||
Thread.sleep(1_200);
|
||||
assertEquals("expired", store.status().state(), "超过有效期且无推送应判过期");
|
||||
|
||||
// 同 revision 重发:仍是 APPLY_OLD(内容幂等),但必须刷新新鲜度
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_OLD, store.apply(message).code());
|
||||
assertEquals("ready", store.status().state(), "重新收到同 revision 后应恢复可用");
|
||||
assertNotNull(store.lookup("v2", "public-key-1"), "持续同步期间必须能取到订阅");
|
||||
}
|
||||
|
||||
/** 超过有效期后仍未收到任何快照,才判过期并停止分发。 */
|
||||
@Test
|
||||
void expiresOnlyAfterNoSnapshotArrivesWithinMaxAge(@TempDir Path directory) throws Exception {
|
||||
SubscriptionSnapshotStore store = new SubscriptionSnapshotStore(directory, SECRET, 3600, 1024 * 1024);
|
||||
// 内容生成于 2 小时前,且重启后再没收到主站推送 → 超期。
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content",
|
||||
System.currentTimeMillis() - 7_200_000L);
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
|
||||
SubscriptionSnapshotStore reloaded = new SubscriptionSnapshotStore(directory, SECRET, 3600, 1024 * 1024);
|
||||
reloaded.load();
|
||||
assertEquals("expired", reloaded.status().state(), "重启后未再收到快照,超期应判过期");
|
||||
assertNull(reloaded.lookup("v2", "public-key-1"), "过期快照不得继续分发");
|
||||
}
|
||||
|
||||
/**
|
||||
* 主站的常规存活信号(例如每 30 分钟的可用性检查)必须刷新新鲜度。
|
||||
*
|
||||
* <p>这是「过期 ⇔ 主站失联」的关键:快照是内容寻址的,内容长期不变时主站没有理由
|
||||
* 重发整份快照;若新鲜度只由「收到快照」驱动,备机会在内容不动的第 7 天误判过期。
|
||||
*/
|
||||
@Test
|
||||
void primaryContactWithoutNewSnapshotKeepsSubscriptionFresh(@TempDir Path directory) throws Exception {
|
||||
SubscriptionSnapshotStore store = new SubscriptionSnapshotStore(directory, SECRET, 1, 1024 * 1024);
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content", System.currentTimeMillis());
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
|
||||
// 超过有效期(1 秒),期间只有主站存活信号、没有新快照
|
||||
Thread.sleep(1_200);
|
||||
assertEquals("expired", store.status().state(), "有效期已过且无任何主站信号,应先判过期");
|
||||
|
||||
store.markPrimaryContact();
|
||||
assertEquals("ready", store.status().state(), "收到主站存活信号后应恢复可用");
|
||||
assertNotNull(store.lookup("v2", "public-key-1"), "主站在线期间必须能取到订阅");
|
||||
// 内容仍是最初那份,说明续期靠的是存活信号,而不是内容变化
|
||||
assertArrayEquals("v2-content".getBytes(StandardCharsets.UTF_8), store.lookup("v2", "public-key-1").content());
|
||||
}
|
||||
|
||||
/** 主站失联(长时间没有任何信号)后仍必须判过期,不能因加入存活信号而永不失效。 */
|
||||
@Test
|
||||
void staysExpiredWhenPrimaryContactStops(@TempDir Path directory) throws Exception {
|
||||
SubscriptionSnapshotStore store = new SubscriptionSnapshotStore(directory, SECRET, 1, 1024 * 1024);
|
||||
SubscriptionSnapshotMessage message = message("public-key-1", "v2-content", "clash-content", System.currentTimeMillis());
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_SUCCESS, store.apply(message).code());
|
||||
store.markPrimaryContact();
|
||||
assertEquals("ready", store.status().state());
|
||||
|
||||
Thread.sleep(1_200); // 之后主站再无任何消息
|
||||
assertEquals("expired", store.status().state(), "主站失联超过有效期必须判过期");
|
||||
assertNull(store.lookup("v2", "public-key-1"), "失联后不得继续分发");
|
||||
}
|
||||
|
||||
private SubscriptionSnapshotMessage message(String publicKey, String v2, String clash, long generatedAt) throws Exception {
|
||||
byte[] v2Bytes = v2.getBytes(StandardCharsets.UTF_8);
|
||||
byte[] clashBytes = clash.getBytes(StandardCharsets.UTF_8);
|
||||
SubscriptionAccountSnapshot account = new SubscriptionAccountSnapshot();
|
||||
account.setAccountId(1);
|
||||
account.setEnabled(true);
|
||||
account.setV2ContentBase64(Base64.getEncoder().encodeToString(v2Bytes));
|
||||
account.setV2Sha256(sha256(v2Bytes));
|
||||
account.setClashContentBase64(Base64.getEncoder().encodeToString(clashBytes));
|
||||
account.setClashSha256(sha256(clashBytes));
|
||||
SubscriptionBindingSnapshot binding = new SubscriptionBindingSnapshot();
|
||||
binding.setPublicKeySha256(sha256(publicKey.getBytes(StandardCharsets.UTF_8)));
|
||||
binding.setAccountId(1);
|
||||
SubscriptionSnapshotPayload payload = new SubscriptionSnapshotPayload();
|
||||
payload.setSchemaVersion(1);
|
||||
payload.setAccounts(java.util.List.of(account));
|
||||
payload.setBindings(java.util.List.of(binding));
|
||||
byte[] json = mapper.writeValueAsBytes(payload);
|
||||
byte[] compressed = gzip(json);
|
||||
SubscriptionSnapshotMessage message = new SubscriptionSnapshotMessage();
|
||||
message.setSchemaVersion(1);
|
||||
message.setRevision(sha256(json));
|
||||
message.setGeneratedAt(generatedAt);
|
||||
message.setPayloadBase64(Base64.getEncoder().encodeToString(compressed));
|
||||
message.setPayloadSha256(sha256(compressed));
|
||||
String input = "1\n" + message.getRevision() + "\n" + generatedAt + "\n" + message.getPayloadSha256();
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
message.setSignature(hex(mac.doFinal(input.getBytes(StandardCharsets.UTF_8))));
|
||||
return message;
|
||||
}
|
||||
|
||||
private static byte[] gzip(byte[] bytes) throws Exception {
|
||||
ByteArrayOutputStream output = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream gzip = new GZIPOutputStream(output)) { gzip.write(bytes); }
|
||||
return output.toByteArray();
|
||||
}
|
||||
|
||||
private static String sha256(byte[] bytes) throws Exception {
|
||||
return hex(MessageDigest.getInstance("SHA-256").digest(bytes));
|
||||
}
|
||||
|
||||
private static String hex(byte[] bytes) { return HexFormat.of().formatHex(bytes); }
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package lion.Service;
|
||||
|
||||
import lion.Domain.GalleryTask;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 任务日志的持久化与写失败回滚。
|
||||
*
|
||||
* <p>这些用例锁定的是一条不变式:内存状态永不领先磁盘。任何一次落盘失败之后,内存里能看到的
|
||||
* 内容必须与磁盘上的最后一份一致,否则重启会丢命令回执,主站重试将重复执行副作用。
|
||||
*/
|
||||
class TaskJournalTest {
|
||||
|
||||
@Test
|
||||
void failedCommitLeavesMemoryAtLastDurableState(@TempDir Path root) throws Exception {
|
||||
TaskJournal journal = new TaskJournal(root);
|
||||
GalleryTask task = task(70001, GalleryTask.DOWNLOADING, 3);
|
||||
journal.bumpAndPutTask(task, "op-1");
|
||||
long durableRevision = journal.revision();
|
||||
|
||||
// 用一个一定失败的目录替换:把日志目录指向一个普通文件,任何写入都会抛 IOException。
|
||||
Path blocked = root.resolve("blocked");
|
||||
Files.writeString(blocked, "not a directory");
|
||||
TaskJournal broken = new TaskJournal(blocked);
|
||||
assertThrows(IOException.class,
|
||||
() -> broken.bumpAndPutTask(task(70002, GalleryTask.DOWNLOADING, 1), "op-2"));
|
||||
assertTrue(broken.tasks().isEmpty(), "写盘失败后内存不得留有未落盘的任务");
|
||||
assertEquals(0, broken.revision(), "失败不得推进 revision");
|
||||
|
||||
// 原日志不受影响,仍然是那份已成功落盘的内容。
|
||||
assertEquals(durableRevision, journal.revision());
|
||||
assertEquals(1, journal.tasks().size());
|
||||
}
|
||||
|
||||
/** 成功的写盘必须真的出现在磁盘上,重新加载后内容一致。 */
|
||||
@Test
|
||||
void reloadReturnsExactlyWhatWasCommitted(@TempDir Path root) throws Exception {
|
||||
TaskJournal journal = new TaskJournal(root);
|
||||
journal.bumpAndPutTask(task(70010, GalleryTask.COMPRESS_COMPLETE, 12), "op-a");
|
||||
journal.putReceipt(TaskJournal.CommandReceipt.of("op-a", "DOWNLOAD", 70010, "name", NodeErrorCode.SUCCESS));
|
||||
|
||||
TaskJournal reloaded = new TaskJournal(root);
|
||||
reloaded.load();
|
||||
|
||||
assertEquals(1, reloaded.tasks().size());
|
||||
assertEquals(70010, reloaded.tasks().values().iterator().next().gid);
|
||||
assertEquals(NodeErrorCode.SUCCESS, reloaded.receipt("op-a").result);
|
||||
assertEquals(journal.revision(), reloaded.revision());
|
||||
}
|
||||
|
||||
/** 高于本程序支持的 schemaVersion 必须拒绝启动,而不是按旧规则硬读。 */
|
||||
@Test
|
||||
void refusesNewerSchemaVersion(@TempDir Path root) throws Exception {
|
||||
Path directory = Files.createDirectories(root.resolve("journal"));
|
||||
Files.writeString(directory.resolve(TaskJournal.STATE_FILE),
|
||||
"{\"schemaVersion\": 99, \"revision\": 5, \"tasks\": {}, \"commandReceipts\": {}, \"pendingReports\": {}}");
|
||||
|
||||
IOException failure = assertThrows(IOException.class, () -> new TaskJournal(directory).load());
|
||||
assertTrue(failure.getMessage().contains("schemaVersion"), "错误信息应指出是版本问题");
|
||||
}
|
||||
|
||||
/** 损坏的日志文件不得被当作空日志吞掉。 */
|
||||
@Test
|
||||
void refusesCorruptJournal(@TempDir Path root) throws Exception {
|
||||
Path directory = Files.createDirectories(root.resolve("journal"));
|
||||
Files.writeString(directory.resolve(TaskJournal.STATE_FILE), "{not json");
|
||||
|
||||
assertThrows(IOException.class, () -> new TaskJournal(directory).load());
|
||||
}
|
||||
|
||||
/** 快照必须拷贝任务字段:之后任务继续变化不得改写已生成的报告内容。 */
|
||||
@Test
|
||||
void reportSnapshotCopiesTaskFields(@TempDir Path root) throws Exception {
|
||||
TaskJournal journal = new TaskJournal(root);
|
||||
TaskJournal.JournalTask stored = journal.bumpAndPutTask(task(70020, GalleryTask.DOWNLOADING, 1), "op-x");
|
||||
|
||||
TaskJournal.ReportSnapshot snapshot = TaskJournal.ReportSnapshot.of(
|
||||
"report-1", stored.revision, List.of(stored), Map.of(70020, "op-x"));
|
||||
|
||||
// 模拟扫描继续推进同一个任务对象。
|
||||
GalleryTask advanced = stored.toGalleryTask();
|
||||
advanced.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
advanced.setProceeding(44);
|
||||
|
||||
TaskJournal.JournalTask inSnapshot = snapshot.tasks.get(0);
|
||||
assertEquals(GalleryTask.DOWNLOADING, inSnapshot.status, "快照内容不得被后续变化改写");
|
||||
assertEquals(1, inSnapshot.proceeding);
|
||||
|
||||
// 而且对外转换出的协议对象也是独立副本。
|
||||
GalleryTask[] wire = snapshot.toGalleryTasks();
|
||||
wire[0].setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
assertEquals(GalleryTask.DOWNLOADING, snapshot.tasks.get(0).status);
|
||||
}
|
||||
|
||||
private static GalleryTask task(int gid, byte status, int proceeding) {
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(gid);
|
||||
task.setName("sample [" + gid + "]");
|
||||
task.setStatus(status);
|
||||
task.setProceeding(proceeding);
|
||||
return task;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,605 @@
|
||||
package lion;
|
||||
|
||||
import io.netty.channel.embedded.EmbeddedChannel;
|
||||
import lion.Config.Config;
|
||||
import lion.Domain.GalleryTask;
|
||||
import lion.ErrorCode.ErrorCode;
|
||||
import lion.Message.Main.AvailableCheckMessage;
|
||||
import lion.Message.Main.DeleteGalleryMessage;
|
||||
import lion.Message.Main.DownloadPostMessage;
|
||||
import lion.Message.Main.DownloadStatusMessage;
|
||||
import lion.Message.Main.IdentityMessage;
|
||||
import lion.Message.Main.ProtocolHelloMessage;
|
||||
import lion.Message.Main.ResponseMessage;
|
||||
import lion.Message.Main.SubscriptionSnapshotMessage;
|
||||
import lion.Message.Main.TaskCommandMessage;
|
||||
import lion.Message.Main.TaskReportAckMessage;
|
||||
import lion.Message.Main.TaskReportMessage;
|
||||
import lion.Service.NodeErrorCode;
|
||||
import lion.Service.SubscriptionSnapshotStore;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
/**
|
||||
* 节点消息处理与状态上报循环。
|
||||
*
|
||||
* <p>{@link storageNode.MyChannelInboundHandlerAdapter} 与 {@link storageNode#mainThread()}
|
||||
* 约 250 行,承载任务下发、删除、探活、订阅快照分发与队列重放,此前只有静态方法
|
||||
* {@code isPrimaryEvidence} 被间接覆盖。这里用 {@link EmbeddedChannel} 把入站消息直接
|
||||
* 喂给真实处理器,断言回包与队列副作用,同时锁定「无可用通道时不得丢弃任务」的语义。
|
||||
*/
|
||||
class StorageNodeMessageHandlingTest {
|
||||
|
||||
@TempDir
|
||||
static Path tempDir;
|
||||
|
||||
static storageNode node;
|
||||
|
||||
@BeforeAll
|
||||
static void startNode() throws Exception {
|
||||
Path storageRoot = Files.createDirectories(tempDir.resolve("gallery"));
|
||||
Path subscriptionRoot = Files.createDirectories(tempDir.resolve("sub"));
|
||||
|
||||
storageNode.storagePath = storageRoot + "/";
|
||||
SubscriptionSnapshotStore store =
|
||||
new SubscriptionSnapshotStore(subscriptionRoot, "test-secret", 3600, 1024 * 1024);
|
||||
// 离线构造:不绑定端口、不唤醒真实主站、不启动调度线程。
|
||||
node = new storageNode(store, tempDir.resolve("task-journal"), false);
|
||||
// 订阅同步默认关闭:快照消息必须被拒绝,也不触发异步落盘。
|
||||
Config.subscriptionSyncEnabled = false;
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stopNode() {
|
||||
// 统一释放:压缩调度、通知、重连与可靠执行器都不得在测试结束后残留线程。
|
||||
node.shutdown();
|
||||
}
|
||||
|
||||
@BeforeEach
|
||||
void resetState() throws Exception {
|
||||
node.queue.clear();
|
||||
node.tempQueue.clear();
|
||||
node.reportSends.clear();
|
||||
// 每个用例用独立的任务日志:共享一份日志会让上一个用例的任务与回执泄漏到下一个。
|
||||
node.reliableTaskService = new lion.Service.ReliableTaskService(
|
||||
Files.createDirectories(tempDir.resolve("journal-" + java.util.UUID.randomUUID())),
|
||||
node.queue, node.downloadCheckService, Path.of(storageNode.storagePath));
|
||||
node.reportRetryIntervalMillis = 5_000L;
|
||||
}
|
||||
|
||||
/**
|
||||
* v2 能力协商:收到声明 version2 的 hello 后,节点回一条同样的 hello,并切换到可靠上报。
|
||||
* 旧主站因此只会在自己先发起协商时才看到 type10。
|
||||
*/
|
||||
@Test
|
||||
void protocolHelloNegotiatesAndRepliesHello() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
|
||||
ProtocolHelloMessage reply = channel.readOutbound();
|
||||
assertNotNull(reply, "确认对端支持 v2 后必须回 hello");
|
||||
assertEquals(ProtocolHelloMessage.VERSION, reply.getProtocolVersion());
|
||||
assertTrue(reply.getFeatures().contains(ProtocolHelloMessage.RELIABLE_TASK_V2));
|
||||
assertTrue(node.reliableTaskService.reporter().reliableV2(), "应切换到可靠任务协议");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 未声明 v2 的 hello(例如更老或未知版本)不得让节点升级,也不得回包或断开连接。
|
||||
* 这条用例覆盖「旧对端发来未知 hello 时不能退出」的兼容要求。
|
||||
*/
|
||||
@Test
|
||||
void legacyHelloDoesNotUpgradeOrBreakConnection() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
ProtocolHelloMessage legacy = new ProtocolHelloMessage();
|
||||
legacy.setProtocolVersion(1);
|
||||
legacy.setFeatures(java.util.List.of("something-else"));
|
||||
|
||||
channel.writeInbound(legacy);
|
||||
|
||||
assertNull(channel.readOutbound(), "未达成 v2 时不应回 hello");
|
||||
assertFalse(node.reliableTaskService.reporter().reliableV2());
|
||||
assertTrue(channel.isActive(), "未知 hello 不得导致连接异常");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 任务命令:先回执,再由可靠服务持久化任务与回执。 */
|
||||
@Test
|
||||
void taskCommandIsAppliedAndReceipted() throws Exception {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
drainOutbound(channel);
|
||||
|
||||
TaskCommandMessage command = new TaskCommandMessage();
|
||||
command.messageId = 77;
|
||||
command.setOperationId("op-node-1");
|
||||
command.setAction(TaskCommandMessage.ACTION_DOWNLOAD);
|
||||
command.setGid(91001);
|
||||
command.setName("sample [91001]");
|
||||
|
||||
channel.writeInbound(command);
|
||||
awaitReliableExecutor();
|
||||
|
||||
ResponseMessage response = readResponse(channel, 77);
|
||||
assertNotNull(response, "命令必须按原 messageId 回执,否则主站无法配对");
|
||||
assertEquals(NodeErrorCode.SUCCESS, response.getResult());
|
||||
assertNotNull(node.reliableTaskService.journal().receipt("op-node-1"),
|
||||
"成功命令必须落盘回执以便重试幂等");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 报告 ACK:压缩完成的任务在收到成功 ACK 后才从队列移除。 */
|
||||
@Test
|
||||
void reportAckClearsCompletedTaskOnlyAfterSuccess() throws Exception {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
drainOutbound(channel);
|
||||
|
||||
GalleryTask done = task(91002);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(91002, done);
|
||||
// 模拟扫描已把最新进度落盘(生产里由 flushReport 先做这一步)。
|
||||
node.reliableTaskService.refreshJournalFromQueue();
|
||||
|
||||
node.flushReport();
|
||||
TaskReportMessage report = readReport(channel);
|
||||
assertNotNull(report, "协商 v2 后应发送可靠报告");
|
||||
assertFalse(node.queue.isEmpty(), "发出报告本身不得移除任务");
|
||||
|
||||
TaskReportAckMessage ack = new TaskReportAckMessage();
|
||||
ack.setReportId(report.getReportId());
|
||||
ack.setResult(TaskReportAckMessage.RESULT_SUCCESS);
|
||||
channel.writeInbound(ack);
|
||||
awaitReliableExecutor();
|
||||
|
||||
assertTrue(node.queue.isEmpty(), "成功后应清理压缩完成的任务");
|
||||
assertFalse(node.reliableTaskService.journal().tasks().containsKey(91002));
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 同一连接上 ACK 丢失时必须能重发:TCP 未断、通道引用未变,也要在重发间隔后再次发送,
|
||||
* 且 reportId 与 revision 保持稳定。
|
||||
*/
|
||||
@Test
|
||||
void pendingReportIsResentOnSameChannelUntilAcked() throws Exception {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
drainOutbound(channel);
|
||||
|
||||
GalleryTask done = task(91003);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(91003, done);
|
||||
node.reliableTaskService.refreshJournalFromQueue();
|
||||
|
||||
node.flushReport();
|
||||
TaskReportMessage first = readReport(channel);
|
||||
assertNotNull(first);
|
||||
|
||||
// 同一通道、ACK 未到:间隔内不重发。
|
||||
node.flushReport();
|
||||
assertNull(readReport(channel), "重发间隔内不应重复发送");
|
||||
|
||||
// 超过间隔后必须重发,且身份不变。
|
||||
node.reportRetryIntervalMillis = 0;
|
||||
node.flushReport();
|
||||
TaskReportMessage resent = readReport(channel);
|
||||
assertNotNull(resent, "超出重发间隔后必须重发,否则 ACK 丢失会永久卡住");
|
||||
assertEquals(first.getReportId(), resent.getReportId(), "重发必须复用同一 reportId");
|
||||
assertEquals(first.getRevision(), resent.getRevision(), "重发必须复用同一 revision");
|
||||
|
||||
// ACK 到达后停止重发。
|
||||
TaskReportAckMessage ack = new TaskReportAckMessage();
|
||||
ack.setReportId(first.getReportId());
|
||||
ack.setResult(TaskReportAckMessage.RESULT_SUCCESS);
|
||||
channel.writeInbound(ack);
|
||||
awaitReliableExecutor();
|
||||
node.flushReport();
|
||||
assertNull(readReport(channel), "确认后不得再重发");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 写入通道失败时必须清除发送登记,使下一轮立即重试。 */
|
||||
@Test
|
||||
void failedChannelWriteAllowsImmediateRetry() throws Exception {
|
||||
FailingChannel channel = new FailingChannel(node.new MyChannelInboundHandlerAdapter(node.tempQueue));
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
drainOutbound(channel);
|
||||
channel.failWrites = true;
|
||||
|
||||
GalleryTask done = task(91004);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(91004, done);
|
||||
node.reliableTaskService.refreshJournalFromQueue();
|
||||
node.reportRetryIntervalMillis = 60_000;
|
||||
|
||||
node.flushReport();
|
||||
channel.runPendingTasks();
|
||||
|
||||
assertTrue(node.reportSends.isEmpty(), "写入失败后不得留下发送登记");
|
||||
assertTrue(node.queue.containsKey(91004), "写入失败绝不能让任务消失");
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/** 通道断开后必须复位协商结果,避免向未协商的对端发送 v2 消息。 */
|
||||
@Test
|
||||
void disconnectResetsNegotiation() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
assertTrue(node.reliableTaskService.reporter().reliableV2());
|
||||
|
||||
close(channel);
|
||||
|
||||
assertFalse(node.reliableTaskService.reporter().reliableV2(), "断线后应重新协商");
|
||||
}
|
||||
|
||||
/**
|
||||
* 队列为空、扫描也无变化时,仍必须重试待确认报告。
|
||||
*
|
||||
* <p>历史缺陷:待发报告的路径都挂在「有任务变化」的分支后面,一旦任务不再变化
|
||||
* (例如下载已结束、或应用刚重启且队列为空),ACK 丢失的报告就再也不会被重发。
|
||||
*/
|
||||
@Test
|
||||
void pendingReportIsRetriedEvenWhenQueueIsEmpty() throws Exception {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
channel.writeInbound(ProtocolHelloMessage.v2());
|
||||
drainOutbound(channel);
|
||||
|
||||
GalleryTask done = task(91005);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(91005, done);
|
||||
node.reliableTaskService.refreshJournalFromQueue();
|
||||
node.reportRetryIntervalMillis = 0;
|
||||
|
||||
node.flushReport();
|
||||
assertNotNull(readReport(channel));
|
||||
|
||||
// 模拟主站已把任务标为完成并从节点队列移除,但 ACK 仍未到达。
|
||||
node.queue.clear();
|
||||
node.mainThread();
|
||||
|
||||
TaskReportMessage resent = readReport(channel);
|
||||
assertNotNull(resent, "队列为空也必须重试待确认报告");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 旧协议路径下,写入失败时不得清空已完成任务。
|
||||
*
|
||||
* <p>旧协议没有 ACK,唯一的安全边界就是「写成功」。若在写失败时照旧把压缩完成的任务
|
||||
* 出队,这份完成状态既没送达主站也不会再重发,任务会永久停在未完成。
|
||||
*/
|
||||
@Test
|
||||
void legacyReportKeepsTaskWhenWriteFails() {
|
||||
FailingChannel channel = new FailingChannel(node.new MyChannelInboundHandlerAdapter(node.tempQueue));
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
drainOutbound(channel);
|
||||
channel.failWrites = true;
|
||||
|
||||
GalleryTask done = task(91006);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(91006, done);
|
||||
|
||||
// 未协商 v2:走旧上报路径。
|
||||
assertFalse(node.reliableTaskService.reporter().reliableV2());
|
||||
node.mainThread();
|
||||
channel.runPendingTasks();
|
||||
|
||||
assertTrue(node.queue.containsKey(91006), "写失败时任务必须保留在队列中等待重发");
|
||||
} finally {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 任务下发:先按 gid 入队并立即回传当前状态,再回执响应。
|
||||
*
|
||||
* <p>回传顺序不能反——主站的单任务重试接口依赖「先收到状态、再收到响应」。
|
||||
*/
|
||||
@Test
|
||||
void downloadPostEnqueuesTaskAndReportsStatusBeforeResponse() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
DownloadPostMessage post = new DownloadPostMessage();
|
||||
post.setGalleryTask(task(90001));
|
||||
post.messageId = 7;
|
||||
|
||||
channel.writeInbound(post);
|
||||
|
||||
DownloadStatusMessage status = channel.readOutbound();
|
||||
assertNotNull(status, "应先把当前任务状态回传给主站");
|
||||
assertEquals(1, status.getGalleryTasks().length);
|
||||
assertEquals(90001, status.getGalleryTasks()[0].getGid());
|
||||
|
||||
ResponseMessage response = channel.readOutbound();
|
||||
assertNotNull(response, "随后应回执响应");
|
||||
assertEquals(7, response.messageId, "响应必须带上下发方的 messageId");
|
||||
assertEquals(0, response.getResult());
|
||||
|
||||
assertTrue(node.queue.containsKey(90001), "任务应已进入待处理队列");
|
||||
assertTrue(node.primaryChannel.usable(), "主站消息应认领通道引用");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 删除任务:既清理待上报队列,也删掉磁盘目录,并回执成功。 */
|
||||
@Test
|
||||
void deleteGalleryRemovesQueuedTaskAndDirectory() throws Exception {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
Path directory = Files.createDirectories(Path.of(storageNode.storagePath, "sample [90002]"));
|
||||
node.queue.put(90002, task(90002));
|
||||
|
||||
DeleteGalleryMessage delete = new DeleteGalleryMessage();
|
||||
delete.setGalleryName("sample [90002]");
|
||||
delete.messageId = 3;
|
||||
channel.writeInbound(delete);
|
||||
|
||||
ResponseMessage response = channel.readOutbound();
|
||||
assertNotNull(response);
|
||||
assertEquals(3, response.messageId);
|
||||
assertEquals(0, response.getResult(), "目录删除成功应回 0");
|
||||
|
||||
assertFalse(node.queue.containsKey(90002), "已删除的任务不得继续上报状态");
|
||||
assertFalse(Files.exists(directory), "磁盘目录应被删除");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 删除不存在的目录时仍要回包,并如实返回 FILE_NOT_FOUND。 */
|
||||
@Test
|
||||
void deleteGalleryMissingDirectoryReportsFileNotFound() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
DeleteGalleryMessage delete = new DeleteGalleryMessage();
|
||||
delete.setGalleryName("never-existed [90003]");
|
||||
delete.messageId = 4;
|
||||
|
||||
channel.writeInbound(delete);
|
||||
|
||||
ResponseMessage response = channel.readOutbound();
|
||||
assertNotNull(response, "无论删除成败都必须回执,否则主站会一直等待");
|
||||
assertEquals(4, response.messageId);
|
||||
assertEquals(ErrorCode.FILE_NOT_FOUND, response.getResult());
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 探活:必须原样带回 messageId,供主站配对。 */
|
||||
@Test
|
||||
void availableCheckEchoesMessageId() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
AvailableCheckMessage probe = new AvailableCheckMessage();
|
||||
probe.messageId = 42;
|
||||
|
||||
channel.writeInbound(probe);
|
||||
|
||||
ResponseMessage response = channel.readOutbound();
|
||||
assertNotNull(response);
|
||||
assertEquals(42, response.messageId);
|
||||
assertEquals(0, response.getResult());
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 主站身份消息认领通道引用;备机身份消息不得认领,否则状态会上报到错误连接。 */
|
||||
@Test
|
||||
void onlyPrimaryIdentityClaimsChannelReference() {
|
||||
EmbeddedChannel primary = nodeChannel();
|
||||
EmbeddedChannel sidecar = nodeChannel();
|
||||
try {
|
||||
sidecar.writeInbound(identity("lionwebsiteside"));
|
||||
assertFalse(node.primaryChannel.usable(), "备机身份不得占用主站引用");
|
||||
assertSame(sidecar, node.node, "备机通道应登记为 node");
|
||||
|
||||
primary.writeInbound(identity("lionwebsite"));
|
||||
assertTrue(node.primaryChannel.usable(), "主站身份应认领引用");
|
||||
assertSame(primary, node.primaryChannel.current());
|
||||
} finally {
|
||||
close(primary);
|
||||
close(sidecar);
|
||||
}
|
||||
}
|
||||
|
||||
/** 未开启订阅同步时,快照消息必须被拒绝。 */
|
||||
@Test
|
||||
void subscriptionSnapshotRejectedWhenSyncDisabled() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
SubscriptionSnapshotMessage snapshot = new SubscriptionSnapshotMessage();
|
||||
snapshot.messageId = 9;
|
||||
snapshot.setSchemaVersion(1);
|
||||
snapshot.setRevision("a".repeat(64));
|
||||
|
||||
channel.writeInbound(snapshot);
|
||||
|
||||
ResponseMessage response = channel.readOutbound();
|
||||
assertNotNull(response);
|
||||
assertEquals(9, response.messageId);
|
||||
assertEquals(SubscriptionSnapshotStore.APPLY_INVALID, response.getResult());
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/** 有待上报任务且通道可用时,mainThread 应上报并清空已完成的压缩任务。 */
|
||||
@Test
|
||||
void mainThreadReportsCompletedTaskAndDrainsQueue() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
drainOutbound(channel);
|
||||
|
||||
GalleryTask done = task(90004);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(90004, done);
|
||||
|
||||
node.mainThread();
|
||||
|
||||
DownloadStatusMessage status = readStatusMessage(channel);
|
||||
assertNotNull(status, "通道可用时必须上报任务状态");
|
||||
assertEquals(1, status.getGalleryTasks().length);
|
||||
assertEquals(90004, status.getGalleryTasks()[0].getGid());
|
||||
assertFalse(node.queue.containsKey(90004), "压缩完成的任务上报后应出队");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 无可用通道时不得上报,也绝不能丢弃任务——队列要保留到主站重连后重放。
|
||||
* 这是「状态永久卡在已提交」的另一半成因。
|
||||
*/
|
||||
@Test
|
||||
void mainThreadKeepsTasksWhenNoChannelIsUsable() {
|
||||
assertFalse(node.primaryChannel.usable(), "前置条件:本用例开始时不应有已认证通道");
|
||||
|
||||
GalleryTask done = task(90005);
|
||||
done.setStatus(GalleryTask.COMPRESS_COMPLETE);
|
||||
node.queue.put(90005, done);
|
||||
|
||||
node.mainThread();
|
||||
|
||||
assertTrue(node.queue.containsKey(90005), "无通道时任务必须保留在队列中等待重放");
|
||||
assertNull(node.primaryChannel.current());
|
||||
}
|
||||
|
||||
/** 队列为空时不得发送任务状态消息(心跳不算)。 */
|
||||
@Test
|
||||
void mainThreadSendsNoStatusWhenQueueIsEmpty() {
|
||||
EmbeddedChannel channel = nodeChannel();
|
||||
try {
|
||||
channel.writeInbound(identity("lionwebsite"));
|
||||
drainOutbound(channel);
|
||||
|
||||
node.mainThread();
|
||||
|
||||
assertNull(readStatusMessage(channel), "没有任务可报时不应发送状态消息");
|
||||
} finally {
|
||||
close(channel);
|
||||
}
|
||||
}
|
||||
|
||||
private static EmbeddedChannel nodeChannel() {
|
||||
return new EmbeddedChannel(node.new MyChannelInboundHandlerAdapter(node.tempQueue));
|
||||
}
|
||||
|
||||
/** 等待可靠执行器把当前排队的任务处理完,使异步回包/落盘对测试可见。 */
|
||||
private static void awaitReliableExecutor() throws Exception {
|
||||
node.reliableExecutor.submit(() -> { }).get(5, java.util.concurrent.TimeUnit.SECONDS);
|
||||
// 执行器是单线程 FIFO:再提交一个空任务并等它返回,即代表之前的任务都已完成。
|
||||
node.reliableExecutor.submit(() -> { }).get(5, java.util.concurrent.TimeUnit.SECONDS);
|
||||
}
|
||||
|
||||
/** 读到指定 messageId 的响应消息,忽略其它出站内容。 */
|
||||
private static ResponseMessage readResponse(EmbeddedChannel channel, int messageId) {
|
||||
Object outbound;
|
||||
while ((outbound = channel.readOutbound()) != null)
|
||||
if (outbound instanceof ResponseMessage response && response.messageId == messageId)
|
||||
return response;
|
||||
return null;
|
||||
}
|
||||
|
||||
/** 读出第一条可靠报告;容忍排在其前面的其它出站消息。 */
|
||||
private static TaskReportMessage readReport(EmbeddedChannel channel) {
|
||||
Object outbound;
|
||||
while ((outbound = channel.readOutbound()) != null)
|
||||
if (outbound instanceof TaskReportMessage report)
|
||||
return report;
|
||||
return null;
|
||||
}
|
||||
|
||||
private static GalleryTask task(int gid) {
|
||||
GalleryTask task = new GalleryTask();
|
||||
task.setGid(gid);
|
||||
task.setName("sample [" + gid + "]");
|
||||
return task;
|
||||
}
|
||||
|
||||
private static IdentityMessage identity(String value) {
|
||||
IdentityMessage message = new IdentityMessage();
|
||||
message.setIdentity(value);
|
||||
return message;
|
||||
}
|
||||
|
||||
/** 读出并丢弃全部出站消息,用于清场。 */
|
||||
private static int drainOutbound(EmbeddedChannel channel) {
|
||||
int count = 0;
|
||||
while (channel.readOutbound() != null)
|
||||
count++;
|
||||
return count;
|
||||
}
|
||||
|
||||
/** 找出第一条任务状态消息;容忍调度线程可能插入的心跳等消息。 */
|
||||
private static DownloadStatusMessage readStatusMessage(EmbeddedChannel channel) {
|
||||
Object outbound;
|
||||
while ((outbound = channel.readOutbound()) != null)
|
||||
if (outbound instanceof DownloadStatusMessage status)
|
||||
return status;
|
||||
return null;
|
||||
}
|
||||
|
||||
private static void close(EmbeddedChannel channel) {
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
|
||||
/**
|
||||
* 可注入写失败的 EmbeddedChannel。
|
||||
*
|
||||
* <p>用于验证「TCP 未断但写入失败」这一分支:写入失败必须清除发送登记,
|
||||
* 让下一轮扫描立刻重发,而不是把报告当成已送达而永久留在待确认状态。
|
||||
*/
|
||||
static class FailingChannel extends EmbeddedChannel {
|
||||
volatile boolean failWrites;
|
||||
|
||||
FailingChannel(io.netty.channel.ChannelHandler... handlers) {
|
||||
super(handlers);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void handleOutboundMessage(Object msg) {
|
||||
if (failWrites)
|
||||
throw new io.netty.channel.ChannelException("injected write failure");
|
||||
super.handleOutboundMessage(msg);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user