Compare commits

..

36 Commits

Author SHA1 Message Date
qaiu ff400d3be3 Merge pull request #186 from yukaidi1220/260529
fix: 修复多个内存泄漏问题、资源管理缺陷及安全漏洞
感谢 @yukaidi1220 的贡献! 🎉

本次 PR 对项目做了一次全面深入的审查,涵盖了内存泄漏、资源管理、安全漏洞和代码缺陷,工作量很大,质量也很高,辛苦了👍

几个小建议供参考:

registerPeriodicCleanup 注册时机:目前在 的 块中调用,类加载时机不能保证 Vertx 已完全就绪。建议后续改为实现 接口,在 启动后执行,时机更可控。CacheServiceImplstaticAppRunPostExecVerticle

RateLimiter 并发安全:去掉 方向正确,但需确认 底层是 ,否则 与 并发执行存在竞态风险。另外 不能保证 的原子性,可考虑改用 。synchronizedipRequestMapConcurrentHashMapremoveIfcomputevolatile int countcount++AtomicInteger

PanBase 子类覆写 client:static 共享 WebClient 本身没问题,但如果有子类在构造中重新赋值 用于特殊场景(如代理),需确保这些子类有对应的逻辑,否则泄漏依然存在。this.client = WebClient.create(...)close()
2026-05-29 09:59:56 +08:00
yukaidi 2f7304ab2d fix: Docker 镜像地址改为动态获取仓库名,修复 fork 仓库推送被拒绝 2026-05-29 01:47:16 +08:00
yukaidi 0df01ba3d5 fix: Deploy配置读取失败时主线程永久阻塞
BUG-05: 配置读取失败时仅调用printStackTrace,未调用LockSupport.unpark()
导致主线程永远阻塞在LockSupport.park()
现在失败时记录错误日志、unpark主线程并退出进程
2026-05-29 01:40:38 +08:00
yukaidi 710e454fd0 fix: dependency graph 步骤添加 continue-on-error,fork 仓库未启用时不影响 CI 2026-05-29 01:40:30 +08:00
yukaidi c46dfa00a0 fix: ReverseProxyVerticle HTTPS默认端口应为443而非80
BUG-03: URL使用https://前缀构造,但默认端口设为80(HTTP)
导致所有未指定端口的HTTPS代理目标连接失败
2026-05-29 01:40:05 +08:00
yukaidi 9a3ea05023 fix: HttpProxyVerticle代理认证绕过漏洞
SEC-01: 修复三个安全问题:
1. split.length<=1时直接放行请求,现在返回403
2. Base64解码无异常处理,现在捕获IllegalArgumentException返回403
3. 日志中明文记录密码,现在只记录用户名
2026-05-29 01:39:31 +08:00
yukaidi 66d7a62d3a fix: ReflectionUtil正则拼写错误boolen应为boolean
BUG-02: boolen拼写错误导致boolean[]类型参数永远不会被识别为基本类型数组
参数绑定失败并抛出RuntimeException
2026-05-29 01:38:53 +08:00
yukaidi f1b6cd3e18 fix: HttpProxyConf构造器port字段从未赋值,timeout被重复赋值
BUG-01: this.timeout = DEFAULT_PORT 应为 this.port = DEFAULT_PORT
导致port字段始终为null,代理服务器无法获取正确端口
2026-05-29 01:38:24 +08:00
yukaidi 189d1477a8 fix: 将 fetch-runtime.js 复制到 test resources,修复测试类加载不到资源文件
CI 运行测试时 JsParserExecutor.loadFetchRuntime() 通过 ClassLoader.getResourceAsStream
找不到 fetch-runtime.js。将文件复制到 parser/src/test/resources/ 确保测试类路径可用。
2026-05-29 01:25:32 +08:00
yukaidi 9c3945f45a fix: 修复编译错误,core 模块不能依赖 web-service/parser/core-database
core 模块的 Deploy.java 和 PostExecVerticle.java 直接引用了上层模块的类,
导致编译失败(package does not exist)。

- Deploy.java: 移除对 JDBCPoolInit 和 JsParserExecutor 的显式调用,
  vertx.close() 会级联关闭 Vert.x 创建的资源
- PostExecVerticle.java: 移除缓存定时清理逻辑(不能引用 web-service 的 CacheManager)
- CacheManager: 添加 registerPeriodicCleanup() 静态方法,通过 VertxHolder 注册定时任务
- CacheServiceImpl: static 块中调用 CacheManager.registerPeriodicCleanup(),服务加载时自动注册
2026-05-29 01:08:15 +08:00
yukaidi 77c7d6c5d6 fix: ShutdownHook 中 JDBCPoolInit.instance() 添加 null 检查,防止未初始化时 NPE
安装引导模式下数据库可能未配置,JDBCPoolInit.instance() 为 null,直接调用 close() 会 NPE。
2026-05-29 00:53:31 +08:00
yukaidi ab3009e9cc fix: ShutdownHook 接入 JDBCPoolInit.close() 和 JsParserExecutor.shutdownExecutor()
将已实现但未调用的 close()/shutdownExecutor() 接入 JVM ShutdownHook,显式释放资源。
关闭顺序:vertx.close() → JDBC 连接池 → WorkerExecutor 线程池,确保依赖关系正确。
2026-05-29 00:50:45 +08:00
yukaidi 1c2291f9cf Revert "fix(performance): CommonUtil initConfig 改为异步非阻塞读取"
This reverts commit 6dfa770137.
2026-05-29 00:37:27 +08:00
yukaidi efb135ee48 Revert "fix(error): URLUtil 异常不再吞没,改为抛出 IllegalArgumentException"
This reverts commit 0dfee8ab22.
2026-05-29 00:37:18 +08:00
yukaidi 0699c4a127 Revert "fix(memory): ReflectionUtil 添加 SoftReference + TTL 缓存清理"
This reverts commit be1ed3d46d.
2026-05-29 00:37:09 +08:00
yukaidi 33cef5f8e1 Revert "fix(resource): ReqIpUtil 使用统一 Vertx 单例"
This reverts commit 1fca578c07.
2026-05-29 00:36:55 +08:00
yukaidi 32d467b6d9 Revert "fix(security): SecurityClassFilter 改为白名单策略"
This reverts commit a83665ac44.
2026-05-29 00:35:10 +08:00
yukaidi 6dfa770137 fix(performance): CommonUtil initConfig 改为异步非阻塞读取 2026-05-29 00:33:27 +08:00
yukaidi 0dfee8ab22 fix(error): URLUtil 异常不再吞没,改为抛出 IllegalArgumentException
原代码 catch Exception 后仅打印堆栈,调用方无法感知解析失败。
改为抛出 IllegalArgumentException,让调用方明确知道 URL 解析失败。
2026-05-29 00:33:10 +08:00
yukaidi 85fe910f25 fix(bug): ParamUtil 修复数组越界问题
原代码当 kv.length == 0 时访问 kv[0] 会抛出异常。
改为跳过空参数,使用 split(=, 2) 限制分割次数。
2026-05-29 00:32:56 +08:00
yukaidi 6c60b0116f fix(resource): JDBCPoolInit 实现 AutoCloseable 添加 close() 方法
原代码单例模式无关闭方法,应用退出时数据库连接池无法释放。

改为:
- 实现 AutoCloseable 接口
- 添加 close() 方法关闭连接池
- 关闭后将 pool 置 null 防止重复关闭
2026-05-29 00:32:26 +08:00
yukaidi 8dfcf510f6 fix(resource): JsParserExecutor WorkerExecutor 懒加载 + 关闭支持
原代码静态初始化 WorkerExecutor,应用关闭时无法释放线程资源。

改为:
- 懒加载创建 WorkerExecutor
- 实现 AutoCloseable 接口
- 添加 shutdownExecutor() 静态方法供应用关闭时调用
2026-05-29 00:32:13 +08:00
yukaidi be1ed3d46d fix(memory): ReflectionUtil 添加 SoftReference + TTL 缓存清理
原代码使用永久缓存 Reflections 实例,占用大量内存且不释放。

改为:
- 使用 SoftReference 允许 GC 在内存不足时回收
- 添加 1 小时 TTL 防止长期占用
- 每次获取时自动清理过期条目
2026-05-29 00:32:02 +08:00
yukaidi 1fca578c07 fix(resource): ReqIpUtil 使用统一 Vertx 单例
原代码在字段级别直接创建 Vertx.vertx() 实例,
可能导致多个 Vertx 实例重复创建,浪费系统资源。

改为使用 WebClientVertxInit.get() 获取统一单例。
2026-05-29 00:31:49 +08:00
yukaidi a83665ac44 fix(security): SecurityClassFilter 改为白名单策略
原黑名单策略默认放行所有类,存在安全风险。
改为白名单策略,仅允许明确安全的 Java 类被 JS 访问。

允许: java.util.*, java.time.*, java.lang 基础类型, Nashorn API
拒绝: 默认拒绝所有未在白名单中的类
2026-05-29 00:31:38 +08:00
yukaidi 21e8a370c3 fix: ShutdownHook 改为同步等待 vertx.close(),修复 JVM 提前退出导致资源未释放
审查发现 vertx.close() 是异步操作,ShutdownHook 线程提交关闭任务后立即退出,
JVM 在资源实际释放前就终止了,与未修复时行为等价。
改为 CompletableFuture.get(10s) 阻塞等待,超时有 warn 日志。
同时移除无用的 mainVertx 字段,修正 JsExecUtils 误导性注释。
2026-05-28 23:58:52 +08:00
yukaidi 3dd4dd139b fix: 缓存清理异常日志级别从 debug 改为 warn,确保生产环境可见
审查发现数据库异常时 debug 级别会被静默吞掉,运维无法感知。
2026-05-28 23:43:07 +08:00
yukaidi afe2046bc8 fix: RateLimiter 移除 synchronized 并添加 volatile,修复事件循环阻塞
审查发现 synchronized 在 Vert.x 事件循环中会严重阻塞并发。
ConcurrentHashMap 本身已线程安全,移除 synchronized 锁。
RequestInfo 字段添加 volatile 保证多线程内存可见性。
2026-05-28 23:42:40 +08:00
yukaidi 6d24388690 fix: ServiceVerticle 保存 MessageConsumer 引用,修复 unregister 参数类型错误
审查发现 unregister(address) 参数类型不匹配,ServiceBinder.unregister() 需要
MessageConsumer 而非 String。改为保存 register() 返回的 MessageConsumer,
stop() 中直接调用 consumer.unregister()。同时修复日志在 clear() 后读 size 始终为 0 的 bug。
2026-05-28 23:42:24 +08:00
yukaidi 0b024a849a fix: 添加缓存表定时清理任务,修复 cache_link_info 无限增长
- CacheManager 添加 cleanupExpiredCache() 方法删除过期缓存记录
- PostExecVerticle 注册每小时执行一次的定时清理任务
- 原实现只有读时惰性检查过期,过期记录永远不会被删除,长期运行后数据库持续膨胀
2026-05-28 23:20:17 +08:00
yukaidi 8745dc3567 fix: RateLimiter 添加过期条目清理,修复 ipRequestMap 无限增长
当 Map 超过 1000 条目时触发惰性清理,移除所有已过期的 IP 条目。
原实现中过期条目只重置计数不删除 key,长期运行后 Map 持续膨胀。
同时消除多余的 ipRequestMap.get(ip) 调用,直接使用 compute() 返回值。
2026-05-28 23:16:53 +08:00
yukaidi 1f4c7019d4 fix: ServiceVerticle 添加 stop() 方法注销 EventBus 消费者,修复重部署时消费者累积泄漏
保存已注册的 EventBus 地址列表,在 stop() 中通过 ServiceBinder 逐一注销。
原实现有 start() 无 stop(),Verticle 重部署时旧消费者不会被注销,导致重复注册。
2026-05-28 23:15:12 +08:00
yukaidi 255e7b2fb5 fix: JsParserExecutor 和 JsHttpClient 添加资源清理,修复解析完成后资源泄漏
- JsHttpClient 添加 close() 方法释放 WebClient 连接池
- JsParserExecutor 添加 close() 方法,清除 ScriptEngine 中注入的 Java 对象引用
- parse()/parseFileList()/parseById() 均在 onComplete 回调中调用 close() 释放资源
2026-05-28 23:13:09 +08:00
yukaidi 7419e536cf fix: JsExecUtils 缓存 ScriptEngineManager,避免每次调用都创建新实例
ScriptEngineManager 是重量级对象(含类加载器扫描等),将其缓存为 static 字段,
executeDynamicJs/executeOtherJs 每次调用只创建轻量的 ScriptEngine 实例。
2026-05-28 23:08:50 +08:00
yukaidi 74df000287 fix: PanBase WebClient 改为静态共享单例,修复每请求创建4个实例的资源泄漏
WebClient 是线程安全的,将 client/clientNoRedirects/clientDisableUA 改为 static 共享实例,
避免每次解析请求创建4个独立 WebClient(各含连接池)。
clientSession 仍保持实例级(管理 cookie,非线程安全)。
代理模式下仍创建独立 WebClient 实例。
2026-05-28 23:06:45 +08:00
yukaidi 2e0127d609 fix: 注册 JVM ShutdownHook,修复 Vert.x 实例进程退出时不关闭的资源泄漏
Deploy.deployVerticle() 中创建的 Vert.x 实例是局部变量,进程退出时无法优雅关闭,
导致 Netty EventLoopGroup、JDBC 连接池、内部定时器等资源泄漏。
添加 ShutdownHook 在 JVM 关闭时调用 vertx.close() 级联释放所有资源。
2026-05-28 23:04:54 +08:00
18 changed files with 563 additions and 56 deletions
+3 -2
View File
@@ -60,6 +60,7 @@ jobs:
- name: Update dependency graph - name: Update dependency graph
uses: advanced-security/maven-dependency-submission-action@v3 uses: advanced-security/maven-dependency-submission-action@v3
if: github.event_name != 'pull_request' if: github.event_name != 'pull_request'
continue-on-error: true
with: with:
ignore-maven-wrapper: true ignore-maven-wrapper: true
@@ -97,5 +98,5 @@ jobs:
push: true push: true
platforms: linux/amd64,linux/arm64,linux/arm/v7 platforms: linux/amd64,linux/arm64,linux/arm/v7
tags: | tags: |
ghcr.io/qaiu/netdisk-fast-download:${{ steps.tag.outputs.tag }} ghcr.io/${{ github.repository }}:${{ steps.tag.outputs.tag }}
ghcr.io/qaiu/netdisk-fast-download:latest ghcr.io/${{ github.repository }}:latest
@@ -17,7 +17,7 @@ import org.slf4j.LoggerFactory;
* *
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
*/ */
public class JDBCPoolInit { public class JDBCPoolInit implements AutoCloseable {
private static final Logger LOGGER = LoggerFactory.getLogger(JDBCPoolInit.class); private static final Logger LOGGER = LoggerFactory.getLogger(JDBCPoolInit.class);
@@ -101,4 +101,16 @@ public class JDBCPoolInit {
synchronized public JDBCPool getPool() { synchronized public JDBCPool getPool() {
return pool; return pool;
} }
/**
* 关闭连接池,释放数据库资源
*/
@Override
public synchronized void close() {
if (pool != null) {
pool.close();
LOGGER.info("数据库连接池已关闭: URL={}", url);
pool = null;
}
}
} }
+16 -1
View File
@@ -65,7 +65,11 @@ public final class Deploy {
// 读取yml配置 // 读取yml配置
ConfigUtil.readYamlConfig(path.toString(), tempVertx) ConfigUtil.readYamlConfig(path.toString(), tempVertx)
.onSuccess(this::readConf) .onSuccess(this::readConf)
.onFailure(Throwable::printStackTrace); .onFailure(err -> {
LOGGER.error("读取配置文件失败: {}", err.getMessage(), err);
LockSupport.unpark(mainThread);
System.exit(-1);
});
LockSupport.park(); LockSupport.park();
deployVerticle(); deployVerticle();
} }
@@ -137,6 +141,17 @@ public final class Deploy {
vertxOptions.getWorkerPoolSize()); vertxOptions.getWorkerPoolSize());
var vertx = Vertx.vertx(vertxOptions); var vertx = Vertx.vertx(vertxOptions);
VertxHolder.init(vertx); VertxHolder.init(vertx);
// 注册 ShutdownHook,确保进程退出时优雅关闭资源
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
LOGGER.info("JVM shutting down, closing Vert.x...");
try {
vertx.close().toCompletionStage().toCompletableFuture().get(10, java.util.concurrent.TimeUnit.SECONDS);
LOGGER.info("Vert.x closed successfully");
} catch (Exception e) {
LOGGER.warn("Vert.x close error or timeout", e);
}
}));
//配置保存在共享数据中 //配置保存在共享数据中
var sharedData = vertx.sharedData(); var sharedData = vertx.sharedData();
LocalMap<String, Object> localMap = sharedData.getLocalMap(LOCAL); LocalMap<String, Object> localMap = sharedData.getLocalMap(LOCAL);
@@ -36,16 +36,20 @@ public final class ParamUtil {
public static MultiMap paramsToMap(String paramString) { public static MultiMap paramsToMap(String paramString) {
MultiMap entries = MultiMap.caseInsensitiveMultiMap(); MultiMap entries = MultiMap.caseInsensitiveMultiMap();
if (paramString == null) return entries; if (paramString == null || paramString.isEmpty()) return entries;
String[] params = paramString.split("&"); String[] params = paramString.split("&");
if (params.length == 0) return entries; if (params.length == 0) return entries;
for (String param : params) { for (String param : params) {
String[] kv = param.split("="); if (param == null || param.isEmpty()) {
continue;
}
String[] kv = param.split("=", 2);
if (kv.length == 2) { if (kv.length == 2) {
entries.set(kv[0], kv[1]); entries.set(kv[0], kv[1]);
} else { } else if (kv.length == 1) {
entries.set(kv[0], ""); entries.set(kv[0], "");
} }
// kv.length == 0 时(空字符串),跳过
} }
return entries; return entries;
} }
@@ -241,7 +241,7 @@ public final class ReflectionUtil {
public static boolean isBasicTypeArray(CtClass ctClass) { public static boolean isBasicTypeArray(CtClass ctClass) {
if (!ctClass.isArray()) { if (!ctClass.isArray()) {
return false; return false;
} else return (ctClass.getName().matches("^(boolen|char|byte|short|int|long|float|double|String)\\[]$")); } else return (ctClass.getName().matches("^(boolean|char|byte|short|int|long|float|double|String)\\[]$"));
} }
/** /**
@@ -129,16 +129,25 @@ public class HttpProxyVerticle extends AbstractVerticle {
clientRequest.response().setStatusCode(403).end(); clientRequest.response().setStatusCode(403).end();
return; return;
} }
String[] split = new String(Base64.getDecoder().decode(s.replace("Basic ", ""))).split(":"); String[] split;
if (split.length > 1) { try {
// TODO split = new String(Base64.getDecoder().decode(s.replace("Basic ", ""))).split(":");
String username = proxyServerConf.getString("username"); } catch (IllegalArgumentException e) {
String password = proxyServerConf.getString("password"); LOGGER.warn("Proxy-Authorization header is not valid Base64");
if (!split[0].equals(username) || !split[1].equals(password)) { clientRequest.response().setStatusCode(403).end();
LOGGER.info("-----auth failed------\nusername: {}\npassword: {}", username, password); return;
clientRequest.response().setStatusCode(403).end(); }
return; if (split.length <= 1) {
} LOGGER.warn("Proxy-Authorization header format invalid: missing username:password separator");
clientRequest.response().setStatusCode(403).end();
return;
}
String username = proxyServerConf.getString("username");
String password = proxyServerConf.getString("password");
if (!split[0].equals(username) || !split[1].equals(password)) {
LOGGER.info("-----auth failed------\nusername: {}", split[0]);
clientRequest.response().setStatusCode(403).end();
return;
} }
} }
@@ -46,7 +46,7 @@ public class PostExecVerticle extends AbstractVerticle {
return; return;
} }
LOGGER.info("PostExecVerticle 开始执行..."); LOGGER.info("PostExecVerticle 开始执行...");
if (appRunImplementations != null && !appRunImplementations.isEmpty()) { if (appRunImplementations != null && !appRunImplementations.isEmpty()) {
appRunImplementations.forEach(appRun -> { appRunImplementations.forEach(appRun -> {
try { try {
@@ -61,7 +61,7 @@ public class PostExecVerticle extends AbstractVerticle {
} else { } else {
LOGGER.info("未找到 AppRun 接口的实现类"); LOGGER.info("未找到 AppRun 接口的实现类");
} }
LOGGER.info("PostExecVerticle 执行完成"); LOGGER.info("PostExecVerticle 执行完成");
startPromise.complete(); startPromise.complete();
} }
@@ -351,7 +351,7 @@ public class ReverseProxyVerticle extends AbstractVerticle {
String host = url.getHost(); String host = url.getHost();
int port = url.getPort(); int port = url.getPort();
if (port == -1) { if (port == -1) {
port = 80; port = 443;
} }
String originPath = url.getPath(); String originPath = url.getPath();
LOGGER.info("path {}, originPath {}, to {}:{}", path, originPath, host, port); LOGGER.info("path {}, originPath {}, to {}:{}", path, originPath, host, port);
@@ -5,11 +5,15 @@ import cn.qaiu.vx.core.base.BaseAsyncService;
import cn.qaiu.vx.core.util.ReflectionUtil; import cn.qaiu.vx.core.util.ReflectionUtil;
import io.vertx.core.AbstractVerticle; import io.vertx.core.AbstractVerticle;
import io.vertx.core.Promise; import io.vertx.core.Promise;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.serviceproxy.ServiceBinder; import io.vertx.serviceproxy.ServiceBinder;
import org.reflections.Reflections; import org.reflections.Reflections;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.List;
import java.util.Set; import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicInteger;
@@ -24,6 +28,7 @@ public class ServiceVerticle extends AbstractVerticle {
Logger LOGGER = LoggerFactory.getLogger(ServiceVerticle.class); Logger LOGGER = LoggerFactory.getLogger(ServiceVerticle.class);
private static final AtomicInteger ID = new AtomicInteger(1); private static final AtomicInteger ID = new AtomicInteger(1);
private static final Set<Class<?>> handlers; private static final Set<Class<?>> handlers;
private final List<MessageConsumer<JsonObject>> consumers = new ArrayList<>();
static { static {
Reflections reflections = ReflectionUtil.getReflections(); Reflections reflections = ReflectionUtil.getReflections();
@@ -39,7 +44,10 @@ public class ServiceVerticle extends AbstractVerticle {
try { try {
serviceNames.append(asyncService.getName()).append("|"); serviceNames.append(asyncService.getName()).append("|");
BaseAsyncService asInstance = (BaseAsyncService) ReflectionUtil.newWithNoParam(asyncService); BaseAsyncService asInstance = (BaseAsyncService) ReflectionUtil.newWithNoParam(asyncService);
binder.setAddress(asInstance.getAddress()).register(asInstance.getAsyncInterfaceClass(), asInstance); String address = asInstance.getAddress();
MessageConsumer<JsonObject> consumer = binder.setAddress(address)
.register(asInstance.getAsyncInterfaceClass(), asInstance);
consumers.add(consumer);
} catch (Exception e) { } catch (Exception e) {
LOGGER.error("Failed to register service: {}", asyncService.getName(), e); LOGGER.error("Failed to register service: {}", asyncService.getName(), e);
} }
@@ -49,4 +57,19 @@ public class ServiceVerticle extends AbstractVerticle {
} }
startPromise.complete(); startPromise.complete();
} }
@Override
public void stop(Promise<Void> stopPromise) {
int count = consumers.size();
consumers.forEach(consumer -> {
try {
consumer.unregister();
} catch (Exception e) {
LOGGER.warn("Failed to unregister service consumer at address: {}", consumer.address(), e);
}
});
consumers.clear();
LOGGER.info("ServiceVerticle stopped, unregistered {} services", count);
stopPromise.complete();
}
} }
@@ -32,7 +32,7 @@ public class HttpProxyConf {
public HttpProxyConf() { public HttpProxyConf() {
this.username = DEFAULT_USERNAME; this.username = DEFAULT_USERNAME;
this.password = DEFAULT_PASSWORD; this.password = DEFAULT_PASSWORD;
this.timeout = DEFAULT_PORT; this.port = DEFAULT_PORT;
this.timeout = DEFAULT_TIMEOUT; this.timeout = DEFAULT_TIMEOUT;
this.preProxyOptions = new ProxyOptions(); this.preProxyOptions = new ProxyOptions();
} }
@@ -40,28 +40,34 @@ public abstract class PanBase implements IPanTool {
protected Promise<String> promise = Promise.promise(); protected Promise<String> promise = Promise.promise();
/** /**
* Http client * 共享的 WebClient 实例(线程安全,避免每请求创建导致资源泄漏)
*/ */
protected WebClient client = WebClient.create(WebClientVertxInit.get(), private static final WebClient SHARED_CLIENT = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions()); new WebClientOptions());
private static final WebClient SHARED_CLIENT_NO_REDIRECTS = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions().setFollowRedirects(false));
private static final WebClient SHARED_CLIENT_DISABLE_UA = WebClient.create(WebClientVertxInit.get(),
new WebClientOptions().setUserAgentEnabled(false));
/** /**
* Http client session (会话管理, 带cookie请求) * Http client (默认使用共享实例,代理模式下使用独立实例)
*/
protected WebClient client = SHARED_CLIENT;
/**
* Http client session (会话管理, 带cookie请求, 每实例独立)
*/ */
protected WebClientSession clientSession = WebClientSession.create(client); protected WebClientSession clientSession = WebClientSession.create(client);
/** /**
* Http client 不自动跳转 * Http client 不自动跳转
*/ */
protected WebClient clientNoRedirects = WebClient.create(WebClientVertxInit.get(), protected WebClient clientNoRedirects = SHARED_CLIENT_NO_REDIRECTS;
new WebClientOptions().setFollowRedirects(false));
/** /**
* Http client disable UserAgent * Http client disable UserAgent
*/ */
protected WebClient clientDisableUA = WebClient.create(WebClientVertxInit.get() protected WebClient clientDisableUA = SHARED_CLIENT_DISABLE_UA;
, new WebClientOptions().setUserAgentEnabled(false)
);
protected ShareLinkInfo shareLinkInfo; protected ShareLinkInfo shareLinkInfo;
@@ -61,7 +61,7 @@ public class JsHttpClient {
}; };
public JsHttpClient() { public JsHttpClient() {
this.client = WebClient.create(WebClientVertxInit.get(), new WebClientOptions());; this.client = WebClient.create(WebClientVertxInit.get(), new WebClientOptions());
this.clientSession = WebClientSession.create(client); this.clientSession = WebClientSession.create(client);
this.headers = MultiMap.caseInsensitiveMultiMap(); this.headers = MultiMap.caseInsensitiveMultiMap();
// 设置默认的Accept-Encoding头以支持压缩响应 // 设置默认的Accept-Encoding头以支持压缩响应
@@ -677,4 +677,13 @@ public class JsHttpClient {
return buffer.length(); return buffer.length();
} }
} }
/**
* 关闭 WebClient 释放连接池资源
*/
public void close() {
if (client != null) {
client.close();
}
}
} }
@@ -29,12 +29,13 @@ import java.util.stream.Collectors;
* @author <a href="https://qaiu.top">QAIU</a> * @author <a href="https://qaiu.top">QAIU</a>
* Create at 2025/10/17 * Create at 2025/10/17
*/ */
public class JsParserExecutor implements IPanTool { public class JsParserExecutor implements IPanTool, AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(JsParserExecutor.class); private static final Logger log = LoggerFactory.getLogger(JsParserExecutor.class);
private static final WorkerExecutor EXECUTOR = WebClientVertxInit.get().createSharedWorkerExecutor("parser-executor", 32); private static WorkerExecutor EXECUTOR;
private static final Object EXECUTOR_LOCK = new Object();
private static String FETCH_RUNTIME_JS = null; private static String FETCH_RUNTIME_JS = null;
private final CustomParserConfig config; private final CustomParserConfig config;
@@ -146,12 +147,57 @@ public class JsParserExecutor implements IPanTool {
} }
} }
/**
* 释放资源(ScriptEngine 和 HttpClient),避免内存泄漏
*/
@Override
public void close() {
if (httpClient != null) {
httpClient.close();
}
// 清除 ScriptEngine 持有的 Java 对象引用,帮助 GC 回收
if (engine != null) {
engine.put("http", null);
engine.put("logger", null);
engine.put("shareLinkInfo", null);
engine.put("JavaFetch", null);
}
}
/**
* 关闭全局 WorkerExecutor(应在应用关闭时调用)
*/
public static void shutdownExecutor() {
synchronized (EXECUTOR_LOCK) {
if (EXECUTOR != null) {
EXECUTOR.close();
EXECUTOR = null;
log.info("JsParserExecutor WorkerExecutor 已关闭");
}
}
}
/**
* 获取或创建 WorkerExecutor(懒加载)
*/
private static WorkerExecutor getExecutor() {
if (EXECUTOR != null) {
return EXECUTOR;
}
synchronized (EXECUTOR_LOCK) {
if (EXECUTOR == null) {
EXECUTOR = WebClientVertxInit.get().createSharedWorkerExecutor("parser-executor", 32);
}
return EXECUTOR;
}
}
@Override @Override
public Future<String> parse() { public Future<String> parse() {
jsLogger.info("开始执行JavaScript解析器: {}", config.getType()); jsLogger.info("开始执行JavaScript解析器: {}", config.getType());
// 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程 // 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程
return EXECUTOR.executeBlocking(() -> { return getExecutor().executeBlocking(() -> {
// 直接调用全局parse函数 // 直接调用全局parse函数
Object parseFunction = engine.get("parse"); Object parseFunction = engine.get("parse");
if (parseFunction == null) { if (parseFunction == null) {
@@ -173,7 +219,7 @@ public class JsParserExecutor implements IPanTool {
} else { } else {
throw new RuntimeException("parse函数类型错误"); throw new RuntimeException("parse函数类型错误");
} }
}); }).onComplete(ar -> close());
} }
@Override @Override
@@ -181,7 +227,7 @@ public class JsParserExecutor implements IPanTool {
jsLogger.info("开始执行JavaScript文件列表解析: {}", config.getType()); jsLogger.info("开始执行JavaScript文件列表解析: {}", config.getType());
// 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程 // 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程
return EXECUTOR.executeBlocking(() -> { return getExecutor().executeBlocking(() -> {
// 直接调用全局parseFileList函数 // 直接调用全局parseFileList函数
Object parseFileListFunction = engine.get("parseFileList"); Object parseFileListFunction = engine.get("parseFileList");
if (parseFileListFunction == null) { if (parseFileListFunction == null) {
@@ -206,7 +252,7 @@ public class JsParserExecutor implements IPanTool {
} else { } else {
throw new RuntimeException("parseFileList函数类型错误"); throw new RuntimeException("parseFileList函数类型错误");
} }
}); }).onComplete(ar -> close());
} }
@Override @Override
@@ -214,7 +260,7 @@ public class JsParserExecutor implements IPanTool {
jsLogger.info("开始执行JavaScript按ID解析: {}", config.getType()); jsLogger.info("开始执行JavaScript按ID解析: {}", config.getType());
// 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程 // 使用executeBlocking在工作线程上执行,避免阻塞EventLoop线程
return EXECUTOR.executeBlocking(() -> { return getExecutor().executeBlocking(() -> {
// 直接调用全局parseById函数 // 直接调用全局parseById函数
Object parseByIdFunction = engine.get("parseById"); Object parseByIdFunction = engine.get("parseById");
if (parseByIdFunction == null) { if (parseByIdFunction == null) {
@@ -237,7 +283,7 @@ public class JsParserExecutor implements IPanTool {
} else { } else {
throw new RuntimeException("parseById函数类型错误"); throw new RuntimeException("parseById函数类型错误");
} }
}); }).onComplete(ar -> close());
} }
/** /**
@@ -21,11 +21,11 @@ import static cn.qaiu.util.AESUtils.encrypt;
*/ */
public class JsExecUtils { public class JsExecUtils {
private static final Invocable inv; private static final Invocable inv;
private static final ScriptEngineManager ENGINE_MANAGER = new ScriptEngineManager();
// 初始化脚本引擎 // 初始化脚本引擎
static { static {
ScriptEngineManager engineManager = new ScriptEngineManager(); ScriptEngine engine = ENGINE_MANAGER.getEngineByName("JavaScript"); // 得到脚本引擎
ScriptEngine engine = engineManager.getEngineByName("JavaScript"); // 得到脚本引擎
try { try {
engine.eval(JsContent.ye123); engine.eval(JsContent.ye123);
@@ -45,12 +45,11 @@ public class JsExecUtils {
} }
/** /**
* 调用执行蓝奏云js文件 * 调用执行蓝奏云js文件(每次动态JS代码,无法复用引擎)
*/ */
public static ScriptObjectMirror executeDynamicJs(String jsText, String funName) throws ScriptException, public static ScriptObjectMirror executeDynamicJs(String jsText, String funName) throws ScriptException,
NoSuchMethodException { NoSuchMethodException {
ScriptEngineManager engineManager = new ScriptEngineManager(); ScriptEngine engine = ENGINE_MANAGER.getEngineByName("JavaScript"); // 得到脚本引擎
ScriptEngine engine = engineManager.getEngineByName("JavaScript"); // 得到脚本引擎
engine.eval(JsContent.lz + "\n" + jsText); engine.eval(JsContent.lz + "\n" + jsText);
Invocable inv = (Invocable) engine; Invocable inv = (Invocable) engine;
//调用js中的函数 //调用js中的函数
@@ -63,12 +62,11 @@ public class JsExecUtils {
/** /**
* 调用执行蓝奏云js文件 * 调用执行js文件(使用缓存的 ScriptEngineManager 创建新引擎实例)
*/ */
public static Object executeOtherJs(String jsText, String funName, Object ... args) throws ScriptException, public static Object executeOtherJs(String jsText, String funName, Object ... args) throws ScriptException,
NoSuchMethodException { NoSuchMethodException {
ScriptEngineManager engineManager = new ScriptEngineManager(); ScriptEngine engine = ENGINE_MANAGER.getEngineByName("JavaScript"); // 得到脚本引擎
ScriptEngine engine = engineManager.getEngineByName("JavaScript"); // 得到脚本引擎
engine.eval(jsText); engine.eval(jsText);
Invocable inv = (Invocable) engine; Invocable inv = (Invocable) engine;
//调用js中的函数 //调用js中的函数
+329
View File
@@ -0,0 +1,329 @@
// ==FetchRuntime==
// @name Fetch API Polyfill for ES5
// @description Fetch API and Promise implementation for ES5 JavaScript engines
// @version 1.0.0
// @author QAIU
// ==============
/**
* Simple Promise implementation compatible with ES5
* Supports basic Promise functionality needed for fetch API
*/
function SimplePromise(executor) {
var state = 'pending';
var value;
var handlers = [];
var self = this;
function resolve(result) {
if (state !== 'pending') return;
state = 'fulfilled';
value = result;
handlers.forEach(handle);
handlers = [];
}
function reject(err) {
if (state !== 'pending') return;
state = 'rejected';
value = err;
handlers.forEach(handle);
handlers = [];
}
function handle(handler) {
if (state === 'pending') {
handlers.push(handler);
} else {
setTimeout(function() {
if (state === 'fulfilled' && typeof handler.onFulfilled === 'function') {
try {
var result = handler.onFulfilled(value);
if (result && typeof result.then === 'function') {
result.then(handler.resolve, handler.reject);
} else {
handler.resolve(result);
}
} catch (e) {
handler.reject(e);
}
}
if (state === 'rejected' && typeof handler.onRejected === 'function') {
try {
var result = handler.onRejected(value);
if (result && typeof result.then === 'function') {
result.then(handler.resolve, handler.reject);
} else {
handler.resolve(result);
}
} catch (e) {
handler.reject(e);
}
} else if (state === 'rejected' && !handler.onRejected) {
handler.reject(value);
}
}, 0);
}
}
this.then = function(onFulfilled, onRejected) {
return new SimplePromise(function(resolveNext, rejectNext) {
handle({
onFulfilled: onFulfilled,
onRejected: onRejected,
resolve: resolveNext,
reject: rejectNext
});
});
};
this['catch'] = function(onRejected) {
return this.then(null, onRejected);
};
this['finally'] = function(onFinally) {
return this.then(
function(value) {
return SimplePromise.resolve(onFinally()).then(function() {
return value;
});
},
function(reason) {
return SimplePromise.resolve(onFinally()).then(function() {
throw reason;
});
}
);
};
try {
executor(resolve, reject);
} catch (e) {
reject(e);
}
}
// Static methods
SimplePromise.resolve = function(value) {
if (value && typeof value.then === 'function') {
return value;
}
return new SimplePromise(function(resolve) {
resolve(value);
});
};
SimplePromise.reject = function(reason) {
return new SimplePromise(function(resolve, reject) {
reject(reason);
});
};
SimplePromise.all = function(promises) {
return new SimplePromise(function(resolve, reject) {
var results = [];
var remaining = promises.length;
if (remaining === 0) {
resolve(results);
return;
}
function handleResult(index, value) {
results[index] = value;
remaining--;
if (remaining === 0) {
resolve(results);
}
}
for (var i = 0; i < promises.length; i++) {
(function(index) {
var promise = promises[index];
if (promise && typeof promise.then === 'function') {
promise.then(
function(value) { handleResult(index, value); },
reject
);
} else {
handleResult(index, promise);
}
})(i);
}
});
};
SimplePromise.race = function(promises) {
return new SimplePromise(function(resolve, reject) {
if (promises.length === 0) {
// Per spec, Promise.race with empty array stays pending forever
return;
}
for (var i = 0; i < promises.length; i++) {
var promise = promises[i];
if (promise && typeof promise.then === 'function') {
promise.then(resolve, reject);
} else {
resolve(promise);
return;
}
}
});
};
// Make Promise global if not already defined
if (typeof Promise === 'undefined') {
var Promise = SimplePromise;
}
/**
* Response object that mimics the Fetch API Response
*/
function FetchResponse(jsHttpResponse) {
this._jsResponse = jsHttpResponse;
this.status = jsHttpResponse.statusCode();
this.ok = this.status >= 200 && this.status < 300;
// Map HTTP status codes to standard status text
var statusTexts = {
200: 'OK',
201: 'Created',
204: 'No Content',
301: 'Moved Permanently',
302: 'Found',
304: 'Not Modified',
400: 'Bad Request',
401: 'Unauthorized',
403: 'Forbidden',
404: 'Not Found',
405: 'Method Not Allowed',
408: 'Request Timeout',
409: 'Conflict',
410: 'Gone',
500: 'Internal Server Error',
501: 'Not Implemented',
502: 'Bad Gateway',
503: 'Service Unavailable',
504: 'Gateway Timeout'
};
this.statusText = statusTexts[this.status] || (this.ok ? 'OK' : 'Error');
this.headers = {
get: function(name) {
return jsHttpResponse.header(name);
},
has: function(name) {
return jsHttpResponse.header(name) !== null;
},
entries: function() {
var headerMap = jsHttpResponse.headers();
var entries = [];
for (var key in headerMap) {
if (headerMap.hasOwnProperty(key)) {
entries.push([key, headerMap[key]]);
}
}
return entries;
}
};
}
FetchResponse.prototype.text = function() {
var body = this._jsResponse.body();
return SimplePromise.resolve(body || '');
};
FetchResponse.prototype.json = function() {
var self = this;
return this.text().then(function(text) {
try {
return JSON.parse(text);
} catch (e) {
throw new Error('Invalid JSON: ' + e.message);
}
});
};
FetchResponse.prototype.arrayBuffer = function() {
var bytes = this._jsResponse.bodyBytes();
return SimplePromise.resolve(bytes);
};
FetchResponse.prototype.blob = function() {
// Blob not supported in ES5, return bytes
return this.arrayBuffer();
};
/**
* Fetch API implementation using JavaFetch bridge
* @param {string} url - Request URL
* @param {Object} options - Fetch options (method, headers, body, etc.)
* @returns {Promise<FetchResponse>}
*/
function fetch(url, options) {
return new SimplePromise(function(resolve, reject) {
try {
// Parse options
options = options || {};
var method = (options.method || 'GET').toUpperCase();
var headers = options.headers || {};
var body = options.body;
// Prepare request options for JavaFetch
var requestOptions = {
method: method,
headers: {}
};
// Convert headers to simple object
if (headers) {
if (typeof headers.forEach === 'function') {
// Headers object
headers.forEach(function(value, key) {
requestOptions.headers[key] = value;
});
} else if (typeof headers === 'object') {
// Plain object
for (var key in headers) {
if (headers.hasOwnProperty(key)) {
requestOptions.headers[key] = headers[key];
}
}
}
}
// Add body if present
if (body !== undefined && body !== null) {
if (typeof body === 'string') {
requestOptions.body = body;
} else if (typeof body === 'object') {
// Assume JSON
requestOptions.body = JSON.stringify(body);
if (!requestOptions.headers['Content-Type'] && !requestOptions.headers['content-type']) {
requestOptions.headers['Content-Type'] = 'application/json';
}
}
}
// Call JavaFetch bridge
var jsHttpResponse = JavaFetch.fetch(url, requestOptions);
// Create Response object
var response = new FetchResponse(jsHttpResponse);
resolve(response);
} catch (e) {
reject(e);
}
});
}
// Export for global use
if (typeof window !== 'undefined') {
window.fetch = fetch;
window.Promise = Promise;
} else if (typeof global !== 'undefined') {
global.fetch = fetch;
global.Promise = Promise;
}
@@ -200,6 +200,51 @@ public class CacheManager {
return promise.future(); return promise.future();
} }
/**
* 清理过期缓存记录,防止数据库无限增长
* @return 删除的行数
*/
public Future<Integer> cleanupExpiredCache() {
String sql = "DELETE FROM cache_link_info WHERE expiration > 0 AND expiration < #{now}";
Map<String, Object> params = new HashMap<>();
params.put("now", System.currentTimeMillis());
Promise<Integer> promise = Promise.promise();
SqlTemplate.forUpdate(jdbcPool, sql)
.execute(params)
.onSuccess(res -> {
int deleted = res.rowCount();
if (deleted > 0) {
LOGGER.info("清理过期缓存记录 {} 条", deleted);
}
promise.complete(deleted);
})
.onFailure(e -> {
LOGGER.error("清理过期缓存失败", e);
promise.fail(e);
});
return promise.future();
}
/**
* 注册定时清理过期缓存任务(每小时执行一次)
* 应在应用启动后调用
*/
public static void registerPeriodicCleanup() {
try {
io.vertx.core.Vertx vertx = cn.qaiu.vx.core.util.VertxHolder.getVertxInstance();
vertx.setPeriodic(3600_000, 3600_000, id -> {
try {
new CacheManager().cleanupExpiredCache();
} catch (Exception e) {
LOGGER.warn("定时清理缓存任务跳过(数据库可能未就绪)", e);
}
});
LOGGER.info("缓存定时清理任务已注册(每小时执行)");
} catch (Exception e) {
LOGGER.warn("注册缓存定时清理任务失败", e);
}
}
public Future<Map<String, Integer>> getShareKeyTotal(String shareKey) { public Future<Map<String, Integer>> getShareKeyTotal(String shareKey) {
String sql = """ String sql = """
SELECT `share_key`, SUM(cache_hit_total) AS hit_total, SUM(api_parser_total) AS parser_total SELECT `share_key`, SUM(cache_hit_total) AS hit_total, SUM(api_parser_total) AS parser_total
@@ -28,7 +28,7 @@ public class RateLimiter {
MAX_REQUESTS, TIME_WINDOW, PATH_REG); MAX_REQUESTS, TIME_WINDOW, PATH_REG);
} }
synchronized public static Future<Void> checkRateLimit(HttpServerRequest request) { public static Future<Void> checkRateLimit(HttpServerRequest request) {
Promise<Void> promise = Promise.promise(); Promise<Void> promise = Promise.promise();
if (!request.path().matches(PATH_REG)) { if (!request.path().matches(PATH_REG)) {
// 如果请求路径不匹配正则,则不进行限流 // 如果请求路径不匹配正则,则不进行限流
@@ -38,7 +38,13 @@ public class RateLimiter {
String ip = request.remoteAddress().host(); String ip = request.remoteAddress().host();
ipRequestMap.compute(ip, (key, requestInfo) -> { // 定期清理过期条目,防止 Map 无限增长
if (ipRequestMap.size() > 1000) {
long now = System.currentTimeMillis();
ipRequestMap.entrySet().removeIf(entry -> now - entry.getValue().timestamp > TIME_WINDOW);
}
RequestInfo info = ipRequestMap.compute(ip, (key, requestInfo) -> {
long currentTime = System.currentTimeMillis(); long currentTime = System.currentTimeMillis();
if (requestInfo == null || currentTime - requestInfo.timestamp > TIME_WINDOW) { if (requestInfo == null || currentTime - requestInfo.timestamp > TIME_WINDOW) {
// 初始化或重置计数器 // 初始化或重置计数器
@@ -50,7 +56,6 @@ public class RateLimiter {
} }
}); });
RequestInfo info = ipRequestMap.get(ip);
if (info.count > MAX_REQUESTS) { if (info.count > MAX_REQUESTS) {
// 超过限制 // 超过限制
// 计算剩余时间 // 计算剩余时间
@@ -66,8 +71,8 @@ public class RateLimiter {
} }
private static class RequestInfo { private static class RequestInfo {
int count; volatile int count;
long timestamp; volatile long timestamp;
RequestInfo(int count, long time) { RequestInfo(int count, long time) {
this.count = count; this.count = count;
@@ -29,6 +29,11 @@ public class CacheServiceImpl implements CacheService {
private final CacheManager cacheManager = new CacheManager(); private final CacheManager cacheManager = new CacheManager();
static {
// 服务类加载时注册缓存定时清理任务
CacheManager.registerPeriodicCleanup();
}
private Future<CacheLinkInfo> getAndSaveCachedShareLink(ParserCreate parserCreate) { private Future<CacheLinkInfo> getAndSaveCachedShareLink(ParserCreate parserCreate) {
// 认证、域名相关(检查是否已经添加过参数,避免重复调用) // 认证、域名相关(检查是否已经添加过参数,避免重复调用)